Tips and Tricks for working with the JWST Pipeline
A few useful tips and tricks for working with the JWST pipeline, and for using the tools that it contains to interact with the JWST pipeline data products, are highlighted in this article.
On this page
Query the WCS using the jwst datamodels
JWST has many instruments with extremely complex and discontinuous distortion solutions that cannot be well described by simple FITS polynomial approximations. World coordinate information computed as part of the calwebb_image2 and calwebb_spec2 steps is described as a series of transformations saved within an ASDF-format extension.
This WCS information can be queried directly using the jwst datamodels infrastructure if the jwst software package has been installed (see the pipeline overview article).
NIRCam example
For instance, open a NIRCam stage 2 calibrated data file "jw01538046001_03109_00001_nrcalong_cal.fits" (in this case, representing a NIRCAM module A long-wavelength exposure from the PID 1538 flux calibration program) and see what WCS reference frames are available:
from jwst import datamodels
model=datamodels.open('jw01538046001_03109_00001_nrcalong_cal.fits')
model.meta.wcs.available_frames
> ['detector', 'v2v3', 'v2v3vacorr', 'world']
These available reference frames typically include "world" coordinates corresponding to R.A., Dec., and (where relevant) wavelength in µm, "detector" coordinates corresponding to detector pixels, "v2v3" coordinates corresponding to the spacecraft V2/V3 internal reference frame, and "v2v3vacorr" corresponding to spacecraft V2/V3 coordinates corrected for velocity aberration.
Transformations between frames are also available. For instance:
model.meta.wcs.transform('detector', 'world', 150, 250)
> (120.05150608392701, -10.758302846616774)
indicates that pixel (x,y) = (150,250) corresponds to (R.A., Dec.) = (120.05150608392701, -10.758302846616774) in decimal degrees. Note that (x, y) pixel values are zero indexed.
MIRI MRS example
As in the NIRCam example above, the MRS distortion model can be opened and queried in the same manner. For example, take calibrated data file "jw01523001001_03101_00001_mirifushort_cal.fits" (in this case, representing a MIRI MRS short-wave (channel 1/2) detector image from the PID 1523 external flat field program):
from jwst import datamodels
model=datamodels.open('jw01523001001_03101_00001_mirifushort_cal.fits')
model.meta.wcs.available_frames
> ['detector', 'alpha_beta', 'v2v3', 'v2v3vacorr', 'world']
model.meta.wcs.transform('detector', 'world', 77, 809)
> (316.75822324023983, 42.23643057583943, 7.460600839320411))
In this case, there is an additional reference frame "alpha_beta" which describes a coordinate frame local to the IFU slicer, and the transform output also includes the wavelength of the pixel (7.4606 µm).
Note that the datamodel interface can be used to interact with the WCS of the final "s3d" and "i2d" rectified data products as well. In this case the "detector" reference frame corresponds to resampled pixels (for imaging data) and voxels (for IFU data cubes) rather than native detector pixels. Take as an example the data cube "jw01523-o001_t002_miri_ch1-shortmediumlong_s3d.fits":
from jwst import datamodels
model=datamodels.open('jw01523-o001_t002_miri_ch1-shortmediumlong_s3d.fits')
model.meta.wcs.available_frames
> ['detector', 'world']
model.meta.wcs.transform('world', 'detector', 316.7580671, 42.2363530, 5.12)
> (20.42698278222596, 33.80277322646009, 274.4998877377921)
This indicates that sky coordinate (R.A., Dec., Wavelength) = (316.7580671, 42.2363530, 5.12) corresponds to cube voxel coordinate (x,y,z) = (20.42698278222596, 33.80277322646009, 274.4998877377921). Note that (x,y) pixel values are zero indexed.
Query CRDS for the latest reference file versions
The JWST pipeline automatically queries CRDS for the best reference files associated with a given context, and users can browse each context directly at https://jwst-crds.stsci.edu/
However, it is also possible to use the pipeline infrastructure to look up and download reference files directly. For instance, here's how to check what CRDS context is being used and download the appropriate "distortion" reference file (see ReadTheDocs list of reference file types) for an input file "jw01538053001_02102_00001_nrcb4_rate.fits":
from jwst import datamodels
from stpipe import crds_client
crds_client.get_context_used('jwst')
> 'jwst_1193.pmap'
model=datamodels.open('jw01538053001_02102_00001_nrcb4_rate.fits')
crds_client.get_reference_file(model.get_crds_parameters(), 'distortion', 'jwst')
> Fetching /Users/bilbo/crds_cache/jwst_ops/references/jwst/nircam/jwst_nircam_distortion_0142.asdf
> /Users/bilbo/crds_cache/jwst_ops/references/jwst/nircam/jwst_nircam_distortion_0142.asdf
Working with DQ flags
As discussed in this section on error and data quality arrays, the JWST pipeline data products encode information about the quality of each science pixel value in a data quality (DQ) array. This DQ array is a series of 32-bit integers; individual bits can be toggled on or off as necessary. For instance, a DQ value of 5 means that bits zero and 2 have been set ( 2^0 + 2^2 = 5). It can be convenient to query the meaning of a given DQ value using the jwst datamodels. Given a value DQ=262657, this can be specified:
from jwst.datamodels import dqflags
value = 262657
dqflags.dqflags_to_mnemonics(value,mnemonic_map=dqflags.pixel)
> {'DO_NOT_USE', 'NON_SCIENCE', 'NO_FLAT_FIELD'
This tells us that this pixel has the "DO_NOT_USE", "NON_SCIENCE", and "NO_FLAT_FIELD" flags set. The most important of these is the zero-bit (2^0 = 1) "DO_NOT_USE" flag, and the others are largely informational (see ReadTheDocs for a summary of all bit flags).
It's also possible to query where in a DQ image a certain flag has been set. For instance, to find every location that a "DO_NOT_USE" flag has been set in the data quality array corresponding to a given file "myfile.fits":
from jwst.datamodels import dqflags
from astropy.io import fits
import numpy as np
hdu=fits.open('myfile.fits')
dqarr=hdu['DQ'].data
thebit=dqflags.interpret_bit_flags('DO_NOT_USE',mnemonic_map=dqflags.pixel)
indx=np.where((dqarr & thebit) != 0)
Query FGS guiding information
Users interested in seeing if any unusual events (e.g., mirror tilt events, poor tracking, etc.) occurred during their observations can download the FGS guider data products from MAST. A community-provided resource to make this process easier is the JWST FGS Spelunker tool; the repository has a README quick start file, and there are some notebooks developed to help with getting started. Note that as a community-provided resource this tool is not supported by STScI.
Creating custom pipeline steps
The JWST pipeline is extensible, allowing users to insert custom steps when reprocessing data. This can be achieved with the use of hooks to attach either before or after an existing pipeline step (see Hook documentation in ReadTheDocs). These custom steps can be as simple or complex as necessary.
For instance, say you wanted to multiply all the fluxes by 2 after the assign_wcs step in calwebb_spec2. Inside the pipeline notebook, you could define a function "XplyStep":
from jwst import datamodels
from jwst.stpipe import Step
class XplyStep(Step):
spec = '''
factor = float(default=2.0) # Multiplicative factor
'''
class_alias = 'xply'
def process(self, input_data):
with datamodels.open(input_data) as model:
result = model.copy()
result.data = result.data * 2
self.log.info('Multiplied everything by two!')
return result
and then add a hook to call this step into the pipeline calling routine:
from jwst.pipeline import Spec2Pipeline
dict = {"assign_wcs": {"post_hooks": [XplyStep]}}
result = Spec2Pipeline.call('myfile.fits', steps=dict, save_results=True)
If the step were defined as a function outside of the notebook environment in the file "xply.py", this could be used instead:
dict = {"assign_wcs": {"post_hooks": ["xply.XplyStep"]}}
While powerful, pipeline hooks currently work only when using the strun or .call() methods of using the pipeline (see Running the JWST Science Calibration Pipeline), and cannot have parameters passed to them. Likewise, examples given here will work only in versions 1.14.0 and later of the JWST pipeline.