Release notes and known issues in build 9.2 of the JWST operational pipeline, released on May 23, 2023, are provided in this article.

This article provides a list of all relevant changes to the operational pipeline that was released with build 9.2. This build focused mainly on updates to the JWST Science Calibration Pipeline, but includes information relevant to changes in other subsystems that might affect what users see in their data. 

Highlights of this build

Build 9.2

• Words in bold are GUI menus/
  panels or data software packages; 
  bold italics are buttons in GUI
  tools or package parameters.

  Memory leaks in the cube_build step for IFU data have been fixed, greatly reducing the amount of memory used by the step and preventing out-of-memory errors.
• The dq_init step has been updated to propagate "DO_NOT_USE" data quality flags from the MASK reference file to both the PIXELDQ and GROUPDQ arrays of the science exposure. With this, all bad pixels are set to a value of NaN in the count rate ("rate"/"rateints") products created by the calwebb_detector1 pipeline.
• Major enhancements and bug fixes have been applied to the near-IR "snowball" and MIRI "shower" flagging algorithms in the jump detection step, which greatly increase the scientific utility of the results. For more information on the algorithms and new step parameters see the jump step documentation. Note that the snowball and shower flagging is currently skipped by default in standard processing, while testing of the new algorithms continues. Once they have been thoroughly vetted, they will be reactivated through parameter reference file updates in CRDS.
• A bug has been fixed in the ramp_fit step to correctly compute the count rate for pixels that only have good (e.g., unsaturated) data in the first group of an integration. Previously, there was an error in the effective integration time that was used to convert the first group value into a count rate. Note that this is distinct from pixels where the "zero frame" value is used to compute a count rate (when all groups are bad). The count rate values from zero frame data were correct.

Known issues

The following issues were not resolved in the jwst calibration pipeline package (version 1.10.1) that was released with build 9.2:

• MIRI MRS and NIRSpec IFU outlier_detection (in the stage 3 calwebb_spec3 pipeline) can have anomalous results, due to several high-level issues, including undersampling. A completely new approach to outlier detection in IFU data is being developed, which will replace the current algorithm in a future build. The current outlier_detection step is skipped by default in standard pipeline processing, in order to avoid the poor results.

Some step and mode-specific issues will be fixed or included in build 9.3:

• The stage 2 and 3 resample step, which uses drizzle, has a bug that can cause regions of the input images to not appear in the resampled image, under certain circumstances.
• The pathloss correction step is accidentally applying the correction twice to NIRSpec fixed slit data sets. This has been fixed for build 9.3.
• Association generator rules will be updated to properly handle NIRCam coronagraphy observations that obtain data from all 4 short-wavelength channel detectors (module A only). Given that only one of the detectors has the coronagraphic mask and target in the field of view, special handling is required so that only those data receive coronagraphic processing in the calwebb_coron3 pipeline, while data from the other detectors are treated as normal imaging and get processed by the calwebb_image3 pipeline.
• A new step, pixel_replace, has been developed for use in the calwebb_spec2 pipeline, which can be used to replace bad pixels in 2-D spectroscopic images with values interpolated from their neighbors. This prevents contamination of extracted 1-D spectra by bad pixels.
• The NIRCam wide field slitless spectroscopy (WFSS) spectral trace and dispersion calibration data currently available in CRDS are not a good match to in-flight data. Updated calibrations have been derived by the NIRCam team, using a more complex form of WCS transforms, which require updates to the calibration pipeline code in order to apply them. Those updates, and the use of the latest in-flight calibrations, will be available in build 9.3.

• If you are working with reprocessed data, you might find that the VELOSYS keyword in the SCI extension header might be missing and the pipeline will be unable to correct for the JWST spacecraft motion during the wavelength calibration step. This will not be noticeable for most of the science cases but it seems to have a larger impact for MIRI MRS data.

• There is a known bug in the calwebb_spec2 background subtraction step for wide field slitless spectroscopy (WFSS) images. Any bad pixels that are set to NaN will cause the output of the step to be an image completely filled with NaNs, as well as all downstream products. This bug has been fixed for build 9.3.

Data affected by this build

Provided below is a list of products in MAST generated by the JWST Science Calibration Pipeline that may be most significantly affected by the updates in build 9.2. The changes listed in the release notes may affect other modes as well, but many of these will be transparent to the MAST user.

The data and pipeline stage affected for data in MAST are:

• Stage 1 pipeline results ("rate", "rateints" products) for most instrument modes can potentially be affected by the updates to:
  • the dq_init step to propagate "DO_NOT_USE" data quality flags from the MASK reference files into the science GROUPDQ arrays
  • the jump step enhancements and bug fixes for near-IR "snowball" and MIRI "shower" detection and flagging^*.
  • the ramp_fit step bug fix for count rate computations of pixels in which only the first group is not saturated and on-board frame averaging was used (e.g., fields with bright stars)
• MIRI MRS stage 2 and 3 products (residual_fringe and straylight step bug fixes)
• MIRI MRS time-series observations (TSO) stage 2 products (assign_wcs step bug fix)
• NIRSpec fixed slit observations stage 2 and 3 products (photom step bug fix)
  
  

Release notes

See also: GitHub jwst project release notes 1.10.0

Build 9.2 of the JWST operational pipeline was released on May 23, 2023 to the Barbara A. Mikulski Archive for Space Telescopes (MAST). This build includes changes to the JWST Science Calibration Pipeline released in version 1.10.1 of the jwst calibration pipeline package via the PyPI repository.

The following fixes or changes were included in build 9.2. Changes to specific steps in the pipeline are listed below, along with descriptions to more general changes made to specific packages and related subsystems. This release also includes changes to the JWST Science Data Processing subsystem (SDP) version JWSTDP-2023.1.1 which creates raw (uncal) products.

assign_wcs step

• The step was fixed to correct a bug in the computation of WCS bounding box for MIRI MRS TSO observations. assign_wcs assumed that the bounding box for all MIRI MRS data was a 2-D image, but this only applies for non-TSO MRS data. The fix now correctly handles the computation of the WCS bounding box for the TSO MRS 3-D data cubes.

cube_build step

• Updated the interface to C extension modules to fix memory allocation and leak issues. This update significantly reduces the amount of memory used when building IFU cubes.

dq_init step

• The step was updated to propagate "DO_NOT_USE" DQ flags from the MASK reference file to both the PIXELDQ and GROUPDQ arrays of the science exposure. This has the effect of preventing slope computations for bad pixels downstream in the ramp_fit step.

extract_1d step

• The step was updated to automatically skip the processing of the NIRSpec fixed slit "rateints" (multi-integration) product because this is not a supported mode.
• The logic for handling the use_source_posn parameter, which indicates if an offset to account for the expected location of the source is applied, was fixed. This parameter can be set in 3 different ways, so the fix now honors the appropriate hierarchy of precedence. The highest to lowest order of precedence is (1) command-line override settings, (2) reference file settings, and 3) internal decisions by the step code of whether or not to use the source position.
• Updated the syntax of unit strings for surface brightness quantities so that they can be properly parsed by astropy.units.

flat_field step

• The step was updated to allow the correct handling of NIRSpec fixed slit for "rateints" products. These type of products are multi-integration, so the update now correctly applies the correction to each integration individually.

jump step

• Several updates to the near-IR "snowball" and MIRI "shower" flagging algorithms were implemented, which significantly increases the scientific utility of the results. Significant changes include the use of ellipses instead of circles in fitting snowball shapes and the inclusion of more stringent tests to prevent incorrect identification of snowballs. The shower detection algorithm is completely new and is now able to find extended emission far below the single pixel signal-to-noise ratio. It also allows detected showers to flag groups after the detection event. These algorithms are currently not applied to the MAST data, but can be activated by users via the "pars_jumpstep" reference file that define the jump step runtime parameters. Instrument teams are in the process to finishing testing of these steps and expect to turn on this algorithm for those modes where the data is correctly flagged.

outlier_detection step

• Updated the step documentation (in ReadTheDocs) to more completely describe the actions of the various parameters that control the step.

pathloss step

• Updated the step to loop over all integrations when given a MIRI LRS fixed slit "rateints" (multi-integration) product as input.

photom step

• Fixed a bug in the handling of NIRSpec fixed slit exposures containing multiple slits, so that each slit gets the proper flux calibration applied depending on whether each slit contains a point or extended source.
• Corrected the units of the data stored in the "photom_uniform" array to MJy/sr (surface brightness), to allow for correct usage in master_background processing.
• Fixed the labeling of NIRISS SOSS flux units to consistently store spectral point source data in units of "MJy".

ramp_fit step

• Fixed a bug in the slope computation for pixels that only have good data (e.g., unsaturated) in the first group of any integration, so that it uses the correct effective exposure time for the first group. This because when on-board frame averaging is in use, the effective integration time of the first group is not the same as the elapsed time between all subsequent groups.

resample/resample_spec steps

• Added the step argument output_wcs, which allows a user to supply a GWCS definition for the resampled output file in the form of an ASDF file.

• Fixed a bug in the use of DQ bits to reject bad pixels when resampling the variance arrays for a given image, so that pixels rejected from the resampled science image are also rejected when constructing the resample variance and error images.

resample_spec step

• The step has been updated automatically skip processing when given a NIRSpec fixed slit "rateints" (multi-integration) product as input, because this is not a supported mode.

residual_fringe step

• Updated the step to allow proper handling of NaN pixel values in the input images.

straylight step

• Updated the step to allow proper handling of NaN pixels values in the input images.

tweakreg step

• Added the function adjust_wcs to the tweakreg.utils module, which can be used to apply additional user-provided rotations and scale corrections to an imaging WCS.
• Added a trap for failures in source catalog construction for each input image, so that an empty catalog is returned instead of the step crashing.
• Added a trap for failures in the alignment of a single image to an absolute reference catalog (e.g., Gaia), due to not enough sources available in the reference catalog.
• Changed the validation of the fit to make the step work for a larger set of step parameters. This provides better information about the magnitude of the corrections and prevents the step from aborting the alignment. 

Pipeline modules

• The calwebb_detector1 pipeline module has been updated to add the undersampling_correction step for all non-TSO data. This step is currently skipped for all instruments.
• The calwebb_spec2 pipeline module has been updated to move the MIRI MRS straylight correction step, so that it occurs before the flat_field step. This change was necessary because the artifact is due to internal reflections within the detector, and it should not take into account any of the pixel area-on-sky normalizations that will be introduced by the flat field step.
• The calwebb_spec2 pipeline module has been updated to make a deep copy of the current data before calling the resample_spec and extract_1d steps, to avoid potential issues with either of those steps modifying the input data in-place.

Product associations

• Updated the "image2" association rules for coronagraphic reference PSF exposures that have linked background observations. With this change, the linked background exposures are now subtracted during calwebb_image2 processing.

General changes

• jwst.datamodels has been moved out of the jwst package and put into the stdatamodels package as stdatamodels.jwst.datamodels. For now, imports from jwst.datamodels will continue to work and will automatically redirect to stdatamodels.
• jwst.transforms was moved out of the jwst package and put into stdatamodels.jwst.transforms.

JWST Science Data Processing subsystem changes

The JSDP subsystem processes data from the telescope and creates raw ("_uncal") products. The following changes have been included in this build:

• The set_telescope_pointing function used to set basic WCS pointing information in raw products has been updated to fill the values of the TARG_RA and TARG_DEC keywords in pure parallel exposures by copying values from RA_REF and DEC_REF. These values are computed from the aperture information, because pure parallel exposures do not carry any target information of their own.
• Fixed a bug in the set_telescope_pointing function to correctly calculate the CRPIX* keywords for MIRI_TACQ exposures with aperture MIRIM_TAMR.
• Fixed a bug in the processing and creation of "uncal" products that was leaving all of the BJD column entries of the "INT_TIMES" table extension set to zero.