Information about current and previous build deliveries of the calibration pipeline software, as well as known issues or shortcomings of each build and planned enhancements, are provided in this article.

The JWST Science Calibration Pipeline used by the Mikulski Archive for Space Telescopes (MAST) to calibrate all JWST data is a subsystem of the JWST Operational Pipeline. It is updated on a regular basis. These updates are only pushed to MAST when there are builds of the operational pipeline, which usually include changes to other subsystems, about every 3 months.

In some cases, users might see patch releases in-between builds of the JWST Operational Pipeline. While these patches may not include changes to the JWST Science Calibration Pipeline, they still impact the calibration products.

The JWST Science Calibration Pipeline can also be updated and released to users before it is integrated into the operational pipeline. These are done via minor releases of the jwst package via conda pip. Such updates will subsequently be included in operations with the next operational build.

This article provides links to release notes for different builds of the operational pipeline and patches affecting the calibration products. When the information is maintained elsewhere, links are provided to that documentation.

---

The current build

The current build of the operational pipeline includes build 9.2 and patches 9.2.1—please refer to the latest release notes for additional details.

Highlights of this build

Build 9.2

• Words in bold are GUI menus/
  panels or data software packages; 
  bold italics are buttons in GUI
  tools or package parameters.

  Memory leaks in the cube_build step for IFU data have been fixed, greatly reducing the amount of memory used by the step and preventing out-of-memory errors.
• The dq_init step has been updated to propagate "DO_NOT_USE" data quality flags from the MASK reference file to both the PIXELDQ and GROUPDQ arrays of the science exposure. With this, all bad pixels are set to a value of NaN in the count rate ("rate"/"rateints") products created by the calwebb_detector1 pipeline.
• Major enhancements and bug fixes have been applied to the near-IR "snowball" and MIRI "shower" flagging algorithms in the jump detection step, which greatly increase the scientific utility of the results. For more information on the algorithms and new step parameters see the jump step documentation. Note that the snowball and shower flagging is currently skipped by default in standard processing, while testing of the new algorithms continues. Once they have been thoroughly vetted, they will be reactivated through parameter reference file updates in CRDS.
• A bug has been fixed in the ramp_fit step to correctly compute the count rate for pixels that only have good (e.g., unsaturated) data in the first group of an integration. Previously, there was an error in the effective integration time that was used to convert the first group value into a count rate. Note that this is distinct from pixels where the "zero frame" value is used to compute a count rate (when all groups are bad). The count rate values from zero frame data were correct.

Known issues

The following issues were not resolved in the jwst calibration pipeline package (version 1.10.1) that was released with build 9.2:

• MIRI MRS and NIRSpec IFU outlier_detection (in the stage 3 calwebb_spec3 pipeline) can have anomalous results, due to several high-level issues, including undersampling. A completely new approach to outlier detection in IFU data is being developed, which will replace the current algorithm in a future build. The current outlier_detection step is skipped by default in standard pipeline processing, in order to avoid the poor results.

Some step and mode-specific issues will be fixed or included in build 9.3:

• The stage 2 and 3 resample step, which uses drizzle, has a bug that can cause regions of the input images to not appear in the resampled image, under certain circumstances.
• The pathloss correction step is accidentally applying the correction twice to NIRSpec fixed slit data sets. This has been fixed for build 9.3.
• Association generator rules will be updated to properly handle NIRCam coronagraphy observations that obtain data from all 4 short-wavelength channel detectors (module A only). Given that only one of the detectors has the coronagraphic mask and target in the field of view, special handling is required so that only those data receive coronagraphic processing in the calwebb_coron3 pipeline, while data from the other detectors are treated as normal imaging and get processed by the calwebb_image3 pipeline.
• A new step, pixel_replace, has been developed for use in the calwebb_spec2 pipeline, which can be used to replace bad pixels in 2-D spectroscopic images with values interpolated from their neighbors. This prevents contamination of extracted 1-D spectra by bad pixels.
• The NIRCam wide field slitless spectroscopy (WFSS) spectral trace and dispersion calibration data currently available in CRDS are not a good match to in-flight data. Updated calibrations have been derived by the NIRCam team, using a more complex form of WCS transforms, which require updates to the calibration pipeline code in order to apply them. Those updates, and the use of the latest in-flight calibrations, will be available in build 9.3.

• If you are working with reprocessed data, you might find that the VELOSYS keyword in the SCI extension header might be missing and the pipeline will be unable to correct for the JWST spacecraft motion during the wavelength calibration step. This will not be noticeable for most of the science cases but it seems to have a larger impact for MIRI MRS data.

• There is a known bug in the calwebb_spec2 background subtraction step for wide field slitless spectroscopy (WFSS) images. Any bad pixels that are set to NaN will cause the output of the step to be an image completely filled with NaNs, as well as all downstream products. This bug has been fixed for build 9.3.

---

Future enhancements for the JWST Science Calibration Pipeline

• Images obtained with NIRCam subarrays, where the number of reset rows is less than the full amplifier size, have a cyclical bias pattern that is currently not completely removed by the superbias subtraction step. This can be addressed by using new reference files for these subarrays, which are in the process of being delivered by the NIRCam Instrument Definition Team (IDT), and corresponding modifications to the pipeline superbias subtraction step for these NIRCam subarrays, which will be implemented in a future release.
  
  
• The jump detection step for cosmic ray removal in calwebb_detector1 currently runs slowly for cases with 2 or more cosmic rays per pixel ramp. Improvements to the relevant parts of this step have been identified for future development and implementation.
  
  
• At very low levels, the NIRCam and NIRISS detectors generally exhibit a behavior referred to as "1/f noise," resulting from slow time-dependent changes that are not fully removed in the reference pixel subtraction step. (Note that NIRSpec detectors are also capable of this behavior, but it is substantially mitigated by use of the IRS2 readout mode which is available for NIRSpec). This effect manifests as slowly varying stripes at low levels along the slow-read direction (e.g., along rows in NIRCam data). Some algorithms already exist to remove this, and are being investigated for eventual inclusion into the pipeline. At the moment, however, these approaches generally depend on effective masking of sources in the image, and are therefore currently made available to be run offline.
  
  Three current algorithms that have been produced by members of the community are as follows:
• • From the NIRCam IDT: Everett Schlawin's "ROEBA" 1/f code (runs on individual groups before ramp fitting):
    ROEBA - Software documentation, code
  • From the NIRISS IDT: Chris Willott's 1/f code (runs on cal.fits files):
    https://github.com/chriswillott/jwst/blob/master/image1overf.py
  • From the CEERS Early Release Science team: Micaela Bagley's 1/f code (runs on rate.fits files):
    https://github.com/spacetelescope/jwebbinar_prep/blob/webbinar13/ceers/nircam/remstriping.py

• Stage 3 processing of large imaging mosaics can take a longer than normal amount of time to process. Updates are planned for the methods used in the tweakreg and resample steps to make them more efficient.

---

Previous builds

Previous build information as a historical record of the ongoing changes to the JWST Science Calibration Pipeline.

• JWST Operational Pipeline Build 9.1 Release Notes
• JWST Operational Pipeline Build 9.0 Release Notes
• JWST Operational Pipeline Build 8.1.2 Release Notes
• JWST Operational Pipeline Build 8.1.1 Release Notes
• JWST Operational Pipeline Build 8.0.1 Release Notes
• JWST Operational Pipeline Build 8.0 Release Notes

---

---