Information about current and previous releases of the operational pipeline, which includes science calibration pipeline software, is provided in this article. Also included are known issues or shortcomings of each build and planned enhancements.

The JWST Science Calibration Pipeline (Bushouse et al. 2023), aka jwst, used by the Mikulski Archive for Space Telescopes (MAST) to calibrate all JWST data, is a subsystem of the JWST Operational Pipeline and is updated on a regular basis. These updates are only pushed to MAST when there are builds of the operational pipeline, which usually include changes to other subsystems, about every 3 months.

In some cases, users might see patch releases in-between major builds of the JWST Operational Pipeline that are not documented here. The reason for not documenting these patches is because they may not include changes to the JWST Science Calibration Pipeline or they do not impact the calibration products. Whenever a patch affects the science data in a considerable way, the changes will be documented here.

Also, note that the JWST Science Calibration Pipeline can be updated and released to users before it is integrated into the operational pipeline. These are done via minor releases of the jwst package via conda pip. Such updates will subsequently be included in operations with the next operational build.

This article provides links to release notes for different builds of the operational pipeline and patches affecting the calibration products. When the information is maintained elsewhere, links are provided to that documentation.

---

The current build

The current build of the operational pipeline includes build 9.3 and patch 9.3.1—please refer to the latest release notes for additional details.

Highlights of this build

Build 9.3 and patch 9.3.1

• Words in bold are GUI menus/
  panels or data software packages; 
  bold italics are buttons in GUI
  tools or package parameters.

  The MIRI MRS flux calibration, applied in the photom step of the calwebb_spec2 pipeline, has been upgraded to include corrections for time-dependent changes in throughput.
• The jump step in the calwebb_detector1 stage of the pipeline  has had several updates and enhancements to make the detection of snowballs/showers more robust and reduce the frequency of "false positives".
• The outlier_detection algorithm for IFU data, applied in the calwebb_spec3 stage of the pipeline, has been completely reworked. It now focuses on finding and flagging pixels that are bad in all input images.
• The extract_1d step has been upgraded to apply a residual fringe correction to MIRI MRS 1-D extracted spectra.
• The photom step in the calwebb_spec2 stage of the pipeline has been updated to calibrate all NIRSpec IFU exposures to units of surface brightness, rather than flux density, for compatibility with the cube_build step's resampling process.
• A new pixel_replace step has been added to the calwebb_spec2 stage of the pipeline, to replace the values of bad pixels with estimates from surrounding good neighbors, in order to avoid issues with dropouts in extracted 1-D spectra.
• The pointing calculations used in populating WCS keywords in raw ("uncal") products have been updated to fix a bug that was causing large offsets in the WCS for observations that use FGS2 for guiding.

Known issues

Issues that affect the JWST data products available from the MAST Archive and that were not resolved with this build are covered in the Known Issues with JWST Data Products article. When there is a workaround for an issue, the relevant information is provided there.

---

Future enhancements for the JWST Science Calibration Pipeline

• All data in general are susceptible to cosmic ray impacts; in most cases these affect only a few pixels at a time and are generally successfully removed by jump detection in the stage 1 pipeline, but occasionally a much more energetic event impacts the detector, which manifests as a bright core surrounded by a large diffuse halo in the NIR instruments (referred to as snowballs) and/or a cascade of smaller events in MIRI (referred to as showers). An initial implementation to remove these is available for offline use in the pipeline code but is not yet enabled in operations until further extensive testing has been completed to determine whether to enable it in operations in future.
   
• Images obtained with NIRCam subarrays, where the number of reset rows is less than the full amplifier size, have a cyclical bias pattern that is currently not completely removed by the superbias step. This can be addressed by using new reference files for these subarrays, which are in the process of being delivered by the NIRCam Instrument Definition Team (IDT), and corresponding modifications to the pipeline superbias subtraction step for these NIRCam subarrays, which will be implemented in a future release.
   
• Several MIRI subarrays are affected by a striping pattern in the data, which is due to spatially and temporally varying noise that originates in the detector electronics (390 Hz noise). In the long term, the MIRI team will likely make changes to the detector readout parameters to completely remove this effect. In the meantime, for data affected by this noise, an algorithm has been developed to characterize and remove this noise, and testing is currently underway to incorporate this into a future pipeline release.
   
• In WFSS observations with NIRISS and NIRCam of fields with large numbers of sources, extracted spectra of targets can often include undesired adjacent "contaminating" sources. Extensive work has been underway to develop pipeline enhancements to develop a new step, wfss_contam, that is designed to account for these sources and exclude them from the extraction, and final testing and implementation is currently underway to include this step in a future pipeline release.
   
• For NIRSpec fixed slit spectroscopy an enhancement to the pipeline association generator and related code is being implemented to combine exposures with the target in the S200A1 and S200A2 slits. Combining more exposures for the final product will improve outlier rejection and signal to noise, as well as providing a final single product with complete wavelength coverage.
   
• At very low levels, the NIRCam and NIRISS detectors generally exhibit a behavior referred to as "1/f noise," resulting from slow time-dependent changes that are not fully removed in the reference pixel subtraction (refpix)step. (Note that NIRSpec detectors are also capable of this behavior, but it is substantially mitigated by use of the IRS² readout mode which is available for NIRSpec). This effect manifests as slowly varying stripes at low levels along the slow-read direction (e.g., along rows in NIRCam data). Some algorithms already exist to remove this, and are being investigated for eventual inclusion into the pipeline. At the moment, however, these approaches generally depend on effective masking of sources in the image, and are therefore currently made available to be run offline.
  Three current algorithms that have been produced by members of the community are as follows:
  • From the NIRCam IDT: Everett Schlawin's "ROEBA" 1/f code (runs on individual groups before ramp fitting):
    ROEBA - Software documentation, code
  • From the NIRISS IDT: Chris Willott's 1/f code (runs on "cal.fits" files):
    https://github.com/chriswillott/jwst/blob/master/image1overf.py
  • From the CEERS Early Release Science team: Micaela Bagley's 1/f code (runs on "rate.fits" files):
    https://github.com/spacetelescope/jwebbinar_prep/blob/webbinar13/ceers/nircam/remstriping.py
     

• Current NIRSpec pipeline products can sometimes exhibit residual 1/f noise and/or a "pedestal/picture frame" effect, likely the result of small thermal instabilities. The effect is more pronounced for the NRS2 detector than NRS1, and is often observed after a change in detector readout mode. This additional noise/pedestal signal can then manifest as flux variations (and "negative" fluxes) in some data sets. An algorithm has been developed (NSClean, Rauscher 2023) which makes use of unilluminated detector pixels to characterize and successfully remove this signal from the data. Evaluation and testing of this algorithm is currently underway for incorporation into a future pipeline release.
   
• Stage 3 processing of large imaging mosaics can take a longer than normal amount of time to process. Updates are planned for the methods used in the tweakreg and resample steps to make them more efficient.

---

Record of builds to JWST Operational Pipeline

Table 1 provides a historical record of ongoing changes to the JWST Science Calibration Pipeline that were installed in the JWST Operational Pipeline. Release notes for a particular build are available via links in the first column. Note that some jwst releases are never installed in operations but can be installed locally via pip. For full information on all jwst releases, please refer to this table of all jwst software releases. (Note: in that table, CRDS_CONTEXT is the testing context; this is a context that, although it might have a number equal to a context in operations, links to a different server and different data. These contexts are not available to outside users.)

Note that testing context, in the CRDS-TEST server, might be different to the operational context and not accessible to outside users.

Table 1. Release notes and associated versions for the JWST Operational Pipeline releases

---