JWST Operations Pipeline Build Information

Information about current and previous releases of the operations pipeline, which includes science calibration pipeline software, is provided in this article. Also included are known issues or shortcomings of each build and planned enhancements.

On this page

Words in bold are GUI menus/
panels or data software packages; 
bold italics are buttons in GUI
tools or package parameters.

The JWST Science Calibration Pipeline (Bushouse et al. 2023), aka jwst, used by the Mikulski Archive for Space Telescopes (MAST) to calibrate all JWST data, is a subsystem of the JWST Operations Pipeline and is updated on a regular basis. These updates are only pushed to MAST when there are builds of the operations pipeline, which usually include changes to other subsystems, about every 3 months.

In some cases, users might see patch releases in-between major builds of the JWST Operations Pipeline that are not documented here. The reason for not documenting these patches is because they may not include changes to the JWST Science Calibration Pipeline or they do not impact the calibration products. Whenever a patch affects the science data in a considerable way, the changes will be documented here.

Also, note that the JWST Science Calibration Pipeline can be updated and released to users before it is integrated into the operations pipeline. These are done via minor releases of the jwst package via conda pip. Such updates will subsequently be included in operations with the next operations build.

This article provides links to release notes for different builds of the operations pipeline and patches affecting the calibration products. When the information is maintained elsewhere, links are provided to that documentation.

Tracking versions in the data products: Several FITS header keywords document the software and reference data versions used to generate data product. These are 2 keywords of note:

  • CAL_VER records the JWST Science Calibration Pipeline version used to process data (e.g., "1.4.6").
  • CRDS_CTX records the context of calibration reference data, in the JWST OPS server, that was used to process data (e.g., "jwst_0857.pmap").



The current build

The current build of the operations pipeline includes build 10.0 and patch 10.0.1—please refer to the latest release notes for additional details.

Highlights of this build

Build 10.0 and patch 10.0.1

  • Words in bold are GUI menus/
    panels or data software packages; 
    bold italics are buttons in GUI
    tools or package parameters.

    The positioning of the 1-D extraction aperture in the extract_1d step, for NIRSpec MOS and fixed slit data, as well as the MIRI LRS fixed slit, has been updated to be more robust. For NIRSpec MOS and FS, the estimated location of the source in the slit, which is provided by the APT and MPT planning tools, is used to set the extraction location, rather than the (often unreliable or inaccurate) source RA/Dec values. Similarly for MIRI LRS FS, the known dither/nod offset applied to an exposure is used to set the extraction location, instead of using the source RA/Dec. This results in much better centering of the extraction aperture on the spectral trace and greatly improved 1-D spectral data.
  • The assignment of SOURCEID values for NIRSpec fixed slit products has been updated so that the data from the slit with the primary target always have SOURCEID = 1. The SOURCEID values for other slits use a 2-digit code, that gives the number of the slit containing the primary target and the number of the secondary slit from which the data were extracted.
  • The stage 3 association rules have been updated to combine data from the NIRSpec S200A1 and S200A2 slits for observations in which the primary target was observed in both slits.
  • The former undersampling_correction step has been renamed to charge_migration, and several enhancements have been added to the step.
  • The algorithms used to find and exclude intermittent bad pixels in the refpix step for NIRSpec IRS2 images have been made more robust, resulting in cleaner images.
  • A problem with duplicate stage 3 product names for NIRCam coronagraphy observations that obtain data using all short-wavelength channel detectors has been fixed.

Known issues

Issues that affect the JWST data products available from the MAST Archive that were not resolved with this build are covered in the Known Issues with JWST Data Products article. When there is a workaround for an issue, the relevant information is provided there.



Future enhancements for the JWST Science Calibration Pipeline

Future enhancements planned for the Science Calibration Pipeline are provided under the mitigation plan in Known Issues with JWST Data Products



Record of builds to JWST Operations Pipeline

Table 1 provides a historical record of ongoing changes to the JWST Science Calibration Pipeline that were installed in the JWST Operations Pipeline. Release notes for a particular build are available via links in the first column. Note that some jwst releases are never installed in operations but can be installed locally via pip. For full information on all jwst releases, please refer to this table of all jwst software releases. (Note: in that table, CRDS_CONTEXT is the testing context; this is a context that, although it might have a number equal to a context in operations, links to a different server and different data. These contexts are not available to outside users.)

Note that testing context, in the CRDS-TEST server, might be different to the operations context and not accessible to outside users.

Table 1. Release notes and associated versions for the JWST Operations Pipeline releases

Operations Pipeline 

Build

Science Calibration

Pipeline Version (jwst)

SDP_VERReleasedOps Install

Operations

CRDS context

10.1/10.0.1

1.12.0 to 1.12.5

2023.3.12023-10-19
2023-12-05 1166
9.3/9.3.11.11.42023.2.12023-08-142023-08-241117
9.21.10.1 
2023.1.12023-03-31 2023-05-231089
JWST Operations Pipeline Build 9.1 Release Notes /9.1.1 /9.1.2 1.9.6

2022.5.0/2022.5.2

2023-03-092023-03-151068
9.01.8.2

2022.4.1/2022.4.0

2022-10-192022-11-171017
8.1.21.7.22022.3.12022-09-122022-09-21984
8.1.11.6.22022.3.02022-07-192022-08-19953
8.0.11.5.32022.2.12022-06-202022-06-30913
8.01.5.22022.2.02022-05-202022-06-16868



References

Bushouse, H., et al. 2023, Zenodo 
JWST Calibration Pipeline

Rauscher, B. 2023, arXiv:2306.03250
NSClean: An Algorithm for Removing Correlated Noise from JWST NIRSpec Images



 
Latest updates
  •  
    Updated for build 10.0

  •  
    Updated for build 9.3, added information about future pipeline enhancements

  •   
    Added Table 1, history of changes to JWST Operations Pipeline

  •  
    added information about build 9.2

  •  
    added information about build 9.1

  •   
    added information about build 9.0

  •  
    added information about build 8.1.2

  •  
    added information about build 8.1.1


  • Added known issues with the pipeline found during commissioning
    Added future enhancements information
Originally published