JWST Operations Pipeline Build Information

Once per quarter, a fixed operational build is produced that corresponds to tagged versions of the JWST Science Data Processing (SDP) system and of the Science Calibration Pipeline, and which points to an updated CRDS reference file context. This Operations Pipeline is installed by the Data Management System (DMS) at STScI and used to reprocess all archival JWST data as well as newly-obtained observations. The data products available for download from the MAST archive (see Accessing JWST Data) are thus tied to these builds and updated periodically.

On this page

This article provides information about what's new in the latest build, what was done in previous builds, and gives a preview of some updates likely to be coming soon in upcoming builds. The information contained here is generally a high-level summary of changes, with links provided to more extensive external documentation.



What's new in the current build?

The current build of the Operations Pipeline is build 10.1. Some highlights are provided here about major relevant changes; please refer to the build 10.1 release notes for additional details.

Please also refer to the calibration notes broken down by instrument mode in the JWST calibration status articles, and the known issues with JWST data articles.

Major pipeline updates in build 10.1

  • A new step, emicorr, has been added to the stage 1 pipeline to remove EMI noise patterns from MIRI exposures (affecting primarily subarray data).
  • A new step, nsclean, has been added to the stage 2 spectroscopic pipeline to remove 1/f noise from NIRSpec exposures.
  • A new step, spectral_leak, has been added to the stage 3 spectroscopic pipeline to remove 2nd-order contamination at 12.2 µm from MIRI MRS point source spectra.
  • Model-based time-dependent flux calibration corrections have been added to the stage 2 imaging pipeline for MIRI imager exposures.
  • The stage 2 spectroscopic pipeline path loss correction for NIRSpec MOS exposures has been enhanced to handle slitlets that have shapes other than the standard 1 × 3 or 1 × 1 shutters.
  • Cosmic ray shower and snowball correction have been enabled by default for NIRSpec, NIRCam, and some MIRI modes (although a bug prevents this from running on SLOWR1 mode MIRI data)

Major reference file updates in build 10.1



Coming soon

The next JWST operations pipeline builds are build 10.2 (expected late spring 2024) and build 11.0 (expected late summer 2024). The following list indicates some of the areas in which development effort will be focused in the coming months.

Pipeline development

JWST Science Calibration Pipeline development work includes:

  • Further treatment of bad pixels and outlier detection in spectroscopic data
  • Expanded treatment of 1/f noise
  • Updates to NIRSpec IRS2 reference pixel correction
  • Including dark current contributions to Poisson noise
  • Improvements to drizzled data quality with different weighting schemes
  • Improvements to photometric calibration of dispersed NIRCam WFSS data
  • Work to reduce runtime of multiple pipeline stages
  • Improved error estimates
  • Initial implementation of PSF-based spectral extraction
  • Improvements for MIRI MRS residual fringe correction and Ch4C 1-D spectral extraction
  • Adding new source finder options for tweakreg
  • Masking persistence after snowballs

Reference files

Major reference file updates currently planned for Spring 2024 include:

  • NIRSpec subarray superbiases and darks
  • NIRISS WFSS spectral trace updates



Record of JWST Operations Pipeline builds

Table 1 provides a historical record of JWST operations builds, along with links to the build release notes and tagged software versions. Note that many intermediate development tags of the JWST Science Calibration Pipeline (indicated by minor version releases of the jwst package) are never installed in STScI DMS operations, but can be installed by end users. For full information on all jwst releases, see the github software version table.

Table 1. Release notes and associated versions for the JWST Operations Pipeline releases

Operations Pipeline 

Build

Science Calibration
Pipeline Version (jwst)

SDP_VERReleasedOps Install

CRDS context

10.1 (ACTIVE)

1.13.3

2023.4.0
2024-01-05 2024-02-281210 
10.0.1

1.12.5

2023.3.12023-10-192023-12-05 1166-1210
9.3.11.11.42023.2.12023-08-142023-08-241107-1166
9.21.10.1 
2023.1.12023-04-13 2023-05-231077-1107
9.1.2 1.9.6

2022.5.2

2023-03-092023-03-151068-1077
9.01.8.2

2022.4.0

2022-10-192022-11-171017-1068
8.1.21.7.22022.3.12022-09-122022-09-21984-1017
8.11.6.22022.3.02022-07-192022-08-19953-984
8.0.11.5.32022.2.12022-06-202022-06-30913-953
8.01.5.22022.2.02022-05-202022-06-16874-913



Reference files and within-build reprocessing

In the early phases of the JWST mission, CRDS reference file contexts (see CRDS overview) are not fixed within a given build in the same manner as the software.  Rather, each build has a certain minimum context, but the context used for data processing can increment over time as new reference files are delivered. Even data within MAST that has been processed with the same build may thus have used different sets of reference files as indicated in Table 1.

As such, STScI will periodically reprocess data within a given build if there have been significant updates in the reference files associated with a given instrument.



Ephemeris information

Note that ephemeris information (the 3-dimentional position and velocity as a function of time) for JWST can evolve between the initial processing of a dataset available through MAST and subsequent reprocessing of that data. The Flight Dynamics Facility (FDF) group at Goddard evaluates the ephemeris for JWST's orbit and releases a weekly update that includes the actual ephemeris up to a specific date, and a predicted ephemeris that is used for planning and executing upcoming observations. Both are stored in a database within minutes of the files being delivered by FDF, and the database is accessed by level 1 data processing in the operational pipeline. Subsequent weekly deliveries by FDF will update the predicted ephemeris database values.

To populate the ephemeris headers in the JWST data products, the JWST Data Management System (DMS) uses the definitive ephemeris values, or if they are not yet available, they use the predicted ephemeris values. Most data are initially processed with the predicted ephemeris values, while later reprocessing will use the definitive values. The header keyword EPH_TYPE lists if Definitive or Predicted was used. The ephemeris files (both definitive and predicted) are also ingested into MAST. The ephemeris is used for aberration correction in the WCS parameters in the science headers, so all science types are affected.



References

Bushouse, H., et al. 2023, Zenodo 
JWST Calibration Pipeline

Rauscher, B. 2023, arXiv:2306.03250
NSClean: An Algorithm for Removing Correlated Noise from JWST NIRSpec Images



 
Notable updates

  • Updated for build 10.1

  • Updated for build 10.0

  • Updated for build 9.3
  • Added information about future pipeline enhancements

 

  • Added Table 1, history of changes to JWST Operations Pipeline

  

  • Added information about build 9.2

 

  • Added information about build 9.1

 

  • Added information about build 9.0

  • Added information about build 8.1.2

  

  • Added information about build 8.1.1

 

  • Added known issues with the pipeline found during commissioning
  • Added future enhancements information
Originally published