Information about what's new in the latest JWST pipeline build, and what was done in previous builds, is provided in this article. It also gives a preview of some updates likely to be coming soon in upcoming builds. The information contained here is generally a high-level summary of changes, with links provided to more extensive external documentation.

Build lifecycle

JWST pipeline development proceeds according to a quarterly build schedule. Each quarter, the active development version of the JWST Science Calibration Pipeline is frozen, combined with other dependent software (e.g., the JWST Science Data Processing system; SDP) and a particular CRDS reference file context, and tagged to produce a candidate build. This candidate build is then tested extensively prior to becoming an operations build installed by the Data Management System (DMS) at STScI and used to reprocess all previous JWST data as well as newly obtained observations. Once a build is no longer operational, it becomes archived.

There are thus four kinds of builds:

• Development (Dev) build:  Changing daily, uses the Latest CRDS context, not recommended for non-developers.
• Candidate build: Development has finished, currently working on testing and documentation. Uses a fixed CRDS context. Available for experienced users.
• Operations build: Installed at STScI and being used to produce data products available in MAST. Uses a fixed CRDS context. Default recommendation for users.
• Archived build: Outdated operations build that is no longer recommended for general use (may still correspond to some MAST data products until reprocessing is completed).
  

Record of JWST Operations Pipeline builds

Table 1 provides a historical record of JWST operations builds, along with links to the build release notes and tagged software versions. The "CRDS Context" column indicates the fixed context used by default by each pipeline version. Note that many intermediate development tags of the JWST Science Calibration Pipeline (indicated by minor version releases of the jwst package) are never installed in STScI DMS operations, but can be installed by end users. For full information on all jwst releases, see the github software version table.

Table 1. Release notes and associated versions for JWST Operations Pipeline releases

What's new in the latest build?

The latest released JWST Calibration Pipeline is build 11.3. Some highlights are provided here about major relevant changes; please refer to the build 11.3 release notes for additional details.

Please also refer to the calibration notes broken down by instrument mode in the JWST calibration status articles, and the known issues with JWST data articles for instrument-specific information.

Major pipeline updates in build 11.3

• Added new clean_showers option to the straylight step to remove residual cosmic ray showers for MIRI MRS data. See Shower and Snowball Artifacts.
• Significantly improved spectral extraction for NIRSpec BOTS data by ensuring the spectral extraction follows the curved spectral trace.
• Updated the clean_flicker_noise step to handle flat field variations for NIRISS data.
• Updated the extract_1d step for MIRI MRS; it is now no longer necessary to set the optional ifu_rfcorr parameter to apply residual fringe correction.  The 1D spectral tables now always contain columns for the extracted spectrum both with (RF_FLUX) and without (FLUX) 1D residual fringe correction applied.  Users should examine both and determine which is more appropriate for their science use case.
• Significantly improved runtime and memory usage of the jump step in calwebb_detector1. Speed gains will be particularly noticeable for time-series observations.
• Added a new algorithm  to the emicorr step for correcting striping artifacts in MIRI data due to electromagnetic interference (EMI). The new algorithm improves runtime and performance for data with fewer than 10 groups, and is most useful for LRS slitless time series observations. This new algorithm is optional in build 11.3 (see emicorr step arguments) and is under study to be made default in upcoming build 12.0.
• Fixed a series of issues that had caused inaccurate S_REGION footprints for MIRI LRS fixed-slit observations.
• Log messages created during pipeline processing are now stored in an extension of the output data files.
• Updated all pipeline software for compatibility with numpy 2.0.

Major reference file updates in build 11.3

Build 11.3 is tied to the CRDS reference file context jwst_1363.pmap. Significant reference file deliveries included in this context include:

• Updated 1D spectral extraction reference files for NIRCam grism TSO data to improve the extraction region (jwst_1327.pmap).
• Updated flat fields for NIRCam longwave imaging and grism data (jwst_1330.pmap).
• Updated multiple reference files for NIRSpec, MIRI, and NIRCam to account for continued evolution in the hot pixel population.
• New NIRSpec parameter reference files for the dark_current step that improve the accuracy of the pipeline-estimated uncertainties for faint targets (jwst_1331.pmap).
• Updated MIRI imaging flat fields to avoid masking out the LRS slit region in TA confirmation images (jwst_1338.pmap).
• Updated the MIRI imaging calwebb_image2 parameter reference file to reduce the amount of residual artifacts from the background subtraction step (jwst_1344.pmap).
• Multiple deliveries in support of upcoming NIRCam DHS observing mode.

Coming soon

The next JWST pipeline builds are build 12.0 (release candidate expected in Summer 2025 and operations build installation Fall 2025) and build 12.1 (release candidate expected Fall 2025). The following list indicates some of the areas in which users can expect to see changes and updates in the coming months.

Pipeline development

Ongoing JWST Science Calibration Pipeline development work in builds 12.0 and 12.1 includes:

• Better side reference pixel correction (reducing 1/f noise) for all NIR instruments.
• Initial implementation of PSF-based spectral extraction
• Improved NIRSpec MOS barshadow correction
• Support for NIRCam dispersed Hartmann sensor observations
• Revisions to data product formats for NIRCam/NIRISS WFSS along with initial support for MIRI WFSS
• Revisions to TSO data product formats and updates to the calwebb_tso3 pipeline

Reference files

Major reference file updates currently planned for build 12.0 include:

• TBD

Reference files

In the early phases of the JWST mission, reference file contexts updated frequently and data within MAST processed with a given build could thus have used different reference files.

As of build 11.1 (December 2024), JWST now uses a fixed CRDS reference file context for each pipeline build (see the CRDS overview and CRDS Migration to Quarterly Calibration Updates).

Ephemeris information

Note that ephemeris information (the 3-dimentional position and velocity as a function of time) for JWST can evolve between the initial processing of a dataset available through MAST and subsequent reprocessing of that data. The Flight Dynamics Facility (FDF) group at Goddard evaluates the ephemeris for JWST's orbit and releases a weekly update that includes the actual ephemeris up to a specific date, and a predicted ephemeris that is used for planning and executing upcoming observations. Both are stored in a database within minutes of the files being delivered by FDF, and the database is accessed by level 1 data processing in the operations pipeline. Subsequent weekly deliveries by FDF will update the predicted ephemeris database values.

To populate the ephemeris headers in the JWST data products, the JWST Data Management System (DMS) uses the definitive ephemeris values, or if they are not yet available, they use the predicted ephemeris values. Most data are initially processed with the predicted ephemeris values, while later reprocessing will use the definitive values. The header keyword EPH_TYPE lists if Definitive or Predicted was used. The ephemeris files (both definitive and predicted) are also ingested into MAST. The ephemeris is used for aberration correction in the WCS parameters in the science headers, so all science types are affected.

References

Bushouse, H., et al. 2023, Zenodo 
JWST Calibration Pipeline