JWST Operations Pipeline Build Information
Information about what's new in the latest JWST pipeline build, and what was done in previous builds, is provided in this article. It also gives a preview of some updates likely to be coming soon in upcoming builds. The information contained here is generally a high-level summary of changes, with links provided to more extensive external documentation.
On this page
Build lifecycle
JWST pipeline development proceeds according to a quarterly build schedule. Each quarter, the active development version of the JWST Science Calibration Pipeline is frozen, combined with other dependent software (e.g., the JWST Science Data Processing system; SDP) and a particular CRDS reference file context, and tagged to produce a candidate build. This candidate build is then tested extensively prior to becoming an operations build installed by the Data Management System (DMS) at STScI and used to reprocess all previous JWST data as well as newly obtained observations. Once a build is no longer operational, it becomes archived.
There are thus four kinds of builds:
- Development (Dev) build: Changing daily, uses the Latest CRDS context, not recommended for non-developers.
- Candidate build: Development has finished, currently working on testing and documentation. Uses a fixed CRDS context. Available for experienced users.
- Operations build: Installed at STScI and being used to produce data products available in MAST. Uses a fixed CRDS context. Default recommendation for users.
- Archived build: Outdated operations build that is no longer recommended for general use (may still correspond to some MAST data products until reprocessing is completed).
Record of JWST Operations Pipeline builds
Due to breaking changes introduced in the gwcs dependent package in late December 2024, users wishing to install builds 11.0 or 11.1 should also downgrade their gwcs version using this command: pip install gwcs==0.21.0
Table 1 provides a historical record of JWST operations builds, along with links to the build release notes and tagged software versions. The "CRDS Context" column indicates the fixed context used by default by each pipeline version. Note that many intermediate development tags of the JWST Science Calibration Pipeline (indicated by minor version releases of the jwst package) are never installed in STScI DMS operations, but can be installed by end users. For full information on all jwst releases, see the github software version table.
Table 1. Release notes and associated versions for JWST Operations Pipeline releases
Operations Pipeline | Science Calibration | Build Status | SDP_VER | Candidate Released | Operations Installation | CRDS context |
|---|---|---|---|---|---|---|
| 11.3 | TBD | Development | TBD | Estimated Spring 2025 | Estimated Summer 2025 | TBD |
| 11.2 | Candidate | TBD | 2025-01-03 | Estimated Mar 2025 | 1321 | |
| 11.1.1 | Operations | 2024.3.1 | 2024-09-20 | 2024-12-03 | 1303 | |
| 11.0.1 | Archived | 2024.2.1 | 2024-06-26 | 2024-08-27 | 1293 |
What's new in the latest build?
The latest JWST Calibration Pipeline in use is build 11.1. Some highlights are provided here about major relevant changes; please refer to the build 11.1 release notes for additional details.
Please also refer to the calibration notes broken down by instrument mode in the JWST calibration status articles, and the known issues with JWST data articles for instrument-specific information.
Major pipeline updates in build 11.1
- Added a new clean_flicker_noise step to the calwebb_detector1 pipeline to correct for 1/f noise in data for all near-IR instrument modes. This is an optional step with a variety of different parameters to optimize performance for various science cases. For further details see 1/f Noise.
- Fix issues with flux conservation for some spectroscopic modes (especially NIRSpec MOS and FS) when resampling to non-default scales
- Updated NIRISS SOSS spectral extraction to improve performance by incorporating the previously standalone PASTASOSS package
- Fixes for cosmic rays saturating in group-2 data for non-IRS2 readout modes
- Fixed an issue with nod-flux oversubtraction for some NIRSpec slit observations with small nod distances
- Further improvements to the runtime of the ramp fitting step for NIRISS data
- Significant runtime improvements to multiple parts of the NIRSpec pipeline, impacting IFU, MOS, and BOTS data
- Improvements to memory management throughout the pipeline, reducing memory pressure for some of the largest steps (e.g., mosaic resampling) and fixing the build 11.0 memory leak in the C-based ramp fitting step
Major reference file updates in build 11.1
Due to the timing of the switch to quarterly calibration file updates, there are few new calibration files associated with build 11.1 compared to build 11.0. These changes include:
- Updated NIRSpec MOS and fixed slit outlier detection parameter reference files to avoid clipping high-SNR sources due to the improved spectral resampling.
Known Issues in build 11.1
There were two previous issues with build 11.0; a memory leak in the C-based ramp fitting step and a bug that prevented the outlier detection step from running as a standalone step outside the pipeline. Both have been fixed with build 11.1.
New issues of note with build 11.1 include:
- The ramp fitting step crashes if users enable multiprocessing for the C-based algorithm for non-NIRISS data. This will be fixed in build 11.2, but the workaround solution is to simply not use multiprocessing for the (default) C-based algorithm. The C-based ramp fitting is sufficiently fast that multiprocessing provides no significant runtime advantage for this step anyway.
- Some NIR programs using non-IRS2 grouped data can experience a small number more breakthrough snowballs in build 11.1 than in build 11.0 due to interaction between the saturation-flagging code and the snowball-detection code. A temporary workaround for any problematic exposures is to set saturation.use_readpatt = False.
- Due to improved resampling fidelity, NIRSpec MOS and FS modes can, in some cases, flag bright pixels as outliers in the outlier detection step. This has been largely mitigated by parameter reference file update, but will be fixed by build 11.2 refactor of outlier detection step.
- MIRI LRS TA confirmation image has no information in the LRS slit region in calibrated "cal.fits" files. Temporary workaround is to use "rate.fits" files for TA confirmation inspection.
- Due to a bug in the moving target WCS code, spectroscopic moving targets with associated background observations cannot be processed through the calwebb_spec3 pipeline with build 11.1. Existing archival products will continue to be based on build 11.0 pipeline reductions, while PIs of new programs upcoming in early 2025 will be contacted individually. For offline reprocessing, users are recommended to use build 11.0 or candidate build 11.2 instead.
- Some large imaging mosaics can appear blurry due to poor performance of the tweakreg step correcting the alignment of individual images. A fix is included in build 11.2 and can be replicated in build 11.1 by rerunning calwebb_image3 with the tweakreg parameters of sharplo, sharphi, roundlo, roundhi set to values of 0.5, 2.0, 0.0, and 0.2 respectively.
Coming soon
The next JWST pipeline builds are build 11.2 (release candidate expected in December 2024 and operations build installation March 2025) and build 11.3 (release candidate expected Spring 2025). The following list indicates some of the areas in which users can expect to see changes and updates in the coming months.
Pipeline development
Ongoing JWST Science Calibration Pipeline development work in builds 11.2 and 11.3 includes:
- Improvements to outlier detection for NIRSpec MOS and FS modes
- Improvements to cosmic ray shower flagging for MIRI
- Improvements to NIRSpec MOS background subtraction
- Better correction of 1/f noise using adjacent reference pixels (where available)
- Continued development and testing of the 1/f correction step
- Initial implementation of PSF-based spectral extraction
Reference files
Major reference file updates currently planned for build 11.2 include:
- MIRI MRS change to produce per-band cubes by default and enable cosmic ray shower correction
- Updated photometric calibration for NIRISS imaging
Reference files
In the early phases of the JWST mission, reference file contexts updated frequently and data within MAST processed with a given build could thus have used different reference files.
As of build 11.1 (December 2024), JWST now uses a fixed CRDS reference file context for each pipeline build (see the CRDS overview and CRDS Migration to Quarterly Calibration Updates).
Ephemeris information
Note that ephemeris information (the 3-dimentional position and velocity as a function of time) for JWST can evolve between the initial processing of a dataset available through MAST and subsequent reprocessing of that data. The Flight Dynamics Facility (FDF) group at Goddard evaluates the ephemeris for JWST's orbit and releases a weekly update that includes the actual ephemeris up to a specific date, and a predicted ephemeris that is used for planning and executing upcoming observations. Both are stored in a database within minutes of the files being delivered by FDF, and the database is accessed by level 1 data processing in the operations pipeline. Subsequent weekly deliveries by FDF will update the predicted ephemeris database values.
To populate the ephemeris headers in the JWST data products, the JWST Data Management System (DMS) uses the definitive ephemeris values, or if they are not yet available, they use the predicted ephemeris values. Most data are initially processed with the predicted ephemeris values, while later reprocessing will use the definitive values. The header keyword EPH_TYPE lists if Definitive or Predicted was used. The ephemeris files (both definitive and predicted) are also ingested into MAST. The ephemeris is used for aberration correction in the WCS parameters in the science headers, so all science types are affected.
References
Bushouse, H., et al. 2023, Zenodo
JWST Calibration Pipeline