CRDS Migration to Quarterly Calibration Updates
In October 2024, as part of the development of build 11.1, STScI changed the way in which CRDS reference file contexts are managed by the JWST Science Calibration Pipeline. The default reference file context used by the JWST operations pipeline will now be updated quarterly and will be associated with specific pipeline builds. New reference files will continue to be available immediately for offline use.
On this page
Overview
Prior to October 2024, the JWST Science Calibration Pipeline automatically used the latest reference files available in CRDS by default. While this ensured that users always had the latest calibration data, this also caused many problems. In addition to results changing unexpectedly for users each time new reference files were delivered, crashes or bad results could also occur if new reference files were incompatible with previous versions of the pipeline software. Efforts to mitigate such incompatibilities often meant that the latest software and reference files could not be released to the astronomical community for months after they were first produced.
To solve these problems, STScI is migrating to quarterly calibration updates. This migration will allow CRDS to be fully treated as a dependency by the JWST Science Calibration Pipeline and version controlled accordingly. By default, each quarterly version of the pipeline will now use a fixed CRDS context (known as a "build context") with all reference files supporting that version. Users running the operational version of the pipeline (see JWST Operations Pipeline Build Information) should thus always get the same results as available in MAST, regardless of any recent updates that may have been made to the calibration files. Likewise, candidate builds and associated reference files will be fully available to the user community immediately. Newer reference files released with the latest contexts will be available immediately for offline use, but require specific selection of the corresponding CRDS context and will not be used by default until the next quarterly pipeline build is released (see the table of JWST pipeline builds).
Similarly, a single CRDS context will be used for processing and reprocessing of all data available in MAST for each quarterly build installation. This new processing schema will go into effect after the DMS build 11.1 installation into operations in December 2024, and will only use reference files under the designated “build” context associated with jwst software version 1.16.0. In preparation for this change, the CRDS reference file context "jwst_1293.pmap" for build 11.0 (Science Calibration Pipeline version 1.15.1) was fixed on September 26, 2024.
Frequently asked questions:
Does this retroactively affect existing pipeline installations?
- Yes; CRDS has been updated with a context mapping for all builds prior to build 11.1 as well.
- This context mapping will work with old pipeline versions if users update their CRDS installation in the appropriate Conda environment (
pip install crds --upgrade
). However, users are recommended to instead install the latest version of the jwst calibration pipeline to ensure that they get the latest software.
When will MAST data products reflect this update?
- Once build 11.1 is installed as an operational build at STScI in December 2024, both new data and the entirety of the MAST data archive will be reprocessed with this build and the corresponding reference files. This processing will take a few weeks to complete, and will be repeated with each new build. Data will not be processed with new reference files under the 'Latest' context until the next build installation.
When should I update my pipeline software?
- Generally, users should update their pipeline installation roughly quarterly. While older versions will continue to work for a period of months or years, updating ensures use of the latest calibration software and reference files.
- As described by JWST Operations Pipeline Build Information, 'operations' builds are the safest and most generally recommended versions while 'candidate' builds provide users with the latest supported version.
How do I use the latest release candidate build?
- As described by JWST Operations Pipeline Build Information, candidate builds represent the latest stable version of the JWST calibration pipeline. This candidate build is typically undergoing testing prior to being used in operations at STScI. Users can install and run this version in the same manner as any other build, and it will default to using the latest reference files available at the time the candidate build was completed.
How do I use the absolute latest versions of calibration reference files?
- By default the pipeline will select the CRDS context with reference files known to be compatible with the pipeline software in use. Users can override this behavior to use specific more recent contexts using the instructions available here. However, users should be aware that using a context other than the default context for a given software version can result in errors or incorrect results due to possible incompatibilities between software and reference file versions.
- CRDS will now feature a banner at the top of the website homepage that lists the build context and associated jwst calibration version currently being used in operations. For versions of the jwst software not actively being used by operations, the full mapping between jwst software version and default context can be accessed in the lower section of the banner (“List of build contexts: build contexts”). Users are encouraged to utilize this table to verify the context they are using is compatible with the version of jwst software selected.
Other issues
- New features and processes are continually being developed to support this transition to CRDS dependency-based processing. Transparency and communication with the user community are main priorities during this process. Any additional questions or concerns that have not been addressed here should be submitted to the JWST Help Desk.