JWST Data Calibration Reference Files
The JWST Science Calibration Pipeline automatically processes data for all available observing modes, and provides science-ready calibrated data for these modes. Details about the science calibration pipeline architecture and algorithms can be found in this Read the Docs webpage.
Each stage of the science calibration pipeline uses a set of instrument-specific reference files that ensure the science calibration pipeline meets its accuracy requirements. All calibration reference files are stored in the Calibration Reference Data System (CRDS).
Before launch, instrument teams used dummy or ground testing data to support development of the science calibration pipeline stages. Now that JWST is in-flight, data taken during commissioning and by Cycle 1 calibration programs will be used to produce the highest quality reference files. Some of these files have been already updated by the instrument teams using commissioning data. More will be coming in the following weeks. The updates to reference files, however, is a continuous process through the lifetime of the mission. Users should follow updates to these files by checking the deliveries to CRDS or subscribing to the mailing lists below.
Reference file types
The detailed software documentation contains a table with the full list of all the different reference file types that are used for various JWST instruments in the pipeline steps, as part of the science calibration pipeline stages. Each reference file includes a set of required keywords documenting the contents of the reference files, how and when these were created, and the reason for the updates. The header keyword
PEDIGREE provides information about the type of data used to create the reference file. Those created before launch can be identified by the
PEDIGREE value of
GROUND. After launch, these files have a
PEDIGREE header keyword value of
INFLIGHT, followed by the dates when the data were taken.
The reference file types table includes links to more details about specific header keywords for each type of file, format, and the instruments for which the reference files are used.
Calibration Reference Data System
Words in bold are GUI menus/
panels or data software packages;
bold italics are buttons in GUI
tools or package parameters.
The information about the contexts used in the calibration of all science is provided in the header keyword
CRDS_CTX of the calibrated data. Comparing contexts allows the identification of changes in rules and files used.
As of November 10, 2022 all observers should use only the standard CRDS OPS server for obtaining the best JWST calibration reference files.
Previously, there were 2 servers available for retrieving calibration reference data:
- JWST OPS (or CRDS-OPS) to support the JWST Operational Pipeline;
- JWST PUB (or CRDS-PUB) to support the latest public release of the jwst calibration pipeline package, which may contain code updates or changes to the reference files that have not been merged into the operational pipeline yet.
The CRDS PUB server was set up in anticipation of rapid reference file updates during commissioning and Cycle 1; however, due to the trouble-free commissioning process and the smooth transition to science operations, it has been decided that the PUB server is no longer needed and it will be decommissioned on March 1, 2023. The institute will retain an internal archive of the CRDS PUB server maps and calibration reference files. Observers who wish to reprocess data using these early CRDS PUB reference files will be able to request those reference files via a JWST Pipeline help desk ticket.
For publications using data reprocessed locally and calibrated before November 10, 2022, be sure to include the name of the CRDS server along with the data reduction software versions in your publication, to ensure that the data reductions and analysis are correctly reproduced.
For observers who are transitioning out of using the CRDS PUB server, we recommend that you rebuild your local cache by first removing all local CRDS caches and then repointing to the CRDS OPS server (https://jwst-crds.stsci.edu/). This will eliminate any possible confusion in the calibration that can lead to errors or cause the pipeline to crash.
The science calibration pipeline pages provide more information on how to specify the desired CRDS context version when running the science calibration pipeline. The CRDS User Manual provides details on how it works and other CRDS tools for exploring the collection of reference files.
Reference files and my data
When updated calibration reference files are delivered to CRDS the data are reprocessed by the JWST Operational Pipeline and re-archived in the Mikulski Archive for Space Telescopes (MAST). This process can take some time; delivery of a reference file and activation of a context in operations does not guarantee the data in MAST have been updated. Users have the option to subscribe to a mailing list to receive notifications about deliveries of all new reference files or subscribe to be notified of when their data has been reprocessed with the updated reference files.
Subscribing to notifications about new reference files
For each instrument, users can be notified about new reference files deliveries by subscribing to a list server notification system. All you have to do is send an email command to the list server. The list server's domain name is maillist.stsci.edu. The list server "subscribe" command for each instrument is listed below:
- MIRI: miri_reffiles_upd-subscribe-request
- NIRCAM: nircam_reffiles_upd-subscribe-request
- NIRISS: niriss_reffiles_upd-subscribe-request
- NIRSPEC: nirspec_reffiles_upd-subscribe-request
- FGS: fgs_reffiles_upd-subscribe-request
To subscribe to notifications, send an email (with no subject and email body) using the address format <subscribe_command>@maillist.stsci.edu. For example, to subscribe to MIRI reference files deliveries, send an email to miri_reffiles_upd-subscribe-request @ maillist.stsci.edu.
The Reference Data for Calibration and Tools Management (ReDCaT) team will authorize your request and you will receive an e-mail with information about the mailing list and how to unsubscribe.
Whenever a delivery of reference files is activated for use with the JWST Operational Pipeline, the ReDCaT Team sends a notification indicating the type of files delivered, the modes affected, and the
USEAFTER dates for the files. It also includes the full list of files, the reason for the delivery, and relevant links.
Using the new context, users can recalibrate their data locally following the documentation on how to set the desired context.
If you prefer to wait for your data to be reprocessed by the operational pipeline, use the MAST Portal to subscribe to observations of interest. You will receive notifications when recalibrated products appear in the Archive. See the Program Subscriptions and Notifications chapter of the Portal Guide for details.