JWST Data Calibration Reference Files

Content Migration

WARNING: This page has been deprecated and the content migrated to JWST Science Calibration Pipeline.  Please update your links accordingly.

The JWST Science Calibration Pipeline automatically processes data for all available observing modes, and provides science-ready calibrated data for these modes. Details about the science calibration pipeline architecture and algorithms can be found in this Read the Docs webpage.

Calibration reference file volatility

Frequent calibration pipeline and reference file updates will occur over the coming months as in-flight data is analyzed

Please refer to the section below titled "Upcoming improvements to calibration reference data" and to the JWST Calibration Pipeline Caveats articles for tracking and details on planned updates.

On this page

See also: JWST Data Calibration Considerations, JWST Science Calibration Pipeline

Each stage of the science calibration pipeline uses a set of instrument-specific reference files that ensure the science calibration pipeline meets its accuracy requirements. All calibration reference files are stored in the Calibration Reference Data System (CRDS).

Before launch, instrument teams used dummy or ground testing data to support development of the science calibration pipeline stages. Now that JWST is in-flight, data taken during commissioning and by Cycle 1 calibration programs will be used to produce the highest quality reference files. Some of these files have been already updated by the instrument teams using commissioning and Cycle 1 data; however, updates to reference files will be a continuous process through the lifetime of the mission.

For information on citing your calibration reference files for publication, see the guidelines in How to Cite JWST Data Processing Versions and Reference Files


Reference file types

The detailed software documentation contains a table with the full list of all the different reference file types that are used for various JWST instruments in the pipeline steps, as part of the science calibration pipeline stages. Each reference file includes a set of required keywords documenting the contents of the reference files, how and when these were created, and the reason for the updates. The header keyword PEDIGREE provides information about the type of data used to create the reference file. Those created before launch  can be identified by the PEDIGREE value of DUMMY, MODEL, or GROUND. After launch, these files have a PEDIGREE header keyword value of INFLIGHT, followed by the dates when the data were taken. Information on how these files were created using commissioning data, their limitations, and the effects these have on the science data are available as JWST technical reports .

The reference file types table includes links to more details about specific header keywords for each type of file, format, and the instruments for which the reference files are used.



Calibration Reference Data System

Words in bold are GUI menus/
panels or data software packages; 
bold italics are buttons in GUI
tools or package parameters.

Calibration reference files are stored and managed within the Calibration Reference Data System (CRDS). CRDS is directly integrated with calibration steps and pipeline stages via reference file mappings, also known as contexts, that are set by default to always access the best reference files for the data, according to certain selection rules (for example, instrument and filter used for an observation). Mappings provide a full view of all the reference files and rules that apply with each version. Accessing the CRDS website allows users to identify the reference files that apply to particular observing modes. The website also provides tabular information for each reference file, such as the "ACTIVATION DATE" for when these files were activated for use in the JWST Operational Pipeline, and their "PEDIGREE", indicating the type of data used to create these files. (To see these columns, click on an instrument, then on one of the reference files listed below.)

The information about the contexts used in the calibration of all science is provided in the header keyword CRDS_CTX of the calibrated data. Comparing contexts allows the identification of changes in rules and files used. 

As of November 10, 2022 all observers should use only the standard CRDS OPS server for obtaining the best JWST calibration reference files.

To setup to use the server, enter the following settings at the start of your session, before activating the jwst environment:


export CRDS_PATH=<locally-accessable-path>/crds_cache/jwst_ops export CRDS_SERVER_URL=https://jwst-crds.stsci.edu

<locally-accessable-path> can be any the user has permissions to use, such as $HOME. Note that the CRDS environment variables need to be defined before any imports, including importing jwst or crds.

For publications using data reprocessed locally and calibrated before November 10, 2022, be sure to include the name of the CRDS server along with the data reduction software versions in your publication, to ensure that the data reductions and analysis are correctly reproduced.

The science calibration pipeline pages provide more information on how to specify the desired CRDS context version when running the science calibration pipeline. The CRDS User Manual provides details on how it works and other CRDS tools for exploring the collection of reference files.

Reference files and my data

When updated calibration reference files are delivered to CRDS the data are reprocessed by the JWST Operational Pipeline and re-archived in the Mikulski Archive for Space Telescopes (MAST). This process can take some time; delivery of a reference file and activation of a context in operations does not guarantee the data in MAST have been updated. Users have the option to subscribe to a mailing list to receive notifications about deliveries of all new reference files or subscribe to be notified of when their data has been reprocessed with the updated reference files.

Subscribing to notifications about new reference files

For each instrument, users can be notified about new reference files deliveries by subscribing to a list server notification system. All you have to do is send an email command to the list server. The list server's domain name is maillist.stsci.edu. The list server "subscribe" command for each instrument is listed below: 

  • MIRI: miri_reffiles_upd-subscribe-request
  • NIRCAM: nircam_reffiles_upd-subscribe-request
  • NIRISS: niriss_reffiles_upd-subscribe-request
  • NIRSPEC: nirspec_reffiles_upd-subscribe-request
  • FGS: fgs_reffiles_upd-subscribe-request

To subscribe to notifications, send an email (with no subject and email body) using the address format <subscribe_command>@maillist.stsci.edu. For example, to subscribe to MIRI reference files deliveries, send an email to miri_reffiles_upd-subscribe-request @ maillist.stsci.edu. 

The Reference Data for Calibration and Tools Management (ReDCaT) team will authorize your request and you will receive an e-mail with information about the mailing list and how to unsubscribe.

Whenever a delivery of reference files is activated for use with the JWST Operational Pipeline, the ReDCaT Team sends a notification indicating the type of files delivered, the modes affected, and the USEAFTER dates for the files. It also includes the full list of files, the reason for the delivery, and relevant links.

Using the new context, users can recalibrate their data locally following the documentation on how to set the desired context.

If you prefer to wait for your data to be reprocessed by the operational pipeline, use the MAST Portal  to subscribe to observations of interest. You will receive notifications when recalibrated products appear in the Archive. See the Program Subscriptions and Notifications chapter of the Portal Guide for details.



Upcoming improvements to calibration reference data

Listed below are planned calibration reference data deliveries as well as planned work to assess the accuracy of the data.

Month

Description

Spring 2024

Assess how field-dependent trace shape affects NIRISS WFSS calibration accuracy.

Dec  2023

Deliver NIRCam distortion reference files for imaging modes.

Dec 2023

Deliver NIRISS reference files that no longer flag some good pixels as bad.

Dec 2023/Feb2024

Delivered a new NIRISS wavemap reference file for the extract_1d stage of the pipeline to support the NIRISS SOSS mode. However, the required calibration pipeline change to use these files will be available in February 2024. The wavelength calibration for the "wavemap" reference file was derived using vacuum wavelengths instead of air wavelengths.

Dec 2023

Deliver new NIRCam distortion reference files for  coronagraphy.

Dec 2023

Deliver improved reference files for MIRI LRS

Winter 2024

Deliver NIRCam flux calibration reference files for time-series weak lens modes.



References 

The following table provides a list of some of the JWST Technical Reports that provide descriptions of the observations taken and the methods used to generate reference files for the calibration of JWST data using commissioning data. Reports from other instruments and modes will be listed here when these become available.

Reference FileInstrumentTechnical Report

Flat field

NIRISS

Taylor, J.; Volk, K.; Willott, C.; et al., 2022, JWST-STScI-008297
NIRISS Commissioning Results: NIS-011a Illumination Flat Fields

NIRCam

Sunnquist, B.; Willmer, C.; Brooks, B.; et al., 2022, JWST-STScI-008304
NIRCam Commissioning Results NRC-10: Flat Fields, Scattered Light, and Backgrounds

Photom

NIRISS imaging

Volk, K. and the NIRISS Team, 2022, JWST-STScI-008269
NIRISS Commissioning Results: Imaging Photometric Calibration

NIRISS SOSSKevin Volk and the NIRISS team, 2022, JWST-STScI-008270
NIRISS Commissioning Results: NIRISS Single Object Slitless Spectroscopy Photometric Calibration




Latest updates
  •  
    Updated upcoming improvements to the calibration reference data table with the new planned dates


  • Updated the upcoming improvements table to remove delivered files and update dates. Updated Warning box.


  • Updated to remove files delivered last week and added MIRI files coming next month

  •  
    Updated incoming updates to calibration reference data table.


  • Added a link on how to cite JWST data and reference files
    Added a list of technical reports on calibration reference files from commissioning  


  • Added note that CRDS PUB will be decommissioned

  • added information about subscription to mailing list for updates to reference files

  •  
    Included explanation about the differences between the CRDS JWST OPS and PUB servers


  • Included information about status of reference files after commissioning.
    How to know which reference files were used with your data.
    How to subscribe to MAST to be notified about new reference files


  • Included more information about CRDS and contexts
Originally published