JWST Calibration Pipeline Caveats

Content Migration

WARNING: This page has been deprecated and the content migrated to Known Issues with JWST Data.  Please update your links accordingly.

JWST Science Calibration Pipeline software is under continual development and improvement. At any time, there may be limitations to the pipeline products for certain instrument modes and science data types. This article provides an overview and links to other articles as more information becomes available.

At this early stage of the mission, with the current operational pipeline and available reference files, calibration products may show a variety of known issues that produce poor registration of images, double spectral lines, or other effects that may limit their usefulness for scientific analysis. Also, users should note that early data available from MAST may have been processed with ground data or model-based reference files; science data will require reprocessing over time to produce the best science-quality results. These issues are being addressed as quickly as possible. Instrument teams will be working in the next weeks to finish updating the reference files with in-flight data obtained during commissioning and later on with data obtained during Cycle 1 observations.

The development team will be working on updates to the JWST Science Calibration Pipeline and will provide information on plans for these fixes in this article: JWST Operational Pipeline Build Information. In the mean time, users should refer to the articles below and contact the JWST Help Desk as needed for assistance.


Tracking the pedigree of data products 

Two important pieces of information about your data should be included in your publications:

  1. The CAL_VER, CRDS_CTX, and SPD_VER keyword values of your JWST data.
    1. CAL_VER records the JWST Science Calibration Pipeline version used to process data (e.g., "1.4.6").
    2. CRDS_CTX records the context of calibration reference data, in the JWST OPS server, that was used to process data (e.g., "jwst_0857.pmap").
    3. SPD_VER records  the JWST Science Data Processing subsystem (SDP) version that was used to generate the raw ("uncal") products.

  2. Publications using data reprocessed locally and calibrated before November 10, 2022, should also provide the CRDS server used to retrieve calibration reference data for reprocessing the data: CRDS-PUB or CRDS-OPS. This is needed because these servers may provide different reference files versions. As of November 10, 2022, the PUB server should not be used. It was fully decommissioned on March 1, 2023.

This information is helpful for future researchers who wish to reproduce the results and retrospectively assess possible calibration issues or reprocessing needs.

Check the JDox General Pipeline Processing Caveats page for known issues in these or subsequent versions. If an issue that impacts your science was addressed in a later version, download the latest version of the data or reprocess the data locally.


On this page

See also: JWST Operational Pipeline Build Information

The JWST Science Calibration Pipeline software performs nominal processing of JWST data for the instruments and instrument modes, making science-ready data products available to users. The pipeline software is expected to be dynamic, with the scheduling of new software builds as experts implement improvements to the current algorithms or address features and artifacts found during in-flight operations. However, even with these improvements, it is impossible to accommodate certain specialized data processing schemes within the context of a standard pipeline processing system, and so, some users may need to adapt various parameters in the software to their particular needs.

The articles mentioned below attempt to convey some of the limitations in standard processing and reference files used, and suggest options that users of various observing modes need to know as they consider the best way to optimize their JWST data processing. They also provide information about the schedule for software updates that are planned to address these limitations, along with known bugs that may be encountered when analyzing data. These articles will be updated regularly as new information becomes available about JWST data.

Check the "Latest Update" box at the bottom of each article to see when the last article updates were made.



JWST Operational Pipeline build information

The JWST Science Calibration Pipeline used by the Mikulski Archive for Space Telescopes (MAST) to calibrate all JWST data is a subsystem of the JWST Operational Pipeline. It is updated on a regular basis. These updates are only pushed to MAST when there are builds of the operational pipeline, which usually include changes to other subsystems, about every 3 months. In some cases, users might see patch releases in-between builds of the JWST Operational pipeline. While these patches may not include changes to the JWST Science Calibration Pipeline, they still impact the calibration products.

Please refer to this article, JWST Operational Pipeline Build Information, for specific information about current and previous calibration pipeline software build deliveries, as well as known issues or shortcomings of each build and planned enhancements.



Non-instrument or mode-specific pipeline caveats 

This article covers pipeline caveats for general data processing that may be applicable to all or multiple instruments and modes: JWST General Calibration Pipeline Caveats



Instrument mode-specific pipeline caveats 

The articles listed below provide information on pipeline caveats for several different instruments and modes:

Additional articles will be available at a later date.



Status of reference files informed by in-flight performance

Each stage of the JWST Science Calibration Pipeline uses a set of instrument-specific reference files that ensure the science calibration pipeline meets its accuracy requirements. Before launch, instrument teams used dummy or ground testing data to support development of the science calibration pipeline stages. During commissioning, teams started updating the reference files using information learned from in-flight performance. This process is ongoing and not all reference files were updated from their pre-flight versions prior to the end of commissioning. Data downloaded from MAST that were calibrated with pre-flight reference files may suffer from poor calibration accuracy.

Users are encouraged to check whether the data downloaded from MAST have been processed with reference files informed by in-flight performance. Calibration reference files are stored and managed within the Calibration Reference Data System (CRDS) and used to reprocess the affected data. Users can check the reference file context for the processing of their data using the CRDS_CTX header keyword. The table below provides the earliest CRDS context version number associated with all in-flight reference files for each observing mode. If the CRDS context in the CRDS_CTX  header keyword is an earlier version, users are encouraged to re-run the science calibration pipeline with the latest CRDS context version, as described in this science calibration pipeline page


Table 1. Minimum CRDS context versions that use reference files informed by in-flight data

MIRI
Observing modeCRDS context in JWST OPS server
Imaging (FASTR1 readout; FULL frame)jwst_0913.pmap
Imaging (SLOWR1 readout; subarrays)reference file updates in progress
Coronagraphic imagingjwst_1058.pmap
Medium resolution spectroscopy (FASTR1 readout)jwst_0913.pmap
Medium resolution spectroscopy (SLOWR1 readout)reference file updates in progress
Low resolution spectroscopy

jwst_0927.pmap

NIRCam
Observing modeCRDS context
Imagingjwst_0995.pmap
Coronagraphic imagingreference file updates in progress
Time-series imagingjwst_0995.pmap
Grism time seriesreference file updates in progress
Wide field slitless spectroscopyreference file updates in progress
NIRISS
Observing modeCRDS context
Wide field slitless spectroscopy

jwst_0982.pmap

Single object slitless spectroscopy

jwst_0923.pmap

Aperture masking interferometryjwst_0895.pmap
Imagingjwst_0895.pmap
NIRSpec
Observing modeCRDS context
Multi-object spectroscopy

jwst_0977.pmap

This context file contains new SFLT and FFLT reference files generated by the NIRSpec team. These hybrid files have been created using the predicted pre-flight throughput combined with an instrument model. Caveat: this is not yet an updated set of flight spectrophotometric calibration files.

IFU spectroscopyjwst_0977.pmap
Fixed slit spectroscopyjwst_0977.pmap
Bright object time-series spectroscopyjwst_0977.pmap



Latest updates
  •  
    Updated minimum CRDS context for MIRI coronagraphic modes

  •  
    Added full list of current caveats pages

  •  
    Added note that CRDS PUB server will be decommissioned

  •  
    Updated CRDS context for NIRCam imaging and TSO imaging

  •  
    Added a sentence that publications should mention the server used to retrieve reference files for processing data

  •   
    Updated CRDS context for NIRISS WFSS

  •  
    Updated CRDS Context for NIRSpec

  •  
    Updated CRDS Context for NIRISS WFSS


  • Updated MIRI LRS reference file context to jwst_0927.pmap  
Originally published