JWST Calibration Pipeline Caveats
Content Migration
WARNING: This page has been deprecated and the content migrated to Known Issues with JWST Data. Please update your links accordingly.
JWST Science Calibration Pipeline software is under continual development and improvement. At any time, there may be limitations to the pipeline products for certain instrument modes and science data types. This article provides an overview and links to other articles as more information becomes available.
At this early stage of the mission, with the current operational pipeline and available reference files, calibration products may show a variety of known issues that produce poor registration of images, double spectral lines, or other effects that may limit their usefulness for scientific analysis. Also, users should note that early data available from MAST may have been processed with ground data or model-based reference files; science data will require reprocessing over time to produce the best science-quality results. These issues are being addressed as quickly as possible. Instrument teams will be working in the next weeks to finish updating the reference files with in-flight data obtained during commissioning and later on with data obtained during Cycle 1 observations.
The development team will be working on updates to the JWST Science Calibration Pipeline and will provide information on plans for these fixes in this article: JWST Operational Pipeline Build Information. In the mean time, users should refer to the articles below and contact the JWST Help Desk as needed for assistance.
Tracking the pedigree of data products
Two important pieces of information about your data should be included in your publications:
- The
CAL_VER,CRDS_CTX, andSPD_VERkeyword values of your JWST data.CAL_VERrecords the JWST Science Calibration Pipeline version used to process data (e.g., "1.4.6").CRDS_CTXrecords the context of calibration reference data, in the JWST OPS server, that was used to process data (e.g., "jwst_0857.pmap").SPD_VERrecords the JWST Science Data Processing subsystem (SDP) version that was used to generate the raw ("uncal") products.
- Publications using data reprocessed locally and calibrated before November 10, 2022, should also provide the CRDS server used to retrieve calibration reference data for reprocessing the data: CRDS-PUB or CRDS-OPS. This is needed because these servers may provide different reference files versions. As of November 10, 2022, the PUB server should not be used. It was fully decommissioned on March 1, 2023.
This information is helpful for future researchers who wish to reproduce the results and retrospectively assess possible calibration issues or reprocessing needs.
Check the JDox General Pipeline Processing Caveats page for known issues in these or subsequent versions. If an issue that impacts your science was addressed in a later version, download the latest version of the data or reprocess the data locally.
On this page
See also: JWST Operational Pipeline Build Information
The JWST Science Calibration Pipeline software performs nominal processing of JWST data for the instruments and instrument modes, making science-ready data products available to users. The pipeline software is expected to be dynamic, with the scheduling of new software builds as experts implement improvements to the current algorithms or address features and artifacts found during in-flight operations. However, even with these improvements, it is impossible to accommodate certain specialized data processing schemes within the context of a standard pipeline processing system, and so, some users may need to adapt various parameters in the software to their particular needs.
The articles mentioned below attempt to convey some of the limitations in standard processing and reference files used, and suggest options that users of various observing modes need to know as they consider the best way to optimize their JWST data processing. They also provide information about the schedule for software updates that are planned to address these limitations, along with known bugs that may be encountered when analyzing data. These articles will be updated regularly as new information becomes available about JWST data.
Check the "Latest Update" box at the bottom of each article to see when the last article updates were made.
JWST Operational Pipeline build information
The JWST Science Calibration Pipeline used by the Mikulski Archive for Space Telescopes (MAST) to calibrate all JWST data is a subsystem of the JWST Operational Pipeline. It is updated on a regular basis. These updates are only pushed to MAST when there are builds of the operational pipeline, which usually include changes to other subsystems, about every 3 months. In some cases, users might see patch releases in-between builds of the JWST Operational pipeline. While these patches may not include changes to the JWST Science Calibration Pipeline, they still impact the calibration products.
Please refer to this article, JWST Operational Pipeline Build Information, for specific information about current and previous calibration pipeline software build deliveries, as well as known issues or shortcomings of each build and planned enhancements.
Non-instrument or mode-specific pipeline caveats
This article covers pipeline caveats for general data processing that may be applicable to all or multiple instruments and modes: JWST General Calibration Pipeline Caveats
Instrument mode-specific pipeline caveats
The articles listed below provide information on pipeline caveats for several different instruments and modes:
- Known Issues with JWST Data Products
- JWST Coronagraphic Pipeline Caveats
- JWST MIRI MRS Pipeline Caveats
- JWST MIRI LRS Pipeline Caveats
- JWST Aperture Masking Interferometry Pipeline Caveats
- JWST NIRSpec IFU Pipeline Caveats
- JWST NIRSpec MOS Pipeline Caveats
- JWST Wide Field Slitless Spectroscopy Pipeline Caveats
- JWST Time Series Observations Pipeline Caveats
Additional articles will be available at a later date.
Status of reference files informed by in-flight performance
Each stage of the JWST Science Calibration Pipeline uses a set of instrument-specific reference files that ensure the science calibration pipeline meets its accuracy requirements. Before launch, instrument teams used dummy or ground testing data to support development of the science calibration pipeline stages. During commissioning, teams started updating the reference files using information learned from in-flight performance. This process is ongoing and not all reference files were updated from their pre-flight versions prior to the end of commissioning. Data downloaded from MAST that were calibrated with pre-flight reference files may suffer from poor calibration accuracy.
Users are encouraged to check whether the data downloaded from MAST have been processed with reference files informed by in-flight performance. Calibration reference files are stored and managed within the Calibration Reference Data System (CRDS) and used to reprocess the affected data. Users can check the reference file context for the processing of their data using the CRDS_CTX header keyword. The table below provides the earliest CRDS context version number associated with all in-flight reference files for each observing mode. If the CRDS context in the CRDS_CTX header keyword is an earlier version, users are encouraged to re-run the science calibration pipeline with the latest CRDS context version, as described in this science calibration pipeline page.
Table 1. Minimum CRDS context versions that use reference files informed by in-flight data
| MIRI | |
|---|---|
| Observing mode | CRDS context in JWST OPS server |
| Imaging (FASTR1 readout; FULL frame) | jwst_0913.pmap |
| Imaging (SLOWR1 readout; subarrays) | reference file updates in progress |
| Coronagraphic imaging | jwst_1058.pmap |
| Medium resolution spectroscopy (FASTR1 readout) | jwst_0913.pmap |
| Medium resolution spectroscopy (SLOWR1 readout) | reference file updates in progress |
| Low resolution spectroscopy | jwst_0927.pmap |
| NIRCam | |
| Observing mode | CRDS context |
| Imaging | jwst_0995.pmap |
| Coronagraphic imaging | reference file updates in progress |
| Time-series imaging | jwst_0995.pmap |
| Grism time series | reference file updates in progress |
| Wide field slitless spectroscopy | reference file updates in progress |
| NIRISS | |
| Observing mode | CRDS context |
| Wide field slitless spectroscopy | jwst_0982.pmap |
| Single object slitless spectroscopy | jwst_0923.pmap |
| Aperture masking interferometry | jwst_0895.pmap |
| Imaging | jwst_0895.pmap |
| NIRSpec | |
| Observing mode | CRDS context |
| Multi-object spectroscopy | jwst_0977.pmap† † This context file contains new SFLT and FFLT reference files generated by the NIRSpec team. These hybrid files have been created using the predicted pre-flight throughput combined with an instrument model. Caveat: this is not yet an updated set of flight spectrophotometric calibration files. |
| IFU spectroscopy | jwst_0977.pmap |
| Fixed slit spectroscopy | jwst_0977.pmap |
| Bright object time-series spectroscopy | jwst_0977.pmap |