JWST Coronagraphic Pipeline Caveats
Content Migration
WARNING: This page has been deprecated and the content migrated to Known Issues with JWST Data and calwebb_coron3. Please update your links accordingly.
Some unique features of the JWST Science Calibration Pipeline for coronagraphic imaging with NIRCam and MIRI, and caveats for users, are described in this article. The information reflects the status for the JWST Science Calibration Pipeline version 1.5.2.
On this page
Summary of specific coronagraphy issues
MIRI coronagraphic imaging
The information in this table about MIRI coronagraphic imaging calibration pipeline issues is excerpted from Known Issues with JWST Data Products.
| Symptoms | Cause | Workaround | Mitigation Plan |
|---|---|---|---|
MR-CI02: Users will want to perform their own PSF subtraction with the data products. | PSF subtraction has too many free parameters to capture them in a one-size-fits-all automated processing pipeline. | The public "Coronagraphy_ExampleNB" repository contains a suite of 6 notebooks that will guide the user through reprocessing coronagraphic data through all steps of the pipeline, as well as for customizing parameters at each step. Users will find notebooks suitable for processing "uncal" files retrieved from MAST through stage 2 to yield background-subtracted photometrically-calibrated data products suitable for use in PSF subtraction. Older notebooks are still available in the "old_pipeline_demos" subfolder. | Updated issue The MIRI team is always considering the scope of what data products the calwebb_coron3 pipeline stage should provide. |
MR-CI03: Tweak stage 1 and 2 parameters to optimize data calibration. | Default parameters may not be optimal for all datasets. | spaceKLIP, provided by the ERS team, can run pipeline steps as well as perform highly tunable PSF subtraction on the data products. There are examples of different pipeline parameter values they have found helpful, which can be found in the MIRI config file under the "tests" folder. Pipeline stages 1 and 2 are shared with the MIRI imager, so users may look there for more examples. | Created issue None at this time. |
| MR-CI04: Current Lyot flat-fielding causes a sharp discontinuity underneath the coronagraphic mask. | A sharp edge in the Lyot coronagraph's flat-fielding reference file creates a sharp discontinuity underneath the coronagraphic mask, in stage 2 pipeline products. This can result in artifacts during certain image processing steps that are sensitive to abrupt changes. | For now, users should be aware that this sharp discontinuity is an artifact of the calibration pipeline and does not represent astrophysical information. They should take this into account during post-processing. | Created issue Mitigation strategies are under investigation. |
| MR-CI01: Absolute flux calibration is incorrect. | An error was made when computing the aperture correction for coronagraphic PSFs. | Multiply fluxes by a factor that will be available by the end of August, 2023. | Updated Operations Pipeline Flux calibration "photom" reference files were updated in October. STScI will reprocess affected data products with updated calibration reference data. Reprocessing of affected data typically takes 2–4 weeks after the update. |
NIRCam coronagraphic imaging
The information in this table about NIRCam coronagraphic imaging calibration pipeline issues is excerpted from Known Issues with JWST Data Products.
| Symptoms | Cause | Workaround | Mitigation Plan |
|---|---|---|---|
| NC-CI01: Dark correction leads to worse outcome than not applying any. | On-sky darks are not high enough in SNR and lead to residuals and erroneous jumps in the "rateints" images. | Skip dark correction during calwebb_detector1. | Created issue Mitigation is not yet scheduled because the data are satisfactory without dark subtraction. |
| NC-CI02: An excessive number of pixels are flagged as outliers in some subarray data. | The coronagraph subarrays do not have reference pixels on all sides. Without a reference pixel correction, the data become noisier and the jump step in calwebb_detector1 sometimes identifies too many pixels as outliers. | The spaceKLIP open-source package (provided by the community) implements a pseudo-reference pixel correction using a few not-illuminated pixels around (at the edge of) the subarray and forces them to be reference pixels. This way, global DC offsets frame to frame are taken care of. | Updated issue Implement something similar to spaceKLIP for calwebb_coron3. Timeline TBD. |
| NC-CI03: Absolute flux calibration is incorrect. | The wavelength dependent throughput of the bar occulters are currently a copy of throughputs of the round occulters, despite a slight difference of about 2% due to the different Lyot stops. In addition, flux calibration for all masks is still from pre-flight expectations. Note: previous text here mistakenly said that the COM (coronagraph optical mount) substrate throughput was not taken into account. That is not the case. | spaceKLIP uses the spectral energy distribution (SED) from published photometry (VO table from VizieR) to assess the expected flux from the star and calibrate the contrast curves, then determines the flux and position of any point source using MCMC and a forward model of the off-axis PSF. | Updated issue Data was reprocessed with an enhanced calibration reference file (photom) in CRDS. An update made on October 31, 2023 takes the full coronagraphic throughput into account using in-flight data. Reprocessing of old data typically takes 2–4 weeks after the update. |
Coronagraphic imaging pipeline and products overview
See also: Getting Started with JWST Data, file header contents
Pipeline stages and steps
Software documentation outside JDox: Pipeline Modules, Associations
JWST coronagraphic data is processed through the pipeline in 3 stages:
The articles listed above provide additional information on the steps in each of these stages, with links to dedicated software documentation. Stages 1 and 2 are generic to imaging observations, with the coronagraphic observation-specific flow set by parameter reference files. Stage 3 of the pipeline, calwebb_coron3, performs coronagraphic-specific processing steps for both NIRCam and MIRI. The input to calwebb_coron3 pipeline is an association file, which lists the various science target and reference PSF exposures to be processed.
Data products
Software documentation outside JDox: Science Products
The pipeline produces a number of additional data products for coronagraphic imaging observations when compared with standard imaging observations. The calwebb_detector1 pipeline produces a "*_rate.fits" file but specifically also a "*_rateints.fits" file. The "rateints" file contains the slope images and other results for each individual integration in the exposure. In the "rateints" file, the science, DQ, error, and variance extensions are 3-D, rather than 2-D, products, with the 3rd dimension matching the number of integrations in the exposure. Similar to calwebb_detector1, calwebb_image2 returns "calints.fits" files, with the 2-D calibrated images for each integration stacked along the 3rd axis of the data cubes (ncols × nrows × nints).
In addition to the "crfints.fits" and "i2d.fits" products produced by other stage 3 processing modules, calwebb_coron3 produces 3 data product types that are particular to coronagraphic processing:
- psfstack: A single 3-D product containing a stack of all the PSF images from the multiple input exposures. The size of the stack will be equal to the sum of the number of integration in each input PSF exposure. The output file name is source-based, using the product name specified in the ASN file, e.g., "jw86073-a3001_t001_nircam_f140m-maskbar_psfstack.fits".
- psfalign: A 4-D data product, where the 3rd axis length is equal to the total number of reference PSF images in the input PSF stack ("_psfstack") product and the 4th axis length is equal to the number of integrations in the input science target product (ncols x nrows x npsfs x nints). Image (n,m) in the 4-D data cube is the nth PSF image aligned to the mth science target integration. The file name is exposure-based, using the input science target exposure name as the root, with the addition of the association candidate ID and the “_psfalign” product type suffix, e.g., "jw8607342001_02102_00001_nrcb3_a3001_psfalign.fits".
- psfsub: A 3-D stack of PSF-subtracted images of the science target, having the same dimensions as the input science target ("_calints") product. The PSF fitting and subtraction has been applied to each integration image independetly. The file name syntax is exposure-based, using the root of the input “_calints” product, with the addition of the association candidate ID and the “_psfsub” product type suffix, e.g. "jw8607342001_02102_00001_nrcb3_a3001_psfsub.fits".
Pipeline caveats
JWST data is processed automatically through the JWST Science Calibration Pipeline once it is transmitted to Earth from the observatory, using the sequence of steps set in the parameter files and default settings that were chosen for each instrument mode by the instrument teams, and the most up-to-date set of calibration reference files. The processed data products will be available in the Archive when you access your data, and these products are "science ready."
However, while the default settings are all well-chosen for your type of observations, they may not be optimal for your particular science measurement. Hence, there may be situations where you want to rerun the pipeline. In addition, in order to gain an in-depth understanding of the data, instrumental systematics, detector noise properties, and how these are treated in the pipeline, you may want to try different settings and parameters, and study their impact on the resulting processed data.
Pipeline reference files
Software documentation outside JDox: Calibration Pipeline Documentation
Many pipeline steps rely on the use of reference files that contain different types of calibration data or information necessary for processing the data. Calibration reference files are stored in the Calibration Reference Data System (CRDS). CRDS is directly integrated with calibration steps and pipelines, and the reference file mappings are set by default to always access the most recently delivered reference files according to certain selection rules (for example, instrument and filter used for an observation).
Parameter reference files can be overridden with custom files that edit the standard data flow. This can be useful to investigate the impact of certain calibration steps on the final measurement accuracy or precision. Instructions on how to implement overrides are provided in the calibration pipeline documentation. The pipeline documentation further contains information on saving output files from a pipeline run, and for creating logging output. Both these strategies are useful for tracking and optimizing the performance of the pipeline for your data.
The align_refs step of calwebb_coron3 uses a "PSFMASK" reference file. The role of the align_refs step is to compute offsets between science target images and reference PSF images, and shift the PSF images into alignment. The "PSFMASK" reference file contains a 2-D mask that’s used as a weight function when computing the shifts between images. The parameters to override the automatic selection of a "PSFMASK" file from CRDS and specify your own file are of the form –override_psfmask. The pipeline documentation contains information on the required keywords for PSFMASK files.