JWST MIRI LRS Pipeline Caveats
Content Migration
WARNING: This page has been deprecated and the content migrated to MIRI LRS Known Issues. Please update your links accordingly.
Features and potential caveats specific to MIRI LRS data processing in the JWST Science Calibration Pipeline are described in this article.
On this page
Summary of specific MIRI LRS pipeline issues
The information in this table about MIRI LRS pipeline issues is excerpted from Known Issues with JWST Data Products.
| Symptoms | Cause | Workaround | Mitigation Plan |
|---|---|---|---|
MR-LRS03: Spectra of bright sources can experience data dropouts at wavelengths where they are brightest (in the "x1d" files). The problem can also be seen in 2-D spectral images, with the peak column flagged as "DO_NOT_USE" in several adjacent rows where the spectrum is brightest. | The jump detection step in calwebb_detector1 can flag pixels with strong signals as "DO_NOT_USE" in the spectral images even though the data are viable. These pixels are "NaN" in the spectral images and their signal is missing from the affected wavelengths in the extracted spectra. | Users can run calwebb_detector1 themselves and adjust the rejection_threshold parameter in the jump step to eliminate (or reduce) improper flagging. A generic workaround is challenging to produce as the occurrence and severity of the issue depends on several factors, e.g., target brightness and number of groups per integration. | Updated issue The jump step algorithm and default parameters are continually being examined and optimized; improvements are expected in future builds (spring 2024 and beyond). The issue likely cannot be entirely eliminated as it differs greatly between observations (and indeed many are not affected). |
| MR-LRS04: Spectra extracted from the LRS slit ("x1d" files) can show major disparities between the nods or with expected values. | The pathloss correction is generating incorrect results due to a poor understanding of the location of the source in the slit. | The "pathloss" reference file has already been modified to apply a correction as though the source were in the center of the slit, no matter where it actually is (or where it is believed to be). Exception: If the source position and telescope pointing indicate that the target is outside of the slit, then no pathloss correction is applied. | Created issue The Science Calibration Pipeline will be modified to allow the user to rerun the pathloss correction with a default position of the center of the slit. The Operations Pipeline will be modified to determine the source position based on the TA verification image. Neither of these solutions are ready for implementation. |
| MR-LRS05: The "ERR" extension in "rate" data products (and other error estimates downstream) is incorrect. | Error values are estimated incorrectly by the pipeline. | Bootstrap uncertainty from science spectra. | Created issue The root cause of incorrect error estimates in the pipeline is being investigated. |
| MR-LRS06: Target acquisition images taken with FASTGRPAVG readout patterns are incorrectly calibrated. | The calibration pipeline does not compute the exposure time for these readout modes when fitting a slope to the uncalibrated data. | The verification images are calibrated correctly; if additional photometry is required, the MIRI team recommends the use these images. | Created issue The issue will be fixed in the calibration pipeline code in early 2024. |
| MR-LRS02: Extracted 1-D spectra ("x1d" files) often show salt-and-pepper noise (i.e., spikes and divots one pixel across). | Pixels flagged as "DO_NOT_USE" appear as "NaN" in the spectral images, and the signal in those pixels is missing from the summed signal at that wavelength in the extracted spectra. | Users can apply the new pixel_replace step in the developmental pipeline, which replaces the pixels flagged "DO_NOT_USE" with pixels in the same column by normalizing nearby spatial profiles in the 2-D spectral images. The best solution is to download the developmental program and just run it. The default for pixel_replace is to use 3 rows above and below the problematic row. This value can be changed by setting the n_adjacent_rows argument. | Updated Operations Pipeline The new pixel_replace step was implemented in the Operations Pipeline, installed on August 24, 2023. STScI reprocessed affected data products, which typically takes 2–4 weeks after the update. |
| MR-LRS01: Spectra extracted from LRS slit and slitless data ("x1d" files) can have little or no signal, or even negative signal, even though the 2-D spectral images look fine. | The pipeline is extracting the spectrum from the wrong location in the 2-D spectral images because the positions it is using to determine the location of the source are inaccurate. | Users can set the location and the width of the extraction apertures themselves when running calwebb_spec2 or calwebb_spec3 manually. A notebook demonstrating the spectral extraction capabilities of the JWST calibration pipeline for MIRI LRS is available in this repository. | Updated Operations Pipeline The pipeline was updated in May 2023 to default to apertures centered on the nominal position of the level 3 spectrum in the aperture. This change is fully in operations, data have been reprocessed. |
About LRS pipeline caveats
See also: MIRI Low Resolution Spectroscopy
Caveats specific to MIRI LRS data processing in the JWST Science Calibration Pipeline are described below. This article is not intended as a how-to guide or as full documentation of individual pipeline steps, but rather to give a scientist-level overview of any known issues that users should be aware of for their science.
MIRI LRS pipeline
Wavelength calibration
See also: JWST Science Calibration Pipeline, JWST Data Calibration Reference Files, Example Jupyter Notebooks - Running the Pipeline
Software documentation outside JDox: Stage 2 Spectroscopic Pipeline, Calibration Reference Data System
Wavelength calibration for the MIRI LRS was updated in October 2022, after an issue was uncovered in the first Cycle 1 science observations. The LRS mode provides nominal coverage from 5 to 12 µm, with the wavelength decreasing with increasing row number on the detector. The true range extends somewhat beyond these limits in both directions though the data calibration becomes more challenging outside of this nominal range. Between 4 and 5 µm, the dispersion steepens with respect to the detector pixels, making it very challenging to fit a solution; beyond 12 µm, the throughput of the prism drops off strongly, limiting the achievable signal to noise ratio. Plots of the dispersion profiles are shown on the MIRI Filters and Dispersers page.
The calibration pipeline originally used a single reference file to fit the dispersion for fixed slit and slitless data taken with the LRS. However, the dispersion was found to be more strongly variable with position in the field than expected. As the location of the spectra for these 2 modes are in different parts of the detector, separate dispersion solutions are needed for the 2 modes. The difference in dispersion between the 2 modes causes errors of up to 0.5 µm in the wavelength calibration. Once the issue was identified, new reference files were produced for both modes, providing calibration accuracies of ~0.010–0.015 µm for slitless spectroscopy, and ~0.02–0.05 µm for observations with the slit. Further observations will be taken in the course of Cycle 1 to further improve the accuracy of the LRS wavelength calibration, with a goal accuracy of 5–10 nm over the full nominal wavelength range.
Words in bold are GUI menus/
panels or data software packages;
bold italics are buttons in GUI
tools or package parameters.
It is important to note that the LRS wavelength calibration does not operate in isolation from other calibration steps. The path loss correction, flat field, photometric calibration, and extraction aperture correction all have a wavelength dependency. Following the update to the wavelength calibration in October 2022, these reference files were also updated. This process will be repeated when further updates are made to the wavelength calibration. Users are encouraged to sign up to the notification email lists to receive prompt notification when new MIRI reference files are delivered.
Users are also encouraged to reach out to the MIRI team via the JWST Helpdesk if they need further clarification on any aspect of this issue.
Spectral extraction
The spectral extraction in the JWST calibration pipeline is not reliably producing science quality spectra when run automatically, i.e., for the products users will find in MAST. Two separate issues affecting the quality of the extracted products have been identified, as explained below.
Errors in source location
This issue particularly affects LRS slit data, where the source is typically moved between 2 different nod positions. The extract_1d step in the calwebb_spec2 pipeline uses the coordinate information of the observation to identify the location of the spectrum in the science aperture, rather than by having the position hard-coded or by applying a source-finding algorithm. This means that any inaccuracies or errors in the coordinate registration of the data will result in the extraction being performed in the wrong part of the aperture, despite the spectrum being in the expected position. If the "x1d" or "x1dints" products do not show the expected spectral shape for a target, despite the spectral images (the "cal" or "s2d" products) showing good quality spectra, this is the likely explanation.
Errors in coordinate registration can occur when guide stars have inaccurate coordinates in the catalog, or if the wrong guide star is identified during the Fine Guidance Sensor's ID process. (The risk of this increases with field crowding.) As long as the offset is small and the target still lands in the target acquisition region of interest, the TA sequence will compensate for such errors, placing the target at the correct position in the slit or the subarray. The assigned coordinates will, however, be offset from what was expected in those cases, and the extraction algorithm will (attempt to) extract the spectrum from the wrong location. As the slit measures just ~4.7" × 0.5", even small coordinate offsets can cause this issue.
The parameters used in the extract_1d step are defined in a JSON-formatted reference file that is available in the CRDS database. As of May 2023, an updated version of this file delivered to CRDS provides a workaround for the coordinate misplacement issue. The new reference file defines the xstart and xstop values to match the expected location of the spectral trace in the nod-combined, resampled level 3 spectral image (the "s2d.fits" product), and explicitly sets the use_source_posn keyword to False, which prevents the pipeline from computing coordinate-based offsets. This strategy should deliver much improved extracted level 3 spectra. We do note that this change will worsen the quality of the level 2b automated extractions, as the geometry of the level 2b spectral images is different from the nod-combined, resampled product.
If extracted spectra from each nod position individually at the level 2b stage are required, such extractions can be accomplished by re-running the extract_1d step offline using a custom "extract_1d" reference file that has been modified to specify the desired extraction limits within the image. This is described in the JWST calibration pipeline documentation. The workaround method is demonstrated in this example notebook, as well as in the materials of the TSO JWebbinar (see video, notebook); both of these demonstrations include examples of custom reference files, and how the pipeline can be instructed to use a custom file.
Bad pixel spikes and dropouts
A second issue with the spectral extraction is bad pixel spikes and dropouts in the extracted spectrum. These appear as single-point dips or spikes in the spectrum that can at times mimic absorption or emission features. The cause of these features is the presence of a flagged or hot pixel in the extraction aperture. When a pixel has been flagged as "DO_NOT_USE" by the pipeline, due to it being a known bad or hot pixel, or being affected by a cosmic ray, the pipeline excludes it from the summation of flux in the extraction aperture. It does not interpolate its value from its neighbouring pixels. This causes the extracted flux along this row in the aperture to drop compared to the rows above and below, i.e., causing a dip. Similarly, hot pixels are included in the summation as they are, causing spikes in the spectrum. This issue affects the cosmetics of the extracted spectrum, and, if located on or near a physical spectral feature, can interfere with the analysis.
As of May 2023, a new step was included in the JWST calibration pipeline, pixel_replace, that will identify these pixels and replace them with an interpolated value. The step is documented in the JWST calibration pipeline documentation. This is particularly valuable for non-dithered observations such as time-series observations with LRS in slitless mode.
Background subtraction (slitless mode)
The JWST calibration pipeline does not currently perform any background subtraction on the spectral images taken with the slitless LRS mode. Work is being done to implement an optimized algorithm for this. Based on experience with commissioning and early science data, the following strategies are recommended:
- A very simple background subtraction can be performed as part of the spectral extraction in the extract_1d step of the calwebb_spec2 pipeline. By creating a custom reference file for the extract_1d step, an off-target background region can be defined alongside the source region, from which a background is computed. This is demonstrated in the materials of the TSO JWebbinar (see video, notebook).
- A better result is obtained by manually computing a background spectrum from off-target regions of the subarray in the "rateints.file", for each individual integration. The best regions of the subarrays from which the background is computed are those at the edges of the subarray (excluding the first 4 columns, which contain the reference pixels). By median-combining different off-target regions to compute an effective background, a 2-D background image can be constructed, and subtracted from the science data. If this operation is performed on the "rateints" product, the same file can then be processed further through the calwebb_spec2 pipeline.