MIRI LRS Known Issues

Known issues specific to MIRI LRS data processing in the JWST Science Calibration Pipeline are described in this article. This is not intended as a how-to guide or as full documentation of individual pipeline steps, but rather to give a scientist-level overview of issues that users should be aware of for their science. 

On this page

Specific artifacts are described in the Artifacts section below. Guidance on using the pipeline data products is provided in the Pipeline Notes section along with a summary of some common issues and workarounds in the summary section.

Please also refer to MIRI LRS Calibration Status for an overview of the current astrometric and flux calibration of MIRI LRS data products.



Artifacts

Scattered light in the LRS

See also: MIRI Low Resolution Spectroscopy

Words in bold are GUI menus/
panels or data software packages; 
bold italics are buttons in GUI
tools or package parameters.

If a very bright target is present in the imager field, its light may scatter into the region where the LRS slit is dispersed. Scattered light artifacts are seen in the detector substrate around bright sources (see the cruciform artifact), so these features can persist despite the presence of a focal plane mask. As the pipeline extracts a cutout region from the array in the later stages of the stage 2 pipeline (the photom and resample_spec steps), it's recommended that users inspect level 2a data (the "rate.fits" file) to check for the presence of such contaminating sources.

Figure 1. Light scattered from the imager field of view into the LRS spectral location

 
Image of a bright source in the imager FOV during an LRS observation. Light from the bright source is dispersed and scatters into the LRS slit spectrum. Image section (a) shows the entire imager FOV, with the slit marked in green, the TA ROI in yellow, and the zoomed region in dashed white. (b) shows the zoomed region from the full FOV. (c) shows the level s2d pipeline product, which shows both LRS along-slit nods subtracted (resulting in the negative and positive spectral images). As mentioned above, the pipeline does not automatically remove this scattered light contamination.

LRS slitless spectral foldover and leak

As both slit and slitless modes use the same dispersing element, the dispersion profile is similar for both, modulo changes due to optical distortion. The nominal spectral range of 5–12 µm is dispersed over ~370 pixels. The dispersion profile however folds over below 4.5 µm (where the prism throughput is very low), superimposing 2 parts of the spectrum onto each other. A dedicated filter is mounted over the slit to block radiation shortward of 4.5 µm to avoid this contamination in the slit. This effect is not mitigated for LRS in slitless mode, causing some spectral contamination at the shortest wavelengths. 



Pipeline notes

Wavelength calibration

See also MIRI LRS Calibration Status

Wavelength calibration for the MIRI LRS was updated in December 2023 for the slit. The last update for slitless was in October 2022.  With these updates, the calibration should be good to ±20 nm in both, with possibly slightly higher discrepancies at the shortest wavelengths. The LRS mode provides nominal coverage from 5 to 12 µm, with the wavelength decreasing with increasing row number on the detector. Below 5 µm, the dispersion steepens with respect to the detector pixels, making it very challenging to fit a solution.  The true range extends to 14 µm, but beyond 12 µm, the throughput of the prism drops off strongly, limiting the achievable signal-to-noise ratio. Plots of the dispersion profiles are shown on the MIRI Filters and Dispersers article. 

At launch, the calibration pipeline used a single reference file to fit the dispersion for fixed slit and slitless data taken with the LRS. However, the dispersion was found to vary more strongly with position than expected. Spectra in the slit and in the slitless mode fall on different parts of the imaging detector, requiring different dispersion solutions for them, which were implemented in October 2022.  Before that update, the wavelength calibration for the slit had errors up to 0.5 µm. After, errors still persisted to about 50 nm at the short wavelength end. The updated calibration for the slit reduced the uncertainties to about ±20 nm.  The goal remains an accuracy of 5–10 nm over the full nominal wavelength range.

It is important to note that the LRS wavelength calibration does not operate in isolation from other calibration steps. The path loss correction, flat field, photometric calibration, and extraction aperture correction all have a wavelength dependency. Following the updates to the wavelength calibration in October 2022 and December 2023, dependent reference files were also updated. This process will be repeated when further updates are made to the wavelength calibration. Users are encouraged to sign up to the notification email lists to receive prompt notification when new MIRI reference files are delivered. 

Users are also encouraged to reach out to the MIRI team via the JWST Helpdesk if they need further clarification on any aspect of this issue. 

Spectral extraction

The spectral extraction in the JWST calibration pipeline is not reliably producing science quality spectra when run automatically, i.e., for the products users will find in MAST. Two separate issues affecting the quality of the extracted products have been identified, as explained below.

Errors in source location

This issue particularly affects LRS slit data, where the source is typically moved between 2 different nod positions. The extract_1d step in the calwebb_spec2 pipeline uses the coordinate information of the observation to identify the location of the spectrum in the science aperture, rather than by having the position hard-coded or by applying a source-finding algorithm. This means that any inaccuracies or errors in the coordinate registration of the data will result in the extraction being performed in the wrong part of the aperture, despite the spectrum being in the expected position. If the "x1d" or "x1dints" products do not show the expected spectral shape for a target, despite the spectral images (the "cal" or "s2d" products) showing good quality spectra, this is the likely explanation. 

Errors in coordinate registration can occur when guide stars have inaccurate coordinates in the catalog, or if the wrong guide star is identified during the Fine Guidance Sensor's ID process. (The risk of this increases with field crowding.) As long as the offset is small and the target still lands in the target acquisition region of interest, the TA sequence will compensate for such errors, placing the target at the correct position in the slit or the subarray. The assigned coordinates will, however, be offset from what was expected in those cases, and the extraction algorithm will (attempt to) extract the spectrum from the wrong location. As the slit measures just ~4.7" × 0.5", even small coordinate offsets can cause this issue. 

The parameters used in the extract_1d step are defined in a JSON-formatted reference file that is available in the CRDS database. As of May 2023, an updated version of this file delivered to CRDS provides a workaround for the coordinate misplacement issue. The new reference file defines the xstart and xstop values to match the expected location of the spectral trace in the nod-combined, resampled level 3 spectral image (the "s2d.fits" product), and explicitly sets the use_source_posn keyword to False, which prevents the pipeline from computing coordinate-based offsets. This strategy should deliver much improved extracted level 3 spectra. Note that this change will worsen the quality of the level 2b automated extractions, as the geometry of the level 2b spectral images is different from the nod-combined, resampled product.  

If extracted spectra from each nod position individually at the level 2b stage are required, such extractions can be accomplished by re-running the extract_1d step offline using a custom "extract_1d" reference file that has been modified to specify the desired extraction limits within the image. This is described in the JWST calibration pipeline documentation. The workaround method is demonstrated in this example notebook, as well as in the materials of the TSO JWebbinar (see video, notebook); both of these demonstrations include examples of custom reference files, and how the pipeline can be instructed to use a custom file. 

Bad pixel spikes and dropouts

A second issue with the spectral extraction is bad pixel spikes and dropouts in the extracted spectrum. These appear as single-point dips or spikes in the spectrum that can at times mimic absorption or emission features. The cause of these features is the presence of a flagged or hot pixel in the extraction aperture. When a pixel has been flagged as "DO_NOT_USE" by the pipeline, due to it being a known bad or hot pixel, or being affected by a cosmic ray, the pipeline excludes it from the summation of flux in the extraction aperture. It does not interpolate its value from its neighbouring pixels. This causes the extracted flux along this row in the aperture to drop compared to the rows above and below, i.e., causing a dip. Similarly, hot pixels are included in the summation as they are, causing spikes in the spectrum. This issue affects the cosmetics of the extracted spectrum, and, if located on or near a physical spectral feature, can interfere with the analysis. 

As of May 2023, a new step was included in the JWST calibration pipeline, pixel_replace, that will identify these pixels and replace them with an interpolated value. The step is documented in the JWST calibration pipeline documentation. This is particularly valuable for non-dithered observations such as time-series observations with LRS in slitless mode. 

Background subtraction (slitless mode)

The JWST calibration pipeline does not currently perform any background subtraction on the spectral images taken with the slitless LRS mode. Work is being done to implement an optimized algorithm for this. Based on experience with commissioning and early science data, the following strategies are recommended:

  • A very simple background subtraction can be performed as part of the spectral extraction in the extract_1d step of the calwebb_spec2 pipeline. By creating a custom reference file for the extract_1d step, an off-target background region can be defined alongside the source region, from which a background is computed. This is demonstrated in the materials of the TSO JWebbinar (see video, notebook). 

  • A better result is obtained by manually computing a background spectrum from off-target regions of the subarray in the "rateints.file", for each individual integration. The best regions of the subarrays from which the background is computed are those at the edges of the subarray (excluding the first 4 columns, which contain the reference pixels). By median-combining different off-target regions to compute an effective background, a 2-D background image can be constructed, and subtracted from the science data. If this operation is performed on the "rateints" product, the same file can then be processed further through the calwebb_spec2 pipeline. 



Summary of common issues and workarounds

The sections above provide detail on each of the known issues affecting MIRI LRS data; the table below summarizes some of the most likely issues that users will encounter along with any workarounds if available. Note that greyed-out issues have been retired, and are fixed as of the indicated pipeline build.

SymptomsCauseWorkaroundFix buildMitigation Plan

MR-LRS03: Spectra of bright sources can experience data dropouts at wavelengths where they are brightest (in the "x1d" files).

The problem can also be seen in 2-D spectral images, with the peak column flagged as "DO_NOT_USE" in several adjacent rows where the spectrum is brightest.

The jump detection step in calwebb_detector1 can flag pixels with strong signals as "DO_NOT_USE" in the spectral images even though the data are viable. These pixels are "NaN" in the spectral images and their signal is missing from the affected wavelengths in the extracted spectra.

Users can run calwebb_detector1 themselves and adjust the rejection_threshold parameter in the jump step to eliminate (or reduce) improper flagging. A generic workaround is challenging to produce as the occurrence and severity of the issue depends on several factors, e.g., target brightness and number of groups per integration.

N/A

Updated issue

The jump step algorithm and default parameters are continually being examined and optimized; improvements are expected in future builds (spring 2024 and beyond). The issue likely cannot be entirely eliminated as it differs greatly between observations (and indeed many are not affected).

MR-LRS04: Spectra extracted from the LRS slit ("x1d" files) can show major disparities between the nods or with expected values.The pathloss correction is generating incorrect results due to a poor understanding of the location of the source in the slit.

The "pathloss" reference file has already been modified to apply a correction as though the source were in the center of the slit, no matter where it actually is (or where it is believed to be). Effective with build 10.0, even if the pipeline thinks a source is out of the slit, it will still be corrected as though it were in the center of the slit.

N/A

Created issue

The Science Calibration Pipeline will be modified to allow the user to rerun the pathloss correction with a default position of the center of the slit.

The Operations Pipeline will be modified to determine the source position based on the TA verification image.

Neither of these solutions are ready for implementation.

MR-LRS05: The "ERR" extension in "rate" data products (and other error estimates downstream) is incorrect.Error values are estimated incorrectly by the pipeline.

Bootstrap uncertainty from science spectra.

 N/A

Created issue

The root cause of incorrect error estimates in the pipeline is being investigated.

MR-LRS06: Target acquisition images taken with FASTGRPAVG readout patterns are incorrectly calibrated.The calibration pipeline does not compute the exposure time for these readout modes when fitting a slope to the uncalibrated data.

The verification images are calibrated correctly; if additional photometry is required, the MIRI team recommends the use these images.
 
For the TA images, count rates can be divided by the FASTGRPAVG frame multiplier to obtain the correct count rate. 

N/A

Created issue

The issue will be fixed in the calibration pipeline code in early 2024. 
 
Note: this affects all FASTGRPAVG TA images taken for MIRI.

MR-LRS02: Extracted 1-D spectra ("x1d" files) often show salt-and-pepper noise (i.e., spikes and divots one pixel across).Pixels flagged as "DO_NOT_USE" appear as "NaN" in the spectral images, and the signal in those pixels is missing from the summed signal at that wavelength in the extracted spectra.

Users can apply the new pixel_replace step in the developmental pipeline, which replaces the pixels flagged "DO_NOT_USE" with pixels in the same column by normalizing nearby spatial profiles in the 2-D spectral images.

The best solution is to download the developmental program and just run it. The default for pixel_replace is to use 3 rows above and below the problematic row. This value can be changed by setting the n_adjacent_rows argument.


 9.3

Updated Operations Pipeline

The new pixel_replace step was implemented in the Operations Pipeline, installed on August 24, 2023. STScI reprocessed affected data products, which typically takes 2–4 weeks after the update.

MR-LRS01: Spectra extracted from LRS slit and slitless data ("x1d" files) can have little or no signal, or even negative signal, even though the 2-D spectral images look fine.
The pipeline is extracting the spectrum from the wrong location in the 2-D spectral images because the positions it is using to determine the location of the source are inaccurate.

Users can set the location and the width of the extraction apertures themselves when running calwebb_spec2 or calwebb_spec3 manually. A notebook demonstrating the spectral extraction capabilities of the JWST calibration pipeline for MIRI LRS is available in this repository

 9.3

Updated Operations Pipeline 

The pipeline was updated in May 2023 to default to apertures centered on the nominal position of the level 3 spectrum in the aperture. This change is fully in operations, data have been reprocessed.




Notable updates


Originally published