MIRI Time Series Observations Pipeline Caveats

Unique features of the JWST Science Calibration Pipeline for time-series observations (TSOs) with the MIRI instrument, and caveats for users, are described in this article. Users should also refer to the TSO pipeline overview for characteristics and caveats that are common to all instruments. This information reflects the status for the JWST pipeline version 1.4.6.

On this page

This article highlights some aspects of the JWST calibration pipeline for MIRI, in particular those where the treatment of TSO exposures differs from "regular" MIRI exposures. It is important also to highlight that early in the mission our understanding of the observatory performance is evolving quite rapidly, and changes to calibration procedures are expected. 

The information presented covers all 3 of MIRI's modes that are available for TSOs: imaging, slitless low-resolution spectroscopy, and medium-resolution spectroscopy. 



Summary of specific MIRI TSO pipeline issues

The information in this table about MIRI time-series observations calibration pipeline issues is excerpted from Known Issues with JWST Data Products.  

SymptomsCauseWorkaroundMitigation Plan
MR-TS01: Spatially and temporally varying noise in subarray TSOs ("390 Hz noise")This noise originates in the detector electronics. This affects subarrays SLITLESSPRISM, SUB64, and SUB128 subarrays.

The noise can be subtracted to a large extent by measuring and subtracting a background for each individual integration. Some residual noise will remain. 

Updated issue

Testing is currently underway on an algorithm that fits and removes the 390 Hz noise signal. This algorithm should be in the Operations Pipeline in early 2024. 

MR-TS02: Discontinuities in persistence behavior are seen in the SLITLESSPRISM subarray at the start of the exposure.This is currently under investigation.

Users should consider increasing the detector settling time from 30 mins to 1 hour. This is particularly important for phase curves. 

Created issue

As soon as the cause of this problem is understood, the MIRI team will make the required changes to mitigate it. 

MR-TS03: The photon count rate and derived flux for MRS TSO data, is lower than predicted at long wavelengths, with maximum deficit roughly a factor of 2 at 28 µm. MRS sensitivity at long wavelengths is decreasing with time.

There is no work around at the moment.

 

Updated Operations Pipeline

The science calibration pipeline was modified, on December 5, 2023, to apply the time-dependent throughput correction for TSO data. STScI will reprocess affected data products with the updated Operations Pipeline; reprocessing of affected data typically takes 2–4 weeks after the update. 



Stage 1 processing for LRS, MRS and imaging

First and last frames

The mid-infrared detectors in MIRI have a wider variety of response behaviours than is typically seen in the optical or near-infrared. The first and last frames in an integration are known to show deviations from their expected response, and they are therefore marked as "DO NOT USE" for linearity correction, jump detection and ramp fitting. The last frame in particular is problematic, as it displays a "pull-down" effect whose magnitude varies between odd and even rows. Including this frame in the ramp fit pulls down the calculated rate, and thus impacts the flux calibration of the target. 

For TSOs, where stability is considered more important than the absolute flux calibration, the first frames of each integration are included in the ramp fitting step (in contrast with non-TSOs). We continue to exclude the last frame, as the impact of the last frame pull-down is more severe. 

Words in bold are GUI menus/
panels or data software packages; 
bold italics are buttons in GUI
tools or package parameters.

Users can always study the impact of including and excluding these frames by toggling the firstframe and lastframe steps of the calwebb_detector1 pipeline. Changing the execution status of pipeline steps is demonstrated in this TSO Webbinar excerpt. (Additional information is available for the full "JWebbinar-11" on time-series observations under the "Materials and Videos" tab at the JWebbinar webpage). Note that these pipeline steps flag the frames as "DO NOT USE", so skipping the steps includes the frames in subsequent steps.  

Reference pixel correction

The MIRI detectors have 4 columns of reference pixels on each side of the detector (8 columns in total). These pixels are read out despite not being part of the illuminated section of the array. The measured signal from these reference pixels can be used to clean up their illuminated counterparts along the same rows. 

However, many of the MIRI subarrays are not connected to the array edges; therefore, the reference pixel signals are not available for many subarray exposures. Insufficient data were available from ground testing to optimize a reference pixel correction method for subarray exposures. For this reason, the reference pixel step is skipped for all subarray exposures (TSO or non-TSO). For full frame imaging or MRS TSOs, the reference pixel step is applied. 



Stage 2 processing

World coordinate system assignment (imaging, LRS, MRS)

The first step in running the calwebb_spec2 or calwebb_image2 pipelines is the assignment of the world coordinate system (WCS) parameters for the data based on the telescope pointing information. For the spectroscopic modes, this includes the wavelength information. This step assumes that the target is perfectly placed at the nominal position in the science aperture (the center of the imaging or MRS fields, the nominal pointing position in the SLITLESSPRISM subarray), and that the guide star coordinates are accurate. Depending on the execution of target acquisition (TA) or the accuracy of the TA procedure, or in the case of offsets specified in the special requirements, there may be small offsets of the target from that nominal position. Small inaccuracies in the guide star catalog coordinates can also introduce coordinate registration errors. This may introduce noise or errors at later pipeline stages. Note also that pointing jitter may add to pointing-related noise in the final science product. LRS slitless observations have mandatory TA, accompanied by a TA verification image taken after the telescope moves the target to the nominal position in the subarray. This allows the target placement to be visualized before the light is dispersed. 

The spectral extraction step in Spec2Pipeline also accesses positional header keywords to locate the target in the subarray. In case of WCS registration issues, the extraction aperture may not be placed in the optimal location. Therefore, verifying the placement of the target in the subarray, and performing a custom extraction manually to ensure the correct placement of the aperture, is recommended.

Note that in Cycle 1, MIRI imaging with target acquisition has not yet been implemented. 

Spectral leak (LRS only)

The spectral dispersion profile of the double prism that is used by the LRS mode turns over on itself around 4 µm; this is challenging to calibrate and not very well reflected in the wavelength calibrations (shown in Figure 1). Wavelengths below 4 µm are dispersed back onto the longer wavelengths. For the case of LRS slit spectroscopy, a filter is mounted on the slit mask blocking all radiation below ~4.5 µm, but for slitless operation, i.e., for all LRS TSOs, this is not the case. LRS TSO spectra can therefore have some contamination of the blue end of the spectrum.

The issue is however mitigated by the sharp drop in prism transmission below 5 µm, as shown in the right panel of Figure 1. Any residual spectral contamination is not currently modeled by the ETC, nor is it corrected in the pipeline. Calibration strategies and reference files will be updated as more in-flight data is gathered, but observers in early observing cycles should be aware of this issue. 

Figure 1. MIRI LRS spectral dispersion profile and photon conversion efficiency plots


Left: spectral dispersion profile of the MIRI LRS mode, for both slit and slitless operation. Right: Photon conversion efficiency comparing the transmission for the slit and slitless modes of the LRS. The PCE drops off very steeply below 5 µm.

 Background subtraction (LRS)

For slitless LRS observations, background subtraction has not yet been implemented in the automated pipeline. The recommended approach based on pre-flight information is to use the background-only columns of the subarray to derive a mean background spectrum, and subtract it manually. The spectral extraction step (extract_1d) can also be configured to perform background subtraction as part of the extraction. This is demonstrated in the TSO JWebbinar materials. 

For TSO imaging, background subtraction is performed in stage 3 of the pipeline, as part of the tso_photometry step. 

Photometric calibration (LRS)

The photometric calibration reference file provides calibration factors as a function of wavelength, which are then attached to the data in the photom step using the wavelength information assigned in the assign_wcs step of the calwebb_spec2 pipeline. This means that any inaccuracies in the initial target placement, or pointing drifts of jitter over the course of the exposure, will result in an inaccurate calibration, or in the case of jitter or drifts, an additional noise source. To test for this issue, or avoid this source of error, the step can be skipped. The subsequent pipeline steps are able to process the data based on DN/s units. 

Cube build and spectral extraction (MRS)

When performing TSOs with the medium resolution spectrometer, pipeline processing ends before the cube building and spectral extraction steps. The final output product are the 2-D, photometrically calibrated spectral images produced by the photom step. Spectral extraction directly from the 2-D focal plane array is not yet supported in the pipeline. Support for MRS TSOs will continue to be improved during early science operations. 



Stage 3 processing

TSO photometry (imaging)

The calwebb_tso3 pipeline produces a photometric time series for MIRI imaging TSOs. This step assumes that the target was placed perfectly at the nominal pointing location in the array or subarray, and does not move between integrations. Time-series imaging with MIRI, however, is not performed with target acquisition. In addition, pointing jitter or drifts can cause further inaccuracies in the placement of the photometric aperture in this step. Target placement should always be checked for each integration for time-series imaging, and it may be beneficial to perform the time-series construction manually. 



References

Kendrew, S., et al. 2015, PASP, 127, 623
The Mid-Infrared Instrument for the James Webb Space Telescope, IV: The Low-Resolution Spectrometer




Latest updates
  •  
    Figure 1 updated with in-flight data
Originally published