JWST Wide Field Slitless Spectroscopy Pipeline Caveats
Specific characteristics of the JWST Science Calibration Pipeline for wide field slitless spectroscopy (WFSS) modes on NIRCam and NIRISS are described in this article. For example codes or more in depth details, the user should refer to the JWebbinar material on github and to the full JWST calibration pipeline documentation, respectively.
Summary of specific WFSS pipeline issues
The information in this table about NIRISS WFSS pipeline issues is excerpted from Known Issues with JWST Data Products.
NR-WFSS01: Spectral traces are offset from where they should be, causing offsets in wavelength zero point.
The current "specwcs" reference file (as of CRDS version jwst_1105.pmap) incorparates the field dependence in the wavelength dispersion silution. However, the shape of the spectral traces also vary as a function of detector position. There may also be a dependence on filter wheel position angle at the time of observation.
In-depth investigation is underway to calibrate the change in shape of the spectral traces as a function of detector position and quantify whether there is a dependence on the filter wheel position angle at the time of observation on the spectral traces.
NR-WFSS02: For direct imaging: when the peak pixel of a star (or other compact object) reaches beyond ~25,000 e- in an integration (after bias and dark subtraction and linearity correction), it starts "spilling" charge to its neighboring pixels. This causes an effective "widening" of the PSF or charge distribution, as well as flux loss for those objects in the combined, resampled products of the calwebb_image3 stage of the Science Calibration Pipeline. This effect is strongest for the most undersampled modes (i.e., filters with pivot wavelength <= 2 μm).
This is due to the so-called "brighter-fatter effect" (BFE) that affects near-IR H2RG detectors, in combination with the current way "jumps" are detected and dealt with in the calwebb_detector1 pipeline stage, and how the latter affects image combination in the calwebb_image3 pipeline stage.
No efficient workaround is available at present.
Apply new Science Calibration Pipeline step charge_migration within the calwebb_detector1 stage. STScI will reprocess affected data products with an updated Operations Pipeline, planned for installation in December 2023. Reprocessing of affected data typically takes 2–4 weeks after the update.
The information in this table about NIRCam WFSS pipeline issues is excerpted from Known Issues with JWST Data Products.
|NC-WFSS01: Adjacent sources are included in extracted spectra.||This occurs in crowded fields with overlapping spectral traces.|
No workaround is available yet. The calwebb_spec2 pipeline includes an option to provide a first-order correction for source contamination with the wfss_contam step, but some discrepancies and issues are still being investigated.
Implement the wfss_contam step in calwebb_spec2. Work on this is ongoing and is planned for completion in 2024.
|NC-WFSS02: The fluxes of some spectra in the field appear to be too low or even negative.||This is caused by over-subtraction of the background. The background subtraction step in calwebb_spec2 assumes the background is constant and does not vary in time, which can be a limiting factor for low signal-to-noise observations.|
Users can manually perform a local background subtraction on extracted 2-D spectra before the 1-D extraction in calwebb_spec2.
A mitigation plan is under development.
|NC-WFSS03: The WFSS pipeline has excessive run times when run locally.||Processing WFSS exposures can be prohibitively resource intensive on individual machines due to the number of sources in these files.|
Adjust the wfss_nbright parameter in the extract_2d step in calwebb_spec2 to limit the number of extracted sources.
Updated Operations Pipeline
Several efficiency improvements have been made at various stages of processing. STScI will continue monitoring issues that get reported.
WFSS pipeline steps
Words in bold are GUI menus/
panels or data software packages;
bold italics are buttons in GUI
tools or package parameters.
The default set of observations in WFSS mode includes direct images before and after the grism exposures (see WFSS observation sequence). These direct images are required to enable proper identification of the objects seen in the grism images. Hence, they should be considered when processing the data with the JWST calibration pipeline. Direct images are processed through the calwebb_detector1, calwebb_image2, and calwebb_image3 stages with the aim of deriving a source catalog, which is required for the extraction of the spectra. The grism data are processed through the calwebb_detector1, calwebb_spec2, and calwebb_spec3 stages to obtain 2-D and 1-D calibrated spectra for every source in the catalog.
Direct imaging - resampling step in calwebb_image3 pipeline stage
Note: This paragraph is only relevant to WFSS data taken prior to July 18, 2022, using a DMS context prior to jwst_0928.pmap (see value of main header keyword
CRDS_CTX). Prior to source catalog creation, the direct images for WFSS (which are taken at 2 or more dither positions per blocking filter) are combined into a "_i2d.fits" file by the calwebb_image3 stage of the pipeline after image alignment, sky matching, outlier detection, and resampling onto an undistorted pixel grid. For the default settings of the calwebb_image3 pipeline, this resampling step involves a pixel-by-pixel weighting procedure, using weights that are proportional to the inverse variance of the read noise (VAR_RNOISE) array stored in each input image
(resample.weight_type = 'ivm'). For NIRISS direct imaging data with the WFSS filters that was run through the JWST pipeline using a DMS context prior to jwst_0928.pmap (which was made active on July 18. 2022), this way of resampling imaging data can lead to flux levels of unsaturated point sources in the resampled image that are too low by up to 30% depending on the size of the measurement aperture and the exact positioning of the star centroid relative to pixel boundaries. For such data, we recommend the following workaround to retain proper flux levels in the combined image for NIRISS WFSS data:
my_asn.json is the association file (ASN) for the direct images taken with a given filter.
JWST's near-infrared HgCdTe detectors have a correlated noise, known as 1/f noise, that is introduced by the detector readout system (Moseley et al. 2010). It results in stripes that can be a limiting factor in how faint of a continuum pipeline processed data can reach when the dispersion direction is parallel to the stripes. If the 1/f noise stripes are perpendicular to the dispersion direction, they create faint fluctuations that can mimic emission lines. Examples on how to correct for the 1/f noise can be found in the CEERS JWebbinar notebook or in the TSO JWebbinar notebook.
A source catalog is required to extract the individual spectra from the grism observations. The calwebb_spec2 stage accepts the source catalog obtained by running the source_catalog step in calwebb_image3 on the direct image. The number of sources extracted during the calwebb_spec2 stage is thus limited by the depth and the covered field of view of the direct imaging. The minimal set of columns that need to be provided in the source catalog to run the extraction includes just label and position (as right ascension and declination in degrees).
Summary of the various steps
The spectral extraction happens in the calwebb_spec2 stage of the JWST Science Calibration Pipeline. Example code and descriptions of the individual steps are provided in the notebook "JWST Spectroscopic Data Calibration: Pipeline Stage 2." For the WFSS mode on NIRISS and NIRCam, the steps in the calwebb_spec2 stage happen in the following order: assign_wcs, background, flat_field, extract_2d, wfss_contam (if enabled), photom, and extract_1d.
- The assign_wcs step associates a world coordinate system (WCS) object with each science exposure. The WCS object transforms positions in the detector frame to positions in the world coordinate frame, ICRS and wavelength.
- The background step subtracts a background reference image from the target exposure. Before being subtracted, the background reference image is scaled to match the signal level of the WFSS image within background (source-free) regions of the image.
- The flat_field step takes an input science dataset and divides it by a flat field reference image. In particular, the SCI array from the flat field reference file is divided into the SCI array of the science dataset, the flat field DQ array is combined with the science DQ array using a bitwise OR operation, and variance and error arrays in the science dataset are updated to include the flat field uncertainty.
- The extract_2d step extracts 2-D arrays from spectral images. The extractions are performed within all of the SCI, ERR, and DQ arrays of the input image model, as well as any variance arrays that may be present. It also computes an array of wavelengths to attach to the extracted data. The extract_2d step uses the source catalog to create the list of objects and their corresponding bounding box. This list is used to make the 2-D cutouts from the dispersed image.
- The wfss_contam step is applied to grism exposures in an attempt to correct effects due to overlapping spectral traces, which often happens in observations of crowded fields. It is not enabled by default. More details are given below.
- The photom step applies flux (photometric) calibrations to a data product to convert the data from units of count rate (DN/s) to surface brightness (MJy/sr). The calibration information is read from a photometric reference file.
- The extract_1d step extracts a 1-D signal from the 2-D cutouts and writes spectral data to an “x1d” product. The extract_1d step collapses the input data from 2-D to 1-D, providing the equivalent of a boxcar extraction. An additional background subtraction can be done if (and only if) bkg_coeff is given in the "EXTRACT1D" reference file. In that case, the background is determined independently for each column (or row, if dispersion is vertical), using pixel values from all background regions within each column (or row).
These extraction steps are best suited to non-crowded fields. The contamination modeling can be used to mitigate the effects of overlapping traces in crowded fields. WFSS images are not corrected for distortion, but the 1-D extraction accounts for variations across the field when computing the spectral traces and the wavelength calibration.
NIRCam and NIRISS WFSS receive only minimal processing by calwebb_spec3. Essentially, in stage 3, the 2-D data are reorganized to convert stage 2 exposure-based data products to stage 3 source-based data products (exp_to_source step). Subsequently, 1-D spectra are re-extracted by the extract_1d step and the combine_1d step computes a weighted average of 1-D spectra to ultimately write the combined 1-D spectra as output. Example code and description of the individual steps are provided in the notebook JWST Spectroscopic Data Calibration: Pipeline Stage 3.
Background subtraction is the second step of the calwebb_spec2 stage of the JWST Science Calibration Pipeline. This step relies on a single master background (in the form of a reference file), which will be updated with Cycle 1 observations, and assumes that the WFSS background does not vary in time (during an exposure) and its spectrum is constant. This can be a limiting factor in the case of low signal-to-noise observations. As a mitigation strategy, the user can perform a local background subtraction on the extracted 2-D spectra before extracting the 1-D spectra.
One of the main challenges in the spectral extraction for WFSS modes on JWST is the contamination of any individual spectrum by the overlapping spectra of nearby sources. The JWST calibration pipeline is best suited for sparse fields where traces do not overlap and the extraction of 2-D and 1-D spectra is robust. However, observed fields are often crowded. Both NIRISS and NIRCam have grisms that disperse light in 2 orthogonal directions, which could be used to help mitigate some of the spectral overlaps. This is because a spectrum strongly contaminated in one spectral direction might not be as contaminated in the other spectral direction. As fainter magnitudes are reached, however, spectral contamination occurs in any direction and, moreover, being able to correct all spectra for contamination before combining them would allow for the full depth of the data to be used.
The JWST calibration pipeline includes an option to provide a first-order correction for contamination with the wfss_contam step. Briefly, source fluxes from a direct image of the field are used to simulate grism spectra for each source. Each source spectrum is then corrected for contamination by subtracting the simulated spectra of nearby sources. More details are provided on the calibration pipeline documentation. The WFSS contamination correction is not performed by default, but it can be enabled when calling calwebb_spec2 as in this example:
This contamination model provides a first-order correction in the extraction of overlapping spectra, but it is based on single-band photometry (i.e., it does not account for flux variations within a photometric band) and it depends on the depth of the segmentation map created from the direct image during calwebb_image3 processing. Also, some discrepancies have been observed between the modeling and the simulated spectra. Updates to the wfss_contam step will be guided by experience with data obtained on-orbit. Pirzkal et a. (2017) provides more information about contamination modeling for wide field slitless spectra.
Moseley, S. H., et al. 2010 SPIE Proceedings Vol. 7742
Reducing the read noise of H2RG detector arrays: eliminating correlated noise with efficient use of reference signals
Pirzkal, N., et al. 2017 ApJ, 846, 84
FIGS—Faint Infrared Grism Survey: Description and Data Reduction