NIRSpec Known Issues
Known issues specific to NIRSpec data processing in the JWST Science Calibration Pipeline are described in this article. This is not intended as a how-to guide or as full documentation of individual pipeline steps, but rather to give a scientist-level overview of issues that users should be aware of for their science.
On this page
Specific artifacts are described in the Artifacts section below. Guidance on using the pipeline data products is provided in the Pipeline Notes section along with a summary of some common issues and workarounds in the summary section.
Information provided in this article affects all NIRSpec observing modes. For mode-specific issues, see the NIRSpec BOTS, fixed slit, IFU, and MOS issues articles. For a detailed discussion of many aspects of the NIRSpec detectors and their impact on science calibration, see NIRSpec Detectors.
Please also refer to NIRSpec Calibration Status.
Artifacts
Spurious features in extracted 1-D spectra
Pixels flagged as "DO_NOT_USE" in the data quality (DQ) array are excluded from spectral extraction, with the possibility of causing divots in the final 1-D spectrum. If the flagged pixel is close to the center of the point spread function (PSF) at a given wavelength, the absence of the signal in the flagged pixel will lead to a divot at that wavelength in the extracted spectrum.
A pixel_replace step has been created for the pipeline, based on an adjacent profile approximation, which generates an average spatial profile from columns immediately adjacent to the one with the bad pixel, normalizes that profile to the good data in the affected column and uses the value of the corresponding pixel in the normalized profile to replace the signal in the bad pixel. Currently this step uses a default-off configuration for all NIRSpec modes. Use of this step can introduce artifacts when pixels are over-corrected by the spatial profile normalization. Bad corrections may have worse impacts on calwebb_spec3 reductions than no corrections.
The NIRSpec team continues to explore options for handling missing pixel data. In the meantime, pipeline users can test the use of the pixel_replace step on their data sets by setting skip = False in the parameters for the pixel_replace step in the stage 2 pipeline.
Snowballs
The detectors used in NIRSpec experience cosmic ray events known as snowballs and showers. Pipeline algorithms to address these artifacts are under development; in the interim, the best mitigation strategy is to dither.
Pipeline notes
See known issues pages for NIRSpec BOTS, fixed slit, IFU, and MOS modes.
Alternating column noise
A pattern of alternating brighter/darker pixel rows over part or all of a detector image is an effect sometimes referred to as alternating column noise (ACN; note that pixel rows in the science data actually represent columns in the detector frame of reference, as the data are rotated to put the dispersion direction horizontally). This is due to the two amplifiers (one for odd and one for even columns) in an output sometimes having slightly different offsets because of drift or a cosmic ray event. Pipeline code testing is underway to mitigate this odd-even column discrepancy.
Figure 1. Example of alternating column noise
The striped pattern of alternating bright/dark pixel rows, including the bright horizontal band, is seen in the lower half of this rate image file.
Uncorrected correlated noise - 1/f noise
JWST's near-infrared HgCdTe detectors have a correlated noise, known as 1/f noise, that is introduced by the detector readout system (Moseley et al. 2010). This noise results in vertical striping, as illustrated in the left panel Figure 2. With NIRSpec's IRS2 readout mode, reference pixels are interspersed with the sampling of normal pixels to mitigate the 1/f noise. However, this noise will still be present in data taken with NIRSpec's traditional detector readout mode; this includes all observations made with NIRSpec's subarrays, where IRS2 is unavailable. In these cases, it may be necessary to use background pixels in the horizontal bands to estimate and subtract the 1/f noise. The accompanying mode-specific pages further discuss 1/f noise.
To address this issue, the JWST Science Calibration Pipeline has integrated options for correcting 1/f noise in calwebb_detector1 in the clean_flicker_noise step and within calwebb_spec2 under the nsclean step (originally named for the "NSClean" algorithm, Rauscher 2024, developed by Bernard Rauscher). Both steps provide the same options for cleaning 1/f noise with the key difference that the clean_flicker_noise step in calwebb_detector1 fits and removes 1/f noise on a group-by-group basis, whereas the nsclean step in calwebb_spec2 performs 1/f cleaning on rate files. The algorithms use dark areas of the detector to fit a model of the 1/f noise either in Fourier space or using the column-wise median of dark regions. It requires an input mask to identify all dark areas of the detector. The more thorough and complete this mask is, the better the background fit. The pipeline creates an on-the-fly mask using default parameters to remove 1/f noise, and the user should refer to the optional parameters for this pipeline step if the default settings are not sufficient for the data. If needed, see the NSClean documentation for suggestions on manually creating a custom mask. The right panel of Figure 2 shows an example of a 1/f cleaned image using clean_flicker_noise.
Further information on cleaning 1/f noise and setting recommended parameters using the JWST Science Calibration Pipeline can be found here: 1/f Noise.
Figure 2. 1/f noise cleaning in NIRSpec/MOS data
Illustration of vertical striping due to 1/f noise seen in NIRSpec/MOS data taken in traditional readout mode (left), and the same data after it has been cleaned by the JWST pipeline clean_flicker_noise step (right).
Pixel rows exhibit anomalous count rates in "rate" images for IRS2 data.
Some reference pixels are "bad" (e.g., excessive noise or "telegraph" behavior) and are flagged as such. However, those flags are not considered in the refpix step of the calwebb_detector1 pipeline, resulting in anomalous corrections for the associated science pixels, leading to image behavior in Figure 3. A flag check has been implemented in the pipeline, along with an improved algorithm to filter out noisy reference pixels.
Figure 3. Excessive noise along rows.
Excessive noise, or 'telegraphing' behavior, is highlighted in the figure for two groups of rows.
No flux or negative flux in FS or MOS extracted 1-D spectrum
The extract_1d step uses an automated centering routine to place the extraction aperture, based on the source coordinates. The extraction aperture can be offset along the spatial axis, for example, due to an error in target coordinates or the world coordinate system. The assumed source position should work well for most NIRSpec sources, but in case the extraction aperture is still poorly placed, it can be manually tweaked by specifying start and stop positions to the extraction step. (Extraction parameters can be set in a reference file in JSON format, described in the pipeline documentation.) Instead of using the sky position, one can use the relative slit position of the target, as planned in APT, to place the aperture. Following a successful target acquisition, this position is typically known with high accuracy (~5–10 mas). The automatic centering can be overridden by setting the optional step argument use_source_posn = False, and modifying the reference file parameters ystart and ystop (found in the pipeline output log file) to define the desired boundary of the extraction aperture (in integer pixels or the use of polynomial for fractional pixels). More on this can be found on the NIRSpec MOS Known Issues page.
Summary of common issues and workarounds
The sections above provide detail on each of the known issues affecting NIRSpec data; the table below summarizes some of the most likely issues users may encounter along with any workarounds, if available. Note that greyed-out issues have been retired, and are fixed as of the indicated pipeline build. Information provided in this table affects all NIRSpec observing modes. For mode-specific issues, see the relevant known issues pages:
| Symptoms | Cause | Workaround | Fix build | Mitigation Plan |
|---|---|---|---|---|
| NS-G03: Extracted 1-D spectrum ("x1d" product) has spurious positive and negative features, typically one pixel wide. | Pixels marked as "DO_NOT_USE" in reference data are not included in the 1-D extracted spectrum. | Workaround: Use the new Science Calibration Pipeline software (jwst 1.11.0 onwards) to estimate the value for each "DO_NOT_USE" pixel from neighboring 2-D profiles before extracting the 1-D spectrum by running the pixel_replace step. This is disabled by default. Testing has shown improvements in many cases, but degradation in data quality has also been seen in certain circumstances; users should exercise caution if they choose to run the step. | N/A | Updated issue The team is evaluating options for further improvements. |
| NS-G06: A pattern of alternating brighter/darker pixel rows over part or all of a detector image. | An effect sometimes referred to as alternating column noise (ACN; note that pixel rows in the science data actually represent columns in the detector frame of reference, as the data are rotated to put the dispersion direction horizontally). This is due to the two amplifiers (one for odd and one for even columns) in an output sometimes having slightly different offsets because of drift or a cosmic ray event. | For traditional readout modes calwebb_detector1 refpix.odd_even_columns is on by default. If alternating column noise is seen in data taken in these modes, it may be improved by turning off refpix.odd_even_columns. For IRS2 readout modes refpix.odd_even_columns is disabled by default. As of jwst 1.14.0 and later refpix.odd_even_columns can be used for IRS2 modes. Testing has shown improvements in some cases, but degradation in data quality has also been seen in certain circumstances; users may try reprocessing with this setting on, but should exercise caution if they choose to do so. | N/A | Updated issue The team is investigating additional improvements to the treatment of alternating column noise in IRS2 modes. |
NS-G09: Uncertainties in the combined background images produced in the background step are not propagated into science data. | The background step does not propagate uncertainties from the background images to the 2-D science arrays | Uncertainties from the the combined background images can be recovered by saving the combined background using background.save_combined_background = True and combining the uncertainties from this image with the science data uncertainties from the background step. | N/A | Created issue Work on a fix is pending an investigation into the necessary implementation. |
References
Moseley, S. H., et al. 2010 SPIE Proceedings Vol. 7742
Reducing the read noise of H2RG detector arrays: eliminating correlated noise with efficient use of reference signals
Rauscher, B. J. 2024 PASP 136:015001
NSClean: An Algorithm for removing Correlated Noise from JWST NIRSpec Images