NIRSpec Known Issues
Known issues specific to NIRSpec data processing in the JWST Science Calibration Pipeline are described in this article. This is not intended as a how-to guide or as full documentation of individual pipeline steps, but rather to give a scientist-level overview of issues that users should be aware of for their science.
On this page
Specific artifacts are described in the Artifacts section below. Guidance on using the pipeline data products is provided in the Pipeline Notes section along with a summary of some common issues and workarounds in the summary section.
Information provided in this article affects all NIRSpec observing modes. For mode-specific issues, see the NIRSpec BOTS, fixed slit, IFU, and MOS issues articles. For a detailed discussion of many aspects of the NIRSpec detectors and their impact on science calibration, see NIRSpec Detectors.
Please also refer to NIRSpec Calibration Status.
Artifacts
Spurious features in extracted 1-D spectra
Pixels flagged as "DO_NOT_USE" in the data quality (DQ) array are excluded from spectral extraction, with the possibility of causing divots in the final 1-D spectrum. If the flagged pixel is close to the center of the point spread function (PSF) at a given wavelength, the absence of the signal in the flagged pixel will lead to a divot at that wavelength in the extracted spectrum.
A pixel_replace step has been created for the pipeline, based on an adjacent profile approximation, which generates an average spatial profile from columns immediately adjacent to the one with the bad pixel, normalizes that profile to the good data in the affected column and uses the value of the corresponding pixel in the normalized profile to replace the signal in the bad pixel. Currently this step uses a default-off configuration for all NIRSpec modes. Use of this step can introduce artifacts when pixels are over-corrected by the spatial profile normalization. Bad corrections may have worse impacts on calwebb_spec3 reductions than no corrections.
The NIRSpec team continues to explore options for handling missing pixel data. In the meantime, pipeline users can test the use of the pixel_replace step on their data sets by setting skip = False in the parameters for the pixel_replace step in the stage 2 pipeline.
Snowballs
The detectors used in NIRSpec experience cosmic ray events known as snowballs and showers. Pipeline algorithms to address these artifacts are under development; in the interim, the best mitigation strategy is to dither.
Pipeline notes
See known issues pages for NIRSpec BOTS, fixed slit, IFU, and MOS modes.
Alternating column noise
A pattern of alternating brighter/darker pixel rows over part or all of a detector image is an effect sometimes referred to as alternating column noise (ACN; note that pixel rows in the science data actually represent columns in the detector frame of reference, as the data are rotated to put the dispersion direction horizontally). This is due to the two amplifiers (one for odd and one for even columns) in an output sometimes having slightly different offsets because of drift or a cosmic ray event. Pipeline code testing is underway to mitigate this odd-even column discrepancy.
The striped pattern of alternating bright/dark pixel rows, including the bright horizontal band, is seen in the lower half of this rate image file.
Uncorrected correlated noise - 1/f noise
JWST's near-infrared HgCdTe detectors have a correlated noise, known as 1/f noise, that is introduced by the detector readout system (Moseley et al. 2010). This noise results in vertical striping, as illustrated in the left panel Figure 2. With NIRSpec's IRS2 readout mode, reference pixels are interspersed with the sampling of normal pixels to mitigate the 1/f noise. However, this noise will still be present in data taken with NIRSpec's traditional detector readout mode; this includes all observations made with NIRSpec's subarrays, where IRS2 is unavailable. In these cases, it may be necessary to use background pixels in the horizontal bands to estimate and subtract the 1/f noise. The accompanying mode-specific pages further discuss 1/f noise.
To address this issue, the JWST Science Calibration Pipeline has integrated options for correcting 1/f noise in calwebb_detector1 in the clean_flicker_noise step and within calwebb_spec2 under the nsclean step (originally named for the "NSClean" algorithm, Rauscher 2024, developed by Bernard Rauscher). Both steps provide the same options for cleaning 1/f noise with the key difference that the clean_flicker_noise step in calwebb_detector1 fits and removes 1/f noise on a group-by-group basis, whereas the nsclean step in calwebb_spec2 performs 1/f cleaning on rate files. The algorithms use dark areas of the detector to fit a model of the 1/f noise either in Fourier space or using the column-wise median of dark regions. It requires an input mask to identify all dark areas of the detector. The more thorough and complete this mask is, the better the background fit. The pipeline creates an on-the-fly mask using default parameters to remove 1/f noise, and the user should refer to the optional parameters for this pipeline step if the default settings are not sufficient for the data. If needed, see the NSClean documentation for suggestions on manually creating a custom mask. The right panel of Figure 2 shows an example of a 1/f cleaned image using clean_flicker_noise.
Further information on cleaning 1/f noise and setting recommended parameters using the JWST Science Calibration Pipeline can be found here: 1/f Noise.
Illustration of vertical striping due to 1/f noise seen in NIRSpec/MOS data taken in traditional readout mode (left), and the same data after it has been cleaned by the JWST pipeline clean_flicker_noise step (right).
Pixel rows exhibit anomalous count rates in "rate" images for IRS2 data.
Some reference pixels are "bad" (e.g., excessive noise or "telegraph" behavior) and are flagged as such. However, those flags are not considered in the refpix step of the calwebb_detector1 pipeline, resulting in anomalous corrections for the associated science pixels, leading to image behavior in Figure 3. A flag check has been implemented in the pipeline, along with an improved algorithm to filter out noisy reference pixels.
Excessive noise, or 'telegraphing' behavior, is highlighted in the figure for two groups of rows.
No flux or negative flux in FS or MOS extracted 1-D spectrum
The extract_1d step uses an automated centering routine to place the extraction aperture, based on the source coordinates. The extraction aperture can be offset along the spatial axis, for example, due to an error in target coordinates or the world coordinate system. The assumed source position should work well for most NIRSpec sources, but in case the extraction aperture is still poorly placed, it can be manually tweaked by specifying start and stop positions to the extraction step. (Extraction parameters can be set in a reference file in JSON format, described in the pipeline documentation.) Instead of using the sky position, one can use the relative slit position of the target, as planned in APT, to place the aperture. Following a successful target acquisition, this position is typically known with high accuracy (~5–10 mas). The automatic centering can be overridden by setting the optional step argument use_source_posn = False, and modifying the reference file parameters ystart and ystop (found in the pipeline output log file) to define the desired boundary of the extraction aperture (in integer pixels or the use of polynomial for fractional pixels). More on this can be found on the NIRSpec MOS Known Issues page.
Summary of common issues and workarounds
The sections above provide detail on each of the known issues affecting NIRSpec data; the table below summarizes some of the most likely issues users may encounter along with any workarounds, if available. Note that greyed-out issues have been retired, and are fixed as of the indicated pipeline build. Information provided in this table affects all NIRSpec observing modes. For mode-specific issues, see the relevant known issues pages:
Symptoms | Cause | Workaround | Fix build | Mitigation Plan |
---|---|---|---|---|
NS-G03: Extracted 1-D spectrum ("x1d" product) has spurious positive and negative features, typically one pixel wide. | Pixels marked as "DO_NOT_USE" in reference data are not included in the 1-D extracted spectrum. | Workaround: Use the new Science Calibration Pipeline software (jwst 1.11.0 onwards) to estimate the value for each "DO_NOT_USE" pixel from neighboring 2-D profiles before extracting the 1-D spectrum by running the pixel_replace step. This is disabled by default. Testing has shown improvements in many cases, but degradation in data quality has also been seen in certain circumstances; users should exercise caution if they choose to run the step. | N/A | Updated issue The team is evaluating options for further improvements. |
NS-G06: A pattern of alternating brighter/darker pixel rows over part or all of a detector image. | An effect sometimes referred to as alternating column noise (ACN; note that pixel rows in the science data actually represent columns in the detector frame of reference, as the data are rotated to put the dispersion direction horizontally). This is due to the two amplifiers (one for odd and one for even columns) in an output sometimes having slightly different offsets because of drift or a cosmic ray event. | For traditional readout modes calwebb_detector1 refpix.odd_even_columns is on by default. If alternating column noise is seen in data taken in these modes, it may be improved by turning off refpix.odd_even_columns. For IRS2 readout modes refpix.odd_even_columns is disabled by default. As of jwst 1.14.0 and later refpix.odd_even_columns can be used for IRS2 modes. Testing has shown improvements in some cases, but degradation in data quality has also been seen in certain circumstances; users may try reprocessing with this setting on, but should exercise caution if they choose to do so. | N/A | Updated issue The team is investigating additional improvements to the treatment of alternating column noise in IRS2 modes. |
NS-G09: Uncertainties in the combined background images produced in the background step are not propagated into science data. | The background step does not propagate uncertainties from the background images to the 2-D science arrays | Uncertainties from the the combined background images can be recovered by saving the combined background using background.save_combined_background = True and combining the uncertainties from this image with the science data uncertainties from the background step. | N/A | Created issue Work on a fix is pending an investigation into the necessary implementation. |
NS-G01: A few pixel rows exhibit anomalous count rates in "rate" images for IRS2 data. | Some reference pixels are "bad" (e.g., excessive noise or "telegraph" behavior) and are flagged as such. However, those flags are not considered in the refpix step of the calwebb_detector1 pipeline, resulting in anomalous corrections for the associated science pixels. | None. | Updated Operations Pipeline A flag check has been implemented, along with an improved algorithm to filter out noisy reference pixels. STScI will reprocess affected data products with an updated Operations Pipeline, planned for installation on . Reprocessing of affected data typically takes 2–4 weeks after the update. | |
NS-G02: No flux or negative flux in FS or MOS extracted 1-D spectrum ("x1d" product), despite obvious flux in 2-D spectrum ("s2d" product). | Extraction aperture is offset along the spatial axis, for example, due to an error in target coordinates or the world coordinate system. For targets with large proper motion, see | For high proper motion targets, if the processing date ( Rerun the extract_1d step with a modified "extract1d" reference file containing the desired extraction aperture location (using either ystart and ystop or src_coef) and the parameter use_source_posn set to False. See this workaround notebook for an example of how to do this using FS data. | Updated Operations Pipeline Updates to spacecraft pointing keywords, which should improve the spectral trace WCS, have been available in the Operations Pipeline since on 24 August 2023. Reprocessing of affected data typically takes 2–4 weeks after the update. An alternate method of extraction aperture centering for MOS and FS modes was implemented, based on the planned slit position of each source, therefore avoiding sky coordinates altogether. This new algorithm was installed in the Operations Pipeline on December 5, 2023. | |
NS-G04: Unexpected flux levels in spectra of sources not centered in a slit. | A bug was found in the code used to interpolate the pathloss correction from the "pathloss" reference data. Affects FS and MOS point source data. | None. | Updated Operations Pipeline A bug fix was implemented and made available in the Operations Pipeline installed on December 5, 2023. Reprocessing of affected data typically takes 2–4 weeks after the update. | |
NS-G05: Wavelengths of spectral features do not match expectation in spectra of sources not centered in a slit. | The calwebb_spec2 pipeline performs a wavelength correction for offset point sources based on their expected position within the slit. However, the correction values are not being propagated to the resample_spec step, so the "s2d" products always have the wavelength scale appropriate for centered sources only. Affects FS and MOS point source data. | Reprocess spectra using the new version of the calibration pipeline software (jwst 1.15.0 or later). The wavelength corrections for point sources are now applied correctly and propagated to the resample_spec step. There are no such corrections applied for extended sources, whose wavelength corrections depend on the source geometry. | Updated Operations Pipeline A fix for this issue has been implemented in the jwst calibration software (jwst 1.15.0 and later) and was made available in the Operations Pipeline installed on August 29, 2024. Reprocessing of affected data typically takes 2–4 weeks after the update. | |
NS-G07: In some data from the NRS2 detector taken with NRSIRS2 readout patterns, users may notice an entire row of pixels with the incorrect DQ flag value, and an excess of good pixels in contiguous areas incorrectly flagged as "hot". | The mask reference file for the NRS2 detector and NRSIRS2/NRSIRS2RAPID readout patterns, covering all data taken after , was incorrectly generated. The snowball correction was not applied to the dark data from which the hot pixel population is determined, leading to many pixels affected by snowball events being inadvertently flagged as "hot". In addition, a file conversion error led to one row of pixels to be flagged incorrectly as reference pixels. Note that there is no effect on the science data, other than that a small excess of pixels will be rejected during stage 3 combination. | Re-process data using the updated calibration reference files in CRDS context jwst_1234.pmap or later. | Updated Operations Pipeline A corrected reference file was delivered to CRDS and was available starting. These files are used with pipeline build 10.2 released mid-June. Reprocessing of affected data typically takes 2–4 weeks after the update. | |
NS-G08: Data with associated background or imprint observations that is reprocessed locally with 1/f cleaning using the nsclean step does not remove 1/f noise. | The nsclean step in calwebb_spec2 currently only runs on the science members in an association. Background and imprint subtraction reintroduces 1/f noise, because these members in the association have not been cleaned of 1/f noise through the nsclean step. | Run calwebb_spec2 with nsclean on all science and background members, save the results of the nsclean step, and rerun calwebb_spec2 on association files that use the 1/f cleaned rate files as inputs. | Updated Operations Pipeline A fix for this issue has been implemented in the jwst calibration software (jwst 1.16.0 and later). Associations reprocessed with nsclean will now run the nsclean step on all science, background, and imprint members in an association. |
References
Moseley, S. H., et al. 2010 SPIE Proceedings Vol. 7742
Reducing the read noise of H2RG detector arrays: eliminating correlated noise with efficient use of reference signals
Rauscher, B. J. 2024 PASP 136:015001
NSClean: An Algorithm for removing Correlated Noise from JWST NIRSpec Images