JWST NIRSpec MOS Pipeline Caveats
An overview of data products specific to the NIRSpec multi-object spectroscopy (MOS) mode is provided in this article. It also outlines known issues and limitations of the JWST spectroscopic pipeline for MOS data, and describe options and workarounds for several use cases.
See also: NIRSpec Multi-Object Spectroscopy
Summary of specific MOS pipeline issues
The information in this table about NIRSpec MOS pipeline issues is excerpted from Known Issues with JWST Data Products.
|NS-MOS01: Significant outliers appear in the 2-D and 1-D extracted spectra.|
The outlier_detection step generally has a hard time finding many outliers. Step parameters need to be tuned to the noise characteristics of each data set, although in many cases outliers are still missed.
For outlier improvement, rerun the outlier_detection step in calwebb_spec3 with different values of the snr parameter.
Updated algorithms are under investigation, possibly for inclusion in the Science Calibration Pipeline in February 2024.
|NS-MOS02: Level 3 extracted spectra have errors that are all "NaN".|
The flat field reference file uncertainties are currently zero. The resultant flat error component calculated from these is "NaN", which propagates to the combined error as "NaN".
Recalculate the combined error using only the read noise and photon noise components. See this worked example for more on how to do this.
Updated issue, new due date for reference file
Reprocess data with an enhanced calibration reference file (flat) in CRDS. An update is planned for early 2024. Reprocessing of old data typically takes 2–4 weeks after the update.
|NS-MOS03: Spectra extracted from slitlets consisting of more than 3 shutters exhibit unexpected wavelength-dependent flux discrepancies.|
A bug in the pathloss step prevents the application of the correction in longer slitlets, for either point or extended sources.
Edit the MSA metafile to break up slitlets longer than 3 shutters into smaller slitlets.
A link to a description of how to do this will be added here by .
An update to the science calibration pipeline code to enable application of the current set of reference files to any slitlet length is expected to be installed in the Operations Pipeline in February 2024.
|NS-MOS04: Negative and/or surplus flux seen in extracted 1-D spectra, typically with an irregular wavelength-dependent undulation.|
Correlated noise from low level detector thermal instabilities, seen as vertical banding in 2-D count rate images, particularly in exposures of the NRS2 detector. While the "IRS2" readout modes reduce this effect, it is not completely eliminated.
No workaround is available yet. It may be possible to improve the noise levels using the NSClean script developed by B. Rauscher on count rate images, using an appropriate mask. However, this has not yet been tested/verified by the team.
A notebook demonstrating the use of the NSClean algorithm is now available.
A workaround notebook is available.
Eventual inclusion of the cleaning algorithm in the pipeline is planned, pending further testing, possibly in February 2024.
|NS-MOS05: Unexpected variations are seen in continuum or line fluxes as a function of field position.|
The flux calibration for MOS is currently based on spectrophotometric observations at a single point in the MOS field of view. The flat field calibration accounts for spatial variations from the MSA aperture plane, through the grating wheel, and at the detector, but not in the OTE or filter wheel. Large variations (>~ 5%) are not expected, but this needs to be confirmed.
Spectrophotometric observations at multiple field points have been obtained and are being analyzed. Updates to the flux calibration, if needed, are planned in fall 2023.
NS-MOS06: Discontinuities in the extracted level 2 spectra and "s2d" images at the upper and/or lower nod positions. These may be as large as ~40%. The effects in level 3 products appear to be much smaller and/or absent.
Errors in rectification when producing the"s2d" image is due to missing WCS information at the edges of CAL image extensions.
Edit the MSA metafile to pretend that an extra shutter at the top and bottom of the slitlet was open. Note that this conflicts with the workaround suggested for NS-MOS03.
Modify the pipeline code to expand the size of the "cal" file cutout and the region with a valid WCS to be large enough to avoid resampling artifacts near the edges. A fix is tentatively planned for installation in operations in early 2024.
NS-MOS07: Lower than expected S/N and/or larger than expected discrepancies between dither positions.
Use of the default “inverse variance weighting” using read noise variance in the resample_spec step (resample_spec.weight_type = ivm) does not appear to be appropriate for high signal-to-noise data
When running calwebb_spec2, set resample_spec.weight_type = exptime, and when running calwebb_spec3, set spec3.weight_type = exptime.
Investigate whether to change the default weighting to exptime using a modified parameter reference file, and/or investigate other algorithms that handle the high S/N limit more gracefully. Also consider extraction and image combination algorithms that don’t require resampling the 2-D data.
MOS pipeline and products overview
Words in bold are GUI menus/
panels or data software packages;
bold italics are buttons in GUI
tools or package parameters.
In brief, the first stage operates as with any other data type, producing full frame count rate images (suffix "rate.fits"). The second stage applies various spectral calibrations including background subtraction (e.g., by subtracting nodded exposures), WCS calibration, and instrument- and slit-related throughput corrections. The 2-D spectral trace corresponding to each source observed in an exposure is extracted, the different throughput corrections are applied to each pixel, and then the 2-D spectra are rectified and 1-D spectra are created by summing over an extraction aperture at each output wavelength. In cases where sources are observed over multiple exposures, the data are combined at stage 3, in which the individual calibrated 2-D unrectified spectra are resampled onto a common rectified grid, and a single 1-D spectrum is extracted.
There are 3 main pipeline products for MOS data:
- calibrated unrectified 2-D spectra (suffix "cal.fits")
- calibrated rectified 2-D spectra ("s2d.fits")
- 1-D extracted spectra ("x1d.fits")
These are organized by source or exposure, depending on the level. For detailed examples, see the JWebbinar series (JWebbinar 1 for generic data product information, and JWebbinar 7 for MOS-specific details).
Metadata for source and slitlet information
One crucial element of the MOS processing at stage 2 and 3 is the MSA metafile. This is a FITS file with binary tables containing information on the sources and open shutters corresponding to each MSA configuration designed for an observation. Metadata such as the source names, coordinates, and open shutter locations are generated by the MSA Planning Tool (MPT) in APT and passed through the ground system to populate the metafile with the specific items needed by the pipeline. Each MOS exposure has a primary header keyword called
MSAMETFL that specifies the name of the metafile to be used for that exposure. Each metafile contains 2 binary FITS tables: "SHUTTER_INFO" specifies all of the slitlets containing one or more open shutters, as designed in MPT; "SOURCE_INFO" contains information for each source in the plan.
Table 1. An example of the "SHUTTER_INFO" table, for a 2-point nod with a single source
There are separate rows for each open shutter specified in the corresponding MSA configuration and planned nod position. In this example, a single 2-shutter slitlet was planned, with 2 exposures obtained by nodding the source between the 2 shutters of the slitlet.
SLITLET_ID: integer number representing each slitlet of one or more open shutters specified in the MSA configuration
MSA_METADATA_ID: integer number corresponding to a particular MSA configuration/MPT plan (typically the same for all rows in a given metafile)
SHUTTER_*: integer ID of the MSA quadrant, row, and shutter for each commanded open shutter in the configuration; note that "column" and "row" are defined in an instrument frame of reference, which is reversed in the science frame (e.g., columns are in the cross-dispersion direction, and vice-versa for rows)
SOURCE_ID: unique integer ID for each source in each slitlet, used for matching to the SOURCE_INFO table
BACKGROUND: boolean indicating whether the shutter is open to background (Y) or contains a known source (N) (for a given nod exposure if the observation includes nodding)
SHUTTER_STATE: generally this will always be OPEN, unless a long slit was used
ESTIMATED_SOURCE_IN_SHUTTER_X/Y: the position of the source within the shutter in relative units (where 0,0 is the bottom left corner and 0.5,0.5 is the center), as planned in MPT
DITHER_POINT_INDEX: integer specifying the index of the nod sequence; matches with the data primary header keyword PATT_NUM
PRIMARY_SOURCE: boolean indicating whether the shutter contains the science source
Table 2. An example of the SOURCE_INFO table
There are separate rows for each source from the parent catalog that lie within any of the planned slitlets.
PROGRAM: program ID
SOURCE_ID: unique integer identifier
SOURCE_NAME: typically a combination of the first two columns
ALIAS: integer identifier from the original source catalog (TBC)
RA/DEC: catalog source coordinates, in decimal degrees
PREIMAGE_ID: name of NIRCam pre-imaging mosaic used to determine the source catalog, if it exists
STELLARITY: DAOphot-style stellarity parameter, where 0 is fully extended and 1 is a perfect point source
The automated JWST pipeline for MOS data processing has been designed around "typical" use cases (e.g., multiple slitlets consisting of one or more open shutters, each containing a single source, with or without nodding). Given the range of configurations possible for the MSA, and many different types of sources, not every observation can be treated optimally by default. Outlined below are several calibration considerations specific to MOS observing, known limitations of the current state of the pipeline, and some aspects that can or should be changed through reprocessing of the data.
Background subtraction: nods vs master background
There are 2 processing flows for background subtraction of NIRSpec data: nod or master background subtraction. In the MOS case, nod subtraction is done automatically when the observation contains 2 or more associated exposures in which the sources have been moved to different shutters in their slitlets for the same MSA configuration. This is a straight subtraction of full frame images, performed before the 2-D spectrum of each source is extracted. For this reason, if nodded background subtraction is desired for a subset of sources on a configuration, the data will have to be processed twice: once with nodding, and again with master background subtraction. The efficacy of this approach depends primarily on the spatial extent of the sources vs. the separation of the nods.
In the master background approach, dedicated background slitlets are opened on areas in the field of view that ideally do not contain sources. The pipeline automatically extracts and combines these into a master 1-D background spectrum, which is then resampled into the 2-D space of each source spectrum and subtracted. This method improves observing efficiency, and is preferred for significantly extended sources; its efficacy depends on the S/N of the master background, the level of uniformity of the background over the field, and the resampling accuracy. The set of slitlets (and particular shutters within slitlets) used to construct the master background can be changed, for example to remove one that contains a previously unknown contaminating source, by editing the MSA metafile and reprocessing. Note that the failed open shutters and fixed slits can also be used to help construct the master background; however, the workflow is more complicated. Users who wish to explore these options should contact the JWST Help Desk for instructions.
Source type determination
The processing flow and some of the throughput corrections that are applied are different depending on whether a source is point or extended. For MOS data, the pipeline determines the source type for each source based on the value of the stellarity parameter in the MSA metafile. For stellarity > 0.75, the source type is set to "POINT"; for stellarity between 0 and 0.75, source type is set to "EXTENDED". The stellarity values are propagated from the source catalog used in MPT; if the catalog did not specify stellarity, then the pipeline defaults to a source type "POINT".
Note that for NIRCam pre-imaging or any other JWST imaging observations, the source catalog generated by the imaging pipeline does not calculate stellarity. Instead, a flag called "IS_EXTENDED" is set for each source depending on its measured concentration indices. In these cases, MPT converts this flag to the corresponding stellarity limits (e.g., IS_EXTENDED = 0 becomes stellarity = 1), and passes those values to the Proposal Planning Subsystem (PPS) database where they are eventually incorporated into the metafile(s).
The source type assignment by the pipeline can be overridden for any given source by changing the stellarity value in the MSA metafile to give the desired result.
For point source processing, several steps depend on the relative position of the source within its shutter (pathloss correction, wavelength zero point correction for offset point sources). The automated pipeline uses the positions as planned in MPT, stored in the "ESTIMATED_SOURCE_IN_SHUTTER_X/Y" columns in the MSA metafile. These positions may be slightly different from the actual values in an exposure, depending on the accuracy of the MSATA. If a confirmation image was taken, the actual position of each source can be measured, converted to relative slit units, the values in the MSA metafile replaced, and the data then reprocessed to improve the calibration accuracy (however, any improvement would depend on the measurement accuracy of the source centroid). A future enhancement is planned to automate such a process in the pipeline, but for now this can only be done manually.
Calibration of extended sources
Two source type-dependent processing flows have been designed for 2 limiting cases: spatially unresolved point sources and uniform illumination of a slit aperture. Most, if not all, extended sources will lie somewhere in between these 2 limits. The calibration of such sources is dependent on the exact source geometry as a function of wavelength, and is thus unique to each source and cannot be fully treated by any automated pipeline. Users will need to customize the reprocessing of the data in these cases, in particular a correction for the geometric slit losses. In commissioning, we characterized the 2-D slit loss function by observing a point source at a grid of positions over the open area of a micro-shutter, for a small subset of shutters. This can then be convolved by the known source geometry (e.g., through forward modeling) to estimate the total slit losses for the integrated spectrum.
Flux calibration considerations
The MOS flux calibration is set by various throughput corrections applied in the flat field step of the pipeline. The current set of reference files used for this are based on pre-launch models and result in systematic flux calibration uncertainties in the range of 15%-40% (typically decreasing with longer wavelengths). Reference files based on in-flight measurements are still under construction, and are expected to significantly reduce these uncertainties.
1-D extraction considerations
The size of the 1-D extraction aperture in the cross-dispersion direction is defined by the reference file for the extract_1d step of the calwebb_spec2 pipeline, via the parameter extract_width (in units of pixels). For MOS observations, when the source type is a point source, the center of the extraction aperture is automatically offset to the expected position of the source within its slitlet. This offset has been shown to be inaccurate in some cases, so users should check the fidelity of the 1-D products and reprocess with different centering if needed. The automatic centering can be overidden by setting the optional step argument use_source_posn to False, and modifying the reference file parameters ystart and ystop to define the desired boundary of the extraction aperture (in pixels). The extract_1d step parameters are global, and currently cannot be modified for different sources in the same exposure. There are other optional reference file parameters that allow specification of a boxcar extraction with height that varies with wavelength (the default is a simple rectangle), as well as a background region for background subtraction at the 1-D extraction stage; these options have not been adequately tested and are not recommended at present. Note that aperture corrections for extraction apertures that are larger or smaller than the default size are not yet available, so in such cases users must calculate their own corrections using point source data.
Unidentified optical shorts
One known characteristic of the MSA is that electrical shorts can occur at a particular shutter location. There is a procedure in place that identifies these shorts using telemetry of electrical currents and masks them in order to prevent the possibility of damage to the array. However, some shorts are too weak to be readily identified via elevated current, but rather manifest themselves by producing a glow caused by heating of the area around the shutter. These "optical" shorts are identified using dark exposures and subsequently masked; while the incidence of new shorts is expected to be very low, there is a possibility that some may appear in between calibration checks during normal operations, which could then contaminate science data. This would manifest in dispersed data as a spectrum with roughly a blackbody shape. We recommend that users check for any unexpected dispersed signal in their images not related to the already-known failed open shutters (which are flagged in the DQ array of the level 2b data products). Nod subtraction should remove this signal, though the noise level will be somewhat elevated.
Long slit processing
Observations using one of the pre-defined long slit configurations (Q4 field point 1 or 2) require somewhat different handling compared to the slitlet case. Because the long slits span the full breadth of 2 MSA quadrants, containing a total of 342 shutters (including some non-operable and vignetted), each shutter needs to be extracted separately in order to preserve wavelength and positional accuracy. This is contrary to the slitlet case, where the calibration is based on the single shutter that contains the source in a given exposure. The full processing flow for the long slit case, particularly in terms of how the metadata should be generated by MPT, is still being defined. For now, users should reprocess long slit data through the calwebb_spec2 pipeline, starting with the level 2a (rate.fits) product, with a modified MSA metafile as described below:
- each shutter in the long slit should be specified as a separate "slitlet" with unique slitlet ID in the SHUTTER_INFO table
- each shutter that is expected to contain light from the target source should be given a placeholder source in the SHUTTER_INFO table, with unique SOURCE_ID, BACKGROUND = N, PRIMARY_SOURCE = Y, DITHER_POINT_INDEX = 1
- each source ID set above should have a corresponding entry in the SOURCE_INFO table, with arbitrary coordinates and stellarity = 0 (to ensure that the extended source processing flow is used)
- each shutter that is expected to contain only background light should be designated a background shutter in the SHUTTER_INFO table (BACKGROUND = Y, PRIMARY_SOURCE = N); these will then be used to generate a master background (if background subtraction is not desired, this can be skipped by specifying sources in all shutters)
The primary data product will then be a calibrated 2-D spectrum for each designated source shutter. In cases where the long slit has been "stepped" across the source in the dispersion direction, the 2-D spectra can in principle be combined to form a data cube analogous to that of the IFU case. However, the pipeline currently has no mechanism for building such a cube from MOS data—adapting the IFU cube_build step for this purpose is a future planned enhancement. Also note that the caveats given above for extended source calibration apply here, so combination of overlapping dithered exposures should be done with extreme caution.
Level 3 spectral combination
Users should treat automated level 3 products with caution. Some important considerations depending on the source type and specific science requirements:
- Pipeline stage 3 processing includes an outlier rejection step that depends on the noise properties of the individual exposure-level input data. The "crf.fits" output of this step should be checked to verify the performance. If too many pixels are being rejected, raise the S/N threshold using the snr step argument until the results look reasonable. The step should be turned off if the number of input exposures is small (~3 or less).
- As mentioned above, accurate combined products for extended sources require custom slit loss corrections that the automated pipeline cannot treat.
- For observations that include widely-separated dithers in which each source is located at very different relative slit locations, the spatial profile of a point source will vary considerably because of the different slit losses. The pipeline does not apply a correction to the spatial profile, so the combination of 2-D data may lead to larger uncertainties in the combined s2d product. For such cases, users may want to combine the exposure-level x1d products instead by running the optional combine_1d step in place of the resample_spec and extract_1d steps of the calwebb_spec3 pipeline.