Release notes for build 9.3 and patch 9.3.1 of the JWST Operations Pipeline, both released on August 24, 2023, are covered in this article.

This article provides a list of all relevant changes to the operations pipeline that was released with build 9.3 and patch 9.3.1. These focus mainly on updates to the JWST Science Calibration Pipeline, but includes information relevant to changes in other subsystems that might affect what users see in their data. 

Highlights of this build

Build 9.3 and patch 9.3.1

• Words in bold are GUI menus/
  panels or data software packages; 
  bold italics are buttons in GUI
  tools or package parameters.

  The MIRI MRS flux calibration, applied in the photom step of the calwebb_spec2 pipeline, has been upgraded to include corrections for time-dependent changes in throughput.
• The jump step in the calwebb_detector1 stage of the pipeline  has had several updates and enhancements to make the detection of snowballs/showers more robust and reduce the frequency of "false positives".
• The outlier_detection algorithm for IFU data, applied in the calwebb_spec3 stage of the pipeline, has been completely reworked. It now focuses on finding and flagging pixels that are bad in all input images.
• The extract_1d step has been upgraded to apply a residual fringe correction to MIRI MRS 1-D extracted spectra.
• The photom step in the calwebb_spec2 stage of the pipeline has been updated to calibrate all NIRSpec IFU exposures to units of surface brightness, rather than flux density, for compatibility with the cube_build step's resampling process.
• A new pixel_replace step has been added to the calwebb_spec2 stage of the pipeline, to replace the values of bad pixels with estimates from surrounding good neighbors, in order to avoid issues with dropouts in extracted 1-D spectra.
• The pointing calculations used in populating WCS keywords in raw ("uncal") products have been updated to fix a bug that was causing large offsets in the WCS for observations that use FGS2 for guiding.

Known issues

Issues that affect the JWST data products available from the MAST Archive and that were not resolved with this build are covered in the Known Issues with JWST Data Products article. When there is a workaround for an issue, the relevant information is provided there.

Data affected by this build

Provided below is a list of products in MAST generated by the JWST Science Calibration Pipeline that may be most significantly affected by the updates in build 9.3. The changes listed in the release notes may affect other modes as well, but many of these will be transparent to the MAST user.

The data and pipeline stage affected for data in MAST are:

• MIRI LRS: Stage 2 and beyond
• MIRI MRS: Stage 2 and beyond
• NIRCam WFSS: Stage 2 and beyond
• NIRSpec fixed slit: Stage 2 and beyond
• NIRSpec IFU: Stage 2 and beyond
• All TSO data

Release notes

See also: GitHub jwst project release notes 1.11.3

Build 9.3 and patch 9.3.1 of the JWST Operations Pipeline were released on August 24, 2023, to the Barbara A. Mikulski Archive for Space Telescopes (MAST). The build  and patch include changes to the JWST Science Calibration Pipeline released in versions 1.11.0 to 1.11.4 of the jwst calibration pipeline package via the PyPI repository.

The following fixes or changes were included mainly in build 9.3. Changes to specific steps in the pipeline are listed below, along with descriptions to more general changes made to specific packages and related subsystems. This release also includes changes to the JWST Science Data Processing subsystem (SDP) version JWSTDP-2023.2.1 which creates raw (uncal) products.

assign_wcs step

• Updated to pass the dispersion relation to NIRCam WFSS row/column transforms, to interpolate against if an analytic inverse transform does not exist. This change fixes the offsets between imaging and WFSS traces so that the pipeline products have the right target in the extraction boxes. This change require  a new set of reference files that will not work with previous versions of the code.

background step

• The introduction of "NaN" values for pixels flagged as "DO_NOT_USE" caused a bug in background subtraction for WFSS images. The step was updated to mask out "NaN" pixels in WFSS images before removing outlier values and calculating the robust mean values from the background regions in the WFSS image and for the same regions in the background reference image.

cube_build step

• Removed deleting the "spaxel_dq" array twice when using a weighting method of either Exponential Modified Shepard Method (emsm) or Modified Shepard Method (msm).
• Updated to read the wavelength range for NIRSpec IFU cubes from the "cubepars" reference file, instead of setting it based on the data. This makes use of new NIRSpec IFU "cubepars" reference files with wavelength arrays included for the drizzle method. Given the barycentric wavelength correction (which can differ between observations) and other factors, the NIRSpec cubes are not always consistent with each other, so determining the default wavelength range dynamically from the data can make the comparisons between cubes hard.

dark_current step

• Fixed the way segmented exposures are handled when processing MIRI exposures, so that the correct integration-dependent data from the dark reference file is applied to the corresponding integrations in the science exposures.

extract_1d step

• Changed the logic for handling NIRSpec IFU data, so that both point and extended sources are treated the same. The units of point sources are changed to surface brightness in the photom step, and in the extract_id step all sources are converted back to flux density.
• Changed IFU source location to floating point from integer. This change allows for better centering and extraction for point sources, which otherwise could result in a mis-centering of the aperture by as much as 0.5 spaxels.
• Added the ifu_autocen option to automatically find point source centroids in IFU cubes using DAOStarFinder, rather than relying on the source coordinates, to estimate its position and center the extraction aperture. This addresses some issues with incorrect extraction locations due to both errors in the guide star derived world coordinates and inaccuracies in the nominal RA/Dec of the target (provided in the proposal).
• Added the ifu_rfcorr option to select whether on not to apply 1-D residual fringe correction to extracted MIRI MRS spectra. The default is false.

flat_field step

• Added log messages to report user-supplied reference file used.

jump step

• Updated the jump detection to use the numpy "sigmaclip" routine to find the actual rms across integrations when there are at least 101 integrations in the exposure. This addresses the changing count rate when there are high contrast bright sources, which leads to the Brighter-Fatter/Charge-Spilling where the bright pixels lose charge to neighbors; i.e., ramps that are curved after the nonlinearity correction. This still allows cosmic rays and snowballs/showers to be flagged without being affected by slope variations due to either "brighter-fatter," charge-spilling, or errors in the nonlinearity correction. Three new parameters were added to the jump step to control this process.
• Updated  the step to compute the cosmic ray rate and the snowball/shower rate and store those values in the new header keywords PRIMECRS and EXTNCRS, respectively.

outlier_detection step

• Updated the detection algorithm applied to IFU data. The new algorithm works exclusively within the 2-D space of IFU slice images, rather than 3-D cubes, in order to find and flag pixels that are deemed to be bad in all input images. This significantly reduces the number of false positive detections that plagued the previous IFU algorithm and enables the flagging of detector pixels that have bad behavior, but are not permanently bad (and hence not flagged in the static bad pixel mask applied in the dq_init step).

pathloss step

• Bug fix for NIRSpec fixed slit data to remove double application of correction factors.

photom step

• To ensure flux conservation after "cal.fits" products are resampled in the cube_build step, the NIRSpec point sources in IFU data are now converted to units of surface brightness (MJy/sr).
• The MIRI MRS has been observed to have a time-variable photometric response that is particularly significant at long wavelengths. Thus far, it appears to be possible to describe this time-dependent throughput using an exponential function that asymptotically approaches a constant value in each band. In order to provide well-calibrated science data, the  photom step now includes a  time-dependent flux calibration correction for MIRI MRS data, which utilizes time-dependent correction factors. The coefficients for each band in which a correction has been determined are stored in the "photom" reference file for MRS mode.

pixel_replace step

• During spectral extraction, pixels flagged as bad are ignored in the summation process. If a bad pixel is part of the point spread function (PSF) at a given wavelength, the absence of the signal in the flagged pixel will lead to a divot at that wavelength in the extracted spectrum. To avoid this defect in the 1-D spectrum, this new step is Implemented in the calwebb_spec2 stage. The step uses a weighted interpolation to estimate and replace flux values for pixels flagged as "DO_NOT_USE". Affected pixels are flagged with the new DQ value "FLUX_ESTIMATED". Note that by default this step is set to be skipped, but it can be enabled through appropriate calwebb_spec2 parameter reference files. Currently it is only applied by default to MIRI LRS exposures. The step has arguments to specify the number of adjacent columns to use in the creation of the source profile.
• Fixed a bug to properly set the step completion status in the S_PXREPL keyword at the end of processing. Only when the step is applied the value will be set to COMPLETE.

ramp_fitting step

• Fixed a bug in the way the number of groups in a segment is computed when applying optimal weighting to line fit segments, which affects the resulting count rate.
• Updated to use the correct effective photon collection time for ramps in which only the first group is usable. The first group time is not the same as the normal elapsed time between successive groups. Count rates previously computed for these cases were too low.

residual_fringe step

• The 2-D residual fringe correction was previously implemented in the JWST Science Calibration pipeline as an optional step in order to help correct for fringes remaining in the data after standard application of a static fringe flat. The same approach can also be applied to 1-D extracted spectra with extremely good results. In this release, the utilities code was updated to add functions for the MIRI MRS residual fringe correction, which is applied to one-dimensional spectra within the extract_1d step.

refpix step

• The NIRSpec subarrays SUB512, SUB512S, and SUB32 don't reach the edge of the detector and as such lack real reference pixels. This can cause significant issues during the ramp step when the pedestal level is variable, and at low count rates can lead to errors in the derived flux of a few tens of percent or more. However, for each of these subarrays, the recommended use (PRISM with SUB512 and SUB512S, MIRROR with SUB32), leaves the pixels along the left and the right edges (in the science frame) unilluminated, and therefore these can be used as reference pixels and resolve the issues seen during the ramp step. With this build, the step was updated to assign reference pixel DQ flags to the last 4 columns of the above NIRSpec subarrays, which are unilluminated pixels. Previously the refpix step was skipped for such subarrays.
• For NIRSpec NRSIRS2RAPID and NRSIRS2 readout pattern, a set of 4 reference pixels are read in an interleaved way for every 16 science pixels, in order to correct for correlated noise and improve detector noise performance in general. However, some of these reference pixels are bad, i.e., show excessive noise and/or "telegraph" behavior, with the latter meaning that the reference pixel is toggling between states. Using these pixels in the time series that is created for IRS² processing is detrimental to the data quality. As the IRS² reference correction is done at the group level before the jump detection step, a bad/toggling reference pixel can also impact the jump detection and lead to streaks of erroneously flagged jumps. The refpixel step uses a reference file to correctly mask bad pixels and with this build, the step was updated to replace intermittently bad pixels with the nearest good reference pixel for NIRSpec IRS² images.

srctype step

• The srctype step in the calibration pipeline attempts to determine whether a spectroscopic source should be considered to be a point or extended object, populating the SRCTYPE keyword with a value of either POINT or EXTENDED. This step was updated so that when the observer sets the source type value in APT, saved in the SRCTYAPT keyword, this takes precedence over PATTTYPE keyword, which provides the name of the primary dither pattern and is assumed to be used only with point-like targets. This affects MIRI LRS, MIRI MRS, NIRCam TSGRISM, and NIRSpec bright object, fixed slit, and IFU modes.

tweakreg step

• Fixed a bug in the tweakreg step that resulted in a TypeError exception when user supplies a custom source catalog  as a member of an ASN file.
• Added a tool transfer_wcs_correction to jwst.tweakreg.utils that applies the same alignment corrections from one file/datamodel to another. It is an analog of the tweakback task in drizzlepac, with the difference that for JWST the tangent-plane correction in WCS is cumulative/total correction. This is not not used in the operations pipeline.
• Added the Gaia Data Release 3 catalog, parameter GAIADR3, available via the MAST Catalogs access services, to the available options for absolute alignment. Also enabled proper motion corrections to the catalog positions based on the epoch of the observation being processed. This catalog has been enabled as the default option.
• Fixed a bug in the "adjust_wcs" function that was preventing passing negative angular arguments in exponential format. Also added support for units for angular arguments to the "adjust_wcs" utility function available within the step module. This is part of the functionality released with build 9.2 to allow an initial solution to be input to tweakreg (shift, rotation, scale) with the option of not having tweakreg do any additional fitting.

Pipeline modules

• Added the new pixel_replace step to the calwebb_spec2 stage of the calibration pipeline.

Product associations

• Updated the stage 2 and stage 3 association generator rules to properly handle NIRCam coronagraphy observations that use all 4 short-wave detectors. A coronagraphic association is created using only the exposures from the detector on which the occulted target is located. An imaging association is created using data from all detectors. Thus two sets of stage 3 products are created: a PSF-subtracted coronagraphic data for the primary target, as well as regular imaging products of the entire field.

General changes

• Documentation updates:
  • Update references to datamodels in all docs to point to stdatamodels, which now provides datamodels for the jwst package.
  • Update the extract_2d step docs to give better descriptions of how to create and use object lists for WFSS grism image extractions.
  • Fixed minor formatting typos in associations documentation.
  • Added note to calwebb_spec2 step table to clarify the swapped order of the photom and extract_1d steps for NIRISS SOSS data processing.
  • Fixed bugs in implementation of pixel_replace documentation.
  • Update tutorial usage of jump.threshold to jump.rejection_threshold.
  • Updated calwebb_spec2 docs to reflect the fact that the MIRI MRS straylight step now comes before the flatfield step.
  • Removed references to deprecated jwst.datamodels.DataModels.
  • Added link to the JWST help desk on the top documentation page.
• Datamodels updates:
  • Removed use of the deprecated stdatamodels.jwst.datamodels.DataModel class from all steps and replaced it with stdatamodels.jwst.datamodels.JwstDataModel.
  • Updated stdatamodels.jwst.datamodels.outlierpars schema to include new parameters needed for the new IFU outlier detection algorithm.
  • Updated the MIRI MRS "photom" reference file data model to support the addition of time-dependent correction factors.
  • Added 2 new header keywords TARGCAT and TARGDESC, to store values supplied by the proposer in the APT.
  • Added the header keyword GSC_VER, to store the version of the Guide Star Catalog used in each observation.
• Other
  • Python 3.8 is not longer supported

JWST Science Data Processing subsystem changes

The JSDP subsystem processes data from the telescope and creates raw (uncal) products. The following changes have been included in this build:

• The algorithm used to do the initial population of WCS-related keywords in raw products contained a bug that led to large offsets (2 arcmin or more) in the pointing information for exposures that use FGS2 for guiding. This bug has been fixed, such that pointing information is consistent between the FGS1 and FGS2 guiders.
• Corrected a bug in the code that computed the barycentric,heliocentric time, and JWST barycentric position keywords in the "PRIMARY" and "SCI" FITS extensions, as well as the time values in the "GROUP" extension table. The bug resulted in an inconsistency between the "INT_TIMES" table and these values. 

• MSA metadata FITS files will now be created for NIRSpec exposures that use "canned" MSA configurations without assigned slitlets. Previously, all exposures with no slitlet data were assumed to be "all closed" and did not trigger metadata file creation, so these data would not process through stage 2 calibration. Since these configurations do not have slitlet information, the "SHUTTER_INFO" extension (extension 2) will use default data for open shutters; since these exposures do not have source information, the "SOURCE_INFO" extension (extension 3) will be empty. Additionally, an issue where the code repeatedly flipped shutter states due to too many rows being retrieved from the database for the same shutter has been fixed.

• Adjusted the calculations for TARG_RA, TARG_DEC to properly account for proper motion in the target and epoch of the coordinates.