JWST Operations Pipeline Build 9.0 Release Notes

Release notes and known issues in patch build 9.0 of the JWST Operations Pipeline, released on November 17, 2022, are provided in this article.

On this page

This article provides a list of all relevant changes to the operations pipeline that was released with patch build 9.0. This build focused mainly on updates to the JWST Science Calibration Pipeline, but includes information relevant to changes in other subsystems that might affect what users see in their data.


Highlights of this build

  • Words in bold are GUI menus/
    panels or data software packages; 
    bold italics are buttons in GUI
    tools or package parameters.

    Updated the stage 2 background subtraction step to properly handle coronagraphic observations that contain background exposures
  • Implemented an initial version of a snowball/shower flagging scheme, to remove affected pixels
  • Fixed a bug in the handling of moving target IFU observations that contain background exposures
  • Updated the tweakreg step to allow user-supplied source catalogs for each input image
  • Updated calwebb_spec2 to include the new MIRI MRS residual fringe correction step


Known issues

The following issues were not resolved in the jwst calibration pipeline package (version 1.8.2) that was released with build 9.0:

  • Updated in-flight reference data for the NIRCam WFSS mode have not yet been delivered to CRDS, so the pipeline is still using pre-flight calibrations. This usually leads to incorrect computations of spectral trace locations and wavelength assignments during stage 2 calibration, rendering the archive products unsuitable for analysis.
  • The computation of the WCS frame for resampled and combined NIRSpec MOS and fixed slit data can sometimes be in error, leading to problems in late stage processing.

Some step and mode-specific issues will be fixed in build 9.1:

  • NIRSpec IFU cubes often contain sampling artifacts. This problem has recently been resolved and will be included in build 9.1.
  • NIRISS SOSS exposures can occasionally fail in the extract_1d step. Algorithm updates are in progress that will resolve these issues and may be included in build 9.1.
  • The computation of bounding boxes for WFSS sources can sometimes be incorrect and lead to crashes in the extract_2d step. These have been fixed for build 9.1.

Data affected by this build

Provided below is a list of products in MAST generated by the JWST Science Calibration Pipeline that may be most significantly affected by the updates in build 9.0. The changes listed in the release notes may affect other modes as well, but many of these will be transparent to the MAST user.

The data and pipeline stage affected for data in MAST are:

  • MIRI MRS - stage 2
  • Coronagraphic level 2 associations with background exposures - stage 2
  • IFU moving targets with backgound observations - stage 2

Release notes

See also: GitHub jwst project release notes 1.8.2

Build 9.0 of the JWST Operations Pipeline was released on November 17, 2022 to the Barbara A. Mikulski Archive for Space Telescopes (MAST). This build includes changes to the JWST Science Calibration Pipeline released in version 1.8.2 of the jwst calibration pipeline package via the PyPI repository.

The following fixes or changes were included in build 9.0. Changes to specific steps in the pipeline are listed below, along with descriptions to more general changes made to specific packages and related subsystems. This release also includes changes to the JWST Science Data Processing subsystem (SDP) version 2022.4.0 and 2022.4.1, which creates raw (uncal) products.

align_refs step

  • Fixed the median image replacement routine to also replace pixels set to "NaN", in addition to pixels flagged as bad or "DO_NOT_USE" in the DQ array, with suitable interpolate values from surrounding good pixels. This prevents the fast Fourier transforms from returning all "NaNs", which in turn led to have the resulting aligned images completely filled with "NaNs".

background step

  • Updated the stage 2 background subtraction step to accept "rateints" (3-D data) files as background exposures and use them to subtract an average background from the science exposure. This allows coronagraphic observations that contain background exposures to process properly.

combine_1d step

  • Fixed a bug to properly exclude input spectra that have only one wavelength bin. This issue was generating an error.

cube_build step

  • When using the option coord_system=IFUALIGN for MIRI MRS cubes, the slice boundaries now align exactly with the cube axes, this will have more general utility to correct for artifacts, which tend to be on slice-by-slice basis.

dark_current step

  • Fixed a bug in the computation of the total number of frames when the science exposure use on-board frame averaging and/or group gaps. The calculation changed from 


    sci_total_frames = sci_ngroups * (sci_nframes + sci_groupgap)

    to

    sci_total_frames = sci_ngroups * sci_nframes + (sci_ngroups - 1) * sci_groupgap

jump step

  • Implemented a first version of snowball/shower flagging, so that pixels affected by those events are excluded from subsequent slope calculations. Note that this option is turned off by default for automatic processing in operations, but can be activated by users for off-line processing.

master_background step

  • Fixed a bug so that IFU observations of moving targets that also have observations of a background target are loaded and handled correctly so that the updated WCS information pertinent to the moving target gets applied correctly when constructing IFU cubes.

resample step

  • Fixed a bug in the construction of the context (CON) array for resampled images so that it properly accommodates combined images containing more than 32 inputs.
  • Fixed a bug that was causing a misalignment, in the world coordinate system (WCS) values of the "i2d" output, of a single resampled image part of a mosaic with that of the combined mosaic image produced in stage 3.
  • Fixed the calculation of the "pixel_scale_ratio" when the pixel_scale parameter value is supplied by the user. Also this is used to update the photometry  PIXAR_SR and PIXAR_A2 keyword values to reflect the change in pixel scale.

resample_spec step

  • Updated the computation of target RA and Dec for slit spectra, in order to better handle input spectral images that contain negative traces resulting from nodded background subtraction.

tweakreg step

  • Relaxed the FITS WCS Simple Imaging Polynomial (SIP) fitting parameters to make the code more robust (and hence reduce failure cases).
  • Updated the step to allow users to specify source catalogs to be used for each input image, rather than having the step do source detection itself.

Pipeline modules

  • The calwebb_spec2 (Spec2Pipeline) module has been updated to include the new MIRI MRS residual fringe correction step. Note that the step is skipped by default in automatic processing in operations, but can be activated by users for off-line processing.

Product associations

  • The "sequence number" field in association file names (i.e. the "MMMMM" in "jwPPPPP-TNNNN_YYYYMMDDtHHMMSS_ATYPE_MMMMM_asn.json") has been expanded from 3 to 5 digits, in order to accommodate observations containing large numbers of exposures.
  • Added a warning when users include a directory path in association member names contained in ASN files, because the use of directory paths is not officially supported.

General changes

  • Added support in the datamodels schemas for the S_RESFRI and R_FRIFRQ keywords tracking, respectively, whether the new MIRI MRS residual fringe correction step (see the Pipeline modules section above) was applied and the name of the "fringe" reference file contains pixel-by-pixel fringing correction values.

JWST Science Data Processing subsystem changes

The JSDP subsystem processes data from the telescope and creates raw (uncal) products. The following changes have been included in this build:

  • The HGA_MOVE, HGA_STRT, and HGA_END keywords that document moves of the JWST high gain antennae should now have proper values when appearing in science data FITS files. They were previously fixed in guider data FITS files, and now the science FITS files should have those same fixes in place.  If there was no antennae movement, the HGA_STRT and HGA_END keywords are left out of the header. HGA_MOVE should always be present indicating "True" if movement occurred during the dataset, and "False" if no movement was seen.

  • The NIRSpec fixed slit dither-related FITS keyword SUBPXPTS value (number of points in a subpixel dither pattern) should no longer come out with an incorrect value = 0. The values in the right column below should now properly be based on the subpixel dither pattern value shown in the left column below.

    APT sub-pixel pattern

    SUBPXPTS


    NONE

    		1

    SPATIAL

    		2
    SPECTRA3
    BOTH

    4



Notable updates


Originally published