JWST Operations Pipeline Build 11.1 Release Notes

A list of all relevant changes to build 11.1 of the JWST Operations Pipeline is provided in this article. 

On this page

Build 11.1 was released as a build candidate in September 2024, and installed as an operations build in December 2024. The information provided here focuses mainly on updates to the JWST Science Calibration Pipeline and associated reference files, but includes information relevant to changes in other subsystems that might affect what users see in their data.

Added January 2025: Due to breaking changes introduced in the gwcs dependent package in late December 2024, users wishing to install build 11.1 should also downgrade their gwcs version using this command: pip install gwcs==0.21.0

Major updates

Words in bold are GUI menus/
panels or data software packages; 
bold italics are buttons in GUI
tools or package parameters.

Major pipeline updates in build 11.1

  • Added a new clean_flicker_noise step to the calwebb_detector1 pipeline to correct for 1/f noise in data for all near-IR instrument modes. This is an optional step with a variety of different parameters to optimize performance for various science cases. For further details see 1/f Noise.
  • Fix issues with flux conservation for some spectroscopic modes (especially NIRSpec MOS and FS) when resampling to non-default scales
  • Updated NIRISS SOSS spectral extraction to improve performance by incorporating the previously standalone PASTASOSS package
  • Fixes for cosmic rays saturating in group-2 data for non-IRS2 readout modes
  • Fixed an issue with nod-flux oversubtraction for some NIRSpec slit observations with small nod distances
  • Further improvements to the runtime of the ramp fitting step for NIRISS data
  • Significant runtime improvements to multiple parts of the NIRSpec pipeline, impacting IFU, MOS, and BOTS data
  • Improvements to memory management throughout the pipeline, reducing memory pressure for some of the largest steps (e.g., mosaic resampling) and fixing the build 11.0 memory leak in the C-based ramp fitting step

Major reference file updates in build 11.1

Due to the timing of the switch to quarterly calibration file updates, there are few new calibration files associated with build 11.1 compared to build 11.0.  These changes include:

  • Updated NIRSpec MOS and fixed slit outlier detection parameter reference files to avoid clipping high-SNR sources due to the improved spectral resampling.

Known Issues in build 11.1

There were two previous issues with build 11.0; a memory leak in the C-based ramp fitting step and a bug that prevented the outlier detection step from running as a standalone step outside the pipeline. Both have been fixed with build 11.1.

New issues of note with build 11.1 include:

  • The ramp fitting step crashes if users enable multiprocessing for the C-based algorithm for non-NIRISS data. This will be fixed in build 11.2, but the workaround solution is to simply not use multiprocessing for the (default) C-based algorithm. The C-based ramp fitting is sufficiently fast that multiprocessing provides no significant runtime advantage for this step anyway.
  • Some NIR programs using non-IRS2 grouped data can experience a small number more breakthrough snowballs in build 11.1 than in build 11.0 due to interaction between the saturation-flagging code and the snowball-detection code.  A temporary workaround for any problematic exposures is to set saturation.use_readpatt = False.
  • Due to improved resampling fidelity, NIRSpec MOS and FS modes can, in some cases, flag bright pixels as outliers in the outlier detection step. This has been largely mitigated by parameter reference file update, but will be fixed by build 11.2 refactor of outlier detection step.
  • MIRI LRS TA confirmation image has no information in the LRS slit region in calibrated "cal.fits" files. Temporary workaround is to use "rate.fits" files for TA confirmation inspection.
  • Due to a bug in the moving target WCS code, spectroscopic moving targets with associated background observations cannot be processed through the calwebb_spec3 pipeline with build 11.1.  Existing archival products will continue to be based on build 11.0 pipeline reductions, while PIs of new programs upcoming in early 2025 will be contacted individually.  For offline reprocessing, users are recommended to use build 11.0 or candidate build 11.2 instead.
  • Some large imaging mosaics can appear blurry due to poor performance of the tweakreg step correcting the alignment of individual images.  A fix is included in build 11.2 and can be replicated in build 11.1 by rerunning calwebb_image3 with the tweakreg parameters of sharplo, sharphi, roundlo, roundhi set to values of 0.5, 2.0, 0.0, and 0.2 respectively.

Data affected by this build

Provided below is a list of products in MAST generated by the JWST Science Calibration Pipeline that may be most significantly affected by the updates in build 11.1. The changes listed in the release notes below may affect other modes as well, but many of these will be transparent to the MAST user.

Provided below is a list of science data products in MAST generated by the JWST Science Calibration Pipeline that may be most significantly affected by the updates in build 11.1. The changes listed in the release notes below may affect other modes as well, but many of these will be transparent to the MAST user.

The data and pipeline stages most affected include:

  • Stage 1 processing for NIRSpec data obtained without the IRS2 readout mode (due to improvements in flagging of saturating cosmic rays).
  • Stage 1 processing for MIRI data with extremely short ramps (n_groups < 5) due to bug fixes to the emicorr step.
  • Stage 1 offline reprocessing for all NIR instruments due to availability of new optional 1/f correction step (clean_flicker_noise).
  • Stage 2 processing for NIRSpec MOS and FS modes due to updates in the spectral resampling code.
  • Stage 3 processing for NIRISS SOSS, NIRSpec MOS, and NIRSpec FS data due to various updates in the spectral extraction.

Release notes

Build 11.1 of the JWST Operations Pipeline was released as a candidate build on September 20, 2024 and installed as an operations build (11.1) for the Barbara A. Mikulski Archive for Space Telescopes (MAST) on December 3, 2024. The build includes changes to the JWST Science Calibration Pipeline released in version 1.16.1 of the jwst calibration pipeline package via the PyPI repository. This release also includes changes to the JWST Science Data Processing subsystem (SDP) version JWSTDP-2024.3.1 which is the DMS sub-system that creates raw (uncal) products used as input to the calibration pipeline.

General changes

  • The pipeline now ensures that science pixels with either a NaN value or a DO_NOT_USE DQ flag have both; this may increase the number of NaN values seen during processing, but it should not affect the quality of the data.
  • Throughout the pipelines, deep copies of NIRSpec WCS objects have been replaced with shallow copies within most loops. This reduces runtime, most noticeably for NIRSpec IFU data.

Pipeline modules

  • Updated calwebb_detector1 to add the optional clean_flicker_noise step between jump and ramp_fit, which models and cleans transient flicker noise (e.g., 1/f noise) from group images in ramp data.
  • Updated calwebb_detector1 to remove unnecessary array copies which reduces memory usage throughout pipeline.
  • Updated calwebb_spec2 to run nsclean on NIRSpec imprint and background association members.
  • Updated calwebb_spec3 to not save the pixel_replace output by default.
  • Updated calwebb_image3 to use ModelLibrary instead of ModelContainer, and added an optional on_disk parameter to govern whether models in the library should be stored in memory or on disk. By default, models will be stored in memory.
  • Updated calwebb_coron3 to tighten tolerance of psf alignment.
  • Log version of jwst pipeline code at the end of processing.

Product associations

  • Restored slit name to level 3 product names for NIRSpec BOTS and background fixed slit targets, fixing a bug introduced in build 11.0.
  • Excluded nearby background candidates from NIRSpec fixed slit associations for S1600A1 with 5-point dithers, to reduce overlap between background nods and science exposure.
  • Added association rule for level 3 image mosaic candidates, allowing for combined processing of imaging exposures with different target IDs.

Datamodels

  • Added ModelLibrary class to allow passing on-disk models between steps in the image3 pipeline.

Step Changelist

Changes to specific steps in the pipelines are listed below. 

align_refs step

  • Compute alignment shifts from the first integration of the science exposure only.
  • Fixed a bug where the aligned PSF was being saved as a 4-D array with the first axis being identical; now it is saved as a 3-D array.

assign_wcs step

  • Revised functions to avoid copying the full NIRSpec WCS within a loop

badpix_selfcal step

  • Fixed a bug for MIRI MRS data in this optional step by subtracting the pedestal dark when constructing min array across selfcal exposures.

clean_flicker_noise step

  • Implemented this new optional step to clean transient flicker noise (e.g., 1/f noise) from group images in ramp data.

cube_build step

  • Allow the user to provide ra and dec shifts to apply for each file to fine tune the WCS.

emicorr step

  • Fixed a bug where step would return NaNs when it was unable to compute a correction.

extract_1d step

  • Updated NIRISS SOSS extraction to utilize pastasoss rotation solution.
  • Fixed an issue with extraction aperture positions for moving target data in MOS and FS modes.

master_background step

  • Either of background or bkg in slit name now defines the slit as a background slit, instead of bkg only.

mrs_imatch step

  • Added a deprecation warning and set the default to skip=True for the step.

nsclean step

  • Changed subarray mode from operating on the entire array at once to operating on sections of the array and smoothly combining these sections. Due to the computational costs of matrix operations, this is a large speedup that has little effect on the results.
  • Merged implementation with the new clean_flicker_noise step. This step can still be called from the calwebb_spec2 pipeline on NIRSpec rate data, but it is now deprecated.

outlier_detection step

  • Fixed failures due to a missing wcs.array_shape attribute when the outlier_detection step was run standalone using strun.
  • Fixed file names and output directory for intermediate resampled spectral images. Intermediate files now have suffix "outlier_s2d" and are saved to the output directory alongside final products.

ramp_fitting step

  • Moved the read noise variance recalculation for CHARGELOSS flagging to the C implementation, when the algorithm is OLS_C.
  • Fixed memory leak in C implementation.

resample step

  • Fixed a typo in load_custom_wcs from array_shape to pixel_shape and changed to use values in the top-level ASDF structure if the values in the WCS are None.

resample_spec step

  • Modified the output NIRSpec spectral WCS to sample the input data linearly in sky coordinates, rather than slit coordinates, in order to conserve spectral flux in default reductions.
  • Updated handling for the pixel_scale_ratio parameter to apply only to the spatial dimension, to match the sense of the parameter application to the documented intent, and to conserve spectral flux when applied.
  • Implemented handling for the pixel_scale parameter, which was previously ignored for spectral resampling.
  • Fixed a bug resulting in incorrect output slit coordinates for NIRSpec moving targets in the calwebb_spec3 pipeline.
  • Separate resample_spec step parameters from resample step parameters so that the spectral resampling step only exposes parameters that are appropriate for spectral data.

saturation step

  • Enable use of the readout pattern information to improve saturation flagging in grouped data.

tso_photometry step

  • Replaced photutils.aperture do_photometry with photutils.ApertureStats to remove NaNs when performing TSO photometry for non SUB64 WLP8 pupil data. For SUB64 WLP8 changed summing data in the aperture to use np.nansum to ignore NaNs.

tweakreg step

  • Updated requirement for tweakwcs to version 0.8.8 which fixes a crash in the tweakreg step due to a "MalformedPolygonError" exception being raised by the spherical_geometry package for some data sets.

JWST Science Data Processing subsystem changes

  • New mt_v2 and mt_v3 columns were added to the MOVING_TARGET_POSITION extension to include position information for all, but particularly for SLIT and COMPOUND, aperture types.
  • New keyword, S_CLNFNS, added to indicate whether the clean_flicker_noise  calibration step was applied.
  • New keyword, S_BPXSLF, added to indicate whether the Bad Pixel Self calibration step was applied.
  • New keyword, R_PSTSOS, added to identify the reference file used by NIRISS SOSS for trace rotation during 1-D spectral extraction.



Notable updates


Originally published