JWST Operations Pipeline Build 11.0 Release Notes

A list of all relevant changes to build 11.0 of the JWST Operations Pipeline is provided in this article. 

On this page

Build 11.0 was released as a build candidate in June 2024, and installed as an operations build in August 2024. The information provided here focuses mainly on updates to the JWST Science Calibration Pipeline and associated reference files, but includes information relevant to changes in other subsystems that might affect what users see in their data.

Due to significant updates in the pipeline and corresponding reference files, all users processing MIRI data are recommended to update their pipeline installations to use build 11.0.


Major updates

Words in bold are GUI menus/
panels or data software packages; 
bold italics are buttons in GUI
tools or package parameters.

Major pipeline updates in build 11.0

  • A major runtime improvement has been made to the calwebb_detector1 pipeline by introducing a new C-based version of the ramp fitting algorithm that can improve the runtime of the ramp_fitting step by as much as a factor of 200 for long integrations (decreasing runtime in one case from over an hour to under 20 seconds). This C-based algorithm is now the default used by the pipeline, although the previous Python-based algorithm can still be used by specifying --steps.ramp_fitting.algorithm='OLS'.  Note, however, that further study in July/August 2024 has revealed a memory leak in the C-based routine that can cause problems for some users with either large observing programs or limited available RAM.  This memory leak will be fixed in the build 11.1 release; in the meantime affected users should continue to use the Python-based ramp fitting algorithm.

  • Multiple improvements have been made to error propagation throughout the pipeline; these benefit all operational modes, but most noticeably improve the uncertainty estimates for MIRI MRS observations. In addition to bug fixes, this latter now includes an approximate treatment of covariance when extracting spectra from the 3-D data cubes. Likewise, all MIRI observing modes now include the Poisson uncertainty in the subtracted dark current, which can be significant for longer observations.

  • Multiple improvements have been made to the flagging of bad pixels and outliers. Most noticeably, this includes a bug fix for NIRSpec IRS2 grouped data in which extremely bright cosmic rays in the second group were not being flagged and rejected. A similar issue still remains for grouped data obtained using non-IRS2 readout modes that will be addressed in build 11.1. Additional changes include a bug fix for flagging outliers in extremely long ramps (more than 256 groups), improved detection of asymmetrical snowballs in near-IR data, and the use of a time-rolling median for detection of outliers in time series observations. The (optional) pixel replacement step to fill in missing pixel values has also been moved from the calwebb_spec2 to the calwebb_spec3 pipeline to run after the outlier detection step.

  • A new convention has been adopted for padding empty areas within mosaiced data products that are outside the instrument field of view.  Previously, areas of mosaiced images and/or data cubes outside the instrument footprint were assigned zero values- these are now assigned NaN (not-a-number) values instead to better communicate that there is no information in these pixels. Note that this may cause problems for some custom photometric extraction routines that did not use the data quality array to identify pixels that should be excluded from analysis.

  • Fixed a bug in the imaging outlier detection step that could sometimes result in large segments of the field of view being flagged as outliers, particularly for MIRI F2550W imaging.

  • Improved the accuracy of the SIP polynomial representation of the world coordinate solutions in imaging observations.

  • Runtime improvements to the MIRI-only emicorr step in calwebb_detector1.

  • Updated NIRSpec MOS source ID assignments.

Major reference file updates in build 11.0

  • Three new MIRI parameter reference files have been delivered that are incompatible with previous pipeline versions. These include (1) an updated dark_current step parameter reference file enabling inclusion of the Poisson error on the dark current in the pipeline uncertainty estimates, (2) LRS calwebb_spec3 parameter reference files enabling use of the now-moved pixel_replace step in calwebb_spec3, (3) MRS extract_1d parameter reference files enabling use of the new code that accounts for spatial covariance in IFU data cubes when extracting one-dimensional spectra.  

    Since these reference files are incompatible with prior pipeline versions all MIRI users are recommended to update to using build 11.0.

  • Updated MIRI MRS photometric calibration reference files. This includes both an update to the time-dependent count rate loss model based on monitoring observations taken in the first half of 2024 (jwst_1238.pmap) in addition to an updated approach to photometric calibration (jwst_1259.pmap). The calibration vector derived from the primary calibrator (O9V star 10 Lac) is now scaled to the average normalization of 10 different O, A, and G type standard stars. The primary effect of this change is to make all MRS observations brighter by about 2% at short wavelengths to 5% at long wavelengths.

  • Updated the NIRSpec IFU "cubepar" reference file to adjust the minimum default wavelength used for cube building to trim a large wavelength range with no spectral coverage in the G140M/F070LP and G140H/F070LP gratings (jwst_1242.pmap).

  • Updated the NIRSpec IFU "sflat" reference files to remove large "bullseye" artifacts in medium resolution grating flat fields, in addition to trimming low-throughput noisy pixels on the edge of IFU slices in all gratings/filters (jwst_1243.pmap, jwst_1253.pmap).

  • Updated NIRISS WFSS F115W and F150W "specwcs" reference files, improving spectral trace characterization (jwst_1244.pmap, jwst_1262.pmap).

  • Updated MIRI dark current reference files to correct several artifacts present in the previous versions, resulting in improvements in the faint source regime (jwst_1260.pmap).  Note that the refpix (reference pixel correction) step should be skipped when using these new reference files, as per a corresponding update to the calwebb_detector1 parameter reference file.

  • Updated MIRI LRS wavelength calibration, aperture correction, and photometric calibration reference files (jwst_1261.pmap). The slitless wavelength calibration adjustment is largest at short wavelengths, with a change of about 180 nm at 5 µm.

  • Updated bad pixel masks for all NIRCam detectors (jwst_1263.pmap). This update both revises the algorithms to identify bad pixels and accounts for evolution in the bad pixel population since the creation of the previous masks.


Known Issues in build 11.0

Issues that affect the JWST data products available from MAST that were not resolved with this build are compiled in Known Issues with JWST Data - High-Level Summary. When there is a workaround for an issue, the relevant information is provided there.

Two new issues of note with build 11.0 include:

  • A memory leak in the new C-based implementation of the ramp_fitting step. Users who encounter issues should revert to using the original Python implementation; a fix will be provided in build 11.1.
  • A bug in the outlier_detection step that prevents it from running as a standalone step outside the calwebb_image3 or calwebb_spec3 pipelines. A fix will be provided in build 11.1.


Data affected by this build

Provided below is a list of products in MAST generated by the JWST Science Calibration Pipeline that may be most significantly affected by the updates in build 11.0. The changes listed in the release notes below may affect other modes as well, but many of these will be transparent to the MAST user.

The data and pipeline stage affected for data in MAST are:

  • Stage 1 processing for all data, particularly MIRI (due to updated dark current reference files) and NIRCam (due to updated bad pixel masks).
  • Stage 2 processing for all data, particularly MIRI, NIRSpec IFU, NIRSpec MOS, and NIRISS WFSS.
  • Stage 3 processing for all data, due to changes in the no-value padding around stage 3 data products from zero to NaN.

Release notes

Build 11.0 of the JWST Operations Pipeline was released as a candidate build on July 8, 2024 and installed as an operations build (11.0.1) for the Barbara A. Mikulski Archive for Space Telescopes (MAST) on August 27 2024. The build includes changes to the JWST Science Calibration Pipeline released in version 1.15.1 of the jwst calibration pipeline package via the PyPI repository.

Changes to specific steps in the pipeline are listed below, along with descriptions of more general changes made to certain packages and related subsystems. This release also includes changes to the JWST Science Data Processing subsystem (SDP) version JWSTDP-2024.1.1 which is the DMS sub-system that creates raw (uncal) products used as input to the calibration pipeline.

ami_analyze step

  • The step can now be run on "_cal" files.

assign_wcs step

  • The routines that load slit and source data for NIRSpec MOS data have been updated to better handle background and virtual slits, assigning appropriate metadata for use downstream during master_background creation and background subtraction.
  • The default SIP approximation has been parametrized to higher order to increase accuracy.
  • Introduced an intermediate alpha-beta slit reference frame in the WCS for MIRI LRS data, between pixel coordinates and the v2/v3 frame.
  • Handle fixed slit sources defined in an MSA metadata file for combined NIRSpec MOS and fixed slit observations. 

badpix_selfcal step

  • Added a new optional step to the calwebb_spec3 pipeline to self-calibrate bad pixels in IFU data.

combine_1d step

  • Corrected a mistake in the computation of errors for combined 1-D spectra.

emicorr step

  • Introduced a new parameter, use_n_cycles, to select the number of cycles used to calculate the phase of the error; the default value of 3 will improve runtime for most processing.

exp_to_source step

  • Modified slit sorting to use source_name as the key, rather than source_id, in order to support changes in source_id handling for NIRSpec MOS exposures that contain background and virtual slits.
  • Update the top-level model exposure type from the slit exposure type, to support processing for combined NIRSpec MOS and fixed slit observations.

extract_1d step

  • Fixed a crash encountered when multiple background or source regions are specified, and the lower and upper limits for one are outside the valid area for some data range.
  • Corrected the output slit name for non-primary slit extraction in the calwebb_spec3 pipeline, for NIRSpec fixed slit mode.
  • Added parameter ifu_covar_scale to help correct for IFU cube covariance.
  • Added propagation of uncertainty when backgrounds are subtracted from source spectra during extraction.
  • Fixed a bug when processing point sources in non-primary slits, when multiple slits are populated with point sources.
  • Added functionality to the phase-based aperture correction object to allow reuse of object across multiple integrations, to improve runtime especially for NIRSpec BOTS data.

extract_2d step

  • Added handling for NIRCam grism time-series pointing offsets.

flat_field step

  • Updated code to ensure pixels with one of SCI=ERR=NaN and DQ is DO_NOT_USE have the complementary values, to ensure NaNs are not used downstream.

master_background step

  • Disabled support for master background correction for NIRSpec MOS slits when called as part of calwebb_spec3/master_background - master background correction for MOS should be performed via master_background_mos, called in calwebb_spec2.
  • Use zero values for master background outside the background wavelength range instead of NaN, to avoid NaN values for entire sets of science data when backgrounds are missing. 

master_background_mos step

  • Updated a check for MOS background slits to use new naming convention: slit.source_name now contains the string "BKG" instead of "background".

nsclean step

  • Improved run time of NSClean.fit() by using a vector rather than a large, sparse matrix to perform linear algebra with a diagonal weight matrix.
  • Added a check for combined NIRSpec MOS and fixed slit products: fixed slit quadrant not automatically masked if fixed slits are defined in MOS product.

outlier_detection step

  • Parameter weight_type now passed to all resampling function calls, such that default weighting can be overridden by the step parameter.
  • Intermediate output files now properly cleaned up after processing if not specified to be saved; association id added to intermediate file names.
  • Removed setting of self.skip=True when the step is skipped due to inappropriate results, such that the step will continue to execute when called again while processing a list of sources.
  • Added option to use a rolling median to detect outliers in TSO data, with user-defined rolling window width via the rolling_window_width parameter.
  • Fixed a bug that led small total flux offsets between input and blotted images if the nominal and actual wcs-computed pixel areas were different.
  • Deprecate nlow and nhigh parameters, which are no longer used.

pathloss step

  • Updated calculations for NIRSpec fixed slit mode to use the appropriate wavelengths for point and uniform sources if the wavecorr wavelength zero-point corrections for point sources have been applied.

photom step

  • Ensure that NaNs in MRS photom files are not replaced with ones for consistency with other modes.

pixel_replace step

  • Changed location in calwebb_spec2 and added to calwebb_spec3 pipeline: in both, it is now executed immediately before resample_spec/cube_build.
  • Added estimated errors and variances for replaced pixels, following the interpolation scheme used for the data.

ramp_fit step

  • Changed data type for several variables to use uint16 rather than uint8, in order to avoid potential overflow problems.
  • Added a C implementation of the existing algorithm written in Python; the default algorithm is now the C implementation, labeled "OLS_C", while the Python version can be selected as "OLS". 

refpix step

  • Use double for fft filter coefficients to improve compatibility across system architectures.

resample step

  • Update variance handling to propagate resampled variance components with weights that match the science weight_type.
  • Change default value of fillval parameter from INDEF to NaN.
  • Removed the use of the "drizpars" reference file.

resample_spec step

  • Populate the wavelength array in resampled Slit and MultiSlit models.
  • Fixed a bug resulting in large WCS errors in the resampled WCS when the slit was closely aligned with the RA axis.
  • Fixed a bug resulting in incorrect output spectral WCS for NIRSpec data when the first input data set is filled with zero or NaN.
  • Change default value of fillval parameter from INDEF to NaN.
  • Removed the use of the "drizpars" reference file.

residual_fringe step

  • Use the DQ plane to exclude pixels marked as DO_NOT_USE in correction.

srctype step

  • Reset source_xpos and source_ypos values to zero for extended sources in NIRSpec FS data to enable assigning those attributes in extract_2d.

saturation step

  • Add a check for saturation bias in group 2 for IRS2 mode nframes > 1.

tweakreg step

  • Output source catalog file now respects output_dir parameter.
  • Changed default settings for abs_separation parameter for the tweakreg step to have a value compatible with the abs_tolerance parameter.
  • Change code default to use IRAF StarFinder instead of DAO StarFinder.
  • Add new step parameters to control SIP approximation in the output FITS WCS, matching the default values used in the assign_wcs step.
  • Added a check for (abs_)separation and (abs_)tolerance parameters, that separation > sqrt(2) * tolerance: when this condition is not satisfied, source confusion is possible during catalog matching. Now, a error message will be printed to the log and step will be skipped.
  • Refactor step to improve performance; improve error handling in absolute alignment.

wavecorr step

  • Changed the NIRSpec wavelength correction algorithm to include it in slit WCS models and resampling. Fixed the sign of the wavelength corrections.
  • Added a check for fixed slits that already have source position information, assigned via a MSA metafile, for combined NIRSpec MOS and fixed slit processing. Point source position is calculated from dither offsets only for standard fixed slit processing.
  • Assign slit source_xpos and source_ypos attributes in extract_2d instead of in wavecorr for NIRSpec FS data.

wfss_contam step

  • Fixed flux scaling issue in model contamination image by adding background subtraction and re-scaling fluxes to respect wavelength oversampling.

Pipeline modules

  • Fixed a bug in the calwebb_spec2 and calwebb_image2 pipelines that was causing them to not respect the output_file parameter.
  • Removed the unused scale_detection argument from the calwebb_tso3 pipeline.
  • Updated the calwebb_spec3 pipeline handling of NIRSpec MOS inputs to comply with the new scheme for slit naming - source ("s"), background ("b"), and virtual ("v") - and the construction of output file names for each type.
  • Added new optional step badpix_selfcal tocalwebb_spec2 pipeline.
  • Added calwebb_spec2 pipeline handling for combined NIRSpec MOS and fixed slit observations. Steps that require different reference files for MOS and FS are run twice, first for all MOS slits, then for all FS slits. Final output products (cal, s2d, x1d) contain the combined products.
  • Added a hook to skip photom step when the extract_1d step was skipped for NIRISS SOSS data.
  • Added hook to the calwebb_tso3 pipeline to skip all subsequent steps if the extract_1d step is skipped.

Product associations

  • Ensure NRS IFU exposures don't make acalwebb_spec2 association for grating/filter combinations where the nrs2 detector isn't illuminated. Add similar optical path constraints for NIRSpec TSO associations.
  • Match NIRSpec imprint observations to science exposures on mosaic tile location and dither pointing, MOSTILNO and DITHPTIN. Default values for these non-header keywords have been added to the "asn_make_pool" script.
  • Updated Level3 rules for new handling of NIRSpec MOS source_id formatting when constructing output file names.
  • Associations for WFSS pure parallel observations are now created.
  • Exclude NIRISS SOSS data taken with uncalibrated filter F277W from calwebb_spec2 and calwebb_tso3 associations.

General changes

  • The pipeline now requires numpy<2.0, until testing and development resolves any existing issues. 
  • Removed deprecated stdatamodels model types "DrizProductModel", "MIRIRampModel", and "MultiProductModel".

JWST Science Data Processing subsystem changes

  • NIRSpec Micro-Shutter Array (MSA) FITS files now have a new fixed_slit column in the SHUTTER_INFO table. This contains the name of the fixed slit, if one is being used in multi-object spectroscopy (MOS) mode, and otherwise contains the value “NONE”.



Notable updates


Originally published