JWST Operations Pipeline Build 10.2 Release Notes

A list of all relevant changes to build 10.2 of the JWST Operations Pipeline is provided in this article. 

On this page

Build 10.2 was released as a build candidate in March 2024, and installed as an operations build in June 2024.  The information provided here focuses mainly on updates to the JWST Science Calibration Pipeline and associated reference files, but includes information relevant to changes in other subsystems that might affect what users see in their data. 

NOTE: A recent (June 2024) update to the Numpy library is incompatible with the calibration pipeline software released for build 10.2 (jwst tag 1.14.0). Users should either install patch release jwst tag 1.14.1 instead (which pins numpy to a compatible version), or manually adjust the numpy version in use in their build 10.2 installation (pip install "numpy<2.0").

Major updates

Words in bold are GUI menus/
panels or data software packages; 
bold italics are buttons in GUI
tools or package parameters.

Major pipeline updates in build 10.2

  • Major upgrades have been made to the stage 3 pipeline for NIRISS aperture masking inteferometry (AMI) mode data. These upgrades include updates to the algorithm used to extract interferometric observables, improved bad pixel corrections, and the use of the "OIFITS" standard for storing interferometry observables in the output data products. This results in new suffixes for the AMI output products, including "ami-oi.fits" and "aminorm-oi.fits".  Previous file suffixes using the old format will be deprecated in the near future. For further details, see the NIRISS AMI pipeline notes.
  • The jump step in calwebb_detector1 has had several upgrades and bug fixes related to the detection and flagging of extended events (i.e., Shower and Snowball Artifacts). These changes significantly reduce the impact of snowball artifacts on near-IR data in particular.
  • Several enhancements have been made to the reference pixel correction step (refpix) to yield better results for NIRSpec exposures using the IRS2 readout pattern.
  • The tweakreg step in calwebb_image3 now has an option for using the "IRAFStarFinder" or photutils "SourceFinder" methods when performing source detection, in addition to the default "DAOStarFinder".  Preliminary testing indicates that the IRAFStarFinder results in improved performance given the structure of the JWST PSF. While build 10.2 retains the DAOStarFinder default, future builds may change this default to IRAFStarFinder.
  • The photom step has been upgraded to properly take the dispersion per pixel into account when flux calibrating NIRCam WFSS and TSGRISM data.
  • The extract_1d step has been upgraded to store the x/y extraction limits that were used for extracting slit-like spectra in new keywords that are added to the "x1d" product header.
  • World coordinate information for pure parallel visits has been fixed using new preprocessing that now populates key pointing-related FITS header keywords with values taken from the prime visits.
  • Added a new keyword TMEASURE to the primary FITS header for all observations; this keyword gives the measurement time most relevant to typical computations of the effective signal-to-noise ratio.

Major reference file updates in build 10.2

  • Updates to multiple NIRSpec MOS reference files, including flat field and photometric calibration, spectral extraction, and pathloss correction. (jwst_1211.pmapjwst_1217.pmap)
  • Updated NIRSpec dark superbias, and bad pixel mask reference files (jwst_1224.pmap - jwst_1226.pmap and jwst_1233.pmap - jwst_1234.pmap)
  • Updated NIRSpec MOS failed shutter reference files (jwst_1227.pmap and jwst_1236.pmap)
  • Delivery of a new NIRISS non-redundant pupil mask reference file (jwst_1222.pmap) and updated throughput reference files (jwst_1223.pmap) for AMI modes
  • Updated NIRISS WFSS reference files improving the trace positions for extraction of spectra from the detector (jwst_1228.pmap)
  • Updated NIRISS tweakreg parameter reference enabling default use of the IRAFStarFinder algorithm for NIRISS imaging data (jwst_1229.pmap)
  • Updated NIRCam WFSS and TSGRISM photometric calibration reference files to work with the updated photometric calibration step (jwst_1232.pmap). Note that these new reference files must be used together with the new build 10.2 code in order to obtain an accurate flux calibration.

Known issues

Issues that affect the JWST data products available from MAST that were not resolved with this build are compiled in Known Issues with JWST Data - High-Level Summary. When there is a workaround for an issue, the relevant information is provided there.

Data affected by this build

Provided below is a list of products in MAST generated by the JWST Science Calibration Pipeline that may be most significantly affected by the updates in build 10.2. The changes listed in the release notes below may affect other modes as well, but many of these will be transparent to the MAST user.

The data and pipeline stage affected for data in MAST are:

  • Stage 1 processing for essentially all data, if users are interested in taking advantage of the new and enhanced options for flagging "snowballs" and "showers" in the jump step of calwebb_detector1.  
  • Stage 1 processing of NIRSpec exposures using the IRS2 readout pattern, due to the enhancements in the refpix step.
  • Stage 2 processing of NIRCam WFSS and TSGRISM observations, due to the update in flux calibration for those modes.
  • Stage 2 and 3 processing for all NIRISS AMI observations.

Release notes

See also: GitHub jwst project release notes 1.14.0

Build 10.2 of the JWST Operations Pipeline was released as a candidate build on March 29, 2024 and installed as an operations build (10.2.1) for the Barbara A. Mikulski Archive for Space Telescopes (MAST) on June 12, 2024. The build includes changes to the JWST Science Calibration Pipeline released in version 1.14.0 of the jwst calibration pipeline package via the PyPI repository.

Changes to specific steps in the pipeline are listed below, along with descriptions of more general changes made to certain packages and related subsystems. This release also includes changes to the JWST Science Data Processing subsystem (SDP) version JWSTDP-2024.1.1 which is the DMS sub-system that creates raw (uncal) products used as input to the calibration pipeline.

 background step

  • The background  step in the stage 2 pipelines has been updated to handle background exposures that have different values of NINTS (number of integrations). Each background exposure is first averaged over integrations to form a single 2-D background image. Then, the multiple background images are average together to form the background that will be subtracted from the science exposure.

charge_migration step

  • The charge_migration step has been updated to flag all subsequent groups beyond the one that crosses the charge loss signal threshold.

cube_build step

  • The cube_build step has been updated to issue a warning message when no valid data are found on the detector.

dark_current step

  • The dark_current step has been updated to store the average_dark_current value in the data model, for later use in the ramp_fitting step. The average_dark_current value can be supplied either via a new step parameter of the same name or read from a DARK reference file. The average dark current will in future be taken into account when estimating Poisson noise in the ramp_fitting step.

emicorr step

  • Updated to skip cleanly when no reference file is found.
  • Added the parameter onthefly_corr_freq , which allows a user to specify a list of frequencies to be corrected and overrides any input from a reference file.
  • Fixed a bug in the retrieval of correction data for exposures with SUBARRAY = 'FULL'.

exp_to_source step

  • Fixed a bug in the handling of MultiSlit data, so that the proper BUNIT keyword values and WCS object information get propagated to the reorganized output models.

extract_1d step

  • Fixed a bug in the calling sequence for the optional MIRI MRS 1-D residual fringe correction that could cause defringing to fail in some cases.
  • Added a hook to skip the step (with a warning) for NIRISS SOSS exposures taken in full frame mode, because the "ATOCA" extraction algorithm is not capable of handling full frame images.
  • Fixed a bug in the NIRISS SOSS "ATOCA" algorithm, which caused failures when nearly all pixels in the 2nd order spectrum are flagged as bad.
  • Fixed the automatic aperture centering method for IFU cubes to only use wavelengths < 26 µm, due to extremely low throughput at the long wavelength end of the MIRI MRS band 4C.
  • Updated to save the extraction aperture limits used for slit-like data to the new keywords EXTRXSTR (extraction X start), EXTRXSTP (extraction X stop), EXTRYSTR (extraction Y start), and EXTRYSTP (extraction Y stop).

extract_2d step

  • Fixed a bug that had caused the step to crash when a user provided an integer value for the slit_name parameter.

jump step

  • Added the capability to flag pixels in the next integration in the saturated cores of "snowballs", including new parameters to control this behavior.
  • Added the new parameter min_diffs_single_pass to control whether iterative detection of jumps within a single pixel should be performed. Above this value, no iterative detection is performed, thus improving performance for integrations with a large number of groups.
  • Updated the flagging of "showers" to mask reference pixels, require a minimum number of groups to trigger the detection, and use all integrations to determine the median value.
  • Enabled the use of multiple integrations to find outliers.
  • Switched the multiprocessing method to use "fork_server" .
  • Fixed a bug in the computation of the number of rows per slice for multiprocessing, which was causing different results when running the step with or without multiprocessing.

outlier_detection step

  • Removed the grow parameter from the step definition, because it's no longer used within the algorithms.
  • Removed all references to and use of the "tophat" kernel in calls to resampling from within the outlier_detection step.
  • Fixed a bug in the removal of intermediate files when the step completes, which was causing original input files to accidentally get deleted.

pathloss step

  • Updated to add the "DO_NOT_USE" DQ flag to all pixels having a value of NaN.

photom step

  • Updated to include spectral dispersion when applying NIRCam WFSS and TSGRISM flux calibration, which is necessary because the dispersion varies with location in the field and wavelength.
  • Updated to retrieve the pixel area values PIXAR_A2 and PIXAR_SR from the AREA reference file, instead of the PHOTOM reference file.
  • Fixed a bug with the propagation of BUNIT values from step input to output files when processing "MultiSlit" data.

pixel_replace step

  • Fixed a bug that caused array size mismatches when the "mingrad" algorithm was applied to NIRSpec data.

ramp_fitting step

  • Initial updates to allow for future use of average_dark_current when calculating Poisson variance; note that more development is required beyond build 10.2.
  • Switched the multiprocessing method to use "fork_server" .
  • Fixed a bug that could yield wrong results for integrations with more than 256 groups.

refpix step

  • Updated the method used to identify and flag bad reference pixels when processing NIRSpec IRS2 exposures, so that it considers values from all groups in a given integration, and robustly replace identified bad pixels from their nearest neighbors.
  • Added the new parameter preserve_irs2_refpix, which allows the interleaved reference pixels contained in NIRSpec IRS2 exposures to be retained in the output file, for calibration and diagnostic purposes.
  • Added the new parameter irs2_mean_subtraction, which allows for subtraction of mean reference pixel offsets per amplifier and detector column—as is done for non-IRS2 modes—before the special IRS2-based reference pixel correction is applied to an exposure.

resample step

  • Updated all instances in the step that make use of exposure time in weighting to switch from using the EFFEXPTM keyword to the new TMEASURE keyword when TMEASURE is available.
  • Removed the population of the TEXPTIME keyword in the combined output files, because it was redundant with the total (combined) exposure time already recorded in other keywords.
  • Updated to use the same value of iscale for resampling all science and variance arrays.
  • Removed all references to the "tophat" kernel for drizzling.

residual_fringe step

  • Fixed a bug in the step that was setting pixels with negative signal values to zero, instead of just ignoring them.

source_catalog step

  • Updated to suppress the warning messages from photutils.background.Background2D regarding NaNs in the input data.

tweakreg step

  • Updated to recompute the S_REGION keyword value after WCS corrections have been applied.
  • Added an option to use "IRAFStarFinder" or "segmentation.SourceFinder" instead of the default "DAOStarFinder" method, and exposed all relevant parameters for each method.
  • Fixed a bug that caused a failure instead of just a warning when no GAIA sources are found within the bounding box of an input image.

Pipeline modules

  • The calwebb_spec2 pipeline has been updated to include "NRS_BRIGHTOBJ" (NIRSpec BOTS) exposures in the list of modes to which the nsclean (1/f noise correction) step gets applied.
  • The calwebb_ami3 pipeline and the constituent steps have had major upgrades to the algorithms and output products:
    • The ami_average step has been removed from the stage 3 processing flow. The updated pipeline now contains only the ami_analyze  and ami_normalize  steps.
    • The ami_analyze  and ami_normalize  steps have been upgraded to create outputs that conform to the "OIFITS" convention.
      • Output product types now include "ami-oi" and "psf-ami-oi" from the ami_analyze step and "aminorm-oi" from the ami_normalize  step
    • A new "NRM" reference file is used to apply a bad pixel correction to the AMI images when calculating complex visibilities.
    • Additional optional input arguments are available for greater processing flexibility.
    • By default all stage 2 and 3 processing of AMI exposures now uses multi-integration products (i.e. "rateints" and "calints"), rather than exposure-averaged (i.e. "rate" and "cal") products.

Product associations

  • Updated the association generator rules for stage 2 ASN files in order to correctly handle NIRSpec MOS nodded observations that include subpixel dithers (at each nod location), so that only exposures from other nod positions are used as "background" members in the association.
  • Updated the association generator rules for stage 3 ASN files in order to exclude NIRISS SOSS exposures with NINTS=1. These are "dummy" exposures inserted into a SOSS observation that are not intended to be used for science.
  • Updated the stage 2 and 3 association generator rules for NIRISS AMI exposures, so that stage 2 ("image2") associations use "rateints" products as input, instead of "rate" files, and stage 3 ("ami3") associations use "calints" products as input, instead of "cal" files.

General changes

  • Documentation updates:
    • Added more detailed information for the scale and snr parameters in the outlier_detection step documents.
    • Corrected the names of the step parameters usigma and lsigma in the skymatch step documents.
    • Updated the description of outlier_detection for IFU data to explain the method more clearly.
    • Added information about the MIRI MRS 1-D residual fringe correction that is applied within the extract_1d step.
  • Datamodels updates:
    • Added IMAGER and ALL_MRS as allowed values for the MRSPRCHN (MIRI MRS primary channel) keyword, in order to support observations using the MIRI imager and MRS in parallel.
    • Removed the TEXPTIME keyword definition from all datamodels, because it's no longer used.
    • Added the NRMModel for the new NIRISS AMI "NRM" reference file.
    • Added the new EXTRXSTR, EXTRXSTP, EXTRYSTR, and EXTRYSTP keywords populated by the extract_1d step.
    • Added the definition of average_dark_current to the DARK reference file data models.

JWST Science Data Processing subsystem changes

The JSDP subsystem processes data from the telescope and creates raw (uncal) products. The following changes have been included in this build:

  • Added new TMEASURE keyword representing the effective integration time of each exposure.

  • Fixed header information related to attitude and world coordinates for pure parallel observations.

Notable updates
  • Updated to reflect jwst software patch 1.14.1 (fixing a numpy dependency issue)
Originally published