JWST Operations Pipeline Build 10.0 Release Notes

Release notes for build 10.0 and patch 10.0.1 of the JWST Operations Pipeline, both released on December 5, 2023, are covered in this article

On this page

This article provides a list of all relevant changes to the Operations Pipeline that was released with build 10.0 and patch 10.0.1. These focus mainly on updates to the JWST Science Calibration Pipeline, but includes information relevant to changes in other subsystems that might affect what users see in their data. 

Highlights of this build

Build 10.0 and patch 10.0.1

  • Words in bold are GUI menus/
    panels or data software packages; 
    bold italics are buttons in GUI
    tools or package parameters.

    The positioning of the 1-D extraction aperture in the extract_1d step, for NIRSpec MOS and fixed slit data, as well as the MIRI LRS fixed slit, has been updated to be more robust. For NIRSpec MOS and FS, the estimated location of the source in the slit, which is provided by the APT and MPT planning tools, is used to set the extraction location, rather than the (often unreliable or inaccurate) source RA/Dec values. Similarly for MIRI LRS FS, the known dither/nod offset applied to an exposure is used to set the extraction location, instead of using the source RA/Dec. This results in much better centering of the extraction aperture on the spectral trace and greatly improved 1-D spectral data.
  • The assignment of SOURCEID values for NIRSpec fixed slit products has been updated so that the data from the slit with the primary target always have SOURCEID = 1. The SOURCEID values for other slits use a 2-digit code, that gives the number of the slit containing the primary target and the number of the secondary slit from which the data were extracted.
  • The stage 3 association rules have been updated to combine data from the NIRSpec S200A1 and S200A2 slits for observations in which the primary target was observed in both slits.
  • The former undersampling_correction step has been renamed to charge_migration, and several enhancements have been added to the step.
  • The algorithms used to find and exclude intermittent bad pixels in the refpix step for NIRSpec IRS2 images have been made more robust, resulting in cleaner images.
  • A problem with duplicate stage 3 product names for NIRCam coronagraphy observations that obtain data using all short-wavelength channel detectors has been fixed.
  • Updated photometric calibration for NIRCam imaging (jwst_1126.pmap), MIRI MRS (jwst_1145.pmap), NIRCam coronagraphy (jwst_1146.pmap), NIRISS SOSS (jwst_1147.pmap), NIRISS imaging (jwst_1143.pmap)
  • Update NIRISS linearity and saturation flagging of bad pixels (jwst_1152.pmap)
  • New NIRSpec darks, biases, and bad pixel masks (jwst_1151.pmap)
  • Update NIRISS SOSS wavelength reference data to vacuum wavelengths (jwst_1144.pmap)

Known issues

Issues that affect the JWST data products available from the MAST Archive that were not resolved with this build are compiled in Known Issues with JWST Data - High-Level Summary. When there is a workaround for an issue, the relevant information is provided there.

Data affected by this build

Provided below is a list of products in MAST generated by the JWST Science Calibration Pipeline that may be most significantly affected by the updates in build 10.0.1. The changes listed in the release notes may affect other modes as well, but many of these will be transparent to the MAST user.

The data and pipeline stage affected for data in MAST are:

  • Stage 1 processing (and beyond) of NIRISS imaging observations
  • Stage 1 processing (and beyond) of any exposures that use frame-averaging (Nframes >1)
  • Stage 1 processing (and beyond) of NIRSpec exposures that use the IRS2 readout pattern
  • Stage 2 processing (and beyond) of NIRSpec MOS observations
  • Stage 2 processing (and beyond) of NIRSpec fixed slit observations
  • Stage 2 processing (and beyond) of MIRI LRS fixed slit observations
  • Stage 2 processing (and beyond) of MIRI MRS observations
  • Stage 3 processing of NIRCam coronagraphic observations that use SUBARRAY = FULL (data obtained from all detectors)

 Note that the snowball and shower correction in the stage 1 jump step is turned off, by default, in the operations pipeline, while further testing is completed. It is available for offline use, as desired.

Release notes

See also: GitHub jwst project release notes 1.12.0, GitHub jwst project release notes 1.12.1GitHub jwst project release notes 1.12.2, GitHub jwst project release notes 1.12.3, GitHub jwst project release notes 1.12.4 GitHub jwst project release notes 1.12.5

Build 10.0 and patch 10.0.1 of the JWST Operations Pipeline were released on December 5, 2023, to the Barbara A. Mikulski Archive for Space Telescopes (MAST). The build  and patch include changes to the JWST Science Calibration Pipeline released in versions 1.12.0 to 1.12.5 of the jwst calibration pipeline package via the PyPI repository.

The following fixes or changes were included mainly in build 10.0. The build 10.0.1 patch contained a single bug fix. Changes to specific steps in the pipeline are listed below, along with descriptions to more general changes made to specific packages and related subsystems. This release also includes changes to the JWST Science Data Processing subsystem (SDP) version JWSTDP-2023.3.1, which creates raw (uncal) products.

assign_wcs step

  • Increased the margin of extra pixels included in the cross-dispersion direction of NIRSpec MOS slitlets, in order to reduce edge effects during resampling of slitlet images. 
  • Applied a simple fix for all NIRSpec imaging products that do not have a defined WCS bounding box, which resulted in incorrect image scale values. In this case, the solution was to create a WCS bounding box that matches the shape of the data array and save it to the WCS object. 
  • Updated the assignment of the SOURCEID values for NIRSpec fixed slit exposures so that the slit containing the primary target always has SOURCEID = 1, rather than using a hardwired relationship between slit name and SOURCEID (which used S200A1=s00001, S200A2=s00002, S400A1=S00003, S1600A1=S00004, S200B1=S00005). In addition, the SOURCEID value for secondary slits now uses a 2-digit value, where the first (tens) digit is the slit number in which the primary target is contained, and the second (ones) digit indicates the secondary slit from which the data were obtained; for example, an exposure that uses S200A1 as primary slit has a SOURCEID of 12 for data in the S200A2 slit. With this approach the SOURCEID maps to the same sky region for point sources and overlapping sky regions for extended sources.
  • Updated the WCS creation for MIRI LRS fixed slit exposures to compute the sky position (RA/Dec) of the center of the slit in each exposure based on the known dither/nod position and store those sky coordinates in the dithered_ra and dithered_dec meta values. These are used downstream in the extract_1d step to position the extraction aperture.

background step

  • Updated the step to allow combined background images to be created from a mixture of full frame and subarray exposures. Note that this capability is intended for use in imaging modes only. While it's possible to apply it to non-WFSS spectroscopic modes, the results may not be suitable for spectra.

charge_migration/undersampling_correction step

  • The former undersampling_correction step has been renamed to charge_migration, which is applied to NIRISS images to help mitigate the effects of charge migration from pixels with high signal levels into its neighboring pixels. In addition to the name change, several other changes have also been incorporated, including:
    • the default signal level threshold for flagging pixels affected by charge migration has been changed from 30,000 DN to 25,000 DN
    • the data quality flag that is applied has had its name changed from "UNDERSAMP" to "CHARGELOSS"
    • pixels that have signal levels above the threshold also have their 4 immediate neighbors flagged with "CHARGELOSS"
    • the order of the stage 1 pipeline steps has been changed so that charge_migration is applied before the jump detection step, rather than after

cube_build step

  • Replaced the scale1 and scale2 step parameters with the single parameter scalexy. This allows only square spaxels in the IFU cube.
  • Corrected the slicer scale and orientation in the output WCS for IFU cubes that are built in the coordinate system of the NIRSpec local IFU slicer plane, i.e., cubes built in the coord_system = internal_cal coordinate frame. Cubes built in the default RA/Dec/lambda coordinate system are not affected.
  • Fixed a bug with memory allocation in the C extension modules when weighting = emsm. This bug resulted in a segmentation error for Mac computers using M1 and M2 chips.
  • Added the step parameters ra_center, dec_center, cube_pa, nspax_x, and nspax_y to allow users to have full control in specifying the desired tangent projection details for an IFU cube.
  • Modified the step infrastructure to keep data models that were opened by the step open until the step is complete, including the updating of meta (keyword) values in the models.

extract_1d step

  • Moved the MIRI MRS residual fringe correction so that it's applied later in the step flow, after normalization by the pixel area of each wavelength plane. Also apply the residual fringe correction to the 1-D extractions for the target that are in units of surface brightness, as well as the extractions for the background region.
  • Added the step parameter ifu_set_srctype that can be used to override the extraction method for the source type given by the SRCTYPE keyword. The SRCTYPE keyword remains unchanged. This only applies to MIRI MRS data.
  • Added the step parameter ifu_rscale, which allows users to scale the extraction radius used for IFU cubes between 0.5 and 3.0 times the FWHM. This only applies to MIRI MRS data.
  • Updated the positioning of the extraction aperture for NIRSpec MOS and fixed slit data to use the source_x and source_y metadata values, which give predicted source locations (within the slit), instead of relying on the source RA/Dec. For MOS data and given the excellent performance of MSATA, the source should be very close to that location in the vast majority of cases, and the metadata values ESTIMATED_SOURCE_IN_SHUTTER_X can be used.

  • Updated the positioning of the extraction aperture for MIRI LRS fixed slit stage 2 products (individual exposures) so that the aperture is centered on the known dither/nod location for each exposure, which is generally very accurate, rather than relying on the source RA/Dec. This addresses a problem found with the placement of the slit, which was error-prone due to its small size. This update does not affect the aperture placement for stage 3 (combined) products. 

 flat_field step

  • Updated to allow NIRSpec spectroscopic data to use the 1-D error vectors associated with the fast variation component and provided in the F-flat reference files. The error array in the reference file is interpolated onto the wavelengths for the observation and added in quadrature to any error from other components. Missing 1-D values expected to come from the S, D, and older F-flats are set to zero for now.
  • For NIRSpec modes, set all "DO_NOT_USE" pixels to "NaN" after applying the flat correction. This addresses issues with the flat fielding process, where some pixels may be left uncorrected because there is no applicable flat data. These are marked in the DQ plane with "NO_FLAT_FIELD" and "DO_NOT_USE" flags, so they are not used in further processing, but they remain in the image. This causes some confusion when looking at the intermediate products before resampling: the scale is so widely different for the corrected and uncorrected pixels that it can throw off automated scaling algorithms, making it look like there is no spectral trace in the image.  

jump step

  • Enabled the detection of "snowballs" that occur in frames that are within a frame-averaged group.

  • Added more allowable selections to the parameter maximum_cores that indicates the number of available cores that will be used for multiprocessing in this step; the default value is 1. Note that the clock time for the step is reduced almost linearly by the number of physical cores used on all machines.

master_background step

  • Updated to allow a user to save the 2-D expanded background image when the 1-D master background spectrum has been provided by the user and the parameter save_background=True.

outlier_detection step

  • Fixed a bug that saved to disk all intermediate files ("blot", "median", "outlier_i2d") at the end of processing, even when the save_intermediate_results parameter was not set.

pathloss step

  • Fixed an interpolation error for point source corrections. In this case the bilinear weights for the pixels to the top and right of the base pixel were swapped, so the pathloss correction retrieved from the reference file was off by up to a pixel.
  • Updated the MIRI LRS fixed slit correction to default to the value appropriate for the center of the slit when the computed target location is outside the slit.
  • Added the step parameter user_slit_loc to allow specifying the source location to be used.

photom step

  • Adapted the MRS time-dependent correction to work with MRS TSO mode data. This correction is applied by default.
  • The step was updated to issue a warning when the PIXAR_SR or PIXAR_A2 pixel area keywords are not found in the "PHOTOM" reference file, rather than proceeding silently without them.

pixel_replace step

  • Added the minimum gradient (algorithm = mingrad) method to be used with MIRI MRS images.
  • Corrected a bug in the spectral profile normalization computation in cases where several adjacent pixels in the core of the spectral trace (at the profile peak) are flagged as bad, so that the replacement values are now much more appropriate.

ramp_fit step

  • Added more allowable selections to the parameter maximum_cores that indicates the number of available cores that will be used for multiprocessing in this step; the default value is 1. Note that the reduction in the elapsed time for the step is proportional to the number of physical cores used on all machines.
  • Updated the variance computation for invalid integrations, as well as the median rate computation by excluding groups marked as "DO_NOT_USE".

refpix step

  • Updated the algorithms used for processing NIRSpec IRS2 readout exposures to be more robust at finding and excluding intermittently bad pixels.

resample/resample_spec step

  • The step now uses the same logic for computing the input range from the input image shape and the bounding box both for the SCI images as well as the ERR and VARIANCE_* images.
  • Updated so that the exposure time keywords XPOSURE, EFFEXPTM, DURATION, and TELAPSE in the output combined/resampled image represent the appropriate sum of the inputs, being careful to not overcount for outputs containing data from multiple detectors (e.g., NIRCam). Note that by design XPOSURE and EFFEXPTM both have the same value and convey the same information.

residual_fringe step

  • The step was updated to replace the use of the "astropy.modeling.Spline1D" function with "scipy.interpolate.BSpline". This change should be transparent to the results of the step.

source_catalog step

  • The step has been updated to issue a warning when the meta (keyword) value for the pixel area is not available in the input image, which is needed for converting image units to and from flux density and surface brightness.

straylight step

  • Modified the algorithm to ensure that the cross artifact correction does not get applied to the pedestal dark signal in MRS images, which could cause small artifacts at the 0.01 DN/s level that were noticeable in some data. Specifically, the pedestal is subtracted first, so that the cross artifact correction isn't applied to light coming from detector glow rather than through the MRS optics.

Pipeline modules

  • Fixed a bug in the calwebb_spec2 pipeline to ensure a "cal" product is created when the pixel_replace step is included in the processing.
  • Updated the calwebb_spec3 pipeline to create output product names for all NIRSpec fixed slit data based on both the SOURCEID and SLTNAME values for each set of slit data so that the product name properly reflects the slit from which the data were taken. Using the standard correlation between source_id and slits (S200A1=s00001, S200A2=s00002, S400A1=S00003, S1600A1=S00004, and S200B1=S00005) the new names for stage 3 products will be like "jw01234-o003_s00031_nirspec_f100lp-g140h-s200a1_x1d.fits", where "s00031" indicates that the target is in slit S400A1 and the product is for slit S200A1. The file names for all the other stage 3 products are the same, with just a different product type suffix (cal, crf, s2d). The stage 3 product for the slit with the target, in this case S400A1, the name will be "jw01234-o003_s00003_nirspec_f100lp-g140h-s200a1_x1d.fits". In cases where the same target is observed in both the S200A1 and S200A2 slits, the combined product will have a file name like "jw01234-o003_s00001_nirspec_f100lp-g140h-s200a1-s200a2_x1d.fits", where "s00001" indicates that the data are for the primary target and "s200a1-s200a2" indicates that the data are taken from both slits.

Product associations

  • Updated the association generator to prevent "group" candidate ID's from generating stage 3 TSO associations. This prevents mismatched exposures from being included together in stage 3 associations.
  • Updated the stage 3 product name construction for NIRCam coronagraphic observations that include data from all of the short wavelength channel detectors. Such observations get processed through both the calwebb_image3 and calwebb_coron3 pipelines, to produce coronagraphic products from the detector data that contain the occulted target, as well as regular imaging products for all the data. The names of the stage 3 products resulting from coronagraphic processing remain the same, while the combined image product resulting from calwebb_image3 has the additional suffix "-image3" in the file name, in order to distinguish it from the combined coronagraphic image.
  • Updated the stage 2 spectroscopic ASN rules to exclude any NIRSpec IFU exposures that use filter/grating combinations that are known to result in no data on the NRS2 detector, so as to avoid having the calwebb_spec2 pipeline abort during an attempted processing run.
  • Updated the stage 3 association rules for NIRSpec fixed slit observations that put the primary target in both the S200A1 and S200A2 slits, so that the exposures from both slits get included in the stage 3 association, resulting in combined products with complete wavelength coverage. The new naming syntax for the slit-specific output products is described above in the related update for the calwebb_spec3 pipeline module.
  • Updated the stage 3 product name template used in "spec3" association files for NIRSpec fixed slit observations, so that both the "source_id" and "slit_name" template fields are left as variables to be populated by the calwebb_spec3 pipeline at execution time.

General changes

  • Documentation updates:
    • Updated the calwebb_spec2 pipeline documentation to indicate that the pixel_replace step is applied to NIRSpec IFU data, and made minor wording updates to the pixel_replace step documentation.
    • Updated the "FilterOffset" reference file documentation to indicate that this reference file type is also used for NIRISS, and increase clarity on how the offsets are applied.
    • Added more details to the cube_build step documentation.
    • Updated the jump detection step documentation to include more information for the step parameters according to the various modes of jump detection.
  • Data models updates:
    • Added support for user-supplied group_id values, either via the "model.meta.group_id" attribute or as a member attribute in an ASN table. Used for grouping images within the tweakreg step.
  • Other
    • Python 3.11 is now the default for the calibration pipeline and SDP processing.
    • Minimum version requirements: asdf 2.15.1 and numpy 1.22
    • Fixed numpy 2.0 deprecation warnings in the cube_build, photom, and wfs_combine steps.

JWST Science Data Processing subsystem changes

The JSDP subsystem processes data from the telescope and creates raw (uncal) products. The following changes have been included in this build:

  • Corrected bugs in the calculation of WCS pointing-related keywords for FGS guiding mode and pure parallel exposures

  • Added a GSC_VER keyword to the header of FITS files to capture the version of the guide star catalog used. The value is something like:


    corresponding to GSC

Notable updates

Originally published