Release notes and known issues in build 8.0 of the JWST Operations Pipeline, released on June 16, 2022, are provided in this article. 

This article provides a list of all relevant changes to the operations pipeline that was released with build 8.0. This build focused mainly on updates to the JWST Science Calibration Pipeline, but included information relevant to changes in other subsystems that might affect what users see in their data.

Highlights of this build

• Weighting scheme was changed for IFU cubes
• Replaced background fitting functions in the residual_fringe step
• Implemented a new extraction algorithm for NIRISS SOSS mode data
• Saturation step now accounts for charge spilling
• Implemented the processing of NIRCam "Frame 0" data
• Users can now set a minimum (faintest) magnitude limit for sources to be used in the background step for WFSS exposures
• A new parameter can now override the source type normally computed when processing exposures that contain a single pre-defined target
• calwebb_spec2 pipeline now provides a way to save an optional intermediate product for WFSS data
• Modified the processing flows of the calwebb_spec2, calwebb_spec3, and calwebb_tso3 pipelines for NIRISS SOSS exposures

Known issues

Release notes

See also: GitHub project release notes 1.5.2

Build 8.0 of the JWST Operations Pipeline was released on June 16, 2022 tothe MAST Archive. This build includes changes to the JWST Science Calibration Pipeline released in 3 versions of the jwst calibration pipeline package via the PyPI repository:

• 1.5.0 released on 05 May 2022 
• 1.5.1 released on 17 May 2022
  
• 1.5.2 released on 20 May 2022 

The following fixes or changes were included in build 8.0. Changes to specific steps in the pipeline are listed below, along with descriptions to more general changes made to specific packages.

align_refs step

• Changed the warning-level messages from the median filter function to be debug level instead.
• Updated the median filter function to exclude pixels flagged as "NON_SCIENCE".

ami_average step

• Updated the step infrastructure to allow the user to specify a list of input files on the command line, rather than always needing to use an  association file as input.

assign_wcs step

• Corrected an issue with the computation of the bounding box for the output WCS that affected all aspects of resampling
• Fixed how the filter offset reference files are parsed so the offset provided in the file is correctly used for MIRI, NIRISS, and NIRCam.

background step

• Added the new step parameter wfss_mmag_extract for use with WFSS exposures, which allows users to set a minimum (faintest) magnitude limit for sources to be used in the step. This is especially helpful for observations of crowded fields that contain thousands of sources.
• Added a check in the step to make sure that a sufficient number of background (source-free) pixels are available in a WFSS image before attempting to calculate background statistics and scale the WFSS background reference image.

cube_build step

• Fixed residual spectral tearing in MIRI MRS multi-band cubes that appeared when combining data from 2 channels—this tearing is due to the slightly different wavelength range that the individual slices have that result in an incomplete spatial coverage at the ends of each slice. 
• Fixed a bug in how the DQ plane of NIRSpec data was set when no valid data are present.
• After a performance analysis, the default calibration processing was changed to use the 3-D drizzle weighting scheme by default, instead of "Exponential Modified Shepard Method" (EMSM), which uses an exponential kernel instead of 1/r or 1/r².
• The Read the Docs documentation was updated to reflect the change.

cube_skymatch step

• Enabled support for several data quality flags. This change now provides the complete set of flags needed to determine which pixels should be included in the computation of the sky value.

dark_subtraction step

• Eliminated unnecessary copies of the input data when the step gets skipped, in order to reduce memory usage.

extract_1d step

• Implemented the new extraction algorithm  "Algorithme de Traitement des Orders ContAminée" (ATOCA) for NIRISS SOSS mode data. Also implemented several low level fixes to that algorithm to trap and handle situations involving bad input data. Note that the ATOCA algorithm does not support exposures using the F277W filter in combination with the grating; the extract_1d step will skip processing for these exposures.
• Updated the use of extract1d reference files to allow specified extraction region limits to be applied to extended sources as well as point sources.
• Updated to propagate non-differentiated errors for IFU mode observations.

flatfield step

• Changed the propagation of DQ flags for NIRSpec data such that if any one of the 3 flat field reference files (D-flat, F-flat, S-flat) has bad data for a given pixel, the output pixel is flagged as "DO_NOT_USE" and "NO_FLAT_FIELD", and no flat field value is applied to that pixel.

jump step

• Enabled multiprocessing options for the step, which reduces processing time when used off-line.

linearity step

• Implemented the processing of NIRCam "Frame 0" data, when present, so that it's properly calibrated when used in the subsequent ramp fitting process.

mrs_imatch step

• Changed the default weighting type method used to create IFU cubes from EMSM to drizzle.

photom step

• Modified to apply flux calibration to 1-D extracted spectra for NIRISS SOSS exposures, rather than the (pre-extraction) 2-D images.

ramp_fitting step

• Added the new step parameter suppress_one_group, which is used to turn off slope calculations for pixels that have only one good (unsaturated) group.
• Implemented the use of NIRCam "Frame 0" data, when present, in order to calculate slopes for pixels that are saturated in all groups. For pixels that have all groups in an integration flagged as saturated, if the zeroframe data for that pixel is not saturated, it is used to estimate a slope for the pixel.
• Modified the logic used to copy the INT_TIMES table extension content to output products, so that the table data appear in all "rateints" products.
• Fixed a bug in handling jumps (CR hits) that occur in the first good group after a set of initial bad groups (e.g., groups flagged by the MIRI firstframe step).

refpix step

• Implemented the processing of NIRCam "Frame 0" data, when present, so that it's properly calibrated when used in the subsequent ramp fitting process.

resample/resample_spec step

• Fixed an output spectrum centering issue in the resample_spec branch for MIRI LRS fixed slit spectra.
• Redesigned the algorithm used to compute the WCS for the output (resampled) frame in the resample_spec branch for NIRSpec fixed slit and MOS spectra.
• Fixed the handling of the weight_type parameter value in the resample_spec branch so that it applies the user-supplied value for that parameter.

reset step

• Fixed a bug in the way segmented input data files are handled.

residual_fringe step

• Replaced background fitting functions from the "BayesicFitting" package with equivalent routines from astropy, in order to remove the dependency on the BayesicFitting package. Note that this leads to slight changes in results, due to numerical differences in how astropy (SciPy based) splines are fitted versus the BayesicFitting splines.

saturation step

• Updated to allow the step to flag neighbors of saturated pixels as saturated in order to account for charge migration (also known as charge spilling). The new step parameter n_pix_grow_sat controls the distance over which neighbors are flagged. The default is 1 and this can also be set to 0 to effectively turn off the feature.
• Implemented the processing of NIRCam "Frame 0" data, when present, so that it's properly calibrated when used in the subsequent ramp fitting process.

skymatch step

• Added a new output keyword BKGMETH, which stores the name of the skymethod used to perform the sky computations.

• Enabled support for mnemonic DQ codes in the skymatch step.
• Changed the default value of the dqbits parameter from "DO_NOT_USE"  to "DO_NOT_USE" + "NON_SCIENCE" (only exclude pixels flagged with those two DQ values).
• Increased the tolerance value for considering two sky polygons identical if their vertices are within a specific proximity to each other in order to avoid crashes due to accuracy loss in the spherical geometry package.

source_catalog step

• Fixed an issue where the SciPy cKDTree function returned "NaN" in the x,y positions of sources, which made the step fail in Linux. The root cause was a "feature" in the scipy.ndimage.zoom interpolation where a zoomed image with all positive values could, in some cases, return an interpolated image with negative values. Photutils 1.4.0 fixes this by adding a keyword to ensure the zoom-interpolated output values do not extend beyond the range of the input values.
• Updated the sharpness and roundness properties (of a star that is cataloged as a source) to use the source centroid instead of the source peak position.

srctype step

• Added a new step parameter source_type, which can be used to override the source type normally computed by the step when processing exposures that contain a single pre-defined target.

superbias step

• Implemented the processing of NIRCam "Frame 0" data, when present, so that it's properly calibrated when used in the subsequent ramp fitting process.

tweakreg step

• Changed the default value of the fitgeometry parameter from general to rshift, to allow for greater chances of success when processing sparsely populated images.
• Changed the default values of the step parameters searchrad, tolerance, separation, and brightest to 2.0, 0.7, 1.0, and 200, respectively.

General changes

• Added alias names for all pipeline steps so that the alias can be used when executing via "strun", instead of the full class name.
• Added the "CRDS_CONTEXT" value in use to the processing log information.

Pipeline modules

• Updated the calwebb_dark pipeline for MIRI data to include the reset correction step.
• Improved memory performance of the calwebb_detector1 pipeline.
• The calwebb_detector1 pipeline was modified so all applicable steps are applied to the NIRCam "Frame 0" (zeroframe) array, when present. The zeroframe array contains the image from the initial frame of each integration. See the ramp_fitting notes for more details on the use of zeroframe.
• Updated the WFSS processing branch of the calwebb_spec2 pipeline to allow it save an optional intermediate product that is a version of the grism image converted to units of e^–/s. The image is saved immediately after the flatfield step, before extract_2d has been applied, so that it is the full grism image before any extractions of spectra have been done. The image data are converted to units of e^–/s by loading the gain reference file, computing the mean gain from the reference data, and multiplying the grism image by the mean gain. The units of the variance and error arrays are also converted. The resulting image is saved as a new "esec" product type. Creating and saving this image is optional and is controlled by the new pipeline parameter save_wfss_esec, which has a default setting of False.
• Modified the processing flows of the calwebb_spec2, calwebb_spec3, and calwebb_tso3 pipelines for NIRISS SOSS exposures, to apply the photom (flux calibration) step after the extract_1d step, because the new SOSS ATOCA extraction algorithm in extract_1d needs to work with uncalibrated data.

Product associations

• Fixed the creation of stage 2 associations for NIRSpec internal lamp exposures by adding logic that is based on the combination of detector, disperser, and lamp in use for a given exposure.
• Added constraints to the creation of stage 2 and stage 3 associations that excludes NIRCam TSGRISM exposures that use the NRCBLONG detector, because TSGRISM is not supported for module B.

Data models

• Updated the comment fields of the V2_REF, V3_REF, FITXOFFS, and FITYOFFS keywords.
• Added the new keyword BKGMETH for use in the skymatch step.
• Removed the FILETYPE keyword from all data models, because it was deemed to be unnecessary and difficult to support.
• Added data models for the new MIRI MRS point source correction reference file, "MirMrsPtCorrModel", and cross-artifact reference file "MirMrsXArtCorrModel".
• Updated to handle the new "MSA_TARG_ACQ" table extension in NIRSpec products.