JWST Operations Pipeline Build 8.0.1 Release Notes
Release notes and known issues in patch build 8.0.1 of the JWST Operations Pipeline, released on June 30, 2022, are provided in this article.
This article provides a list of all relevant changes to the operations pipeline that was released with patch build 8.0.1. This build focused mainly on updates to the JWST Science Calibration Pipeline, but included information relevant to changes in other subsystems that might affect what users see in their data.
Highlights of this build
- Improvements in Level 1 TSO processing
- Calibration updates for MIR MRS stray light
- Improvement in the resample step for NIRSpec FS and MOS
- Bug fixes
Words in bold are GUI menus/
panels or data software packages;
bold italics are buttons in GUI
tools or package parameters.
The following issues were not resolved in the jwst calibration pipeline package (version 1.5.3) that was released with patch build 8.0.1:
- The calwebb_image3 and calwebb_spec3 pipelines can occasionally fail with memory allocation errors when processing observations containing very large image mosaics or many hundreds of spectra. Improvements in memory management are underway and at least an initial phase of improvements will be released in build 8.1.
MIRI MRS observations of a background target run into trouble during processing due to the fact that the 1-D spectra extracted from the cubes of each single exposure (containing two wavelength channels of data) end up with a range of wavelengths not covered by either channel and hence the flux in the background spectrum is zero. When this is combined in the calwebb_spec3 master background step, it leads to incorrect results for those wavelengths. This will be resolved in build 8.1.
- When running a pipeline and providing a custom parameters reference file for some of the steps, the software does not seem to pay attention to the user-supplied parameters reference file and continues to use the values from the parameters reference file in CRDS. This bug will be fixed with build 9.0 later this year.
- The reported update to propagate non-differentiated errors for IFU mode observations was omitted from build 8.0.1. It will be included in build 8.1
- The source_catalog step in calwebb_image3 is reporting error values that are much larger than expected due to a bug which leaves the error values in units of MJy/sr instead of converting them to Jy, as is done with the science data. This will be resolved in build 8.1, but the fix is currently available in the development version of the jwst package.
Also, the following infrastructure-related issues affect this build:
- Due to a software bug, the NIRCam level-1b products for TSO grism data require manual developer intervention, which causes the timing information to be incorrect and prevents processing through stage 1 of the pipeline. This is targeted to be resolved in build 8.1.
- WFSS program data can produce a significant number of files, so ingestion of data into MAST may be delayed by a few days after completion of the program.
See also: GitHub project release notes
Patch build 8.0.1 of the JWST Operations Pipeline was released on June 30, 2022 to the Barbara A. Mikulski Archive for Space Telescopes (MAST). This build includes changes to the JWST Science Calibration Pipeline released in version 1.5.3 of the jwst calibration pipeline package via the PyPI repository. This patch release also includes changes to the JWST Science Data Processing subsystem (SDP) version JWSTDP-2022.2.1, which creates raw (uncal) products.
The following fixes or changes were included in patch build 8.0.1. Changes to specific steps in the pipeline are listed below, along with descriptions to more general changes made to specific packages and related subsystems.
- Fixed a bug that added an empty "SCI" extension to the output product of AMI data, which should not be present in these products.
- The documentation of this step was updated to include information about all of the available step arguments.
- In order to handle images of different sizes, the step was updated to use the minimum size of the fitted (modeled) images to be the size of the averaged image(s). When computing the average, extra rows and columns are trimmed from the edges of inputs that are larger.
- Corrected an issue with the computation of the bounding box for the output WCS that affected all aspects of resampling.
- Fixed how the filter offset reference files are parsed so the offset provided in the file is correctly used for MIRI, NIRISS, and NIRCam.
- Fixed incorrect variance propagation or computation of errors for the background spectra in IFU data.
- Changed the propagation of DQ flags for NIRSpec data such that if any one of the 3 flat field reference files (D-flat, F-flat, S-flat) has bad data for a given pixel, the output pixel is flagged as "DO_NOT_USE" and "NO_FLAT_FIELD", and no flat field value is applied to that pixel.
- Modified the method of estimating sampling wavelengths in resample_spec to make it faster at the expense of not reproducing more accurately sampling wavelengths of the input image and reference image.
- Fixed the default handling for missing Science Instrument Aperture Files (SIAF) values accessed via the python package pysiaf
- Reduced memory usage when running the skymatch step when input to the step is an association file. This is accomplished by saving large image and mask arrays to temporary files and retrieving them on demand. This fix does not work when input to the step is a list already loaded into memory data models
- Fixed a bug in this step that prevented it from running as a standalone step with a filename as an input.
- Added a check to verify that the input data are IFU images before trying to apply this step. This ensures this step is applied only to non-TSO data.
- MIRI MRS ground test data indicated that the "stray light" feature seen around bright sources may have significant spatial structure that was not captured by the existing smooth stray light model. In this patch, the algorithm was updated to use the cross-artifact model, in order to adequately model and subtract the artifacts and to enable derivation of proper flux calibration and aperture correction factors. For more details, please refer to Read the Docs documentation
- Updated the documentation for calwebb_detector1 pipeline, as well as the linearity, refpix, ramp_fit, saturation, and superbias steps to include information on the handling of NIRCam "Frame 0" data.
- Updated the refpix step documentation to clarify the roles of the odd_even_rows and odd_even_columns parameters in the MIRI and NIR data. It also clarifies which parameters are used and which are ignored, and logs an explanatory message while running the refpix step.
- Many of the data models that are used to create new reference files were updated to include current calibration pipeline selectors in the schema structure.
- Fixed a bug in Spec3Pipeline and Tso3Pipeline that skips the photom step when the extract_1d step is skipped for SOSS data taken through the F277W filter.
- Updated the calwebb_image3 pipeline to only use science members from the input association file and ignore any other type of observation. This prevents the module from trying to combine science and background observations that are great distances from one another on the sky.
- Create level 3 association files for background images, and allows background target observations to be used during level 2 procession of image associations files to be used for background subtraction.
- Updated documentation to explain when and how the CRDS PUB server should be used when running the JWST Science Calibration Pipeline
JWST Science Data Processing subsystem changes
The JSDP subsystem processes data from the telescope and creates level-1b (uncal) products. In this version, the following changes were made to the data:
The FITROFFS keyword, used in the x_rot_rel and y_rot_rel calculations for NIRSpec MSATA data, changed sign. This keyword is defined as the roll offset value (arcsec) from the least squares fitting algorithm. The new formulas used to calculate these values are now:
xrot_rel = x_rel*cos(FITROFFS) - y_rel*sin(FITROFFS)
yrot_rel = x_rel*sin(FITSROFFS) + y_rel*cos(FITSROFFS)
x/y_relare values from other columns in the table.
- We stopped generating "Discovered whole program associations." Those were associations specified by the letter "a", followed by a number starting at 3001. These associations were created by associating all data from a program, using the same target, instrument, and optical elements (e.g., filter or grating).
- The keyword structure of the the "TARG_ACQ_EXT" extensions in the level-1b (uncal) data was updated.
GUIDERwas changed to
TA_ITERSwas changed to
V2/3_RESIDwas changed to
- Invalid timestamps between segments for TSO were corrected.