JWST Time Series Observations Pipeline Caveats

Content Migration

WARNING: This page has been deprecated and the content migrated to Key Differences for JWST Time Series ObservationsKnown Issues with JWST Data, and pages therein.  Please update your links accordingly.

Unique features of the JWST calibration pipeline for time-series observations (TSOs), and caveats for users, are covered in this article. 

On this page

Summary of general time-series observations calibration pipeline issues

The information in this table about general time-series observations calibration pipeline issues is excerpted from Known Issues with JWST Data Products.  

SymptomsCauseWorkaroundMitigation Plan

GI-TS01: For time-series data (for all instruments), FITS primary header keywords are different from the "INT_TIMES" extension. Particularly, this concerns the start/end times (BSTRTIME and BENDTIME) and the barycentric correction (BARTDELT) keyword.

"INT_TIMES" are based on the group times directly read into the engineering data. This is not the case with the header keywords, which do not account for electronic shifts on the reading of the data.

Use "INT_TIMES".

Updated Operations Pipeline

A change to the JWST Science Data Processing subsystem to correctly compute the barycentric and heliocentric time, and JWST barycentric position keywords was part of the updated Operations Pipeline, installed on August 24, 2023. STScI reprocessed affected data products, which  typically takes 2–4 weeks after the update.



More about TSO pipeline caveats

This information reflects the status based on jwst calibration pipeline package 1.4.6. Updates will be made as appropriate.

In time-series observations, we continuously observe a time-varying phenomenon. This covers periodic phenomena such as eclipsing binary stars or transiting exoplanets, or other types of variable sources such as flares from accreting black holes. The goal of such observations is typically to obtain relative (spectro-)photometry to characterize the variations with high precision.

TSOs are marked in APT with the time-series observation special requirement, which allows the observations to be executed under conditions optimal for this type of performance, and processed with a dedicated pipeline configuration. As a result, the pipeline output products differ in format from "regular" observations. 

This article describes the data products produced by the pipeline for TSOs and some unique features and caveats for the calibration pipeline when working with TSO data. The following information applies only to the dedicated TSO modes onboard JWST, namely:



TSO pipeline and products overview

See also: Getting Started with JWST DataFile Header Contents
Software documentation outside JDox: Science Product Structures and Extensions 

Time-series observations are processed and calibrated with the same calibration pipeline as "regular" exposures, albeit with dedicated configuration files and algorithmic pathways that are specialized for these observations. The main difference between TSOs and other types of observations is that TSOs are typically executed in single, long duration exposures consisting of multiple integrations. Whereas regular observations may include multiple integrations in an exposure to optimize signal-to-noise ratio over a wide bandwidth (e.g., to increase SNR on faint features without saturating the bright regions), each integration in a TSO is treated as a separate time sample. The duration of an integration therefore sets the time cadence with which the time series is sampled. TSOs can include exposures that are much longer than the 10,000 s to which other exposures are limited. 

File segmentation

Words in bold are GUI menus/
panels or data software packages; 
bold italics are buttons in GUI
tools or package parameters.

Because of this long single exposure execution strategy, TSO exposures can often have very large file sizes. This makes the initial pipeline stages, in particular the calwebb_detector1 stage, which performs detector-level calibrations and converts ramps into slopes, very computationally demanding. To manage this, TSO exposures are segmented by the data management system into segments with a maximum size of 2 GB. This segmentation is performed after the exposure—the exposure itself was recorded continuously by the observatory. 

This means a single TSO exposure may consist of multiple files. The segments can be recognized by "segNNN" in the filename, where NNN is a 3-digit number (e.g., "seg001" indicates the first segment of an exposure). Only segmented files contain this "segNNN" field in their exposure filenames. Header keywords in the primary FITS header record first and last integration packaged in the particular segment file, as follows:

  • EXSEGNUM: The segment number of the current product
  • EXSEGTOT: The total number of segments
  • INTSTART: The starting integration number of the data in this segment
  • INTEND: The ending integration number of the data in this segment

Each segment is processed independently in stages 1 and 2 of the pipeline. In stage 3, the segments are combined to produce a single time-series output product from the exposure. 

Imaging

Pipeline stages and steps

See also: Data Calibration Reference Files
Software documentation outside JDox: Pipeline Modules, Associations

Imaging TSOs follow the pipeline in 3 stages:

The stages listed above are linked to articles that provide further information on the steps in each of them, with additional links to dedicated software documentation. Stages 1 and 2 are generic to imaging observations, with the TSO-specific flow set by parameter reference files. These can be viewed and downloaded from the Calibration Reference Data System (CRDS). 

Stage 3 of the pipeline, calwebb_tso3, performs TSO-specific processing steps for both spectroscopy and imaging. For the case of imaging, 2 steps are executed in stage 3 of the pipeline:

  1. Residual outliers are flagged.
  2. Aperture photometry is performed and the resulting fluxes are written to a flux-calibrated photometric light curve output file.

If the original exposure file was segmented, the data will at this stage be merged to produce a single light curve.  

As in all stage 3 modules, the input to the calwebb_tso3 pipeline is an association file, which brings together the multiple segments.

Data products

Software documentation outside JDox: Science Products 

The pipeline generates a number of additional products for TSOs compared with standard imaging observations. The calwebb_detector1 pipeline produces a "rate.fits" file, but specifically also a "rateints.fits" file. The "rateints" file contains slope images for the exposure, or exposure segment, for each individual integration. In the "rateints.fits" file, the science, DQ, error, and variance extensions are 3-D, instead of 2-D, products, with the 3rd dimension matching the number of integrations in the exposure or exposure segment. 

An additional extension, "INT_TIMES", contains a table listing the beginning, middle, and end time stamps for each integration in the exposure, or exposure segment.

The calwebb_tso3 pipeline stage produces 2 output products. The first, with file extension "crfints.fits", is produced from the outlier detection step, and contains updated outlier flags in the DQ array. No changes are made to the science or error extensions. Finally, the photometric time series is captured in the "phot.ecsv" file, which is an ecsv file. This file consists of a source catalog containing photometry results from all of the "crfints" products, organized as a function of integration time stamps. 

Spectroscopy

Pipeline stages and steps

See also: Data Calibration Reference Files cSoftware documentation outside JDoxPipeline Modules, Associations 

Spectroscopic TSOs follow the pipeline in 3 stages:

The stages listed above are linked to articles that provide further information on the steps in each of them, with additional links to dedicated software documentation. Stages 1 and 2 are generic to spectroscopic observations, with the TSO-specific flow set by parameter reference files. These can be viewed and downloaded from the Calibration Reference Data System (CRDS). 

Stage 3 of the pipeline, calwebb_tso3, performs TSO-specific processing steps for both spectroscopy and imaging. For the case of spectroscopy, 3 steps are executed in stage 3 of the pipeline:

  1. Residual outliers are flagged.
  2. The segments are merged and spectra are re-extracted and packaged into a single file, using the same parameters as in the calwebb_spec2 pipeline.
  3. The spectrum is summed in wavelength space to produce a white light curve.

If the original exposure file was segmented, the segments will at this stage be merged to produce a single extracted spectrum per integration and a single white light curve for the entire exposure. The white light curve is produced by summing the flux between pre-defined wavelength boundaries in the spectrum in each extracted spectrum, yielding a single flux data point per integration. 

As in all stage 3 modules, the input to the calwebb_tso3 pipeline is an association file which brings together the multiple segments.

Data products

Software documentation outside JDox: Science Products 

The pipeline generates a number of additional products for TSOs compared with standard spectroscopic observations. The calwebb_detector1 pipeline produces a "rate.fits" file, but specifically also a "rateints.fits" file. The "rateints" file contains slope images for the exposure, or exposure segment, for each individual integration. In the "rateints.fits" file, the science, DQ, error and variance extensions are 3-D, instead of 2-D, products, with the 3rd dimension matching the number of integrations in the exposure or exposure segment. 

An additional extension, "INT_TIMES", contains a table listing the beginning, middle, and end time stamps for each integration in the exposure, or exposure segment.

Similar to calwebb_detector1, calwebb_spec2 returns 2-D spectro-photometrically calibrated images as "calints.fits" files, and 1-D extracted spectra are returned as "x1dints.fits" files. At this stage, segmented files are processed separately and each exposure segment will produce its own "x1dints.fits" file. 

The calwebb_tso3 pipeline stage produces 3 spectroscopic output products for each additional step as described above, merging the exposure segments for segmented observations. The first, with file extension "crfints.fits", is produced from the outlier detection step, and contains updated outlier flags in the DQ array. No changes are made to the science or error extensions in this step. Second, the exposure segments will be merged, and the spectra are re-extracted as a single product (with the segments merged together), producing a new "x1dints.fits" file. The white light curve is captured in an ecsv file with suffix "wtlt.ecsv". This file lists the integrated white light flux as a function of time, based on the integration time stamps.



Pipeline caveats

JWST data is processed automatically through the JWST calibration pipeline once it is transmitted to Earth from the observatory, using the sequence of steps set in the parameter files and default settings that were chosen for each instrument mode by the instrument teams, and the most up-to-date set of calibration reference files. The processed data products will be available in the Archive when you access your data, and these products are "science ready." However there may be situations where you want to re-run the pipeline and/or perform additional analyses between pipeline steps.

The default settings are all well-chosen for your type of observations, but they may not be optimal for your particular science measurement. In addition, in order to gain an in-depth understanding of the data, instrumental systematics, detector noise properties, and how these are treated in the pipeline, you may want to try different settings and parameters, and study their impact on the data.

Pipeline reference files

Software documentation outside JDox: JWebbinars, Calibration Pipeline Documentation 

Calibration reference files are stored in the Calibration Reference Data System (CRDS). These files can be viewed and downloaded via a convenient web interface

Of particular interest are the parameter reference files that specify the pathway each instrument/mode should take through the calibration pipeline. These files have filenames starting with "pars-" and are in ASDF format, which is readable with a standard text editor. Dedicated parameter reference files exist for TSO data, for each pipeline stage. Guidelines for using the CRDS web interface can be found at this webpage.

Parameter reference files can be overridden with custom files that edit the standard data flow. This can be useful to investigate the impact of certain calibration steps on the final measurement accuracy or precision. Examples of how to implement such overrides can be found in tutorials such as the JWebbinar materials. Instructions are also provided in the calibration pipeline documentation. The pipeline documentation further contains information on saving output files from a pipeline run, and for creating logging output. Both these strategies are useful for tracking and optimizing the performance of the pipeline for your data.

Examples of many of these strategies are captured in the TSO JWebbinar materials, available online. 

General caveats

1/f noise

See also: JWST Time-Series Observations Noise Sources

JWST detectors exhibit different types of frame-to-frame noise produced by the readout electronics. One which is of particular interest for TSOs and which is not completely corrected by the current version of the pipeline is the so called 1/f noise. The most obvious visual feature of this noise is an apparent "banding" of the background pixels in a frame/group. Removing the impact of 1/f noise on the data, especially for the near-infrared instruments, is quite complex given the stochastic nature of the effect (every frame will have its own, unique signature—however, they will all share similar statistical properties). While the reference pixel correction helps diminish its effect in some cases, in general, extra steps may need to be taken in an attempt to correct for it. One simple way of removing part of its effect is to use background pixels along the "banding" to partially correct for 1/f noise on timescales longer than the column/row-read time. However, care must be taken to handle real background flux structure when performing such corrections.

Time stamps across the JWST detectors

See also: NIRCam Detector Readout PatternsNIRISS Detector Readout PatternsNIRSpec Detector Readout

The detectors used by JWST instruments do not read all their pixels at the same time. For the near-infrared detectors, the reading process controlled by the SIDECAR ASIC sequentially reads pixels in the "fast-read" direction. In NIRISS/SOSS frames, for instance, the reading process starts in one of the corner pixels and moves along the fast-read direction, reading one pixel at a time at a cadence of 10 microseconds per pixel. When all the pixels in a given "fast-read" column are read, the detector has a wait time of 120 microseconds before moving to the next one. This process is repeated until the entire subarray is read, after which extra wait time (which depends on the exact subarray configuration) is spent before considering the frame "read". Note that for this NIRISS/SOSS example, this means not all columns are read at the same times, but instead, some are read earlier than the rest. In practice, in NIRISS/SOSS this means that the longer wavelengths are effectively read by the detector at different times than the shorter wavelengths—the difference being about 5 s in total per frame. This is currently not accounted for in the pipeline. Users should study whether this offset might impact their science cases, and consider making corrections to their observations based on this so every wavelength has the correct time stamp associated with it.

The time constants may be different for each instrument, and dependent on the subarray and read mode. To understand these timings, please refer to the instrument detector information pages.

Pointing jitter or drift

See also: JWST Pointing PerformanceJWST Communications SubsystemJWST Attitude Control Subsystem

The JWST calibration pipeline currently assumes that the pointing of the telescope is perfect and constant over the course of an observation. Only the target acquisition procedure includes a source-finding and centroiding step; once the target has been placed into the science aperture, the pipeline assumes that its location is fixed. In reality, the initial target placement may be imperfect, and over many hours the pointing may be subject to drifts, jitter, or jumps. The high gain antenna repointing, for example, has been observed during commissioning to have some impact on the telescope pointing when it is moved. This might have some consequences for time-series observations. 

For imaging, the automated photometry step in calwebb_tso3 assumes a fixed location of the target in the field of view. If this location changes significantly over the course of the time-series observation, the photometry measurement will be performed off-center and return inaccurate results. 

For spectroscopy, the wavelength calibration and photometric calibration factors are applied to the data assuming perfect and constant target placement in the field. Any drifting behavior or jitter, therefore, might introduce systematic noise in the final time series. There is currently no mitigation for this in the calibration pipeline; any pointing changes must therefore be measured and corrected manually in the time series. A suggested workaround is to skip the photometric calibration step by modifying the execution parameters for the calwebb_spec2 stage since for many science needs it is the relative variations that are of most interest. Further pipeline work in this area is planned. 

How this issue might manifest in the calibration process for different instruments is highlighted in the instrument-specific sections below. In practice, a good recommendation is to monitor the JWST guide star data which can be retrieved via MAST, and use it to study any significant pointing aberrations that might be impacting a given TSO. Details on how to perform this data retrieval can be found in JWST Time-Series Observations Noise Sources.

Jump step

Several steps in the pipeline are dependent on actual on-orbit performance and noise properties of the instruments; among these, the jump step is one of the more important to be wary of. The goal of this processing step is to detect the anomalous jumps in flux between groups due to cosmic rays. A specific threshold is used to detect such jumps. Pre-launch, this step had only been tested on simulated data, and the method for simulating cosmic rays may or may not be fully representative of conditions on orbit.

The performance of this step can have a dramatic effect on the derived SNR for reduced observations. One recommendation is to be mindful of the thresholds used to detect jumps, and test different values on your dataset to find the optimum value; bear in mind that different jump thresholds can result in different outcomes further downstream in the pipeline. Information on where jumps were flagged by this processing step can be found in the "DQ" (data quality) extension of the "rateints.fits" file. In Python, this can be achieved with the following code:

from jwst import datamodels

data = datamodels.open(‘file_rateints.fits’)
whereJump = data.dq & 2**2

calwebb_tso3

The 3rd stage of the pipeline performs some final operations on the data to produce higher-level output products. It does not currently perform any additional calibration steps; as such, the output of the calwebb_spec2 pipeline constitutes the "science-ready" data. The main additional operation is that file segments for segmented exposures are merged into single output products; for example, the extract_1d step returns a single "x1dints.fits" file containing the extracted spectra for all integrations. 

For imaging, time-series photometry will be performed, returning a "phot.ecsv" output file tabulating fluxes as a function of julian date of the observation. 

For spectroscopy, a white light curve will be produced by integrating the spectra over a particular wavelength range (which can be adjusted in the step parameters), returned in a "whtlt.ecsv" file. 




Latest updates


Originally published