JWST Data Rate and Data Volume Limits
JWST can store at least 58.8 Gbytes of science data. Science data downlinks occur in two 4-hr contacts per day where each contact can transmit at least 28.6 Gbytes of recorded science data. APT simplifies these limits to 58 Gbytes and 29 GBytes, respectively. Users must be mindful of these limits when designing their observations.
See also: Solid State Recorder
Data volume defines the total amount of data (58.8 GBytes) that can be stored on the Solid State Recorder (SSR) at any given time. Data rate defines the speed with which science data can be written to the SSR, which is ultimately regulated by the ISIM Command and Data Handling subsystem (ICDH). JWST downlinks science data to Earth in two contacts per day, where each contact can transmit at least 28.6 GBytes of recorded science data to the ground.
Managing data rate
Both data rate and data volume are driven by the observing plan. To facilitate the scheduling of the observations, users are encouraged to keep the data volume under 29 Gbytes in a 12 hour period. The reason for this is to prevent the SSR from filling before the next downlink. The Astronomer's Proposal Tool (APT) does prevent observations from breaking the ICDH data rate limits. While APT (as of version 25.4) will generate an error if a visit exceeds 58 GBytes in total data volume, it does not warn the user when the data volume fills at a rate that exceeds 58 GBytes per 24 hours. Users are therefore encouraged to check the information provided by APT to keep the volume under that rate value.
Users should keep in mind that data volume and data rates issues can only be fully identified downstream; the Visit Scheduling Subsystem and the Visit Planning Subsystem are designed to take these issues into consideration. Accepted programs may have to be modified to comply with data volume and data rate limits. Proposers should understand the data rate of their program, and if necessary, take steps to reduce the data rate.
General advice for reducing data volume
Reduce the number of groups and integrations
See also: Understanding Exposure Times
Many of the observing modes that exceed the data volume limit are only problematic if executed for long periods of time (e.g., >12 hours). The simplest method for reducing your program's total data volume in a 12 hour period is to minimize the total number of groups per integration and number of integrations per exposure.
Select a different readout pattern, mode, and/or subarray
Using different readout patterns can enable longer exposures with a reduced amount of saved data. For example, choosing a larger numbers of frames averaged per group reduces data volume (and yields a more precise average). For MIRI, this can be achieved using either fast or slow modes.
Use overheads to your advantage
Factor in potential overheads that will decrease your observing efficiency, but at the same time potentially alleviate any data volume concerns.
There are known scenarios for each instrument that may exceed the data volume limit. Some of these scenarios and suggested solutions are described below.
See also: NIRCam Detector Readout Patterns
A raw 2048 × 2048 pixel detector frame is ~8 MB. Near-Infrared Camera (NIRCam) has 10 detectors, so this implies at least ~80 MB of data for each group within an exposure depending on the readout pattern. Neglecting all overheads and assuming 24 hours of continuous data taking on all 10 detectors (both short- and long-wave channels), these are the limiting cases for the 9 readout patterns:
- DEEP2 and DEEP8: time between groups is ~200s (~34GBytes/day)
- MEDIUM2 and MEDIUM8: time between groups is ~100 s (~ 68 GBytes/day).
- SHALLOW2 and SHALLOW4: time between groups is ~50 s (~136 GBytes/day).
- BRIGHT1 and BRIGHT2: time between groups is ~20 s (~340 GBytes/day).
- RAPID: time between groups is ~10 s (~680 GBytes/day).
In addition to the steps outlined above, NIRCam observers may want to also consider the following options for reducing their data volume.
Use only one NIRCam Module: It is possible to use only one module for an observation as a method to reduce data volume by selecting a single module in APT rather than ALL. The module options in APT will vary depending on the observing mode (e.g., module A for coronagraphy or module B for imaging).
Change the number of outputs: In the case of grism time series observations, a proposer may want to change the number of output amplifiers. Readout of the full NIRCam detector (2048 × 2048 pixels) is performed with 4 outputs simultaneously (Noutputs = 4), each delivering a stripe of data (2048 pixel rows × 512 pixel columns), and taking 10.7 s altogether. Smaller subarrays are read out more quickly, and most are read out through a single output (Noutputs = 1). Noutputs is pre-defined for most subarrays, but observers are given a choice between Noutputs = 1 or 4 in the grism time-series observing mode. Choosing 1 output reduces the frame rate by a factor of four (for the 4 amplifiers).
The Mid-Infrared Instrument (MIRI) has only three detectors, but obtaining simultaneous imaging with both the imager and spectrograph can potentially exceed the data volume limit. MIRI observers should consider the general advice described above if their planned observations exceed the data volume limit.
The Near-Infrared Spectrograph (NIRSpec) has two detectors, which have four readout patterns spread out split over two readout modes. The IRS2 NRSIRS2RAPID readout mode (~1.95 MBytes/s) results in higher data volume than the traditional NRSRAPID readout mode (~1.56 MBytes/s) because interspersed reference pixels and outputs are also saved. Users are recommended to use these RAPID readout patterns (number of frames per group equal to one) for cases that have no data volume issues reported in APT because of the improved performance of cosmic ray rejection. Despite the higher data volumes, the IRS2 patterns are also to be generally preferred over the normal modes due to the reduction in correlated read noise.
NIRSpec observing options that could result in APT data volume errors include:
- Deep, full-frame FS/IFU/MOS long exposures with NRSRAPID or NRSIRS2RAPID readout (with exposure times beyond about 500 seconds).
- NIRSpec MOS + NIRCam parallels (data volume mostly driven by NIRCam).
The general solution is to use one of the group averaging patterns NRS or NRSIRS2. Both the traditional and IRS2 grouped NIRSpec patterns with four (NRS) and five (NRSIRS2) frames averaged per group have significantly smaller data rates and will be within limits.
See also: NIRISS Detector Overview
Since the Near-Infrared Imager and Slitless Spectrograph (NIRISS) has only one Hawaii 2RG detector, it is not expected to exceed data volume or data rate limitations when it is used as the “prime” instrument. The wide field slitless spectroscopy or imaging modes of NIRISS can be used in parallel with other instruments. In these cases, the full-frame readout format will generally be used with the NIS readout pattern to produce data at a rate of 0.195MBytes/s. In rare cases (e.g., with bright targets), the NISRAPID readout pattern could also be used, producing 0.782 Mbytes/s. These rates and the accumulated data volumes are typically small compared with the other instruments, but must still be considered.
This article uses the S.I. definitions of gigabyte and megabyte: 1 Gbyte = 109 bytes, and 1 Mbyte = 106 bytes.