JWST Data Rate and Data Volume Limits
JWST can store at least 58.8 Gbytes of science data. Downlinks for recorded science data occur in 4-hr contacts twice a day, with each contact transmitting at least 28.6 Gbytes. APT simplifies these limits to 58 Gbytes and 29 GBytes, respectively. Users must be mindful of these limits when designing their observations.
See also: Solid State Recorder
Data rate defines the speed with which science data can be written to the solid state recorder (SSR), which is ultimately regulated by the ISIM Command and Data Handling subsystem (ICDH). The Astronomer's Proposal Tool (APT) prevents observations from breaking the ICDH data rate limits.
Data volume defines the total amount of data (58.8 GBytes) that can be stored on the SSR at any given time. JWST downlinks science data to Earth in 2 contacts per day, where each contact can transmit at least 28.6 GBytes of recorded science data to the ground.
Managing data volume
APT places some limits on data volume, and users are encouraged to abide by the following limits for each visit:
- <58 GB (exceeding it generates an APT error)
- <29 GB (exceeding it generates an APT warning)
- <0.65 MB/s (no warning issued if rate is exceeded)
The first limit prevents the SSR from exceeding capacity. The second limit ensures the data volume can be downlinked in one contact. The third limit may be required for schedulability. Users are encouraged to check the ratio of Data Volume to Total Charged Time in APT for each observation and for the total program. If this ratio exceeds 0.65 MB/s for a total time of ~12 hours or more (thus exceeding 28 Gbytes), then the program will be difficult or impossible to schedule. APT does not issue a warning in this case.
Users should keep in mind that data volume and data rates issues can only be fully identified downstream; the visit scheduling subsystem and the visit planning subsystem are designed to take these issues into consideration. Accepted programs may have to be modified to comply with data volume and data rate limits. Proposers should understand the data rate of their program, and if necessary, take steps to reduce the data rate.
General advice for reducing data volume
Reduce the number of groups and integrations
See also: Understanding Exposure Times
Many of the observing modes that exceed the data volume limit are only problematic if executed for long periods of time (e.g., >12 hours). The simplest method for reducing your program's total data volume in a 12 hour period is to minimize the total number of groups per integration and number of integrations per exposure.
Select a different readout pattern, mode, and/or subarray
Using different readout patterns can enable longer exposures with a reduced amount of saved data. For example, choosing a larger numbers of frames averaged per group reduces data volume (and yields a more precise average). For MIRI, this can be achieved using either fast or slow modes.
Use overheads to your advantage
Factor in potential overheads that will decrease your observing efficiency, but at the same time potentially alleviate any data volume concerns.
There are known scenarios for each instrument that may exceed the data volume limit. Some of these scenarios and suggested solutions are described below.
See also: NIRCam Detector Readout Patterns
A raw 2048 × 2048 pixel detector frame is ~8 MB. Near-Infrared Camera (NIRCam) has 10 detectors, so this implies at least ~80 MB of data for each group within an exposure depending on the readout pattern. Neglecting all overheads and assuming 24 hours of continuous data taken with all 10 detectors (both short- and long-wave channels), these are the limiting cases for the 9 readout patterns:
Table 1. Data volume per day generated by NIRCam using all 10 detectors
|Readout pattern||Time between groups||Data Volume|
|DEEP2, DEEP8*||~200s||~34 GB/day||0.39 MB/s|
|MEDIUM2, MEDIUM8||~100s||~68 GB/day||0.79 MB/s|
|SHALLOW2, SHALLOW4||~50s||~136 GB/day||1.6 MB/s|
|BRIGHT1, BRIGHT2||~20s||~340 GB/day||3.9 MB/s|
|RAPID||~10s||~680 GB/day||7.8 MB/s|
* Bold italics style indicates words that are also parameters or buttons in software tools (like the APT and ETC). Similarly, a bold style represents menu items and panels.
Note that for long programs of ~12 hours or more, only the DEEP readout patterns abide by the 0.65 MB/s limit.
In addition to the steps outlined above, NIRCam observers may want to also consider the following options for reducing their data volume.
Use only one NIRCam module: It is possible to use only one module for an observation as a method to reduce data volume by selecting a single module in APT rather than ALL. The module options in APT will vary depending on the observing mode (e.g., module A for coronagraphy or module B for imaging).
Change the number of outputs: In the case of grism time-series observations, a proposer may want to change the number of output amplifiers. Readout of the full NIRCam detector (2048 × 2048 pixels) is performed with 4 outputs simultaneously (Noutputs = 4), each delivering a stripe of data (2048 pixel rows × 512 pixel columns), and taking 10.7 s altogether. Smaller subarrays are read out more quickly, and most are read out through a single output (Noutputs = 1). Noutputs is pre-defined for most subarrays, but observers are given a choice between Noutputs = 1 or 4 in the grism time-series observing mode. Choosing one output reduces the frame rate by a factor of 4.
The Mid-Infrared Instrument (MIRI) has only 3 detectors, but obtaining simultaneous imaging with both the imager and spectrograph can potentially exceed the data volume limit. MIRI observers should consider the general advice described above if their planned observations exceed the data volume limit.
The Near Infrared Spectrograph (NIRSpec) has 2 detectors, which have 4 readout patterns spread out split over 2 readout modes. The IRS2 NRSIRS2RAPID readout mode (~1.95 MBytes/s) results in higher data volume than the traditional NRSRAPID readout mode (~1.56 MBytes/s) because interspersed reference pixels and outputs are also saved. We recommend the use of these RAPID readout patterns (number of frames per group equal to one) for cases that have no data volume issues reported in APT because of the improved performance of cosmic ray rejection. Despite the higher data volumes, the IRS2 patterns are also to be generally preferred over the normal modes due to the reduction in correlated read noise.
NIRSpec observing options that could result in APT data volume errors include:
- Deep, full frame FS/IFU/MOS long exposures with NRSRAPID or NRSIRS2RAPID readout (with exposure times beyond about 500 s).
- NIRSpec MOS + NIRCam parallels (data volume mostly driven by NIRCam).
The general solution is to use one of the group averaging patterns NRS or NRSIRS2. Both the traditional and IRS2 grouped NIRSpec patterns with 4 (NRS) and 5 (NRSIRS2) frames averaged per group have significantly smaller data rates and will be within limits.
See also: NIRISS Detector Overview
Since the Near Infrared Imager and Slitless Spectrograph (NIRISS) has only one Hawaii 2RG detector, it is not expected to exceed data volume or data rate limitations when it is used as the “prime” instrument. The wide field slitless spectroscopy or imaging modes of NIRISS can be used in parallel with other instruments. In these cases, the full frame readout format will generally be used with the NIS readout pattern to produce data at a rate of 0.195 MBytes/s. In rare cases (e.g., with bright targets), the NISRAPID readout pattern could also be used, producing 0.782 Mbytes/s. These rates and the accumulated data volumes are typically small compared with the other instruments, but must still be considered.
This article uses the S.I. definitions of gigabyte and megabyte: 1 Gbyte = 109 bytes, and 1 Mbyte = 106 bytes.