NIRCam Data Rate and Data Volume Limits

JWST NIRCam's 10 detectors generate high data volumes that can be reduced if needed with the strategies given here.

On this page

JWST downlinks data in 4-hour contacts (twice per day and approximately 12 hours apart), during which it can transmit at least 28.6 Gbytes of recorded science data. The onboard Solid State Recorder (SSR) can hold at least 58.8 Gbytes of science data, which is written to the SSR at a maximum sustained data rate of about 48 Mbits per second. The JWST planning system verifies that the data rates within each visit are acceptable, and that the data volume accumulated between contacts does not exceed the downlink capacity or the SSR capacity. The Astronomer’s Proposal Tool (APT 25.4) will generate an error if the data rate or the total data volume of the observation exceeds the SSR's maximum capacities, and the user will be required to change the observing strategy to comply with the limits. However, APT does not generate an error if the total data volume exceeds the recommended downlink volume of 28.6 Gbytes. Because of this limitation, it is important to understand the data volume limits for NIRCam, as there are several observing options that can exceed the volume requirements depending on the readout pattern. Observations that exceed the downlink volume will be more difficult to schedule and should be justified within the proposals.



Parallel observations and data rate limits

Data rate limitations are generally only a concern for observations including parallels, due to the amount of data generated per second for the two instruments. When NIRCam and another instrument are used in parallel, the RAPID readout pattern is always disallowed, and the BRIGHT1 readout pattern is disallowed when both modules are used in “full-frame" mode (i.e., all 10 detectors). The BRIGHT2 readout pattern is disallowed when NIRSpec multi-object spectroscopy is done in parallel with NIRCam imaging. These configurations will automatically generate data rate errors in APT 25.4.



Data volume limits

NIRCam detectors are read out multiple times non-destructively, sampling the data while conserving the charge in each pixel to facilitate cosmic-ray rejection, reduce noise, and increase the dynamic range of the image. Each pixel containing 2 bytes of data is read out, in turn, in 10 µs, which could potentially generate 338 Gbytes of data in 12 hours  (not accounting for overheads) when using short- and long-wavelength detectors with 4 simultaneous outputs each. This exceeds the recommended downlink volume of 28.6 GBytes in a 12 hour period (and the total volume limit of the SSR).

Several NIRCam observing options can exceed these data volume limitations depending on the readout pattern adopted by the observer. Additionally, detector overheads (such as the initial reset cycle) become more important as the group frequency increases. The approximate maximum data volumes for each contact for NIRCam are listed for the 9 readout patterns in Table 1, assuming 12 hours of continuous data taking in both the short- and long-wave channels. The conversion factor used is 2 bytes per pixel; so for example, for RAPID and FULL frame mode: Data Volume = 10 (SCAs) × 2048 × 2048 (pixels) × 2 (bytes/pixel) × 43200 (sec/contact) / 10.7 (sec/group for RAPID) / 1e9 (bytes/Gbyte) ~ 338 GByte/contact. The following sections provide suggestions for reducing the data volume for an observation.

Limits for RAPID and BRIGHT2 Readout Patterns


Table 2. NIRCam readout patterns and maximum data volumes

Readout PatternTGROUP (sec)NGROUPSNFRAMES (per group)NSKIP (per group)

 Max Integration Time 

Data Vol. (Gbyte/contact)

DEEP8

212

1–20

8

12

4165.9

17

DEEP2

1–20

2

18

4101.4 / 21.5

MEDIUM8

106

1–10

8

2

1052.2

34

MEDIUM2

1–10

2

8

987.8

SHALLOW4

53

1–10

4

1

526.1

68

SHALLOW2

1–10

2

3

504.6

BRIGHT2 **

21.2

1–10

2

0

214.7

171

BRIGHT1

1–10

1

1

204.0

RAPID **

10.7

1–10

1

0

107.4

338

** APT (25.4) limits RAPID and BRIGHT2 readout patterns with both modules in “full-frame" mode (i.e., all 10 detectors) to a maximum number of 4 groups/integration. This limit increases to 10 groups when using a single module.



Reducing the Data Volume

Observers are encouraged to keep the data volume under 28.6 Gbytes in a 12 hour period in order to assure that their observations are easily schedulable.  APT 25.4 does not issue an error or a warning when the data volume exceeds this limit (28.6 Gbytes/12 hour period) so observers are encouraged to check the information provided by APT to try to keep the volume under that value. Generally, if the ratio (total data volume / total charged time) > 0.654 MB/s, the program might exceed the maximum downlink volume. The sections below outline a few suggestions to reduce data volume.

Select a different readout pattern

Using different readout patterns enables longer exposures with reduced data rates. Each readout pattern includes up to 20 groups, with each group yielding a single saved image obtained by averaging as many as 8 of the reads. Any remaining reads are then discarded. Choosing longer groups that average more frames reduces data volume (and yields a more precise average), but it also allows more time for a potential cosmic ray (CR) impact, in which case the entire group may need to be discarded. To reduce the data volume, the general guideline is to choose a readout pattern with the largest number of average frames/group, preferably while still allowing for a least 3 to 5 groups per integration.  

Decrease the number of groups or integrations

Adding a second or third integration to an observation effectively doubles or triples the total data volume. One possible solution to decrease the volume is to use the minimal number of groups and integrations required to achieve the desired signal-to-noise ratio. Becoming familiar with the readout patterns available and testing combinations in the ETC can help observers decide the best number of groups or integrations needed to achieve their science goals without exceeding data volume limits.

Use only one module

NIRCam consists of 2 modules with nearly identical optics and detectors. When both are used in tandem, they double the NIRCam field of view and, consequently, the data volume for an observation. It is possible to use only one module for an observation as a method to reduce data volume by selecting a single module in APT rather than ALL. The module options in APT will vary depending on the observing mode (e.g., module A for coronagraphy or module B for imaging).

Choose the right subarray

When using one module, observers may opt to either read out all five detectors completely (FULL) or the smaller detector subarrays. Using a different subarray may help reduce the overall data volume. Table 4 shows the approximate maximum data volumes per contact for the 10 imaging and grism subarrays, assuming 12 hours of continuous data taking in RAPID mode using both the short- and long-wave channels. Subarrays are read out more quickly than the full detector (each pixel is read out  in 10 µsec) allowing for shorter integration times and providing brighter saturation limits for a given number of reads. For most subarrays, pixels are read out through a single amplifier (Noutputs = 1), which helps mitigate data volume. For subarrays spanning the full width of the detector (2048 pixels), 4 parallel output channels can be utilized for faster read out times (Noutputs = 4). The smallest subarrays available for science (64 × 64 pixels) can be read out in 49.4 mas, which is the shortest possible integration time for an observation (though multiple groups of readouts are recommended: NGROUPS > 1).


Table 4. Maximum data volumes for NIRCam imaging and grism subarrays using the RAPID readout pattern

Subarray names

Subarray size

(Nrows x Ncolumns)

Number of Detectors

Max Integration Time

Data Vol. (Gbyte/contact)

SUB64P

64 × 64

1 SW + 1 LW

0.494

14
SUB160

160 × 160

4 SW + 1 LW

2.7692

40
SUB160P

160 × 160

1 SW + 1 LW2.769216
SUB320 

320 × 320

4 SW + 1 LW

10.6572

41
SUB400P

400 × 400

1 SW + 1 LW

16.5212 

17
SUB640

640 × 640

4 SW + 1 LW

41.7932

42
FULL

2048 × 2048

4 SW + 1 LW

107.4

168
SUBGRISM6464 × 20482 SW + 1 LW3.406 (4 outputs)99
SUBGRISM128128 × 20482 SW + 1 LW6.7596 (4 outputs)100
SUBGRISM256256 × 20482 SW + 1 LW13.4668 (4 outputs)101

Change the number of outputs

Another way to mitigate data volume in the case of grism time series observations is to change the number of amplifiers used to read the detector. Readout of the full NIRCam detector (2048 × 2048 pixels) is performed with 4 outputs simultaneously (Noutputs = 4), each delivering a stripe of data (2048 pixel rows × 512 pixel columns), and taking 10.7 sec altogether. Smaller subarrays are read out more quickly, and most are read out through a single output (Noutputs = 1).  Noutputs is pre-defined for most subarrays, but observers are given a choice between Noutputs = 1 or 4 in the grism time series observing mode. Choosing 1 output reduces the frame rate by a factor of four (for the 4 amplifiers).

Reduce the number of dithers

NIRCam has several dither options to improve sky coverage and image quality. While the options were designed with specific types of observations in mind (e.g., FULL primary dithers were designed for mosaics and small-grid dithers for coronagraphy), there is some flexibility allowed. If depth can be sacrificed for an observation, choosing a smaller number of primary dithers and small-grid or subpixel dither positions can significantly decrease the total data volume for an observation.



Note: This article uses the S.I. definitions of gigabyte and megabyte: 1 Gbyte = 109 bytes, and 1 Mbyte = 106 bytes.


References

Robberto, M., 2009, JWST-STScI-001721, SM-12
NIRCAM Optimal Readout Modes

Robberto, M., 2010, JWST-STScI-002100, SM-12
NIRCAM Optimal Readout II: General Case (Including Photon Noise)

JWST technical documents



Published

(enter date using //)

Latest updates
  • (enter date using //)
    Briefly describe change