Data artifacts due to 1/f noise can produce striping in the near-IR detector images.  This article describes the methods available to correct for this striping in the JWST calibration pipeline.

The reference voltage for the readout amplifiers on the near-IR cameras experience unstable fluctuations each time the detector is read out; this can imprint additive offsets into the recorded pixel values. These fluctuations are temporally well described by a 1/f power spectrum, and manifest as striping in the fast-read direction of the detectors (Figure 1). These effects are most significant for ramps with very few readout groups as the coherent noise patterns in each group are obvious. Such patterns can be less obvious for exposures with large numbers of readout groups, as the ramp fitting process can average over the fluctuations and manifest as higher stochastic noise rather than a systematic noise component.

Some general-purpose corrections may, however, be possible in the future through the use of an improved reference pixel subtraction currently being tested for upcoming builds 11.2 and 11.3.

Pipeline implementation

Usage

The clean_flicker_noise step is an optional (turned off by default) step in the calwebb_detector1 pipeline that attempts to measure and remove 1/f noise immediately prior to the ramp fitting step for an individual exposure (i.e., after nonlinearity correction, cosmic rays flagging, etc.). It can be enabled in the calwebb_detector1 pipeline by setting:

steps.clean_flicker_noise.skip=False

Note that the syntax for enabling this step will change slightly depending on whether the pipeline is being run from the command line, or in a notebook via the .call() or .run() methods (see Running the JWST Science Calibration Pipeline).

Algorithms

There are two available algorithms, controlled by the fit_method parameter.

1. The first (and currently default) algorithm is a simple column-based median approach based on the "image1overf" algorithm, developed by Chris Willott, and available on GitHub at chriswillott/jwst. This algorithm masks source pixels, measures the median remaining pixel value in a given column (or row, depending on the fast readout direction), and subtracts that value from the data. Assorted options are available for masking sources and measuring the underlying background signal so that the background is not subtracted off along with the 1/f noise.
2. The second algorithm is the NSClean method (Rauscher 2024). In brief, this uses a fourier-based approach to measure the 1/f signal using unmasked pixels.

Key Parameters

Detailed documentation for the clean_flicker_noise step and its available parameters can be found on ReadTheDocs. Some basic guidance to the most important available parameters, along with notes on which may be of most use to different science cases, is provided below.

fit_by_channel: True or False (default)

• 1/f noise can vary between readout amplifiers, and thus in ideal situations, the most accurate correction should be made for each individual amplifier (fit_by_channel = True).
• However, in many cases, the differences between amplifiers can be small and using individual amplifiers to derive the correction can lead to larger systematic errors in cases where there is significant astrophysical structure in the scene that must be masked.
• Typically this is left as False by default, but might be useful to enable if significant differences in the striping pattern between individual amplifiers are observed in the data.

background_method: 'median' (default), 'model', or None

• A uniform astrophysical background is entirely degenerate with 1/f noise, but a 1/f correction should ideally not subtract this component. In order to do so, the clean_flicker_noise routine attempts to subtract off only variations about the typical background level rather than the raw signal in a given row/column.
• The 'median' method assumes that there is a uniform background across the field, estimated as the median across all unmasked pixels in a given group. All subtracted 1/f noise is determined relative to this median. This method generally works best in low background cases and relatively empty scenes.
• The 'model' method fits the background across the detector with a low-order model to try and preserve intrinsic background variations within the field. This method generally works best in crowded scenes with large bright objects; the smoothing box size for the background model can be further controlled with the background_box_size parameter.
• The None method assumes that there is no background across the field and determines 1/f noise directly from the unmasked data on the detector. This method is only appropriate for data with sufficiently large unilluminated regions on the detector, for example NIRSpec data.

mask_science_regions: True or False

• NIRISS, NIRCam, and NIRSpec are each different in terms of how the detectors are illuminated.  For NIRCam and NIRISS, all detector pixels are illuminated by the sky, while for NIRSpec many pixels are intrinsically dark and unilluminated by the science scene.
• NIRISS and NIRCam should thus always use mask_science_regions = False, as otherwise there would be no pixels left on the array with which to measure 1/f.
• For NIRSpec data however, setting mask_science_regions = True can be helpful for measuring 1/f noise as it will mask out all pixels that might have been illuminated by the sky and use only "dark" pixels to determine the 1/f noise model. This can be helpful in very crowded scenes for which it is difficult to disentangle 1/f from science signal, but can result in poorer performance in faint scenes due to the low number of unilluminated pixels that only sparsely sample the detector.

user_mask: String

• The 1/f correction routine relies upon a mask to determine source pixels to ignore when measuring the 1/f signal. By default, this is done automatically using either sigma-clipping or a histogram-based approach applied to the scene.
• Users can also provide a custom mask to use instead, by providing a path to an image file with the same dimensionality as the exposure being processed.

Examples

Some examples of the pipeline clean_flicker_noise step applied to JWST science data are shown in Figure 2. These examples include:

• 4-group NIRCam RAPID dark frame. This was processed using mostly default parameters (fit_method = 'median', background_method = 'median', mask_science_regions = False) and fit_by_channel = True.
• 5-group NIRCam MEDIUM8 sparse field. This was processed using entirely default parameters.
• 5-group NIRCam BRIGHT1 crowded star field. This was processed using mostly default parameters, except for background_method = 'model'.

Specific guidance and further examples for individual JWST NIR instruments can be found in the sections below.

Advice for NIRCam Observations

Example 1/f correction cases and general advice for working with NIRCam data are provided in the NIRCam 1/f Noise Removal Methods page.

Advice for NIRSpec Observations

Investigating the optimal parameters for 1/f noise cleaning is still in progress. However, the settings below provide initial starting points for cleaning 1/f noise in NIRSpec data. The pipeline defaults are recommended for any parameters not explicitly listed here. 

1/f noise fitting method 

The pipeline offers two fitting methods for addressing 1/f noise in NIRSpec data:  

• 'fit_method' = 'fft': The Fast Fourier Transform ('fft') method is generally recommended, as it often provides a better fit to the 1/f noise patterns.
• 'fit_method' = 'median':  
  • The median method can be beneficial in specific scenarios:
    • MOS Observations: 
      For closely packed targets illuminating a significant portion of the NIRSpec detector, the 'median' method reduces the risk of overfitting the data.

    • Subarray BOTS Data: 

      If processing runtimes are prohibitively long with the 'fft' method, the 'median' can be a sufficient alternative.  

Masking options

• 'n_sigma' = 2: This setting typically produces effective masks, capturing source flux and any remaining cosmic rays without masking the peaks of the 1/f pattern. The optimal value of 'n_sigma' may depend on the data and source fluxes.
• 'mask_science_regions' (Mode dependent): This parameter masks any area on the detector that falls within the WCS-defined regions for the observed science case (e.g., open MOS slitlets, IFU slices, or the slits for the FS and BOTS modes) as illuminated in the mask. However, the WCS-defined regions often include a buffer near the edges of the open apertures, which can lead to over-masking the available detector space, and leaving less area to measure the 1/f pattern.
  • Recommendations by mode:
    • MOS observations with densely packed sources: Set 'mask_science_regions' = False to avoid over-masking and leave more detector area available for 1/f fitting.
    • FS/BOTS subarray data: Set 'mask_science_regions' = False to prevent masking the entire subarray detector area.
    • IFU data or sparsely populated MOS observations: Setting 'mask_science_regions' = True typically returns reasonable masks. However, turning it off may improve results in some cases. 

 Background Treatment

• 'background_method' = None: The NIRSpec detectors are not fully illuminated and contain intrinsically dark regions. This setting ensures that no unnecessary background corrections get applied to these naturally dark areas. 

Cleaning 1/f noise: clean_flicker_noise in 'calwebb_detector1' vs. nsclean in 'calwebb_spec2' 

For NIRSpec data, users can perform 1/f noise-cleaning at two stages in the JWST pipeline:

• clean_flicker_noise in 'calwebb_detector1': Cleans 1/f noise group-by-group. This method is typically recommended for cleaning 1/f noise because the 1/f noise pattern changes from group to group. However, runtimes will be longer than cleaning at the rate file stage using 'nsclean' in 'calwebb_spec2'.
• nsclean in 'calwebb_spec2': Cleans 1/f noise by fitting the pattern in the rate files.
• Both steps use the same algorithms and access the same parameters. However, the default parameters for the two steps are currently different. 

Advice for NIRISS Observations

Example 1/f correction cases and general advice for working with NIRISS data are provided in the NIRISS 1/f Noise Removal article.

References

Rauscher, B. J., 2024, PASP, 136, 015001
NSClean: An Algorithm for Removing Correlated Noise from JWST NIRSpec Images