Starting with build 11.1, the JWST Calibration pipeline contains the clean_flicker_noise step, which is intended to mitigate 1/f noise. The step runs on each group of a given integration, immediately prior to the ramp-fitting step.

This article shows several examples of clean_flicker_noise results on NIRCam data. As the figures below show, the quality of the 1/f mitigation is highly dependent on the scene and the parameter values used in the call to the step. The examples below show clean_flicker_noise results in several typical scenes, including sparse and crowded fields, as well as a field with a large, extended source. 

JWST SIDECAR ASICs (detector readout electronics) generate significant 1/f noise during detector operations and signal digitization. In NIRCam data, this 1/f noise appears as horizontal banding that spans the entire width of the image, and varies from row to row across all NIRCam exposures. If not handled properly, it can impact the data in many ways, such as obscuring  faint targets, artificially inflating the statistical significance of a detection in imaging data, or introducing systematic errors in time-series observations. More discussion of 1/f noise is provided in NIRCam Detector Performance and NIRCam Known Issues. 

The reference pixel correction step of the stage 1 pipeline can partly mitigate 1/f noise. But, with the limited number of reference pixels available in NIRCam observations (four pixels on each edge of the full array images), the 1/f noise cannot be completely removed. 1/f correction is expected to be even less effective in subarray observations, which typically contain no or few reference pixels.

Example cases

The examples below show the rate files for these particular exposures both with and without running the clean_flicker_noise step, and using different options for the background_method and fit_by_channel parameters (default values were used for all other parameters). Results for each example case are shown in a table. Each column of the table blinks two rate file images: the first from a case where clean_flicker_noise was not run, and the second where clean_flicker_noise was run, while using the background method described in the column header. The bottom row shows a difference image between the two rate files. For a given example, all rate images are shown on the same image scale for easier comparison, as are the difference images; however, the rate images and difference images are scaled differently from each other due to the large difference between source brightness and the level of the 1/f noise.

Imaging mode

Table 1. A summary of the clean_flicker_noise results for five files shown in the examples below

Program 1345 - Sparse field

In this example, the image is a sparse field of mostly background galaxies. Here, the four cases of 1/f subtraction have fairly similar results. The most notable difference is in the channel-based median background case. Here the slightly elevated background level in the top half of the center left channel results in elevated median values and a slight over-estimation of the 1/f noise in this area, as seen in the difference image. The other three difference images are similar to one another. In cases like this, with lots of background-only pixels, the channel-based corrections have the best chance of working well. Even with the area of elevated background, the channel-based model background case returns a good estimate of the 1/f noise.

Program 2221 - Sparse Field

This is another example of a relatively sparse field. In this case the image is from a narrowband filter, and the rate file without a 1/f correction shows significant 1/f noise, along with channel to channel variations in the background level. The full-image correction scheme that uses the median background largely removes the 1/f noise and reduces the channel-to-channel differences in background level. The full-image model based background correction successfully removes high frequency (thin bands) 1/f noise, but leaves behind lower-frequency 1/f noise. The channel-based median background correction does the best job, removing virtually all of the 1/f noise as well as correcting the channel-to-channel background differences. Results from the channel-based model background correction are very similar to the full-image model background correction. This case, in combination with the program 1345 case above, shows that even in the relatively straightforward case of a sparse field, the best parameters to use in the clean_flicker_noise step will not always be the same.

Program 1334 - Crowded Field

This example shows the clean_flicker_noise results when working on a crowded field. In this case, the limited number of background-only pixels in some rows can result in poor background estimation and therefore over-estimation of the 1/f noise. This is most obvious in the channel-based median case. The source density increases from left to right across the image, resulting in the larger over-estimations of the 1/f noise from left to right. The channel-based model background case does a slightly better job, but the calculated 1/f is still higher on the right compared to the left side. The results from the full image cases are better. There appears to be a slight under-estimation of the 1/f noise along the bottom of the detector in the full-image median background case (left-most column). As with the extended source case below, data like these, with a high fraction of source pixels (and therefore limited background pixels) will most likely benefit the most from one of the full-image correction methods.

Program 2731 - Extended emission

In this example, clean_flicker_noise results are shown for a case where the image contains extended emission across most of the detector. The channel-based corrections in the two right-most columns clearly do not work well for this science scene. In many rows, the large scale features cover all or almost all of the pixels in individual channels, resulting in inaccurate calculated background levels. The full detector-width corrections appear to do a better job. However, when using the median value for the background (left-most column), there are still areas where there are few, if any, true background pixels. This results in an over-estimation of 1/f noise, as can be seen in the bright central band across the difference image. The full detector model background case seems to do the best job subtracting the 1/f noise, with no obvious signs of the real signal in the difference image.

Program 1783 - Check skymatch results

This example examines any effects the clean_flicker_noise correction has on later pipeline stages. Specifically, you should check to see if the corrected files make sky matching more difficult when combining multiple images into a final mosaic image. The scene in this case is an extended nearby galaxy that spans several mosaic tiles, along with surrounding background areas. When correcting with the full image median background, some of the tile-to-tile differences in background level are reduced compared to the case where no correction is made. However, some of the larger tile-to-tile differences remain, as are apparent along the right side and lower left corner of the mosaic. Similar to what was seen in the program 2731 example above, the 1/f noise is over-estimated and therefore over-subtracted in the center of the mosaic, in the tiles where the galaxy spans all or most columns. This is shown in the elevated signal in the center of the difference image in the bottom row.

The correction is more subtle for the case of the full image model based background. In this case, tile-to-tile background differences are not corrected to the extent that they are in the median background case. However, with the model background, the over-estimation of the 1/f noise in the central portion of the mosaic is much lower.

Wide field slitless spectroscopy mode

The examples below show the results of using clean_flicker_noise on WFSS images. While there may be images where some form of the correction improves the data, in general clean_flicker_noise does not work well, especially for data using the GRISMR element, which orients the spectral traces along rows. With source signal often stretching across a significant fraction of the pixels in a given row, the correction over-estimates the 1/f signal, and over-subtracts. This results in dark streaks across the image, and a loss of signal from the spectral traces. In GRISMC data the traces are oriented along columns. In cases where the scene is not too crowded, clean_flicker_noise may be able to give a good result, although careful checking is necessary.

Table 2. A summary of the clean_flicker_noise results for four files shown in the examples below

Program 2883 - GRISMC

In this example, clean_flicker_noise was run on two GRISMC images. The first used the F360M filter, and the second used the F480M filter. When the GRISMC is inserted into the beam, scenes vary with the crossing filter. Using some filters, such as F360M, the coronagraphic optics are shifted and dispersed into the field of view. 

In the F360M data, when using the median background, the high frequency 1/f features are effectively removed. However, the coronagraphic neutral density squares, which are visible along the top of the image, result in incorrect row-based median values. This causes an incorrect 1/f estimate and subtraction, as can be seen by the brighter background in the regions between the neutral density areas compared to the areas below. This effect is not present when using the model based background. The 1/f correction in that case, seen in the right-most column below, appears better. Note that in the clean_flicker_noise step, there is an option for the user to provide a mask file. Pixels flagged in this file will be ignored when calculating the background and 1/f signal. By providing a mask file that forces clean_flicker_noise to ignore the dispersed neutral density areas, the results may be improved.

In the F480M data, the lower ~1/3 of the image is not usable due to the dispersion geometry. Focusing on the top ~2/3 of the image, the median background case appears to slightly over-estimate the 1/f signal in the center of the image, due to the large number of sources in those rows. The model based background case appears to give better results, without over-subtracting signal.

Program 4092 - GRISMR

The next two examples show the results of clean_flicker_noise processing on GRISMR images. As with GRISMC images, the amount of usable area on the detector varies with crossing filter when using GRISMR. This will again affect the calculations of median and model-based backgrounds. In the table below, which contains an image from the module A detector, both median and model background corrections perform similarly. In rows where spectral traces exist, the 1/f signal is often overestimated. After the subsequent over-subtraction, these rows can show up as darker lines across the brighter background. The same effect can happen when using channel-based corrections, which are not shown here. STScI recommends not using the clean_flicker_noise step in its current form for GRISMR data.

The table below shows the clean_flicker_noise results when working on module B with F335M data. The results  are similar to the module A case above. This shows again why STScI recommends not using the clean_flicker_noise step in its current form for GRISMR data.

Coronagraphic mode

In this example, clean_flicker_noise was run on a coronagraphic exposure (jw04050030001_03106_00001_nrcalong_rate.fits). The results are similar to the GRISMC case above, where the presence of the neutral density squares causes errors in the background estimates. This is shown in both difference images in the table below, where there is an under-estimation of the 1/f signal in the bottom ~1/3 of the field of view. Additional tests, not shown on this page, have revealed that the wings of bright sources also may not be masked when estimating the background. This can lead to over-estimations of the background. As in the GRISMC case, a user-provided mask file that flags the neutral density areas and source pixels may improve this 1/f correction. STScI recommends not using the clean_flicker_noise step in its current form on coronagraphic data without careful examination of the masked pixels, and only in the case where there are no objects of interest between the neutral density filters.

Future work

Additional work is required to test clean_flicker_noise against more datasets and perform quantitative analysis before considering turning the step on in the JWST science calibration pipeline. Additional methods and algorithms are being investigated for some modes, such as time series observations and WFSS.

Other community tools have been developed to help mitigate 1/f noise in JWST data, and may be helpful for some of the science scenes where clean_flicker_noise is not adequate. The performance of some of these tools for NIRCam data is documented in the Alternative 1/f Noise Software Packages article. Note that the background fitting and median cleaning algorithm used in clean_flicker_noise are based on the image1overf tool, detailed on the community tools page.