The algorithms for CALWEBB_DETECTOR1, the 1st stage of the JWST calibration pipeline, are described. These algorithms process the data from raw ramps to uncalibrated slopes.
The CALWEBB_DETECTOR1 module is the 1st stage in the calibration pipeline and processes data taken with all instruments and modes. The input to this stage is the raw non-destructively read ramps and the output is uncalibrated slope images. A number of instrument signatures are accounted for in this stage and slopes are fit to the corrected ramps. The steps are listed in Figure 1 with the flow from the top to the bottom.
Unless otherwise stated, the algorithms described are the baseline version.
NIR/MIR common steps
Data quality initialization
The data quality flags on the pixels are initialized. The pixel data quality flags are initialized based on detector specific reference files and designate permanent conditions that affect all groups for that pixel. Examples of such flags are dead pixels, hot pixels (showing excess dark signal), and low quantum efficiency pixels. The group data quality flags are initialized to zero and designated conditions that only affect some groups for a pixel. Both the pixel and group flags are updated during subsequent pipeline steps as needed.
The analog-to-digital (ADU) values for each group are checked for saturation. The saturation level is set by detector specific reference files. The group data quality flag is set if the DN value in the group is above the reference file set saturation level.
The baseline version of this algorithm does not account for the case where not all of the frames in a group are saturated (some are and some are not). An optimal version of this algorithm that does account for this case is currently under investigation.
The uncertainties are propagated in the pipeline using a noise model. The uncertainty from each step that contributes noise to the final measurement is separately propagated through the pipeline. Different uncertainty sources behave in different ways. Some noise sources (e.g., photon noise) are independent between integrations and others (e.g., flat field noise) are not. In addition, the spatial covariance of different sources varies. By propagating each term through the pipeline, the use of each term can be customized for the processing. For example, the use of the flat field noise term is different between non-dithered and dithered observations. For the former, the noise does not reduce with the addition of more integrations while for the latter it does.
Reference pixel correction
All detectors have reference pixels that have the same readout electronics as regular pixels, but are not sensitive to light. These pixels are located at the edge of the detector arrays and are read out by the same amplifiers as the regular pixels. Thus, reference pixels track drifts in the readout electronics. The average reference pixel level for each readout amplifier is subtracted from the regular pixels for that same amplifier.
All detectors suffer from persistence giving rise to faint "afterimages" of previous exposures that are seen in the current exposure. The persistence decays exponentially. The persistence is corrected using a trap based model of the persistence with the model details given by detector specific calibration reference files. In addition to correcting the persistence, a pixel flag is set indicating that persistence was corrected at a detectable level.
All the detectors show ramp jumps where the ADU level between two consecutive groups is large relative to those between other consecutive pairs of groups. These ramp jumps are most often caused by cosmic rays that instantaneously deposit large amounts of charge in a pixel. These ramp jumps can be detected and flagged. The number of sigmas above the noise threshold is given as a parameter.
For the baseline algorithm, the two-point difference method is used as it is computationally fast and sufficient for measurements in the photon dominated regime (see Anderson & Gordon 2011). An optimal version of this algorithm that detects smaller ramp jumps in the read noise regime is under investigation.
The slope for each ramp is determined by performing a weighted linear least squares fit. If a ramp has flagged ramp jumps, the fit is done for each jump free segment and the resulting slopes averaged to produce a single slope per ramp. The weighting is done using the Fixsen et al. (2000) "optimal weighting" method.
An optimal version of this algorithm that uses generalized least squares and a covariance matrix is under investigation.
Bad ramp flagging
There will be ramps that are significantly nonlinear even after all the corrections in CALDETECTOR1 have been done. Ramps that are significantly nonlinear are flagged. The threshold above which to flag a ramp as nonlinear is given in a calibration reference file.
NIR specific steps
Group scale correction
In rare cases, groups can be taken where the the number of frames in a group is not a power of 2. This results in the on-board software not dividing by the appropriate number of frames as the on-board software does averaging by simple bit-shifting. The information on the number of actual frames in a a group and the on-board assumed number of frames in the group is known. This step multiplies all the raw DN values by the appropriate ratio to provide the correct DN values for the actual number of frames averaged per group.
The overall DN level in each pixel and group is offset by a bias level. The nonlinearity in a ramp is relative to this bias level. Thus, the bias level in each group for each pixel is subtracted based on detector specific calibration reference files to provide the correct DN level for the nonlinearity correction step.
The NIR detectors show a nonlinearity that is due to the gain changing. This non-linearity is well fit with a low order polynomial fit versus relative DN where relative is referenced to the bias level. The correction is done using the fitted polynomials whose parameters are provided by detector specific reference files.
For the baseline algorithm, grouped data is treated the same as non-grouped data and the error due to this assumption is small. An optimal version of this algorithm that accounts for the effects of grouping on the nonlinearity correction is under investigation.
The NIR detectors show excess signal in dark exposures. This excess signal is subtracted group-by-group using detector specific reference files.
MIR specific steps
First frame correction
The 1st frame of MIR data has a transient that has not been fully characterized.
For the baseline algorithm, this 1st frame is flagged and not used further in the pipeline. An optimal version of this algorithm that corrects for this transient is under investigation.
Last frame correction
The MIR detectors show a transient in the last frame that is caused by the reset pattern. The last frame of the MIR detectors is a read-reset frame where two rows are read and then reset. Then the next two rows are processed in the same manner. The transient is caused by reset strongly changing the values in the to-be-read pixels due to the coupling between pixels.
For the baseline algorithm, the last frame is flagged and not used further in the pipeline.
An optimal version of this algorithm is under development. The amplitude of the transient seems to be well characterized by a polynomial fit to the value in the last frame itself. The correction under development will use polynomial parameters provided in detector specific reference files.
The MIR detectors show a non-linearity that is due to the quantum efficiency changing. This non-linearity is well fit with a low order polynomial fit versus absolute DN. The non-linearity is wavelength dependent at wavelengths above approximately 21 microns. The correction is done using the fitted polynomials whose parameters are provided by detector and wavelength specific reference files.
The Reset Switch Charge Decay is a transient seen in the 2nd and subsequent integrations in a MIR exposure. If uncorrected, this transient results in the slopes in the 2nd+ integrations to be larger than the 1st integration. This transient is well described by a decaying single exponential that is proportional to the counts at the end of the previous integration. The correction involves subtracting this exponential from the 2nd+ integration ramps where the parameters of the exponential are set by detector specific reference files.
The MIR darks show a dependence on the integration number in an exposure. The dark subtraction is done group-by-group using detector and integration specific reference files.
Anderson & Gordon 2011, PASP, 123, 1237
Optimal Cosmic-Ray Detection for Nondestructive Read Ramps
Fixsen et al. 2000, PASP, 112, 1350
Cosmic-Ray Rejection and Readout Efficiency for Large-Area Arrays
Published July 11, 2017