Primer and Tutorials
This article presents a high-level summary of how users can run the various JWST calibration pipeline modules, giving examples of several types of common operations.
The JWST calibration pipeline modules and individual calibration steps can be executed as command-line tasks or from within Python (either a simple python shell or something like a Jupyter notebook). There are 3 main stages of pipelines required to completely process a set of exposures for a given observation.
- Stage 1: Apply detector-level corrections to the raw data for individual exposures and produce count rate (slope) images from the "ramps" of non-destructive readouts
- Stage 2: Apply physical corrections (e.g. slit loss) and calibrations (e.g. absolute fluxes and wavelengths) to individual exposures
- Stage 3: Combine the fully calibrated data from multiple exposures
Some of these stages use different pipeline modules depending on the observation mode, as shown in Table 1. The sections below describe and provide examples for running some of these pipeline stages.
Table 1. Pipeline modules used in stages 1, 2, and 3
(imaging and spectroscopy)
|3||Aperture Masking Interferometry||calwebb_ami3|
calwebb_detector1 pipeline applies detector-level corrections to individual exposures, such as dark subtraction and linearity correction, and produces corrected countrate images as its final product. Complete details of the
calwebb_detector1 pipeline are available in the pipeline reference documents. Briefly, the
calwebb_detector1 pipeline takes raw data in the form of an
_uncal.fits product as its input, which contains the original 4D data ramps of an exposure, applies a large list of detector-level corrections to the raw ramps, and then finally computes slopes from the ramps of each pixel to produce countrate images. The final output is a
_rate.fits product. A
_rateints.fits product is also created if the exposure contained multiple integrations. The
calwebb_detector1 pipeline is designed to be applied to raw data from all instruments and all observing modes (imaging, spectroscopy, coronagraphy, etc.).
The pipeline can be run on an individual exposure from the command line by using the
strun task like this:
$ strun calwebb_detector1.cfg jw96090001003_03101_00001_nrca3_uncal.fits
where the first argument is the name of the pipeline configuration file and the second is the name of the input data file.
You can also run the pipeline from within Python by loading and executing the Detector1Pipeline class from the jwst.pipeline package:
> from jwst.pipeline import Detector1Pipeline
> result = Detector1Pipeline.call('jw96090001003_03101_00001_nrca3_uncal.fits', config_file='calwebb_detector1.cfg')
In this case the countrate image product ("result") is passed back to the Python shell, so that you can do anything you want with it at that point, including saving it to a file or passing it along as input to subsequent processing. One convenience of processing from within Python is the ability to apply processing to many exposures at once, such as:
> from glob import glob
> raw_files = glob('jw96090*_uncal.fits')
> for file in raw_files:
... result = Detector1Pipeline.call(file, config_file='calwebb_detector1.cfg')
... result.save(file[:file.rfind('uncal')] + 'rate.fits')
CALWEBB_IMAGE2 and CALWEBB_SPEC2
The second stage of calibration processing is applied using the calwebb_image2 or calwebb_spec2 pipelines. calwebb_image2 is designed for use with data from any imaging-like observation mode, such as direct imaging, coronagraphy, Aperture Masking Interferometry (AMI), and all time-series variants of imaging. calwebb_spec2 is to be applied to all spectroscopic modes, including MIRI LRS and MRS, NIRCam and NIRISS Wide-Field Slitless Spectroscopy (WFSS), NIRISS Single Object Slitless Spectroscopy (SOSS), and NIRSpec fixed-slit, MOS, and IFU modes. These two pipeline modules use the _rate.fits or _rateints.fits product from calwebb_detector1 as input, and apply various calibrations to the countrate images, including populating all World Coordinate System (WCS) information, flat-fielding, and attaching or applying flux calibration information. The stage 2 pipelines produce calibrated _cal.fits or _calints.fits products as their output. Complete details on the steps in these pipelines are available in the pipeline reference documents.
These pipelines can be run on individual countrate exposure products (e.g. _rate.fits files) or on an Association of _rate.fits (or rateints.fits) products. An association of countrate products will be used as input, for example, when dedicated off-target background exposures have been obtained as part of the observation and the background exposures are to be subtracted during calwebb_image2 or calwebb_spec2 processing. An ASN file is necessary as input in this case in order to specify the list of multiple exposures (both target and background) and also identify which exposures are to be used as background.
Running calwebb_image2 or calwebb_spec2 on a single exposure can be done from the command-line like this:
$ strun calwebb_image2.cfg jw96090001003_03101_00001_nrca3_rate.fits
or from within python like this:
> from jwst.pipeline import Spec2Pipeline
> result = Spec2Pipeline.call('jw96090001003_03101_00001_nrca3_rate.fits', config_file='calwebb_spec2.cfg')
Processing an entire association of exposures through these pipelines, using an association (ASN) file as input, can be done like this:
$ strun calwebb_image2.cfg jw96090-a3001_20170327t121212_image2_001_asn.json
> Spec2Pipeline.call('jw96090-a3001_20170327t121212_image2_001_asn.json', config_file='calwebb_spec2.cfg')
The ASN file lists one or more output products, each of which is defined to contain one input product (member), hence each input exposure will generate an individual output (calibrated) product. More details on Associations are available in the calibration pipeline package documents.
Stage 3 pipelines
As shown above, there are 5 different stage 3 pipelines, tailored for different observing modes. All stage 3 pipelines must be executed using an Association (ASN) file as input, listing the calibrated exposures to be combined into final products. Details on each of these pipelines are available in the pipeline section of the calibration pipeline package documentation. To run the calwebb_image3 pipeline, for example, you can do:
$ strun calwebb_image3.cfg jw96090-a3001_20170327t121212_image3_001_asn.json
or from within python:
> from jwst.pipeline import Image3Pipeline
> Image3Pipeline.call('jw96090-a3001_20170327t121212_image2_001_asn.json', config_file='calwebb_image3.cfg')