JWST Pipeline Notebooks

Jupyter notebooks, a JSON-based interactive Python interface, is a powerful and convenient way to run the JWST pipeline that can be easily configured and shared. This article provides links to a selection of such notebooks that help demonstrate how to reduce JWST data using the JWST Science Calibration Pipeline.

On this page

Curated pipeline notebooks

The primary reference examples for how to run the JWST calibration pipeline for various different instruments and modes can be found on the jwst-pipeline-notebooks GitHub repository.  These notebooks are designed to be easily configurable and allow users to run the JWST pipeline on data from a wide variety of science programs with minimal need for editing while still exposing many of the optional parameters that they might be most interested in modifying for a given instrument mode. These notebooks are also lightweight, running the entire pipeline end-to-end without stepping through individual stages or producing large quantities of intermediate plots, and are thus intended more for day-to-day use than for teaching users about individual pieces of the pipeline.

By default, the pipeline notebooks run in "demo" mode, processing example data downloaded from the MAST archive.  If "demo" mode is disabled, they can instead process locally available data associated with any corresponding program.

Notebooks are actively curated and kept up-to-date with recent versions of the JWST pipeline software; while not always backwards-compatible, they generally work with more-recent versions of the pipeline than formally advertised.

Notebooks for individual observing modes will be added as they become available.  Current notebooks include

Observing modeJWST BuildPipeline versionNotebook linkFirst available
MIRI Imaging11.01.15.1JWPipeNB-MIRI-Imaging.ipynbNov. 2024
MIRI MRS11.01.15.1JWPipeNB-MIRI-MRS.ipynbJul. 2024
NIRCam Imaging11.01.15.1JWPipeNB-nircam-imaging.ipynbNov. 2024
NIRISS Imaging11.01.15.1JWPipeNB-niriss-imaging.ipynbSep. 2024



Pipeline caveat notebooks

As discussed in Known Issues with JWST Data, some of the existing JWST data issues can be worked around using dedicated offline notebooks while awaiting long-term pipeline fixes. The majority of these notebooks are located in the jwst-caveat-examples GitHub repository, although some are maintained in alternate locations.  The best way to discover these notebooks is through the tables of common issues and workarounds on the known issues articles for each JWST instrument mode.


JDAT and Jdaviz notebooks

Multiple notebooks have been made available for post-pipeline analysis and inspection of JWST data products. These notebooks do not focus on running the JWST pipeline, but rather on working with the data products that it produces. See JWST Post-Pipeline Data Analysis for further details.



Outdated notebooks

Many demonstration notebooks were produced early in the JWST mission (many in the context of JWebbinars) to introduce the community to the JWST data products, pipeline structure, and individual steps therein. While the notebooks below are no longer maintained and may no longer reflect the latest JWST pipeline environment, they can nonetheless be an excellent place to get started in learning about the pipeline and the data products that it produces.

Outdated data product notebooks

Notebook (in PDF and Github)ContentVideo demonstration (when available)

Using an uncalibrated (raw) JWST exposure:

Using JWST datamodels:

  • Explore stage 1 data products (detector corrections)
  • Examine stage 2 imaging and spectroscopic data products (calibrated individual exposures)
  • Take a closer look at WCS information for JWST data
  • JWST associations


Using the final data products from the pipeline:

  • Take a look at a JWST source catalog
  • Examine our final mosaicked image
  • Look at our final spectral data product



Outdated pipeline notebooks

Notebook in PDF and Github versionContent

Video demonstration

Working with the stage 1 calibration pipeline:

  • Look at the different ways to call the pipeline
  • Examine exactly what each pipeline step does to the science data
    • The Data Quality Initialization step
    • The Saturation Flagging step
    • The Superbias Subtraction step
    • The Reference Pixel Subtraction step
    • The Linearity Correction step
    • The Persistence Correction step
    • The Dark Current Subtraction step
    • The Cosmic Ray Flagging step
    • The Ramp_Fitting step

Working with the stage 2 calibration pipeline for Imaging:

  • Look at the different ways to call the pipeline
  • Examine exactly what each pipeline step does to the science data
    • The Assign WCS Information step
    • The Flat Fielding step
    • The Photometric Calibration step
    • The Resample step


Working with the stage 3 calibration pipeline for imaging:

  • Look at the different ways to call the pipeline
  • Examine exactly what each pipeline step does to the science data
    • The WCS refinement
    • The Sky Matching step
    • The Outlier Detection step
    • The Resample step
    • The Source Catalog step


w

Working with the stage 2 calibration pipeline for spectroscopy:

  • Look at the different ways to call the pipeline
  • Examine exactly what each pipeline step does to the science data
    • The Assign WCS step
    • The Background Subtraction step
    • The Imprint Subtraction step
    • The MSA Flagging step
    • The 2D Extraction step
    • The Source Type step
    • The Master Background step
    • The Wavelengh Correction step
    • The Flat Field step
    • The Stray Light Correction step
    • The Fringe Correction step
    • The Pathloss Correction step
    • The Barshadow Correction step
    • The Photometric Correction step
    • The Resampling spec step
    • The Cube Build step
    • The 1D Extraction step

Working with the stage 3 calibration pipeline for spectroscopy:

  • Look at the different ways to call the pipeline
  • Examine exactly what each pipeline step does to the science data
    • The Assign Moving Target WCS step
    • The Master Background Subtraction step
    • The Exposure to Source Conversion step
    • The MIRI MRS Sky Matching step
    • The Outlier Detection step
    • The Resampling step
    • The Cube Build step
    • The 1D Extraction step
    • The 1D Combination step




Notable updates

  •  
    Updated to point to new JWST pipeline notebooks repository

Originally published