JWST Post-Pipeline Data Analysis

The JWST post-pipeline data analysis ecosystem consists of several components (i.e., analysis software, visualization tools, and Jupyter notebooks) that can be integrated for seamless viewing and analyzing of JWST data.  

On this page

JWST post-pipeline data analysis tools are individual packages included within AstroConda to assist observers in viewing and analyzing their JWST data. The tools are generally written in Python and work with Astropy. Development is ongoing1

The suite of post-pipeline data analysis tools is intended to help astronomers with the often iterative and interactive workflow involved in converting these calibration pipeline data products into meaningful scientific results. This involves tasks such as:

  • inspecting data and data quality information;
  • masking or flagging data and using those annotations to guide later steps in the analysis;
  • using the results of interactive analysis to guide a custom run of the pipeline (e.g., tweaking spectral extraction parameters or background estimates);
  • performing optimized spectral extraction techniques; 
  • combining data sets in various ways, with careful attention to astrometry, PSF matching, and other issues;
  • source detection and photometry using different choices or algorithms than those used in the pipeline;
  • measuring lines and continuum in spectral data;
  • fitting models to data or otherwise testing hypotheses.

A typical workflow involves highly interactive exploratory analysis on small portions of the data, followed by the development of custom scripts to automate the analysis on larger data sets.  Online training is available via the JWebbinar series.

1  All software is open source and community contributions are welcome in the form of suggestions, bug reports, or actual code. Further details on how to contribute can be found at the Data Analysis Tools Development Forum.


Please note that all of the packages described on this page are in active stages of development, including Astropy and glue.  There will be significant attention given to backwards compatibility as the APIs of the different modules evolve.



JWebbinars

See also: JWebbinar Landing Page

Beginning in Spring 2021, STScI will be hosting JWebbinars to teach the JWST community about tools and methods to analyze data from the James Webb Space Telescope. Each JWebbinar will provide virtual, hands-on instruction on common data analysis methods for JWST observations.  All tutorials will be recorded and resources will be made available to the community online at a later date.

JWebbinar topics will include:

  • Overview of data analysis tools and data products
  • The JWST pipeline for imaging
  • The JWST pipeline for spectroscopy
  • Analyzing IFUs and spectral cubes
  • Analyzing MIRI photometry
  • Analyzing time series spectroscopy

Previous user training workshops are archived an available for proposers to review, too.  These are not as up-to-date as the JWebbinar series.



Software packages

Table 1 provides links to further information about the tools. 


Table 1.  Software tools information

Package

Purpose

GitHub RepositoryDocumentation

Astropy

A community python library for astronomy

GitHub

ReadTheDocs

glue

A Python library to explore relationships within and among related datasets

GitHub

ReadTheDocs

ginga

A toolkit designed for building viewers for scientific image data in Python

GitHub

ReadTheDocs

photutils

Tools for detecting and performing photometry of astronomical sources

GitHub

ReadTheDocs

specutilsTools for performing spectroscopic analysis of astronomical sourcesGitHubReadTheDocs

astroimtools

Convenience tools for working with astronomical images

GitHub

ReadTheDocs

imexam

Tool for simple image examination, and plotting, with similar functionality to IRAF's imexamine

GitHub

ReadTheDocs

jdaviz

An interactive analysis tool, including Specviz, Cubeviz, and Mosviz:

  • Specviz is a 1D spectra analysis tool with similar functionality to IRAF's splot
  • Cubeviz is an analysis tool for 3D spectroscopy
  • Mosviz is an analysis and visualization tool for multi-object spectroscopy
GitHubReadTheDocs

asdf

Advanced Scientific Data Format is a next generation interchange format for scientific data 

GitHub

ReadTheDocs

gwcs

Generalized World Coordinate System tools for dealing with image and spectral geometries

GitHub

ReadTheDocs

synphot

Synthetic photometry toolkit for building model spectra and estimating count-rates

GitHub

ReadTheDocs



Jdaviz

See Also: Jdaviz Landing Page

The JWST Data Analysis Visualization (Jdaviz) Tool is a specific component of the broader analysis tool ecosystem designed to work seamlessly with astropy within a Jupyter notebook. The visualization tools also may be operated as standalone desktop applications or as embedded windows within a website.  The tool currently has three configurations:

  • Specviz is a tool for visualization and quick-look analysis of 1D astronomical spectra. 
  • Mosviz is a visualization tool for viewing multi-object spectrograph data (e.g., JWST NIRSpec MOS mode), and includes viewers for 2D and 1D spectra as well as contextual information like on-sky views of the spectrograph slit.
  • Cubeviz provides access to and manipulation of spectroscopic data cubes (like those produced by JWST MIRI MRS or NIRSpec IFU), along with 1D spectra extractions from the cube.

  The Jdaviz tools have demonstration videos available:



JWST Jupyter notebooks

See Also: Notebooks Landing Page, Notebooks GitHub Page

A series of Jupyter notebooks have been created to illustrate workflows for analyzing JWST data.  The notebooks utilize astropy machinery, including the jdaviz visualization tools described above in Table 1.  The notebooks are likely to be useful for analyzing data from other observatories as well.  These notebooks can be downloaded and executed by cloning the GitHub repository to your local computer.  Most of the notebooks rely on packages that are available in astroconda, although a few rely on packages that should be installed using pip. The version dependencies are listed in the environment.yaml and in the requirements file in each notebook folder.  You can also just view rendered versions of the notebooks.


Table 2.  JWST Jupyter notebooks

Notebook Description
General
asdf
  • Use case: create ASDF (Advanced Scientific Data Format) file from FITS file.
  • Data: CANDELS image of the COSMOS field.
  • Tools: asdf, gwcs, astrocut.
  • Cross-instrument: all instruments.
Background Estimation
  • Use case: estimate the sky background in complex scenes and evaluate the quality of the sky estimation.
  • Data: images with pathological background pattern created in the notebook.
  • Tools: photutils.
  • Cross-instrument: all instruments.
Redshift Cross-Correlation
  • Use case: reproduce the workflow of the IRAF task XCORFIT to measure redshift with a cross correlation algorithm.
  • DataLEGA-C spectra and galaxy template spectra; optical rest-frame.
  • Tools: specutils.
  • Cross-instrument: all instruments.
Querying MAST
  • Use caseHow to submit a NIRSpec MAST Query using python, and how to perform checks for potential duplication issues with any given targets by comparing with pre-existing MAST Data 
  • Data
  • Tools: mast, astroquery
  • Cross-instrument: created for NIRSpec, but good for all instruments
Specviz GUI Interaction
  • Use caseHow to inspect spectra in Specviz, export spectra from the GUI in the notebook, select regions in the GUI and in the notebook, and measure the redshift of a source in the GUI 
  • Data: NIRISS simulation of a generic scene with extended sources generated with the code MIRAGE
  • Tools: specutils, jdaviz
  • Cross-instrument: created for NIRISS, but good for all instruments
NIRCam
MultiBand Aperture Photometry
  • Use case: measure galaxy photometry in a field. Related to JDox Science Use Case #22.
  • Data: JWST simulated NIRCam images from JADES JWST GTO extragalactic blank field.
  • Tools: photutils
  • Cross-instrument: potentially MIRI
Crowded Field Aperture Photometry
  • Use caseCrowded field imaging with Aperture-fitting photometry
  • Data: JWST simulated NIRCam images from MIRAGE, run through the JWST calibration pipeline; LMC astrometric calibration field.
  • Tools: jwst, photutils
  • Cross-instrument: potentially MIRI
PSF Photometry
  • Use case: PSF photometry of crowded star field.
  • Data: JWST simulated NIRCam images from MIRAGE, run through the JWST calibration pipeline; LMC astrometric calibration field.
  • Tools: photutils, webbpsf.
  • Cross-instrument:
PSF (Matched) Photometry
  • Use case: PSF photometry of crowded star field.
  • Data: JWST simulated NIRCam images from MIRAGE, run through the JWST calibration pipeline; LMC astrometric calibration field.  In this case, the PSF-matching is used to correct photometry measured in the Long Wavelength images (redder than F200W).
  • Tools: photutils, webbpsf.
  • Cross-instrument:
NIRISS
WFSS Galaxy Extraction and Analysis
  • Use case: optimal extraction of grism spectra; redshift measurement; emission-line maps.
  • Data: JWST simulated NIRISS data from MIRAGE, run through the JWST calibration pipeline; galaxy cluster.
  • Tools: specutils.
  • Cross-instrument: NIRSpec.
MOS Spectroscopy
  • Use case: emission-line measurements and template matching on 1D spectra.
  • DataLEGA-C spectra and galaxy template spectra; optical rest-frame.
  • Tools: specutils.
  • Cross-instrument: NIRSpec.
SOSS Transiting Exoplanet
  • Use case: primary transit of an exoplanet.
  • Data: JWST simulated SOSS data from awesimsoss simulator.
  • Tools: jwst, juliet.
  • Cross-instrument:
NIRISS AMI Binary Star
  • Use case: Find the binary parameters of AB Dor.
  • Data: MIRAGE simulations for a binary point source AB Dor and calibrator HD37093 using the Aperture Masking Interferometry (AMI) mode on JWST NIRISS
  • Tools: jwst, nrm_analysis
  • Cross-instrument:
NIRSpec
IFU Analysis (Continuum Fitting)
  • Use case: continuum and emission-line modeling of AGN; 1.47-1.87um.
  • DataNIFS on Gemini; NGC 4151.
  • Tools: specutils, cubeviz.
  • Cross-instrument: potentially MIRI.
MOS Optimal Extraction
  • Use case: optimal spectral extraction; method by Horne (1986).
  • DataJWST simulated NIRSpec MOS data; point sources.
  • Tools: jwst, custom functions.
  • Cross-instrument: any spectrograph.
MOS Pre-Imaging w/ NIRCam
  • Use case: simulation of NIRCam pre-imaging for NIRSpec.
  • DataJWST simulated NIRCam data from MIRAGE; LMC.
  • Tools: mirage, jwst.
  • Cross-instrument: NIRCam.
BOTS Transiting Exoplanet
  • Use case: Bright Object Time Series; extracting exoplanet spectra.
  • DataJWST simulated NIRSpec data from ground-based campaign; GJ436b spectra from the Goyal et al. (2018).
  • Tools:
  • Cross-instrument:
IFU Optimal Extraction
  • Use case: optimal spectral extraction; method by Horne (1986).
  • Data: faint (quasar) point source was simulated using the NIRSpec Instrument Performance Simulator (IPS), then run through the JWST Spec2 pipeline.
  • Tools: jwst, scipy, specutils, jdaviz, photutils, astropy.io, astropy.wcs, astropy.stats, astropy.utils, regions
  • Cross-instrument:
MIRI
IFU Cube Fitting
  • Use case: continuum and emission-line modeling of galaxy IFU spectra; PAH wavelength region.
  • Data: Spitzer IRS on Messier 58.
  • Tools: specutils, custom functions.
  • Cross-instrument: NIRSpec.
LRS Optimal Extraction
  • Use case: extract spectra with different locations, extraction apertures, and techniques.
  • DataJWST simulated MIRI data.
  • Tools: jwst, gwcs.
  • Cross-instrument: any spectrograph.
MRS IFU Cube Analysis 1
  • Use case: extract spatial-spectral features from IFU cube and measure their attributes.
  • Data: KMOS datacube of point sources in the LMC.
  • Tools: specutils, spectral_cube, photutils.
  • Cross-instrument: any IFU.
MRS IFU Cube Analysis 2
  • Use caseuse photutils to automatically detect point sources and extract photometry in a given 3D cube, then  use specutils to find the important lines to first identify and analyze spectral lines.
  • DataALMA 13CO data cubes
  • Tools: specutils, spectral_cube, photutils.
  • Cross-instrument: any IFU.



Community contributions

Astropy is a community library, and the JWST Analysis Tools' success relies on community members, like you, to engage in the development process via bug reports (most effectively filed as github issues, but the JWST help desk is fine as well), or by code contributions through github pull requests. Use of the development versions of the code straight from github comes with the following caveats: at any given time, the code may not actually run or return correct results, and the documentation may be inconsistent with the code. Users who are not interested in contributing to the development software should use the versions in AstroConda.




Latest updates
  •  
    Added section on Jdaviz, updated notebooks for Build D release, added section on JWebbinars

  •  
    Added links to JDAT Jdaviz tools.

  •  
    Updated Software Package and Notebook status to be consistent with Build C for Data Analysis Notebook delivery.  Removed additional resources section.
Originally published