JWST Post-Pipeline Data Analysis
Some of the tools available to work on fully processed data products, including a variety of community-based astronomical analysis packages and custom software developed by STScI to work specifically with the JWST data formats, are described in this article.
On this page
The JWST Data Analysis Tools ecosystem
The JDAT ecosystem is based on Python and includes core libraries, visualization tools, notebook tutorials demonstrating common workflows, and documentation of these software packages. The long-term objective is to provide common, well-tested tools that will accelerate scientific discovery, improve the accuracy and reproducibility of data analysis, and lower barriers for those with fewer resources, experience or scientific connections. The tools are built to serve JWST data analysis, but in such a way that they can be used with astronomical data from many observatories. The highest priorities are:
- Producing and delivering Jupyter notebooks to demonstrate the functionality of the tools, find places where the tools are missing functionality, and test that the tools work as expected (using continuous integration).
- Supporting the visualization tools and the whole suite of plugins to work in Jupyter notebooks, in cloud platforms using Jupyter Lab, as standalone applications, and in the MAST archive.
- Maintaining the adequate level of performance.
- Maintaining a clear documentation to encourage external contributions.
Because much of the functionality exposed by the notebooks has been integrated in the libraries or in the JWST calibration pipeline, a common workflow is to rerun the calibration pipeline with custom parameters before resorting to other tools. For specific cases, the Tools from the community page provides a list of publicly available tools developed by the community to support proposal planning, data processing, and analysis.
Available software tools
There are a variety of JWST post-pipeline data analysis tools that can assist observers in viewing and analyzing their JWST data (see Table 1). These tools have been developed by a variety of members of the astronomical community, but are generally written in Python and work with Astropy. Each package has its own installation instructions, but most can be installed with standard pip commands.
Note that Astropy is a community library, and the JWST analysis tools' success relies on community members, like you, to engage in the development process via bug reports (most effectively filed as GitHub issues, but the JWST help desk is fine as well), or by code contributions through GitHub pull requests. Use of the development versions of the code straight from GitHub comes with the following caveats: at any given time, the code may not actually run or return correct results, and the documentation may be inconsistent with the code. Users who are not interested in contributing to the development software should use fully released versions (a standard pip install). Details regarding the difference in the installation instructions can be found in the respective documentation for each package.
These post-pipeline data analysis tools can help astronomers with the often iterative and interactive workflow involved in converting science calibration pipeline data products into meaningful scientific results. This typically involves tasks such as:
- Inspecting data and data quality information
- Masking or flagging data and using those annotations to guide later steps in the analysis
- Using the results of interactive analysis to guide a custom run of the pipeline (e.g., tweaking spectral extraction parameters or background estimates)
- Performing optimized spectral extraction techniques
- Combining data sets in various ways, with careful attention to astrometry, PSF matching, and other issues
- Source detection and photometry using different choices or algorithms than those used in the pipeline
- Measuring lines and continuum in spectral data
- Fitting models to data or otherwise testing hypotheses
A typical workflow may thus involve highly interactive exploratory analysis on small portions of the data, followed by the development of custom scripts to automate the analysis on larger datasets.
Table 1. Tools for working with JWST data
Package | Purpose | GitHub Repository | Documentation |
|---|---|---|---|
Astropy | A community python library for astronomy | GitHub | |
glue | A Python library to explore relationships within and among related datasets | GitHub | |
ginga | A toolkit designed for building viewers for scientific image data in Python | GitHub | |
photutils | Tools for detecting and performing photometry of astronomical sources | GitHub | |
| specutils | Tools for performing spectroscopic analysis of astronomical sources | GitHub | ReadTheDocs |
| specreduce | A toolkit designed to reduce and calibrate spectroscopic data. | GitHub | |
astroimtools | Convenience tools for working with astronomical images | GitHub | |
imexam | Tool for simple image examination, and plotting, with similar functionality to IRAF's imexamine | GitHub | |
jdaviz | An interactive analysis tool designed to be integrated into a Jupyter notebook science workflow. Included Imviz, Specviz, Specviz2d, Cubeviz, and Mosviz:
| GitHub | ReadTheDocs |
asdf | Advanced Scientific Data Format is a next generation interchange format for scientific data | GitHub | |
gwcs | Generalized World Coordinate System tools for dealing with image and spectral geometries | GitHub | |
synphot | Synthetic photometry toolkit for building model spectra and estimating count-rates | GitHub |
Jdaviz
The JWST Data Analysis Visualization (Jdaviz) tool is a specific component of the broader analysis tool ecosystem that has been developed by STScI to work explicitly with JWST data products using Astropy functionality. The JDAviz tool may be operated as a standalone desktop applications, within a Jupyter notebook environment, or as embedded windows within a website.†
†Jdaviz is built using using the glue visualization framework. For users who are interested in using glue in contexts beyond those provided for in jdaviz, please visit https://glueviz.org for download links and more information.
Jdaviz currently has 5 available configurations:
- Imviz is a 2D image viewer.
- Specviz is a tool for visualization and quick-look analysis of 1D astronomical spectra.
- Specviz2d is a tool for visualization and extraction of 2D spectra.
- Cubeviz provides access to and manipulation of spectroscopic data cubes (like those produced by JWST MIRI MRS or NIRSpec IFU), along with 1D spectra extractions from the cube.
- Mosviz is a visualization tool for viewing multi-object spectrograph data (e.g., JWST NIRSpec MOS mode), and includes viewers for 2D and 1D spectra as well as contextual information like on-sky views of the spectrograph slit.
For further information see the Jdaviz ReadTheDocs page and an overview of available Jdaviz Tutorials.
JWebbinars
Starting in 2021, STScI organized and hosted a variety of community training webinars on various aspects of the JWST Science Calibration Pipeline and on post-pipeline data analysis tools. These sessions, called JWebbinars, provided users with general background and supporting materials such as Jupyter notebooks and overview PowerPoint presentations.
Information can be found on the JWebbinars page.
JWST data analysis (JDAT) notebooks
A series of example Jupyter notebooks have been built to demonstrate expected science workflows using various JWST observing modes. While focused on JWST, they show useful workflows for analyzing data from other observatories as well. The notebooks utilize Astropy packages and the Jdaviz visualization tools when possible. These notebooks can be downloaded and executed by cloning the GitHub repository to your local computer. The documentation contains detailed instructions for installing the notebooks, as well as rendered versions of the notebooks that allow you to view them in your browser without downloading. See the table on the landing page of the JDAT Notebooks documentation for a complete and up-to-date list of notebooks.