Data Discovery

The MAST Discovery Portal is the primary web interface for discovering, visualizing, assessing, and retrieving calibrated data products from the JWST calibration pipeline. The portal also serves the associated calibration guide-star, and engineering data products.

On this page

Accessing JWST data

There are a variety of ways for researchers to access JWST data of interest, including the MAST Discovery Portal (hereafter, Portal) which is the primary web interface for discovering, visualizing, assessing, and retrieving archived data. The portal also provides access to ancillary and engineering data related to the observations. Users may evaluate contemporaneous calibration reference data that were used to remove the instrumental signature from the science data products. They may also discover data through a programmatic (software-based) interface such as the MAST Applications Programming Interface (API), and through various community tools. The Portal also provides a subscription service that allow users to be notified as data from new JWST observations become available through MAST.

Data product types

The JWST Data Management System (DMS) produces many products for each JWST observation, including the science files produced by the data reduction pipeline. The exact type and number of products depends upon the instrument, its configuration, and operating mode. Consult the Data Products article for a detailed description of each science product and the concomitant data. Most of the science data files are images or tables in FITS format (Pence, et al. 2010), while others are in structured or unstructured ASCII. Table 1 contains a short summary of the data product types that may be included with each data set along with the semantic content of the various data products, including some that are produced outside the science data reduction pipeline.


Table 1. Summary of data product types that may be included with each JWST observation

Product typeFormatDescription
Science dataFITS MEF imagesScience data products for imaging, spectroscopy, coronagraphy, and (early stages of) time-series observations. The image extensions contain multi-dimensional science pixel arrays, concomitant data quality flags, and variance arrays.
FITS binary tableExtracted spectra are stored as tables, and include fields for spectral coordinates, flux, and concomitant quality information. AMI fringe parameters are also stored as binary tables of coefficients.
ECSVSource catalogs, as well as TSO light curves are stored in an ASCII enhanced character-separated variable format.
ASDFMetadata for images or spectra are stored in ASDF format, but are packaged as BYTE arrays in a FITS extension of type BINTABLE.
Guide-star dataFITS MEF imagesData products from associated guide-star activities: identification, acquisition, tracking and guiding. The image extensions are analogous to those for science data.
AssociationsECSVA collection of meta data for all exposures that match a combination of instrument, program ID, and observation from which associations are generated based on program information.
JSONEnumeration of relationships between all levels of science data products, and related calibration reference products, that were used during the course of calibration pipeline processing.
WSSFITSWavefront-sensing Optical Path Difference (OPD) images that describe the state of the mirror segment alignments for a particular epoch. The OPD files can be used with a tool such as WebbPsf to determine the estimated PSFs for any given instrument aperture.
Engineering dataCSVEngineering data are provided in the form of a time series of values, in CSV format, with one mnemonic per file.
CAL reference filesASDF, ECSV, FITS, JSONReference files used in calibration processing come in various formats.
PreviewsJPEG or PNGPreview and thumbnail images for presenting data products in the Portal.

Minimum recommended data products

Of the many different data products produced by the calibration pipeline, a subset has been identified as essential for extracting the intended science from the data. These are termed the "minimum recommended products" (MRP). The selection of data products that are included in this set depends upon the instrument used to obtain the data, and its configuration and operating mode. Generally, products in the MRP include the lowest-level calibrated science product (i.e., those for which the instrumental signature has been removed) along with the associated data products, but exclude calibration reference files, preview images, and guide-star data products.

MRP Checkbox

The MRP checkbox in the Download Manager must be de-selected in order to retrieve raw or intermediate-level data products, and ancillary products.



MAST Discovery Portal

MAST implements various protocols of the Virtual Observatory (VO) including those for image, table, and spectral data access.  As part of these protocals, MAST core services operate using the Common Archive Observation Model (CAOM).  As a result, MAST data can be searched and retrieved by VO-aware applications. 

Archived data

The MAST Portal offers great flexibility in customizing queries to identify datasets from several hosted NASA missions to explore and retrieve. See the Data Exploration article for a detailed example of a search for JWST science data using the Portal. The following tutorials may be helpful for new Portal users:

Engineering data

Data associated with the thousands of engineering telemetry points on JWST are stored in the Engineering Database. The data take the form of timeseries, and they may be searched with the AUI by means of an identifier, or mnemonic. Most of them can be queried, visualized, and downloaded by general users through a MAST Portal interface. Users may access the engineering database via a direct link to the query interface or, after querying the Portal for science data, through a link on each row of a search results table. See the JWST Engineering Database Browser for details.

Engineering data are obtained contemporaneously, but are not packaged with, the science data. Rather, EDB data are searched by mnemonic and a time range which spans the interval of the science exposure(s) of interest. The data files are provided in CSV format, which can be read by a wide variety of software applications, including MS Excel. The following tutorial may help first-time users of the EDB interface:

Anticipated data

At any given time some observations in a program may have been executed, archived, and become available to the community; some archived observations may temporarily be restricted to those with exclusive access; while still other observations may remain to be obtained. Investigating teams and the broader community each have an interest in data availability. In order to encourage the greatest possible use of JWST data in MAST, a subscription service will notify registered and subscribed users when one of the following observation-related events occur:

  • new observations are archived,
  • archived data have been reprocessed, or
  • restricted-access data become available to the public.

Users may tune the notifications by mission, program ID, event type, and science product. Users may establish or cancel their subscription through the MAST Discovery Portal, change the media and frequency of notifications, and change the selection criteria for notifications. See the following tutorial:



Planned Observations

MAST provides the capability to compare the position of one or more user targets against extant and planned observations to identify potentially duplicate observations. Duplicated observations are, in general, not allowed so it is in the proposer's best interest to perform this check prior to submitting a proposal for review. Read the article Identifying Potential Duplicate Observations to see how to query planned observations.



Virtual observatory tools

Many community software applications are capable of accessing remote data using protocols developed for the Virtual Observatory. You may have used them before without being fully aware of how such data were obtained. Table 2 provides an incomplete list of community, VO-aware applications for visualizing and exploring archived astronomical images, spectra, and catalogs:


Table 2. An incomplete list of community, VO-aware applications for visualizing and exploring archived astronomical data

ApplicationPrimary StrengthDescription
AladinAnalysisInteractive sky atlas, capable of overlaying archived images and source catalogs
DataScopeDiscoveryPosition-based search engine for astronomical archives
SAOimage/DS9AnalysisImage display tool, capable of overlaying archived source catalogs
SIMBADRetrievalAstronomical catalog search engine
SkyViewDiscoveryAstronomical image search engine
SpecViewAnalysis1-D spectrum visualization and analysis
TOPCATAnalysisCatalog cross-match, filtering, sub-setting, and visualization tool
VizieRRetrievalAstronomical catalog search engine
astropy.vo.clientDiscoveryPython packages for using VO services, including accessing VO registries, as well as catalog manipulation in software

 


References

Pence, W. D., et al. 2010, A&A, 524, A42
Definition of the Flexible Image Transport System (FITS), version 3.0

MAST Discovery Portal




Published

 

Latest updates
  •  
    Additional information on identifying duplicate observations, including a link to the "how-to" page
    Rewritten subsection on the Engineering Database interface, with a link to Data Exploration article
  • Added information about guide-star data, and simplified the description of the engineering database.