Understanding Data Files

JWST data files share a basic structure with slight variations that depend on observing mode or instrument used. Their particular format depends on the stage of the JWST Data Reduction Pipeline where they were created. 

On this page

The following articles provide more information on specific aspects of understanding JWST data files:

General overview

See also: Getting Started with JWST Data (V1)File Header Contents
ReadTheDocs (External Links): File Naming Conventions, Data Product Types, Science Product Structures and Extensions, Data File Associations

Working with JWST data requires an understanding of JWST file types, including FITS files, Advanced Scientific Data Format (ASDF) files, JSON files, and Enhanced Character Separated Values (ECSV) files.  In general, all the science data include FITS and ASDF extensions, but these formats can also be used independently for calibration reference files.   FITS format extensions, on top of the science pixel values, provide relevant information about the science data pixels; e.g., uncertainty or variance estimates for each pixel. ASDF format extensions are used to provide more complex information about that data, like data model metadata or world coordinate system information.  ECSV files are products generated in the last stages of calibration, for certain observing modes, and which contain a catalog of sources or specific photometric information derived by the calibration pipeline. Finally, JSON files contain information regarding the way the science data should be associated by the calibration software and according to the rules for each of the observing templates defined by the different instrument teams.

Once JWST data are obtained, telemetry data from the recorded science data files are received by the Data Management System (DMS) in the same binary format that is stored on the JWST Solid State Recorder (SSR), and then they are processed to extract the science data and relevant detector and exposure information. The initial FITS header will contain keywords required by the FITS standard and those required for identification, naming of the files, data structure definition, and calibration of the science data by the calibration pipeline. These keywords are populated with telemetry packet and science image information; proposal, planning, and scheduling information; spacecraft position; time conversions; pointing information; and selected engineering parameters. For science data the corresponding transformation from the detector positions to a world coordinate frame (ICRS and wavelength) is provided via distortion and spectral models stored in ASDF format extensions of the FITS file. These data are then processed through the different stages of the calibration pipeline, where the JSON file provides the list of data to be associated, and finally archived in MAST

JWST data products can be divided into two main types, depending on the level of processing the file contains; data products from single exposures produced during stage 1 and stage 2 of the calibration pipeline, and data products that result from the combination of these exposures into a single integrated product in stage 3. Within the two categories, there are different ways in which a set or subset of exposures is combined, and each of these corresponds to a unique association. The way exposures are combined depends on the instrument and observing mode and is determined from information passed to DMS from the  submitted APT files.




Published

 

Latest updates