Understanding Data Files


JWST data files are FITS files with unique names that map to the original proposal, observation, visit, and instrument and detector used. Their particular format depends on the stage of the JWST Data Reduction Pipeline where these were created.

See also: File Naming Conventions and Data ProductsHeader Keywords and RelationshipsUnderstanding AssociationsJWST Data StructureCoordinate Systems and Transformations


All JWST data shares a basic data structure with slight variations that depend on the observing mode or instrument used. Being able to work with JWST data requires an understanding of JWST FITS files,  Advanced Scientific Data Format (ASDF) files, and JSON files. FITS format files contain the science data pixels, ASDF files contain the world coordinate system information, and the JSON files contain information regarding the way the science data is associated. 

JWST data are FITS format files generated by the Data Management System (DMS) via the Science Data Processing (SDP) and the Calibration Pipeline. The telemetry data from the Recorded Science Data files that are received by DMS, is in the same binary format as stored on the JWST Solid State Recorder (SSR).  These files will come in compressed packets that will be read by SDP to extract science data and relevant detector and exposure information. 

The initial FITS header will contain keywords required by the FITS standard and keywords required for identification, naming of the files, data structure definition, and for the correct calibration of the science data by the calibration pipeline. These keywords are populated with telemetry packet headers and Science Image headers; proposal, planning, and scheduling information; spacecraft position; time conversions; pointing information; and select engineering parameters (JWST-STScI-002111).

The corresponding transformation from the detector positions to a world coordinate frame (ICRS and wavelength) for the science data are provided via distortion and spectral models stored in ASDF format extensions to the FITS file.

The JSON file provides the list of data to be associated. JWST data products can be divided into two main types of data; data products from single exposures produced during stage 1 and stage 2 of the calibration pipeline and data products that result from the combination of these exposures into a single one produced in stage 3. The difference between these two types of products can be easily told apart by their File Naming Conventions or their Header Keywords. Within an association, the exposure level data are combined into what we call stage 3 and stage 4 data. Within these two categories, there are different ways in which a set or subset of exposures is combined, each of these corresponds to a unique association. The way in which these are combined is determined by the information coming from the Astronomer Proposal Tool, matching the proposed organization of the observations via dithers or mosaics, and any other special requirement associating the data into a single set.

Science exposures are part of an association, however, when considered necessary, an association will also include data that needs to be kept together for calibration purposes or for post-calibration analysis by experienced observers.




References

JWST-STScI-002111



Last updated

Published July 31, 2017