JWST Data Formats
Various JWST data products are produced in one of several standardized data formats, as described in this article.
See also: Understanding JWST Data Files
Working with JWST data requires an understanding of JWST file types and steps in the JWST Science Calibration Pipeline that produce them.
JWST file formats
- FITS format extensions, on top of the science pixel values, provide relevant information about the science data pixels, e.g., uncertainty or variance estimates for each pixel, data quality flags for each pixel, integration times for each exposure, metadata for data groups or for data that was combined into a single product.
- ASDF format extensions are used to provide more complex information about that data, like data model metadata or world coordinate system information.
- ECSV files are products generated in the last stages of calibration and for certain observing modes. These contain a catalog of sources or specific photometric information derived by the science calibration pipeline.
- Finally, JSON files contain information regarding the way the science data should be associated by the calibration software and according to the rules for each of the observing templates defined by the different instrument teams.
Once JWST data are obtained, telemetry data are received by the Data Management System (DMS) and processed to extract the science data. The FITS header will contain keywords required by the FITS standard and keywords relevant to the observation that are extracted from the telemetry packet headers and science image headers.
Multi-extension FITS format
Flexible Image Transport System (FITS) is a standard format for exchanging astronomical data, independent of the hardware platform and software environment. FITS format files consist of a series of header data units (HDUs), each containing 2 components: an ASCII text header and binary data. The header contains a series of keywords that describe the data in a particular HDU; the data component may immediately follow the header.
For JWST FITS data, the first HDU, or primary header, only contains header information in the form of keyword records with an empty data array, which is indicated by the occurrence of
0 in the primary header. The primary header may be followed by one or more HDUs called extensions, which may take the form of images, binary tables, ASDF files, or ASCII text tables. The data type for each extension is recorded in the
XTENSION header keyword.
The Advanced Scientific Data Format (ASDF) is a next generation, human-readable, hierarchical metadata structure made up of basic dynamic data types such as strings, numbers, lists, and mappings. Data is saved as binary arrays. It is primarily intended as an interchange format for delivering information about the science instruments or how products were created to scientists or, for example, between stages of the JWST Science Calibration Pipeline. ASDF files are added to certain calibration pipeline products and are part of the reference data used by the software. As an example, distortion and spectral models needed to transform detector positions to a world coordinate frame are in ASDF format.
The Enhanced Character Separated Values (ECSV) format, which is standard for the interchange of tabular data in a text-only format, is used to store a catalog of derived data for sources identified in some steps of the calibration pipeline. This file includes a header section with the definition, data type, and description for the columns, and a data section with as many rows as sources identified in an image. Besides a simple comma-separated delimited text file reader, these files can also be read, modified, or created using the Astropy ECSV code.