JWST Data Retrieval
There are a variety of options for retrieving data of interest from MAST, including direct download through the browser, via the MAST Portal download manager, or via custom scripts. Certain community tools are also able to search for and retrieve public data from MAST.
Retrieving JWST science data from MAST
Words in bold are GUI menus/
panels or data software packages;
bold italics are buttons in GUI
tools or package parameters.
The MAST Portal interface requires that pop-up blockers be disabled for the domain https://mast.stsci.edu/ in order to download data. Check the documentation for your browser to determine how this is done, and be alert for notifications from the browser that popups are being blocked. In many cases it simply requires giving approval for popups when you are asked.
There are multiple ways in which data of interest may be identified. These include:
- a search with the Portal,
- following a link in a notification message (via email, or a Portal notification icon:subscription to program data, ) as generated with a
- by using a direct URL that was saved from a prior search by you or a colleague (e.g., by using the link icon),
- by performing a search with a script using a MAST programmatic interface (API)
Data retrieval options
Science observations and ancillary data of interest, identified with the MAST Portal, may be retrieved using one of the methods summarized below. Retrieving engineering data via the Portal is analogous. For detailed instructions, see the following references:
- JWST Archive Manual: Data Retrieval (for a summary)
- Portal Guide: the Retrieving Data chapter (for details)
- JWST Archive Manual: Using the Engineering Data Portal
This download from the Portal Results table is the quickest and simplest choice for single entries in the results table: simply click the disk icon Minimum Recommended Products. It is also possible to save the search results table to a CSV file by clicking the file save icon ; this allows preserving the results of a complex search (with, say, many applied filters) and later uploading to the Portal to reproduce the search.in the table row corresponding to the entry of interest to retrieve the
The download basket (click this icon:) is appropriate for retrieving multiple entries at once. The basket contents can be customized in the download manager to select specific product types (science, auxiliary, and preview), particular file types (FITS, ECSV, etc.), as well as the associated calibration reference files and contemporaneous Guide Star data. There are two options for retrieval: streaming download of a file bundle (i.e, a zip or tar file) through the browser, and staging data for later retrieval.
Some Observations (i.e., rows in the Portal Search Results table) are associated with a very large number of files. Loading too many such Observations into the Basket at once can result in an error. This problem is particularly likely for Observations with many extracted spectra (e.g., NIRCam wide field slitless spectra), or Observations containing lots of spatial dithers or other spacecraft maneuvers (as for an extended mosaic of a field or for moving targets). In these cases a MAST API is your only choice for data retrieval. There are tutorials linked from the article Using MAST APIs.
Immediate (streaming) download: The selected data files will be bundled into a .tar, .tar.gz, or .zip file and transferred to the user's machine.
Data retrievals via immediate download are limited to about 20,000 files. Larger transfers require an asynchronous retrieval method: a cURL script, staging data for FTP retrieval, or the MAST API.
cURL script: A Bash script can be created and downloaded to your machine. Simply execute the Bash script in a Unix shell, and provide login credentials when prompted, to initiate the secure transfer of all selected files. The script contains a sequence of cURL commands, which are relatively robust against connection interruptions and other foibles of internet file transfers.
There are 2 methods of batch retrieval: staging data on the MAST FTP server for subsequent retrieval by the user, and having data written to hard media that is later shipped to the user. Note that fulfilling requests for data on hard media requires special effort, and will involve a significant delay. These options are summarized in Table 1, below.
Table 1. Batch retrieval methods
Note that a valid email address must be provided for all batch data retrieval requests, though an STScI MyST ID is not required for public data.
Batch retrieval is not supported for JWST engineering data.
What's in the box
All data products for all selected observations will be bundled together for delivery. When the .zip or .tar file is unpacked, data for each observation will appear in a separate subdirectory. For each subdirectory, the data bundle includes by default the highest-level data products, plus all parent data (unless those data have been de-selected in the download manager) For example, if an observation/visit/exposure/detector combination resulted in level-2 data products, all level-1 products would automatically be included unless the user explicitly chooses otherwise. Note: level-3 products will appear in separate directories.
The download manifest
The zip (or tar) file will include a file called "MANIFEST.HTML" which lists each file name, a short description, and whether access is restricted. It will also note any files that could not be downloaded and the reason why (e.g., if you do not have permission to retrieve them).
The MAST programmatic API
It is possible to retrieve science data using scripts in various programming languages, such as Unix shell scripts or Python, through the Mashup API for MAST. The most popular interface is the astroquery.mast Python package. See Programmatic Interfaces for details.
Retrieval of JWST engineering data using cURL scripts is possible, but it may be simpler to use a Python program. See the Using MAST APIs chapter of the JWST Archive Manual for details.
A variety of community applications can retrieve data from astronomical archives, including MAST. These tools use Virtual Observatory protocols to discover and retrieve data files. Table 2 provides a few examples of such tools.
Table 2. List of applications and capabilities
|Aladin||Sky atlas tool|
|SAOImage/DS9||Image and catalog display tool|
|TOPCAT||Catalog joins, subsetting, editing, and visualization|
|VizieR||Extensive Virtual Observatory catalog offerings|
|VO Software||Standalone, VO-aware software for querying and downloading data from Virtual Observatory archives|
These tools currently lack the ability to provide user authentication credentials to archive services, and so cannot be used to retrieve data archived within the exclusive access period.
MAST Portal Guide