Data Retrieval

Data identified with a search may be retrieved in a number of ways, including via the MAST Portal download manager, browser download, ftp, and with user software. The best method of retrieval is largely a matter of convenience. Only authenticated and authorized users may retrieve data within the exclusive access period.

On this page

Retrieving JWST science data from MAST

Science data that have been identified through a Portal search may be retrieved in a number of ways. The best method of retrieval is largely a matter of user convenience, although fulfilling requests for very large volumes of data on hard media requires special effort. Anonymous users may search for and retrieve public datasets in MAST without registering and without logging in, but only authenticated and authorized users may retrieve data within the applicable exclusive access period. See the Data Access Policy for details and information on data access restrictions.



Science data retrieval options

Users who have identified science data of interest within the MAST Data Discovery Portal may retrieve the files (or collections of files) using one of the methods summarized below. For detailed instructions, see the following tutorial:

Direct download

This download (through the browser) is the quickest and simplest choice for single entries in the results table: simply click the disk icon in the table row corresponding to the entry of interest. It is also possible to save the search results table itself to a CSV file by clicking the file save icon; this allows preserving the results of a complex search (with, say, many applied filters) and later uploading to the Portal to reproduce the search.

Download Basket

The download basket is appropriate for retrieving multiple entries at once. The basket contents can be customized in the download manager to select specific product types (science, auxiliary, and preview), as well as the associated calibration reference files. There are two options for retrieval: immediate download through the browser, and staging data for later retrieval.

Immediate download: The selected data files will be bundled into a .tar, .tar.gz, or .zip file and transferred to the user's machine.

Size Limitation

Data retrievals via immediate download are limited to about 5 GB. Larger transfers require an asynchronous retrieval method: a cURL script, staging data for ftp retrieval, or the MAST API.

cURL script: A cURL script can be created and downloaded to the user's machine. Then simply execute the script in a Unix shell, and provide login credentials when prompted, to initiate the secure transfer of all selected files. Alternatively, users with poor internet connectivity may request data via hard media.

Batch Retrieval

There are two methods of batch retrieval: staging data on the MAST ftp server for subsequent retrieval by the user, and having data written to hard media that is later shipped to the user. These options are summarized in Table 1, below.


Table 1. Batch retrieval methods

Staging
  • Data will be staged to an STScI ftp server.
  • The user will be notified via e-mail when the transfer is complete and will be given the server address and path to the staging directory.
  • Mac users with OS version 10.13 and higher will need to use a third-party ftp client (such as Cyberduck).
Shippable Media
  • The user must provide a full mailing address for shipment.
  • The options include a USB-mountable drive, such as a thumb drive or (for large requests) a drive that a user ships to MAST.
  • Users must certify that they cannot retrieve the data through other means, such as staging or via the API.

Note that a valid email address must be provided for all batch data retrieval requests, though an STScI MyST ID is not required for public data.

What's in the box

All data products for all selected observations will be bundled together for delivery. When the .zip or .tar file is unpacked, data for each observation/visit/exposure/detector will appear in a separate subdirectory. For each subdirectory, the data bundle includes by default the highest-level data products, plus all parent data (unless those data have been de-selected in the download manager) For example, if an observation/visit/exposure/detector combination resulted in Level-2 data products, all Level-1 products would automatically be included unless the user explicitly chooses otherwise. Note: Level-3 products will appear in separate directories.

The download manifest

The zip (or tar) file will include a file called MANIFEST.HTML which lists each file name, a short description, and whether access is restricted. It will also note any files that could not be downloaded and the reason why (e.g., if you do not have permission to retrieve them).

MAST portal download examples

Figure 1. The result of a MAST Portal query of the target M51, showing results for HST data


Figure 2. Selected images are added to the download basket with Minimum Recommended Products still enabled




The MAST programmatic API

It is possible to retrieve data using scripts in various programming languages, such as Unix shell scripts or Python, through the Mashup API for MAST. See Programmatic Interfaces for details.



Community tools

A variety of community applications can retrieve data from astronomical archives, including MAST. These tools use Virtual Observatory protocols to discover and retrieve data files. Table 2. proviedes with a list of some of them.


Table 2. List of applications and cabilities

ToolCapabilities
AladinSky atlas tool
SAOImage/DS9Image and catalog display tool
TOPCATCatalog joins, subsetting, editing, and visualization
VizieRExtensive VO catalog offerings
VOClientStandalone software for querying and downloading VO data from USVAO

These tools currently lack the ability to provide user authentication credentials to archive services, and so cannot be used to retrieve data archived within the exclusive access period.

VizieR retrieval examples

Figure 6. Searching the hst/obscore catalogue in VizieR for the same obsID as the MAST Portal example


Figure 7. VizieR is able to find the same M51 data found through MAST Portal


Figure 8. The resulting VOTable downloaded for the _drz file



References

MAST Portal user's guide

Programmatic data retrieval



Published

 

Latest updates