Synopsis
Download public data by ObsId from the Chandra archive.
Syntax
download_chandra_obsid obsid download_chandra_obsid obsid1,..,obsidN download_chandra_obsid obsid filetype download_chandra_obsid obsid1,..,obsidN filetype1,..,filetypeN The supported file types are: adat aoff aqual arf asol bias bpix cntr_img dtf eph0 eph1 evt1 evt1a evt2 flt fov full_img msk mtl oif osol pbk pha2 plt readme rmf soff src2 src_img stat sum vv vvref The --exclude flag allows you to skip a file type. The -m or --mirror flags allow you to use a mirror of the Chandra Data Archive. The -h or --help flags displays information on the command-line options. The -q or --quiet flags is used to turn off screen output.
Description
The download_chandra_obsid tool will download the data for the given Chandra Observation Id (ObsId) from the public archive. It can not be used to download proprietary data. Multiple values can be given - for both the ObsId and filetype lists - either as a comma-separated list or in a separate file (using the "@" syntax as described in 'ahelp stack').
Where are the files placed?
The files are stored in the current working directory, using the Observation Id as a directory name: so the data for ObsId 1843 would be placed in the directory 1843/. Within this directory the files are placed as they are in the archive, namely a top-level directory containing the oif.fits and V&V report, with the remaining data in the primary/ and secondary/ sub-directories.
How do I download a specific file?
The optional second argument to download_chandra_obsid allows you to restrict the download to one or more files. A comman-separated list (or stack) of file types can be given, so
unix% download_chandra_obsid 1843 evt2,asol,bpix,fov
will just download the evt2, asol, bpix, and fov files for ObsId 1843.
What happens when files already exist?
The tool will skip downloading any files that already exist on disk as long as they are the correct size. If the on-disk version is smaller than the archive version, then download will be resumed rather than started again. A warning message will be displayed if the on-disk version is larger than the archive version.
The behavior described above means that you can interrupt a download and re-start it and will not have to re-download existing data.
Screen output
Unless the -q or --quiet flag was used, each file that is downloaded will be displayed on screen, giving the "type", format, file type before a progress bar of # marks (each # mark indicates 5% of the file). Once the file has been downloaded an average rate is displayed for the download. Examples of the output for two files are shown below:
Type Format Size 0........H.........1 Download Time Average Rate --------------------------------------------------------------------------- evt2 fits 17 Mb #################### 2 s 7170.3 kb/s vv pdf 78 Kb #################### < 1 s 843.1 kb/s
If a file has already been downloaded then the message "already downloaded" will be displayed instead of the progress bar.
Examples
Example 1
unix% download_chandra_obsid 1842
This will download all the data files for ObsId 1842 to the directory 1842/. The screen output for the run will look something like the following (where many lines have been excluded, indicated by the "..." lines):
Downloading files for ObsId 1842, total size is 81 Mb.
Type Format Size 0........H.........1 Download Time Average Rate
------------------------------------------------------------------------
evt1 fits 36 Mb #################### 46 s 818.4 kb/s
evt2 fits 19 Mb #################### 23 s 809.1 kb/s
...
readme ascii 11 Kb #################### < 1 s 91.7 kb/s
eph1 fits 6 Kb #################### < 1 s 109.7 kb/s
fov fits 6 Kb #################### < 1 s 108.1 kb/s
msk fits 5 Kb #################### < 1 s 62.0 kb/s
pbk fits 4 Kb #################### < 1 s 81.4 kb/s
Total download size for ObsId 1842 = 81 Mb
Total download time for ObsId 1842 = 1 m 45 s
Example 2
unix% download_chandra_obsid 1842 evt1,bpix,asol,fov,readme
Here we restrict the download to just the evt1, bpix, asol, fov, and readme files from ObsId 1842.
Example 3
unix% download_chandra_obsid 1842 --exclude vvref
Download all but the vvref2.pdf file for obsid 1842. This file contains the detailed V&V data for the observation and can be large.
Example 4
unix% download_chandra_obsid 1842,1843
This will download the data for ObsIds 1842 and 1843 to the directories 1842/ and 1843/ respectively.
Example 5
unix% cat obsids 1842 1843 unix% download_chandra_obsid @obsids
Here we use the support for stacks (see 'ahelp stack' for more information) to download the data for the ObsIds listed in the file obsids, in this case 1842 and 1843.
Example 6
unix% download_chandra_obsid 1842,1843 evt1,asol -q
Here we download only the evt1 and asol files from the two ObsIds and use the -q flag to turn off the screen output.
Example 7
unix% download_chandra_obsid -m https://cxc.cfa.harvard.edu/cdaftp/ 1842
Download ObsId 1842 from the given CDA mirror site (in this example we have used the location of the actual CDA archive). If the observation is not found at the mirror site then the script will skip it (i.e. it does not fall back to trying the CDA site).
An alternative is to use the CDA_MIRROR_SITE environment variable, so that the above could also be written as:
unix% setenv CDA_MIRROR_SITE https://cxc.cfa.harvard.edu/cdaftp/ unix% download_chandra_obsid 1842
Mirror sites of the Chandra Data Archive
If you have set up, or have access to, a mirror of the Chandra Data Archive (it can be only a partial archive), then you can use the --mirror command-line argument, or the CDA_MIRROR_SITE environment variable, to tell the script to use this mirror instead. The mirror option takes precedence over CDA_MIRROR_SITE if both are set.
If an ObsId can not be found on the mirror site then it is skipped; i.e. the script will not also check the CDA site for the data.
The mirror location should include the URL of the HTTP or HTTPS site (the leading https:// or http:// is required) and the path up to, but not including, the byobsid/ directory. So, for the Chandra Data Archive itself you would use
https://cxc.cfa.harvard.edu/cdaftp/
(although obviously in this case you do not need to use the mirror option). Please see the documentation for the mirror site to find out the correct path to use.
Changes in the scripts 4.13.1 (March 2021) release
Validation and Verification files
The Chandra archive contains two V&V files for an observation:
- axaff<obsid>N<n>_VV001_vv2.pdf
- secondary/axaff<obsid>N<n>_VV001_vvref2.pdf.gz
where the first file is intended for users, and the second is only needed in rare situations. The vvref filetype has been added to select this file, and the vv filetype now only refers to the vv2 file.
Excluding file types
The --exclude flag has been added to allow you to ignore certain file types, so to avoid downloading the vvref2 file you can say
download_chandra_obsid 1843 --exclude vvref
Changes in the scripts 4.13.0 (December 2020) release
The script should be more robust to changes made to the Chandra Data Archive web site.
Changes in the scripts 4.12.2 (April 2020) release
Archive has moved
The script now uses the HTTPS service provided by the Chandra Data Archive as the FTP service is being retired. The archive is now available at https://cxc.cfa.harvard.edu/cdaftp/
Order by file size
For each observation, the files are now sorted by decreasing file size before being downloaded.
Support for ARF and RMF access
The next reprocessing of the Chandra archive will add grating ARF and RMF products, so "arf" and "rmf" have been added to the list of supported file types. Note that it will take some time before these files appear in the archive (as the reprocessing has not started yet). Please review the Archive status page for information on data products.
Changes in the scripts 4.9.3 (May 2017) release
Updated to use ftp://cda.cfa.harvard.edu as the default FTP server.
Changes in the scripts 4.8.1 (December 2015) release
The code has been updated to avoid warning messages from NumPy version 1.9. There is no difference to how the script behaves.
Changes in the scripts 4.6.5 (June 2014) release
Added evt1a and adat file types
It is now possible to select evt1a and adat (PCAD Level 1 ACA image) files using the filetype argument. The adat files can be used with the monitor_photom script.
Changes in the scripts 4.5.4 (August 2013) release
Support for CDA mirror sites
The --mirror command-line argument and support for the CDA_MIRROR_SITE environment variable has been added to allow data access from a mirror of the Chandra Data Archive site.
Support for the 00README files
The new 00README files are included in the download, and can be explicitly selected using the readme filetype.
Notes
This script is not an official part of the CIAO release but is made available as "contributed" software via the CIAO scripts page. Please see this page for installation instructions.
Bugs
There are no known bugs for this tool.
See Also
- contrib
- cda_data, cda_search
- tools::download
- download_obsid_caldb, find_chandra_obsid, obsid_search_csc, search_csc
- tools::utilities
- splitobs