CSC Command-line Interface

As an alternative to CSCview, tables of catalog data may be downloaded directly through a URL with Unix command-line tools such as cURL and Wget, using the ADQL query synytax or the IVOA-compliant VO Cone Search service.

ADQL Queries

All source properties contained in the catalog may be accessed directly from the following URL in an ADQL command-line query:

http://cda.cfa.harvard.edu/csccli/getProperties

This URL takes five parameters:

Output: A Tab Separated Values (TSV) or VOTable TABLEDATA format table containing the sources and associated source properties requested in the catalog query.

Crossmatch queries

When the user supplies a table for crossmatch via the UPLOAD parameter, the data are represented in a table called user_table in ADQL. An association table between the user_table and the CSC master_source table is called cross_match.

The user_table table contains columns named:

      usrid  - a unique identifier generated by the upload code
      ra     - a floating point column
      dec    - a floating point column
      radius - a floating point column
      sigma  - a floating point column
      objid  - an integer or character data column based on the datatype
      of the user-input table
      

The cross_match table contains columns named:

      usrid       - a key to join on the user_table.usrid column
      msid        - a key to join on the master_source.msid column
      separation  - a floating point column representing the separation of the
                    two sources in arcseconds
      probability - a floating point column representing the probability that
                    the two sources are the same  
    

Data product queries

The CSC data products - e.g. spectra, light curves, and images - may be accessed at the command line, but only after interactively querying the catalog within CSCview. After submitting a query in the GUI and selecting the desired data products, users are brought to the Products tab where they are given the option of downloading the selected data products via a download script or tar file. The download script contains a list of Wget commands, one for each file, which can be executed on the Unix command line for a batch download. See the CSC thread Retrieving Data Products to learn how to access catalog data products.

ADQL syntax

An ADQL SELECT statement returns a result set of records from one or more tables of astronomical data, located by the FROM clause. The available tables are 'master_source m' (Master Sources Table), 'obi_source o' (Source Observations Table), 'master_obi_assoc a' (Master Sources/Source Observations Associations), 'dataset d' (Data Products), and 'crossmatch c' (Crossmatch) for the CSC. Some of the optional clauses of a SELECT statement include:

• TOP specifies the number of rows to retrieve.
• WHERE specifies which rows to retrieve, according to the search criteria.
• ORDER BY specifies an order in which to return the rows.

ADQL SELECT, TOP, FROM, WHERE, ORDER BY statements may also be entered into the ADQL window in the Query tab of the data access GUI CSCview. This view is accessed by selecting the menu item View->Query->Show Language while the Query tab is open.

Examples

1. Perform a basic source property query.

# curl --form query='SELECT TOP 50 m.name, m.significance, m.flux_aper_b, m.alpha FROM master_source m WHERE (m.significance > 10.0 AND m.pileup_flag = 0 AND m.hard_hs > 0.7)' 'http://cda.cfa.harvard.edu/csccli/getProperties?outputFormat=votable'

# wget -O out.file 'http://cda.cfa.harvard.edu/csccli/getProperties?query=SELECT TOP 50 m.name, m.significance, m.flux_aper_b, m.alpha FROM master_source m WHERE (m.significance > 10.0 AND m.pileup_flag = 0 AND m.hard_hs > 0.7) &outputFormat=votable' 

This query accesses from the catalog the master source name, significance, broad-band aperture energy flux, and power-law model best-fit photon index for the first 50 master sources found with a significance greater than 10.0, a hard-to-soft hardness ratio greater than 0.7, and pileup fraction less than 10%. The result set is returned in VOTable TABLEDATA format.

2. Perform a basic cone search query.

# curl --form query='SELECT m.name, m.ra, m.dec, m.flux_aper_b FROM master_source m WHERE dbo.cone_distance(m.ra,m.dec,83.77333,-5.68464)<=10' 'http://cda.cfa.harvard.edu/csccli/getProperties'

# wget -O out.file 'http://cda.cfa.harvard.edu/csccli/getProperties?query=SELECT m.name, m.ra, m.dec, m.flux_aper_b FROM master_source m WHERE dbo.cone_distance(m.ra,m.dec,83.77333,-5.68464)<=10' 

This query accesses from the catalog the master source name, R.A and Dec., and broad-band aperture energy flux for all sources located within 10 arcminutes of the celestial position R.A.=83.77333, Dec.=-5.68464 (in decimal degrees).

3. Upload a query from a file to the URL.

# curl --form query=@query.txt 'http://cda.cfa.harvard.edu/csccli/getProperties'

The file should contain a query written in ADQL 2.0, e.g. a query save file output by CSCview, like the one shown here.

4. Specify how a catalog value of NULL should appear in the output table (instead of a blank space).

# curl --form nullAppearance=NULL --form query="SELECT DISTINCT o.obsid, o.obi, o.region_id, o.theta, o.mjr_axis_raw_s FROM obi_source o WHERE (o.instrument = 'ACIS' AND o.theta < 0.5 AND o.edge_code = 0 AND o.detect_significance_b > 10.0) ORDER BY theta DESC" 'http://cda.cfa.harvard.edu/csccli/getProperties'

# wget -O out.file "http://cda.cfa.harvard.edu/csccli/getProperties?nullAppearance=NULL&query=SELECT DISTINCT o.obsid, o.obi, o.theta, o.mjr_axis_raw_s FROM obi_source o WHERE (o.instrument = 'ACIS' AND o.theta < 0.5 AND o.edge_code = 0 AND o.detect_significance_b > 10.0)"

This query accesses from the catalog the ObsID, OBI, off-axis angle, and (full) source ellipse major axis length for all distinctly identified ACIS-observed sources found in the Source Observations Table with an off-axis angle less than 0.5, a detect significance greater than 10.0, and which do not dither off chip boundaries during the observation. Here, what normally would appear as an empty character in the output table will appear as "NULL".

5. Specify the output R.A. and Dec. coordinate format. Default is sexagesimal.

unix% curl 'http://cda.cfa.harvard.edu/csccli/getProperties?query=SELECT TOP 1000 m.name, m.ra, m.dec FROM master_source m WHERE (m.significance > 10.0 AND m.pileup_flag = 0 AND m.hard_hs > 0.7) &coordFormat=decimal' 

unix% wget -O out.file 'http://cda.cfa.harvard.edu/csccli/getProperties?query=SELECT TOP 1000 m.name, m.ra, m.dec FROM master_source m WHERE (m.significance > 10.0 AND m.pileup_flag = 0 AND m.hard_hs > 0.7) &coordFormat=decimal'

This query accesses from the catalog the master source name, R.A. and Dec. for the first 1000 sources with a master source significance greater than 10.0, a hard-to-soft hardness ratio greater than 0.7, and a pileup fraction less than ~10%. The R.A. and Dec. values are output in decimal degrees.

6. Specify that floating point numbers should appear in native format.

unix% curl 'http://cda.cfa.harvard.edu/csccli/getProperties?query=SELECT top 10 o.obsid,o.obi,o.targname,o.ra_targ,o.dec_targ,o.ra_pnt,o.dec_pnt,o.roll_pnt,o.ra_nom,o.dec_nom,o.roll_nom,o.gti_start,o.gti_stop,o.gti_elapse,o.gti_end,o.gti_obs,o.gti_mjd_obs,o.mjd_ref,o.instrument,o.grating,o.datamode,o.readmode,o.cycle,o.exptime,o.timing_mode,o.ascdsver,o.caldbver,o.crdate,o.ao FROM obi_source o&floatFormat=native'

This query accesses from the catalog many observation-specific properties for the first 10 sources found in the Source Observations Table, with source property values output in native floating point format (the "native" catalog format includes numbers with many significant digits, exactly as they are output by the catalog processing pipeline).

7. Use the current database view as opposed to a specific release view.

unix% curl "http://cda.cfa.harvard.edu/csccli/getProperties?query=SELECT top 5 m.name,m.ra,m.dec,m.err_ellipse_r0,o.obsid,o.theta,o.flux_significance_b,o.conf_code,o.extent_code,o.sat_src_flag,o.mjr_axis_raw_b,o.mnr_axis_raw_b,o.pos_angle_raw_b,o.cnts_aper_b,o.src_rate_aper_b,o.src_rate_aper_hilim_b,o.src_rate_aper_lolim_b,o.flux_aper_b,o.flux_aper_hilim_b,o.flux_aper_lolim_b,o.hard_hm,o.hard_ms,o.var_index_b,o.livetime,o.detector,o.posid FROM master_obi_assoc a , master_source m , obi_source o WHERE ((a.match_type = 'u')) AND o.posid=a.posid AND m.msid=a.msid ORDER BY name ASC&version=current"

This query accesses source properties from both the Master Sources and Source Observations tables, for the first five unambiguously identified sources in the current catalog database (as opposed to a static 'release' version of the database, frozen in time). The results are sorted on the master source name column; an example of a CSC master source name is 'CXO J080238.1-395249.'

8. Find CSC counterparts to sources of interest by performing a cross-match against the CSC using an input table of source positions.

unix%  curl \
    --request POST \
    --form 'query=select u.usrid, u.objid, m.msid, u.ra, u.dec, m.ra, m.dec, c.separation, c.probability from cross_match c, user_table u, master_source m where u.usrid = c.usrid and c.msid = m.msid' \
    --form RA=ra \
    --form DEC=dec \
    --form UPLOAD=user_table,param:table1 \
    --form name='@//data/somewhere/table.xml;filename=table1' \
    http://cda.cfa.harvard.edu/csccli/getProperties

This query accesses from the catalog the separation in arcseconds between the matched catalog master sources and the corresponding sources in a user-input table, along with the names and RA & Dec. positions associated with the catalog sources in the CSC Master Sources Table. The probability value of each source match is also returned, where a value of 1.0 indicates that the returned match is very likely a true match (down to many siginificant digits in source position), and a value of 0.0 represents a very unlikely true match.

VO Cone Search

A VO Cone Search service is available for conducting a search on source position in the CSC. It follows the IVOA cone search recommendations, with a fixed-call syntax which does not get combined with CSC cone searches in ADQL. The command-line query specifies a sky position and an angular distance to be used as the search radius, defining a cone on the sky. The query returns a list of astronomical sources from the catalog whose positions lie within the cone, in VOTable format.

The base URL for the VO cone search is http://cda.cfa.harvard.edu/cscvo/coneSearch

Examples

unix% curl 'http://cda.cfa.harvard.edu/cscvo/coneSearch?RA=83.77333&DEC=-5.68464&SR=.233&VERB=2'

unix% wget -O out.vot 'http://cda.cfa.harvard.edu/cscvo/coneSearch?RA=83.77333&DEC=-5.68464&SR=.233&VERB=2 '

See example output in 'out.vot' produced by the Wget example above.

In the VO Cone Search, one searches on RA, DEC and SR (search radius) in decimal degrees, and receives one of the three available result sets, based on the value of the parameter VERB (verbosity):

• verbosity = 1 reports the source name, RA and DEC values from the Master Sources Table for each matching source resulting from the cone search
• verbosity = 2 reports the master source summary results and Space Time Coordinate (STC) metadata
• verbosity = 3 reports all the columns in the Master Sources Table

The master source summary returned by verbosity level 2 is the result set of one of the standard queries available in CSCview. It includes the following CSC master source properties:

• name
• ra
• dec
• err_ellipse_r0
• conf_flag
• sat_src_flag
• significance
• flux_aper_b, flux_aper_lolim_b, flux_aper_hilim_b
• flux_aper_w, flux_aper_lolim_w, flux_aper_hilim_w
• extent_flag
• hard_hm, hard_hm_lolim, hard_hm_hilim
• hard_ms, hard_ms_lolim, hard_ms_hilim
• var_intra_index_b
• var_inter_index_b
• var_intra_index_w
• var_inter_index_w