Last modified: 13 August 2013

URL: http://cxc.harvard.edu/csc/threads/convert/

Using a CSC Save File in CIAO

CSC Threads


Overview

Synopsis:

The results of a catalog search may be written to a file by selecting the "Save" toolbar button while the the Results tab of CSCView is open, or by clicking the "Save Results to File" box in the Query tab before actually submitting a query. The save files output in Tab Separated Values (TSV) format consist of headers commented with '#', which contain definitions of the CSC columns in the table, as well as data type and format information. There is a row of CSC column names beneath the header, and beneath that are fixed format, tab-separated columns of data values. TSV files are compatible with Vizier and Aladin, and also with the CIAO Data Model.

This file can now be used directly by CIAO tools as well as being imported into SAOImage ds9.

Last Update: 13 Aug 2013 - Rewritten to highlight that CIAO can now read TSV format directly.


Contents


Identify ASCII format

The CXC Datamodel can read and write various ASCII file formats. Many of these can be automatically recognized by first few lines in the file; however, the Tab-Separated-Values (TSV) returned by CSCView needs to be explicitly identified.

To use any CSCView TSV format file users need to specify a kernel option which takes the form:

[opt kernel=text/tsv]

So for example if the output from CSCView was saved as m81.tsv, then users can use this in CIAO using the full Virtual File Syntax would be

"m81.tsv[opt kernel=text/tsv]"

Quotes are likely going to be needed when commands are run on the UNIX command line but should be omitted when a CIAO tool prompts for the value.

[NOTE]
Note

Note: The VOTable (XML) format output is not supported by CIAO tools; however one can use the Python XML parsing routines or other third-party modules to read such files into Python applications. One example is the astropy.io.votable subpackage of Astropy.


Using with CIAO tools

With the virtual file specification discussed above, all the CIAO utilities are now available for use.

Suppose we have done a CSCView search, specifying that we wanted all the results for ObsID 4396, one of the observations of Orion and have saved the TSV format file as 4396.tsv. We can now use the various CIAO tool to inspect and analyze these data.

We can start with simply running dmlist to see what columns we selected:

unix% dmlist orion.tsv"[opt kernel=text/tsv]" cols 
--------------------------------------------------------------------------------
Columns for Table Block orion.tsv
--------------------------------------------------------------------------------
 
ColNo  Name                 Unit        Type             Range
   1   name                              String[20]                          Source name in the format 'CXO Jhhmmss.s +/- ddmmss'
   2   ra                   deg          Real4          -Inf:+Inf            Source position, ICRS right ascension
   3   dec                  deg          Real4          -Inf:+Inf            Source position, ICRS declination
   4   instrument                        String[4]                           Instrument used for the observation; 'ACIS' or 'HRC'
   5   obsid                             Int4           -                    Observation identifier
   6   obi                               Int4           -                    Observation interval number
   7   region_id                         Int4           -                    Source region identifier
   8   match_type                        String[1]                           A Source Observation association with a single Master Source is unique ('u'), wi
   9   cnts_aper_b          counts       Int4           -                    Total counts observed in the modified source region aperture; ACIS broad energy 
  10   flux_aper_b          erg/s/cm^2   Real4          -Inf:+Inf            Aperture-corrected net energy flux inferred from the source region aperture, cal
  11   flux_significance_b               Real4          -Inf:+Inf            Significance of the source determined from the ratio of the source flux to the e
  12   ks_prob_b                         Real4          -Inf:+Inf            Kolmogorov-Smirnov variability probability; ACIS broad energy band
  13   theta                arcmin       Real4          -Inf:+Inf            Source aperture position, off-axis angle

unix% dmlist orion.tsv"[opt kernel=text/tsv]" counts
1040

In the first example we see which columns were selected, including the units and comments, as well as the number of rows (ie number of sources).

If we wanted to check to see how many sources are unambiguous (ie not confused with other sources in other observations) we can simply use the dmfiltering syntax.

unix% dmlist "orion.tsv[opt kernel=text/tsv][match_type=u]" counts
1028   

which shows us that 1028 out of 1040 sources are not confused.

If we wanted to do a population study of the sources, ie a Log(N)/Log(S) analysis, we can use the TSV file with several CIAO tools as shown below

unix% dmstat "orion.tsv[opt kernel=text/tsv][cols flux_significance_b]"
flux_significance_b
    min:	1.9400000572 	      @:	606 
    max:	232.27000427 	      @:	516 
   mean:	19.150874997 
  sigma:	23.648265448 
    sum:	19916.909997 
   good:	1040 
   null:	0 
% dmextract "orion.tsv[opt kernel=text/tsv][bin flux_significance_b=1:250:1]" src_sig.fits op=generic clob+
# dmextract (CIAO 4.5): WARNING: Keywords LIVETIME and EXPOSURE not found in file 'orion.tsv'
		LIVETIME set to default 1.0

unix% sherpa

-----------------------------------------------------
Welcome to Sherpa: CXC's Modeling and Fitting Package
-----------------------------------------------------
CIAO 4.5 Sherpa version 1 Tuesday, December 4, 2012

sherpa-1> load_data("src_sig.fits[cols flux_significance_b,counts]")
sherpa-2> set_source(exp.ee)
sherpa-3> ignore(0,3)
sherpa-4> set_method("moncar")
sherpa-5> fit()
Dataset               = 1
Method                = moncar
Statistic             = chi2gehrels
Initial fit statistic = 686.011
Final fit statistic   = 52.5056 at function evaluation 1550
Data points           = 247
Degrees of freedom    = 244
Probability [Q-value] = 1
Reduced statistic     = 0.215187
Change in statistic   = 633.506
   ee.offset      0.0441532   
   ee.coeff       -0.087585   
   ee.ampl        104.326     
sherpa-6> plot_fit()
sherpa-7> log_scale(XY_AXIS)
sherpa-8> limits(Y_AXIS,0.1,AUTO)
sherpa-9> set_plot_xlabel("Broad Band Source Significance")
sherpa-10> set_plot_ylabel("Number of Sources")
sherpa-11> set_plot_title("CSC Results for ObsId 4396")

Here we have determined the range of the broad band source significance values using dmstat, used that range to extract a histogram of the number of sources with flux significances between 1 and 250 with dmextract and then input the data into Sherpa to fit the profile with an exponential model, as shown in Figure 1.

Figure 1: The result of plot_fit()

[Thumbnail image: The exponential fit describes the overall shape but not the bright end.]

[Version: full-size]

[Print media version: The exponential fit describes the overall shape but not the bright end.]

Figure 1: The result of plot_fit()

Obviously this is not a great fit to these data, but the purpose of this example is to illustrate the types of analysis users can perform.

We can also use the prism file viewer to display the contents of the TSV file, as shown in Figure 2.

Figure 2: Using prism

[Thumbnail image: The contents of the orion.tsv file are displayed.]

[Version: full-size]

[Print media version: The contents of the orion.tsv file are displayed.]

Figure 2: Using prism

unix% prism "orion.tsv[opt kernel=text/tsv]" &

Using with SAOImage ds9

The Tab-Separated-Value file can be import into ds9's catalog tool either on the command line:

unix% ds9 4396/primary/acisf04396N003_cntr_img2.fits.gz -catalog import tsv orion.tsv

or by going to Analysis → Catalog Tool ..., then in the Catalog Tool window goto File → Import → Tab-Separated-Value → orion.tsv. The result is shown in Figure 3

Figure 3: Using ds9

[Thumbnail image: ds9 displaying the image with sources overlain and a tabular view.]

[Version: full-size]

[Print media version: ds9 displaying the image with sources overlain and a tabular view.]

Figure 3: Using ds9

The catalog tool can then be used to select sources, create plots of source properties and participate in SAMP enabled analysis sessions with other SAMP aware applications.


History

06 Oct 2008 original version
01 Jun 2009 updated to include recent changes to CSCview
11 Aug 2010 updated for CSCview version 1.1: conversion script updated to accept TSV format save files in place of RDB format; 'cone_distance' catalog parameter has been changed to 'separation'.
13 Aug 2013 Rewritten to highlight that CIAO can now read TSV format directly.