Last modified: 12 January 2023

URL: https://cxc.cfa.harvard.edu/csc/threads/convert/

Using a CSC Save File in CIAO

CSC Threads


Overview

Synopsis:

The results of a catalog search may be written to a file by selecting the Save toolbar button while the the Results tab of CSCView is open, or by clicking the Save Results to File box in the Query tab before actually submitting a query. The save files output in Tab-Separated-Values (TSV) format consist of headers commented with '#', which contain definitions of the CSC columns in the table, as well as data type and format information. There is a row of CSC column names beneath the header, and beneath that are fixed format, tab-separated columns of data values. TSV files are compatible with Vizier and Aladin, and also with the CIAO Data Model.

This file can now be used directly by CIAO tools as well as being imported into SAOImage DS9. DS9 requires RA and Dec coordinates to be in colon-separated sexagesimal format or decimal degrees while CSCView sexagesimal formatting is whitespace-separated. To change the ouput format in CSCView, go to EditPreferences:Output Coordinate FormatDecimal

Last Update: 12 Jan 2023 - Change Sherpa and prism sections to show matplotlib (rather than ChIPS) and use the DS9 version of prism respectively.


Contents


Identify ASCII format

The CXC Datamodel can read and write various ASCII file formats. Many of these can be automatically recognized by first few lines in the file; however, the Tab-Separated-Values (TSV) returned by CSCView needs to be explicitly identified.

To use any CSCView TSV format file users need to specify a kernel option which takes the form:

[opt kernel=text/tsv]

So for example if the output from CSCView was saved as m81.tsv, then users can use this in CIAO using the full Virtual File Syntax would be

"m81.tsv[opt kernel=text/tsv]"

Quotes are likely going to be needed when commands are run on the UNIX command line but should be omitted when a CIAO tool prompts for the value.

[NOTE]
Note

The VOTable (XML) format output is not supported by CIAO tools; however, one can use the Python XML parsing routines or other third-party modules to read such files into Python applications and can be imported by DS9. One example is the astropy.io.votable subpackage of Astropy.


Using with CIAO tools

With the virtual file specification discussed above, all the CIAO utilities are now available for use.

Suppose we would like to select all CSC 2.0 master sources from the Orion Nebula Cluster as observed in the ACIS broad band (0.5-7.0 keV). Suppose further that we have used the CSC2.0 WWT application to identify the particular ACIS-based master source stack ID acisfJ0535165m052323_001, which includes all the sources we want to consider. If we search for all master sources within this stack using CSCView, we get a table of 1539 unique sources in this region, which we have saved in the tab-separated-value file orion.tsv. We can now use the various CIAO tools to inspect and analyze these data.

We can start with simply running dmlist to see the number of rows:

unix% dmlist "orion.tsv[opt kernel=text/tsv]" counts
1539

and the available columns:

unix% dmlist "orion.tsv[opt kernel=text/tsv]" cols

--------------------------------------------------------------------------------
Columns for Table Block orion.tsv
--------------------------------------------------------------------------------
 
ColNo  Name                 Unit        Type             Range
   1   name                              String[22]                          Source name in the format '2CXO Jhhmmss.s{+|-}ddmmss[X]'
   2   match_type                        String[1]                           Type of match between master source and stacked observation detection; u = uniqu
   3   ra                   deg          Real4          -Inf:+Inf            Source position, ICRS right ascension
   4   dec                  deg          Real4          -Inf:+Inf            Source position, ICRS declination
   5   err_ellipse_r0       arcsec       Real4          -Inf:+Inf            Major radius of the 95% confidence level position error ellipse
   6   err_ellipse_r1       arcsec       Real4          -Inf:+Inf            Minor radius of the 95% confidence level position error ellipse
   7   err_ellipse_ang      deg          Real4          -Inf:+Inf            Position angle (ref. local true north) of the major axis of the 95% confidence l
   8   significance                      Real4          -Inf:+Inf            Highest flux significance (S/N) across all stacked observations and energy bands
   9   likelihood_class                  String[20]                          Highest detection likelihood classification across all stacked observations and 
  10   conf_flag                         String[5]                           Source may be confused (source regions overlap in one or more contributing stack
  11   sat_src_flag                      String[5]                           Source is saturated in all observations; source properties are unreliable
  12   streak_src_flag                   String[5]                           Source is located on an ACIS readout streak in all observations; source properti
  13   src_cnts_aper_b      count        Real8          -Inf:+Inf            Aperture-corrected detection net counts inferred from the source region aperture
  14   src_cnts_aper_lolim_b count        Real8          -Inf:+Inf            Aperture-corrected detection net counts inferred from the source region aperture
  15   src_cnts_aper_hilim_b count        Real8          -Inf:+Inf            Aperture-corrected detection net counts inferred from the source region aperture
  16   flux_aper_b          erg/s/cm^2   Real4          -Inf:+Inf            Aperture-corrected net energy flux inferred from the source region aperture, bes
  17   flux_aper_lolim_b    erg/s/cm^2   Real4          -Inf:+Inf            Aperture-corrected net energy flux inferred from the source region aperture, bes
  18   flux_aper_hilim_b    erg/s/cm^2   Real4          -Inf:+Inf            Aperture-corrected net energy flux inferred from the source region aperture, bes
  19   flux_significance_b               Real4          -Inf:+Inf            Significance of the stacked-observation detection determined from the ratio of t

If we wanted to check to see how many sources are unambiguous (i.e. not confused with other sources in the stack) we can simply use the dmfiltering syntax.

unix% dmlist "orion.tsv[opt kernel=text/tsv][match_type=u]" counts
1515    

which shows us that 1515 out of 1539 sources are not confused.

If we wanted to do a population study of the sources, i.e. a \(\log{(N)}/\log{(S)}\) analysis, we can use the TSV file with several CIAO tools as shown below:

unix% dmstat "orion.tsv[opt kernel=text/tsv][cols flux_significance_b]"
flux_significance_b
    min:	0 	      @:	10 
    max:	460.8999939 	      @:	771 
   mean:	28.99595175 
  sigma:	39.520637864 
    sum:	43261.960011 
   good:	1492 
   null:	47 

unix% dmextract "orion.tsv[opt kernel=text/tsv][bin flux_significance_b=1:146:1]" src_sig.fits op=generic clob+
# dmextract (CIAO 4.16.0): WARNING: Keywords LIVETIME and EXPOSURE not found in file 'orion.tsv'
		LIVETIME set to default 1.0

Here we have determined the range of the broad band source significance values using dmstat, and used that range to extract a histogram of the number of sources with flux significances between 1 and 461 with dmextract.

As an example of what can be done, we use Sherpa to fit the profile with an exponential model:

sherpa In [1]: load_data("src_sig.fits[cols flux_significance_b,counts]")

sherpa In [2]: ignore(hi=3)
dataset 1: 1.5:145.5 -> 3.5:145.5 x

sherpa In [3]: set_source(exp.emodel)

sherpa In [4]: fit()
Dataset               = 1
Method                = levmar
Statistic             = chi2gehrels
Initial fit statistic = 873.193
Final fit statistic   = 94.7485 at function evaluation 80
Data points           = 143
Degrees of freedom    = 140
Probability [Q-value] = 0.998761
Reduced statistic     = 0.676775
Change in statistic   = 778.444
   emodel.offset   19.6592      +/- 0.683624    
   emodel.coeff   -0.0525964   +/- 0.00272563  
   emodel.ampl    25.854       +/- 0           

sherpa In [5]: plot_fit(xlog=True, ylog=True)
   
sherpa In [6]: plt.xlim(3, 120)
(3, 120)

sherpa In [7]: plt.ylim(0.1, None)
(0.1, 107.05022000970477)

sherpa In [8]: plt.xlabel("Broad Band Source Significance")
Text(0.5, 18.922222222222217, 'Broad Band Source Significance')

sherpa In [9]: plt.ylabel("Number of Sources")
Text(31.529033135308154, 0.5, 'Number of Sources')

sherpa In [10]: plt.title("CSC ACIS Stacked Results: Orion Nebula Cluster")
Text(0.5, 1.0, 'CSC ACIS Stacked Results: Orion Nebula Cluster')

sherpa In [11]: plt.savefig("plot.png")

The result is shown in Figure 1.

Figure 1: The result of plot_fit()

[The exponential fit describes the overall shape but not the bright end.]
[Print media version: The exponential fit describes the overall shape but not the bright end.]

Figure 1: The result of plot_fit()

Obviously this is not a great fit to these data, but the purpose of this example is to illustrate the types of analysis users can perform.

We can also use the table-viewer capability of DS9, named prism, to view the contents:

unix% ds9 -prism import tsv orion.tsv &

Figure 2: Using prism (via DS9)

[Thumbnail image: The contents of the orion.tsv file are displayed.]

[Version: full-size]

[Print media version: The contents of the orion.tsv file are displayed.]

Figure 2: Using prism (via DS9)


Using with SAOImage DS9

Returning to our CSCView search, we can select one or more of the sources under the Search Results tab to find and download CSC 2.0 data products, and in this case we choose a Stack Events Image including all of these sources, namely acisfJ0535165m052323_001N020_b_img3.fits. We can then import our orion.tsv table file into DS9's catalog tool either on the command line:

unix% ds9 acisfJ0535165m052323_001N020_b_img3.fits -catalog import tsv orion.tsv &

or by going to AnalysisCatalog Tool ..., then in the Catalog Tool window go to FileImportTab-Separated-Valueorion.tsv. The result is shown in Figure 3

Figure 3: Using DS9

[Thumbnail image: DS9 displaying the image with sources overlain and a tabular view.]

[Version: full-size]

[Print media version: DS9 displaying the image with sources overlain and a tabular view.]

Figure 3: Using DS9

DS9 displaying the image with sources overlain and a tabular view.

The catalog tool can then be used to select sources, create plots of source properties and participate in SAMP enabled analysis sessions with other SAMP aware applications.

A CIAO-compatible format REGION file may be generated from orion.tsv using some strategy with dmlist and an awk command as follows:

unix% dmlist "cscresults.tsv[opt kernel=text/tsv][cols RA,DEC]" data,clean | tail -1539 | awk '{print "circle("$1":"$2":"$3","$4":"$5":"$6",2.5\")"}' > orion_all.reg

where dmlist has been used to dump the RA and DEC columns in a clean format, tail has been used to eliminate the column header text line, and awk has been employed to put the RA and Dec into CIAO-compatible region file format to specify circles centered on each RA and Dec with 2.50 arcsecond radii.


History

06 Oct 2008 original version
01 Jun 2009 updated to include recent changes to CSCview
11 Aug 2010 updated for CSCview version 1.1: conversion script updated to accept TSV format save files in place of RDB format; 'cone_distance' catalog parameter has been changed to 'separation'.
13 Aug 2013 Rewritten to highlight that CIAO can now read TSV format directly.
11 Apr 2019 Updated with CSC 2.0 results.
09 Oct 2019 Figure and text updates.
12 Jan 2023 Change Sherpa and prism sections to show matplotlib (rather than ChIPS) and use the DS9 version of prism respectively.