Search https://cxc.harvard.edu/ciao/
Contact the CXC HelpDesk
Last modified: 13 Jan 2022

URL: https://cxc.cfa.harvard.edu/ciao/threads/dm_intro/
/ ciao / threads / dm_intro /
Return to Threads Page: Top | All | Intro

Making Images & Filtering Data: an Introduction to the Chandra Data Model

CIAO 4.17 Science Threads

Overview

Synopsis:

The CIAO Data Model is a versatile interface used to examine standard format datafiles (FITS, IMH); it also allows powerful filtering of datafiles. The filtered output can either be saved or input directly to an analysis task as a "virtual file." A brief introduction to the Data Model interface and examples on how to use the four core tools are provided here.

Related Links:

Last Update: 13 Jan 2022 - Reviewed for CIAO 4.14. No changes.

Contents

Get Started

For illustration, this thread utilizes the ObsID 1843 (ACIS-I, G21.5-0.9) data that was downloaded in the How to Download Chandra Data from the Archive thread.

If this is your first time using CIAO, please read the Starting CIAO thread to ensure that your environment is configured properly.

Data Model Tools

The four most frequently used (aka "core") tools are:

  • dmlist - List contents or structure of a file
  • dmcopy - Filter and bin tables and images
  • dmextract - Make a histogram table file (e.g. PHA or PI file) for spectral fitting or a radial profile for spatial fitting.
  • dmgti - Define custom good time intervals (GTIs)

In addition, there are a number of more specialized tools (partial list):

  • dmappend - Append multiple blocks/extensions to an existing output file
  • dmarfadd - Add multiple ARF files together, weighting by exposure time
  • dmcontour - Make contour regions from a 2-D image
  • dmcoords - Convert between Chandra instrumental coordinate systems
  • dmdiff - Compare values in two FITS files
  • dmfilth - Replace pixel values in source regions of an image with interpolated values
  • dmgroup - Group a specified column in a table
  • dmgroupreg - Translates ds9 regions and groups to CIAO format
  • dmhedit - Edit Data Model file headers
  • dmhistory - Extracts processing history from file header
  • dmimg2jpg - Make a color JPEG image from three image files
  • dmimgcalc - Perform arithmetic on images
  • dmimghist - Make histogram of values in a 2-D image
  • dmimgpick - Maps image values to rows in a table
  • dmimgthresh - Set low exposure regions of an image to zero
  • dmjoin - Join two files whose sampling is different
  • dmkeypar - Retrieve information about a keyword from an input file
  • dmmakepar - Write header keywords to a parameter file
  • dmmakereg - Create a FITS region file from an ASCII region description
  • dmmerge - Merge two or more compatible tables into one
  • dmpaste - Add new columns to a table
  • dmreadpar - Add parameters from a .par file to a file header
  • dmregrid - Rebin a stack of two-dimensional images
  • dmsort - Sort a table block on a given column
  • dmstat - Compute standard statistics for the column in a table or image
  • dmtcalc - Define new table columns as functions of old ones
  • dmtype2split - Create a type 1 file for specified rows of a type 2 file

Running Data Model Tools

Tools are usually executed from the command-line prompt: 

unix% dmcopy "acisf01843N004_evt2.fits[EVENTS]" acis_events.fits

Alternatively, the parameters can be input to the parameter file before the tool is run: 

unix% punlearn dmcopy
unix% pset dmcopy infile=acisf01843N004_evt2.fits
unix% pset dmcopy outfile=acis_events.fits
unix% dmcopy
Input dataset/block specification (acisf01843N004_evt2.fits): 
Output dataset name (acis_events.fits):

The parameter file that was used can be examined with plist dmcopy. Note that the Data Model automatically uses the "interesting" block (e.g. EVENTS for an event file, SPECTRUM for a PHA file) when one is not provided, so we chose not to specify it in the second example.

Virtual Files

The Data Model offers an easy and powerful means of filtering data. As mentioned, the filtered file can be directly input to an analysis task without writing it to disk first; this is known as a "virtual file." The virtual file, which can also be referred to as a subspace, is simply a means of defining a subset of interest in the dataset.

The basic syntax of a virtual file has four arguments: 

filename[block][filter][columns/binning][newblock]
  • block - the section of the file to use
  • filter - the filter to be applied
  • columns/binning - either the columns from a table to be included in an output table or the binning
  • newblock (optional) - the name for the new block in the output file, default is the block used from the input file

See ahelp dmfiltering for more information on filtering syntax.

Examples of virtual files:

  • Select the first three columns of the EVENTS block by number: 
    acisf01843N004_evt2.fits[EVENTS][cols #1,#2,#3]
    or by name: 
    acisf01843N004_evt2.fits[cols time,ccd_id,node_id]
  • Select certain rows from a FITS file: 
    acisf01843N004_evt2.fits[#row=1:4]
  • Bin an event file to create a PI spectrum (using dmextract): 
    acisf01843N004_evt2.fits[EVENTS][bin pi=1:1024:1]
    or an image (using dmcopy): 
    acisf01843N004_evt2.fits[EVENTS][bin x=3200:4800:4,y=3200:4800:4]
    More information on binning data is available from ahelp dmbinning.

The Core Tools

These examples show just a few of the uses for each tool - see the ahelp files for more options and examples.

1. dmlist

FITS files generally contain several different blocks, each containing multi-dimensional data. The dmlist tool allows the user to inspect all or part of a data file by column, row, etc.

  • To list the blocks in a file: 
    unix% dmlist acisf01843N004_evt2.fits blocks
 
--------------------------------------------------------------------------------
Dataset: acisf01843N004_evt2.fits
--------------------------------------------------------------------------------
 
     Block Name                          Type         Dimensions
--------------------------------------------------------------------------------
Block    1: PRIMARY                        Null
Block    2: EVENTS                         Table        15 cols x 490795   rows
Block    3: GTI7                           Table         2 cols x 1        rows
Block    4: GTI6                           Table         2 cols x 1        rows
Block    5: GTI3                           Table         2 cols x 2        rows
Block    6: GTI2                           Table         2 cols x 1        rows
Block    7: GTI1                           Table         2 cols x 1        rows
Block    8: GTI0                           Table         2 cols x 1        rows
  • To list the columns of the events block in a file, use the cols option (note the required quotes): 
    unix% dmlist "acisf01843N004_evt2.fits[events]" cols

--------------------------------------------------------------------------------
Columns for Table Block EVENTS
--------------------------------------------------------------------------------
 
ColNo  Name                 Unit        Type             Range
   1   time                 s            Real8          84270902.1685000062: 84280645.8188659996 S/C TT corresponding to mid-exposure
   2   ccd_id                            Int2           0:9                  CCD reporting event
   3   node_id                           Int2           0:3                  CCD serial readout amplifier node
   4   expno                             Int4           0:2147483647         Exposure number of CCD frame containing event
   5   chip(chipx,chipy)    pixel        Int2           1:1024               Chip coords
   6   tdet(tdetx,tdety)    pixel        Int2           1:8192               ACIS tiled detector coordinates
   7   det(detx,dety)       pixel        Real4          0.50:     8192.50    ACIS detector coordinates
   8   sky(x,y)             pixel        Real4          0.50:     8192.50    sky coordinates
   9   pha                  adu          Int4           0:36855              total pulse height of event
  10   pha_ro               adu          Int4           0:36855              total read-out pulse height of event
  11   energy               eV           Real4          0:  1000000.0        nominal energy of event (eV)
  12   pi                   chan         Int4           1:1024               pulse invariant energy of event
  13   fltgrade                          Int2           0:255                event grade, flight system
  14   grade                             Int2           0:7                  binned event grade
  15   status[4]                         Bit(4)                              event status bits

--------------------------------------------------------------------------------
World Coord Transforms for Columns in Table Block EVENTS
--------------------------------------------------------------------------------

ColNo    Name
5:    CPC(CPCX) = (+0)[mm] +(+0.0240)* (chip(chipx)-(+0.50))
         (CPCY)   (+0)      (+0.0240)  (    (chipy) (+0.50))
7:    MSC(PHI  ) = (+0)[deg] +TAN-P[(+0.000136667)* (det(detx)-(+4096.50))]
         (THETA)   (+0)             (+0.000136667)  (   (dety) (+4096.50))
8:    EQPOS(RA ) = (+278.0479)[deg] +TAN[(-0.000136667)* (sky(x)-(+4096.50))]
           (DEC)   (-10.5704 )           (+0.000136667)  (   (y) (+4096.50))
  • To list the first 5 events in a file, use the data option and rows parameter together: 
    unix% dmlist acisf01843N004_evt2.fits data row=1:5
 
--------------------------------------------------------------------------------
Data for Table Block EVENTS
--------------------------------------------------------------------------------
 
ROW    time                 ccd_id node_id expno
	chip(chipx,chipy) tdet(tdetx,tdety) det(detx,dety)           sky(x,y)
	pha        energy       pi     fltgrade grade status[4]


 
     1  84272491.7915800959    6    0          4    (162,185)  (3037,1887) (     2999.5632324219,     6754.0478515625) (     1516.4484863281,     2857.8146972656)       1413       1360        5585.42968750    383      0    0  00000000000000000000000000000000
     2  84272491.7915800959    6    1          4    (364,292)  (3239,1994) (     3201.9252929688,     6646.9384765625) (     1612.3209228516,     3065.7360839844)       3259       3095        12443.2968750    853    104    6  00000000000000000000000000000000
     3  84272491.7915800959    6    2          4    (521,626)  (3396,2328) (     3358.3686523438,     6312.3784179688) (     1937.8156738281,     3240.25781250)       1100        898      4318.5424804688    296     16    4    00000000000000000000000000000000
     4  84272491.8326200992    7    2          4     (520,19)  (4437,1721) (     4400.3793945312,     6917.9926757812) (     1276.0732421875,     4247.5571289062)        377        362      1778.7789306641    122     16    4  00000000000000000000000000000000
     5  84272491.8326200992    7    0          4    (238,116)  (4155,1818) (     4118.5268554688,     6822.0991210938) (     1387.2509765625,     3971.3754882812)       2755       2698     12297.6308593750    843     10    6  00000000000000000000000000000000
  • To list the file header: 
    unix% dmlist acisf01843N004_evt2.fits header
 
--------------------------------------------------------------------------------
Header keys for block EVENTS
--------------------------------------------------------------------------------
 
 --  COMMENT                This FITS file may contain long string keyword values that are
 --  COMMENT                continued over multiple keywords.  The HEASARC convention uses the &
 --  COMMENT                character at the end of each substring which is then continued
 --  COMMENT                on the next keyword which has the name CONTINUE.
0001 ASOLFILE             pcadf084271087N004_asol1.fits  String
0002 THRFILE              acisD1996-11-01evtspltN0002.fits String
 --  COMMENT                character at the end of each substring which is then continued
 --  COMMENT                on the next keyword which has the name CONTINUE.
0003 ORIGIN               ASC                            String       Source of FITS file
0004 CREATOR              cxc - Version DS 7.6.10        String       tool that created this output
0005 ASCDSVER             7.6.10                         String       ASCDS version number
0006 MJD_OBS                  51789.3576639870           Real8        Modified Julian date of observation
0007 DS_IDENT             ADS/Sa.CXO#obs/01843           String       dataset identifier
0008 TLMVER               P008                           String       Telemetry revision number (IP&CL)
0009 REVISION             2                              Int4         Processing version of data
0010 CHECKSUM             8FqAAEn71EnA8En7               String       HDU checksum updated 2007-06-06T15:05:05
0011 DATASUM              773790139                      String       data unit checksum updated 2007-06-06T15:03:28
0012 CONTENT              EVT2                           String       What data product
...
0022 DATE                 2007-06-06T15:03:09            String       Date and time of file creation
0023 DATE-OBS             2000-09-02T08:35:02            String       Observation start date
0024 OBS_MODE             POINTING                       String       Observation mode
0025 DATE-END             2000-09-02T11:17:25            String       Observation end date
0026 TIMESYS              TT                             String       Time system
0027 MJDREF                   50814.0                    Real8        MJD zero point for times

.
. (etc.)
.
  • All of the above examples may be combined into one command that yields the greatest possible amount of information: 
    unix% dmlist acisf01843N004_evt2.fits full
    which is shorthand for: 
    unix% dmlist "acisf01843N004_evt2.fits[events]" blocks,header,cols,subspace,data
  • To save the output to a text file: 
    unix% dmlist acisf01843N004_evt2.fits blocks outfile=blocks.txt
unix% cat blocks.txt
 
--------------------------------------------------------------------------------
Dataset: acisf01843N004_evt2.fits
--------------------------------------------------------------------------------

     Block Name                          Type         Dimensions
--------------------------------------------------------------------------------
Block    1: PRIMARY                        Null
Block    2: EVENTS                         Table        15 cols x 490795   rows
Block    3: GTI7                           Table         2 cols x 1        rows
Block    4: GTI6                           Table         2 cols x 1        rows
Block    5: GTI3                           Table         2 cols x 2        rows
Block    6: GTI2                           Table         2 cols x 1        rows
Block    7: GTI1                           Table         2 cols x 1        rows
Block    8: GTI0                           Table         2 cols x 1        rows

2. dmcopy

The dmcopy tool can be used to manipulate data. Unlike dmlist, which produces text output, this tool produces a new data file in one of the supported formats.

  • Copy the events from the central region of a file into a new FITS file: 
    unix% dmcopy "acisf01843N004_evt2.fits[events][x=3600:4000,y=3800:4200]" \
      acis_center.fits
  • Generate a blocked image:

    A convenient way to display the full field of view of an event file is to bin it into an image. In this example, the full range of sky coordinates is blocked by a factor of 4: 

    unix% dmcopy "acisf01843N004_evt2.fits[events][bin x=::4,y=::4][IMAGE]" \
      acis_img.fits

    Since the image file is a binned version of the event file, only the selected variable (i.e. sky coordinates) is retained. All other information (photon arrival times, energy, etc.) is lost. 

    unix% dmlist acis_img.fits cols

--------------------------------------------------------------------------------
Columns for Image Block IMAGE
--------------------------------------------------------------------------------

ColNo  Name                 Unit        Type             Range
   1   IMAGE[2048,2048]                  Int2(2048x2048) -

--------------------------------------------------------------------------------
Physical Axis Transforms for Image Block IMAGE
--------------------------------------------------------------------------------

Group# Axis#
   1   1,2    sky(x) = (+0.50) +(+4.0)* ((#1)-(+0.50))
                 (y)   (+0.50)  (+4.0)  ((#2) (+0.50))

--------------------------------------------------------------------------------
World Coordinate Axis Transforms for Image Block IMAGE
--------------------------------------------------------------------------------

Group# Axis#
   1   1,2    EQPOS(RA ) = (+278.0479)[deg] +TAN[(-0.000136667)* (sky(x)-(+4096.50))]
                   (DEC)   (-10.5704 )           (+0.000136667)  (   (y) (+4096.50))
  • Filter using a region defined in sky coordinates: 
    unix% dmcopy "acisf01843N004_evt2.fits[sky=ellipse(1628,4116,92,160,0)]" \
      source.fits

    Note that the name of the column in the filter must match the name within the file itself; in Chandra data, "sky" is shorthand for "(x, y)."

3. dmextract

The dmextract tool is similar to dmcopy. It is used to bin tables into images, but writes the binned data to a table instead of creating an image.

  • Make a histogram from a table column: 
    unix% dmextract "acisf01843N004_evt2.fits[bin pha=1:2048:2]" histogram.pha
  • Extract the PI spectrum of a source in sky coordinates: 
    unix% dmextract "acisf01843N004_evt2.fits[sky=region(ds9.reg)][bin pi]" \
      spectrum.fits
    where 
    unix% more ds9.reg
# Region file format: CIAO version 1.0
ellipse(1628,4116,92,172,0)

    More examples of region syntax are available from ahelp dmregions.

  • Use a stack input:

    This tool can take a stack as input, output, or both. To combine both of the previous examples into one command: 

    unix% more infiles.lis
acisf01843N004_evt2.fits[bin pha=1:2048:2]
acisf01843N004_evt2.fits[sky=ellipse(1628,4116,92,172,0)][bin pi]

unix% more outfiles.lis
histogram_lis.pha
spectrum_lis.fits

unix% dmextract infile=@infiles.lis outfile=@outfiles.lis

4. dmgti

Pipeline processing of Chandra data uses dmgti to calculate good time intervals (GTIs) based on input MTL files. Although this is what the tool was designed for, dmgti will work equally well on any GTI filter as long as the first column of the input file is time.

A look at a lightcurve file (created with dmextract and these parameters) shows background flares where the count rate reaches values much higher than the mean (several rows were omitted for the sake of space): 

unix% dmextract \
      "acisf01843N004_evt2.fits[(x,y)=field()-ellipse(1628,4116,100,140,0)][bin time=::3.24104]" \
      bkg_lc.fits opt=ltc1

unix% dmlist "bkg_lc.fits[cols time,count_rate,stat_err]" data 
 
--------------------------------------------------------------------------------
Data for Table Block LIGHTCURVE
--------------------------------------------------------------------------------
 
ROW    TIME                 COUNT_RATE           STAT_ERR            
 
     1  84270903.7890200019                    0                    0
     2  84270907.0300600231                    0                    0
     3  84270910.2711000144                    0                    0
.
.
   490  84272488.6575800180                    0                    0
   491  84272491.8986200094         6.5624999873         4.5825756950
   492  84272495.1396600008         10.312499980         5.7445626465
   493  84272498.3807000220         7.1874999861         4.7958315233
.
.
   740  84273298.9175800085        95.9374998141        17.5214154679
   741  84273302.1586199999        94.3749998171        17.3781471970
   742  84273305.3996600211        87.1874998311        16.7032930885
   743  84273308.6407000124       123.7499997602        19.8997487421
   744  84273311.8817400038        86.8749998317        16.6733320005
   745  84273315.1227799952        98.1249998099        17.7200451467
.
.
  3005  84280639.8731800020                    0                    0
  3006  84280643.1142200232                    0                    0
  3007  84280646.3552600145                    0                    0

The tool dmsort is used to sort the count_rate column in descending order, to show more clearly the times of high background: 

unix% dmsort bkg_lc.fits sorted_bkg_lc.fits keys=-count_rate

unix% dmlist "sorted_bkg_lc.fits[cols time,count_rate,stat_err]" data | less
 
--------------------------------------------------------------------------------
Data for Table Block LIGHTCURVE
--------------------------------------------------------------------------------
 
ROW    TIME                 COUNT_RATE           STAT_ERR            
 
     1  84274245.3012599945       185.6249996403        24.3721152139
     2  84274261.5064600110       184.0624996434        24.2693221990
     3  84274248.5423000157       175.9374996591        23.7276210354
     4  84274242.0602200031       166.5624996773        23.0867927612
.
.
   537  84280341.6975000203        77.8124998492        15.7797338381
   538  84280114.8246999979        77.8124998492        15.7797338381
   539  84280108.3426200151        77.8124998492        15.7797338381
   540  84280095.3784600198        77.8124998492        15.7797338381
.
.
  2324  84272556.7194200158         8.1249999843         5.0990195136
  2325  84272537.2731800079         8.1249999843         5.0990195136
  2326  84273039.6343800128         7.8124999849                  5.0
.
.
  3005  84270910.2711000144                    0                    0
  3006  84270907.0300600231                    0                    0
  3007  84270903.7890200019                    0                    0

If we wanted to exclude the times when the count rate was less than 40.0 and greater than 100.0: 

unix% dmgti infile=bkg_lc.fits outfile=acis_gti.fits \
      userlimit="(count_rate>40.0)&&(count_rate<100.0)"

unix% dmlist "acis_gti.fits[gti]" data
--------------------------------------------------------------------------------
Data for Table Block GTI
--------------------------------------------------------------------------------
 
ROW    START                STOP                
 
     1  84273206.5479400158  84273209.7889800072
     2  84273213.0300200284  84273232.4762600064
     3  84273242.1993800104  84273307.0201800168
.
.
    84  84279860.4030600190  84279879.8493000269
    85  84279883.0903400183  84279892.8134600222
    86  84279896.0545000136  84280443.7902600169

The event file may now be filtered on the newly calculated GTI: 

unix% dmcopy "acisf01843N004_evt2.fits[@acis_gti.fits]" acis_filtered_evt2.fits

Doing a dmlist on each of the files shows how the number of events and the GTI blocks are affected by this filter: 

unix% dmlist acisf01843N004_evt2.fits blocks
 
--------------------------------------------------------------------------------
Dataset: acisf01843N004_evt2.fits
--------------------------------------------------------------------------------
 
     Block Name                          Type         Dimensions
--------------------------------------------------------------------------------
Block    1: PRIMARY                        Null
Block    2: EVENTS                         Table        15 cols x 490795   rows
Block    3: GTI7                           Table         2 cols x 1        rows
Block    4: GTI6                           Table         2 cols x 1        rows
Block    5: GTI3                           Table         2 cols x 2        rows
Block    6: GTI2                           Table         2 cols x 1        rows
Block    7: GTI1                           Table         2 cols x 1        rows
Block    8: GTI0                           Table         2 cols x 1        rows

unix% dmlist acis_filtered_evt2.fits blocks
 
--------------------------------------------------------------------------------
Dataset: acis_filtered_evt2.fits
--------------------------------------------------------------------------------
 
     Block Name                          Type         Dimensions
--------------------------------------------------------------------------------
Block    1: PRIMARY                        Null
Block    2: EVENTS                         Table        15 cols x 416088   rows
Block    3: GTI7                           Table         2 cols x 86       rows
Block    4: GTI6                           Table         2 cols x 86       rows
Block    5: GTI3                           Table         2 cols x 87       rows
Block    6: GTI2                           Table         2 cols x 86       rows
Block    7: GTI1                           Table         2 cols x 86       rows
Block    8: GTI0                           Table         2 cols x 86       rows


Parameters for /home/username/cxcds_param/dmcopy.par


        infile = acisf01843N004_evt2.fits Input dataset/block specification
       outfile = acis_events.fits Output dataset name
       (kernel = default)         Output file format type
       (option = )                Option - force output type
      (verbose = 0)               Debug Level
      (clobber = no)              Clobber existing file
         (mode = ql)              
    

Parameters for /home/username/cxcds_param/dmextract.par


        infile = acisf01843N004_evt2.fits[(x,y)=field()-ellipse(1628,4116,100,140,0)][bin time=::3.24104] Input event file 
       outfile = bkg_lc.fits      Enter output file name
          (bkg = )                Background region file or fixed background (counts/pixel/s) subtraction
        (error = gaussian)        Method for error determination(gaussian|gehrels|<variance file>)
     (bkgerror = gaussian)        Method for background error determination(gaussian|gehrels|<variance file>)
      (bkgnorm = 1.0)               Background normalization
          (exp = )                Exposure map image file
       (bkgexp = )                Background exposure map image file
      (sys_err = 0)               Fixed systematic error value for SYS_ERR keyword
          (opt = ltc1)            Output file type 
     (defaults = ${ASCDS_CALIB}/cxo.mdb -> /soft/ciao/data/cxo.mdb) Instrument defaults file
         (wmap = )                WMAP filter/binning (e.g. det=8 or default)
      (clobber = no)              OK to overwrite existing output file(s)?
      (verbose = 0)               Verbosity level
         (mode = ql)

History

23 Dec 2004 reviewed for CIAO 3.2: no changes
01 Dec 2005 updated for CIAO 3.3: default value of dmextract error and bkgerror parameters is "gaussian"; updates to screen output in dmgti example
01 Dec 2006 updated for CIAO 3.4: updates to screen output in dmgti example
09 Jan 2008 updated for CIAO 4.0: filenames and screen output updated for reprocessed data (version N002 event file)
02 Jan 2009 reviewed for CIAO 4.1: no changes
25 Jan 2010 reviewed for CIAO 4.2: no changes
11 Jan 2011 reviewed for CIAO 4.3: no changes
11 Apr 2011 changed title from "Introduction to the Data Model" to "Making Images & Filtering Data with the Data Model"
03 Jan 2012 reviewed for CIAO 4.4: no changes
03 Dec 2012 Review for CIAO 4.5; update file versions
26 Apr 2013 changed title from "Making Images & Filtering Data with the Data Model" to "Making Images & Filtering Data: an Introduction to the Chandra Data Model"
25 Nov 2013 Review for CIAO 4.6. No changes.
17 Dec 2014 Reviewed for CIAO 4.7; no changes.
03 Jan 2017 Reviewed for CIAO 4.9; minor edits only.
02 Nov 2020 Remove reference to peg and the analysis menu which have been withdrawn from CIAO 4.13.
18 Oct 2021 Updated for Repro-5 version number.
13 Jan 2022 Reviewed for CIAO 4.14. No changes.
Return to Threads Page: Top | All | Intro
/ ciao / threads / dm_intro /