Jump to: Description · Examples · Changes in CIAO 4.5 · Bugs · See Also

AHELP for CIAO 4.5

crates

Context: crates

Synopsis

CRATES - A high-level I/O interface for Python

CRATES is a software package developed by the Chandra X-Ray Center (CXC) to provide a high level I/O interface for use by Python. The CIAO applications ChIPS and Sherpa use CRATES to read input data for use in plotting and fitting and to write output data files. It supports reading and writing both FITS and ASCII files.

The classes and methods allow a user to easily access and manipulate general data files. There are specialized classes for specific data products (e.g. PHA Type I and II and RMF). CRATES also provides convenient access to metadata information, such as keywords and WCS transforms, which may be associated with a table column or image.

It can be used from Python by saying

from pycrates import *

The Sherpa and ChIPS environments automatically load CRATES at startup. There are help files for each of the CRATES commands; a full list can be seen with the following command

unix% ahelp -c crates

Example 1

chips> cr = read_file("evt2.fits")
chips> print(get_keyval(cr, "OBJECT"))
ELAIS-S1 A
chips> print(get_keyval(cr, "EXPOSURE") / 1000)
31.7432921583

Here we get the values of the OBJECT and EXPOSURE keywords from the header of evt2.fits. The return value from get_keyval() uses the appropriate Python type, so OBJECT is a string and the exposure time is a number.

Example 2

>>> from pycrates import *
>>> cr = read_file("evt2.fits[ccd_id=7,energy=300:7000]")
>>> e = copy_colvals(cr, "energy")
>>> emin = e.min() / 1000
>>> emax = e.max() / 1000
>>> emean = e.mean() / 1000
>>> print("Mean E = {0} min= {1} max= {2}".format(emean, emin, emax))
Mean E = 3.10554851929 min= 0.300000335693 max= 6.99996289063

Here we read in the energy column from an event file and calculate the minimum, maximum and mean energy in keV.

Example 3

>>> import numpy as np
>>> import pycrates as *
>>> import pychips.all as pychips
>>> cr = read_file("0.5-7.0_bin2.img")
>>> img = copy_piximgvals(cr)
>>> pychips.add_image(np.log10(img))
Warning: divide by zero encountered in log10
>>> pychips.add_image("2.3_bin2.expmap", ["wcs", "logical", "alpha",
[0.2,0.2]])

Here we display the logarithm of the pixel values using the ChIPS add_image() command and then overlay the exposure map for the observation, using transparency to ensure that the image can still be seen.

To display the data using the EQPOS (or SKY) coordinate system, you could copy the transformed pixel values back into the crate and use that; for example

>>> pychips.clear()
>>> pychips.add_window(8, 8, "inches")
>>> set_piximgvals(cr, np.log10(img))
Warning: divide by zero encountered in log10
>>> pychips.add_image(cr)
>>> pychips.add_image("2.3_bin2.expmap", ["alpha", [0.2,0.2]])

The divide-by-zero warning

The warning about divide by zero comes from NumPy and can be turned off using the seterr routine; for example

>>> olderr = np.seterr(divide='ignore')

Example 4

chips> x = np.arange(1, 10)
chips> y = np.sin(x) * np.cos(x)
chips> z = x**2 + y**2
chips> xcol = CrateData()
chips> xcol.name = "x"
chips> xcol.values = x
chips> ycol = CrateData()
chips> ycol.name = "y"
chips> ycol.values = y
chips> zcol = CrateData()
chips> zcol.name = "zz"
chips> zcol.values = z
chips> cr = TABLECrate()
chips> add_col(cr, xcol)
chips> add_col(cr, ycol)
chips> add_col(cr, zcol)
chips> write_file(cr, "out.fits")
chips> write_file(cr, "out.dat[opt kernel=text/simple]")
chips> !cat out.dat
#TEXT/SIMPLE
# x y zz
1 4.5464871341284e-01 1.206705452608
2 -3.7840124765396e-01 4.143187504226
3 -1.3970774909946e-01 9.019518255158
4 4.9467912331169e-01 16.24470743504
5 -2.7201055544468e-01 25.07398974227
6 -2.6828645900022e-01 36.07197762408
7 4.9530367784744e-01 49.24532573329
8 -1.4395165833253e-01 64.02072207994
9 -3.7549362338584e-01 81.14099546120

Here we create two files containing the contents of the x, y, and z arrays in columns called "x", "y", and "zz" respectively. The file out.fits is a FITS binary table whereas out.dat is an ASCII file (see ahelp dmascii for more information on ASCII support in the CIAO Data model).

Example 5

chips> ivals = np.arange(12).reshape(3, 4)
chips> cd = CrateData()
chips> cd.values = ivals
chips> cr = IMAGECrate()
chips> add_piximg(cr, cd)
chips> cr.name = "SIMPLE"
chips> write_file(cr, "img.fits")

Here we create an image, with dimensions of 4 (x) by 3 (y), add it to an IMAGECrate using add_piximg (note that add_image is a ChIPS call that displays an image), and write it out to the file img.fits. After these commands we have:

chips> print(cr)
   Crate Type:        <IMAGECrate>
   Crate Name:        SIMPLE
   Crate Number:      0

chips> !dmlist img.fits blocks
 
--------------------------------------------------------------------------------
Dataset: img.fits
--------------------------------------------------------------------------------
 
     Block Name                          Type         Dimensions
--------------------------------------------------------------------------------
Block    1: SIMPLE                         Image      Int4(4x3)
chips> !dmstat img.fits centroid-
SIMPLE
    min:	0 	      @:	( 1 1 )
    max:	11 	      @:	( 4 3 )
   mean:	5.5 
  sigma:	3.4520525295 
    sum:	66 
   good:	12 
   null:	0

Example 6

unix% cat qconv.py
#!/usr/bin/env python
import sys
import pycrates as pyc
cr = pyc.read_file(sys.argv[1])
sky = cr.get_transform("SKY")
x = float(sys.argv[2])
y = float(sys.argv[3])
lcoord = [[x, y]]
scoord = sky.apply_transform(lcoord)
print("Logical {0} -> SKY {1}".format(lcoord[0], scoord[0]))

This simple Python script (which has no sanity checks such as ensuring the number of command-line arguments is correct) will convert the logical coordinates given on the command line to SKY coordinates using the transform taken from the first argument. An example run (which assumes the executable bit is set) is given below:

unix% ./qconv.py img.fits 1 1
Logical [1.0, 1.0] -> SKY [ 2009.5 3149.5]

Changes in CIAO 4.5

Memory-management improvements

Memory is now correctly freed up when a variable containing a Crate goes out of scope. This means that the work-around

cr.get_dataset().snip()

is no longer needed (but can be retained, since the snip() routine does nothing in CIAO 4.5).

The Crates object

The get_filename attribute of a Crate now includes any Data-Model filter:

chips> cr = read_file("evt2.fits[energy=500:1000][bin sky=8]")
chips> cr.get_filename()
       'evt2.fits[energy=500:1000][bin sky=8]'

where the return would have just been 'evt2.fits' in CIAO 4.4.

New routine

The set_key() routine has been added, which simplifies adding a keyword to a Crate.

Improved transform support

The transform library (ahelp transform) has been updated to support 1D linear transformations.

Bugs

Write access is not robustly checked. (02 Apr 2013)

Users need to be careful when trying to modify a file in place as the checks on whether or not a file is writeable are not robust. Simply specifying mode="rw" will not produce an error nor an exception if the file is not actually writeable.

unix% chmod 400 myfile.fits
unix% python
>>> from pycrates import *
>>> rr = read_file("hrcf01801N006_evt2.fits", mode="rw")
>>> # do something to modify file
>>> rr.write()

The above sequence will not geneate an exception even though the file is not writeable.

Trying to write a file with a Byte data-type in the subspace fails. (29 Mar 2013)

HRC event files have a byte type (8 bit unsigned integer) SUB_MJF subspace column present. Trying to read in the file into a crate and writing it back out will fail.

unix% dmlist hrcf01801N006_evt2.fits subspace
 
--------------------------------------------------------------------------------
Data subspace for block EVENTS: Components: 1 Descriptors: 33 
--------------------------------------------------------------------------------
 
...
  31 ENDMNF               Int4                0:127 
  32 SUB_MJF              Byte                0:64 
  33 CLKTICKS             Int4                0:1091567608 
unix% python
>>> from pycrates import *
>>> rr = read_file("hrcf01801N006_evt2.fits")
>>> write_file(rr, "/tmp/copy")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/export/ciao-4.5/lib/python2.7/site-packages/pycrates/hlui.py", line 985, in write_file
    crate.write(outfile=filename, clobber=clobber)
  File "/export/ciao-4.5/lib/python2.7/site-packages/pycrates/tablecrate.py", line 294, in write
    backend.write( self, outfile=outfile )
  File "/export/ciao-4.5/lib/python2.7/site-packages/pycrates/io/dm_backend.py", line 534, in write
    self.__write_block( crate, close_flag )
  File "/export/ciao-4.5/lib/python2.7/site-packages/pycrates/io/dm_backend.py", line 578, in __write_block
    self.__write_subspace(block, crate)
  File "/export/ciao-4.5/lib/python2.7/site-packages/pycrates/io/dm_backend.py", line 626, in __write_subspace
    ssitem.unit, ssitem.range_min, ssitem.range_max )
TypeError: dmSubspaceColCreate() mins or maxes argument is not a supported type.

The only workaround is to remove the SUB_MJF subspace before reading it into a crate.

unix% dmcopy "hrcf01801N006_evt2.fits[subspace -SUB_MJF]" copy_evt.fits
unix% python
>>> from pycrates import *
>>> rr = read_file("copy_evt.fits")
>>> write_file(rr, "copy2")

Writing a crate with a string column into a file increases the string length by one. (08 Oct 2012)

Writing a crate with a string column into a file increases the string length by one. Repeatedly reading and writing a file will cause the string length to grow with each iteration.

chips> crate = read_file('tmp.fits')
chips> crate.write('tmp1.fits')
chips> exit()

unix%  dmlist tmp.fits cols |grep MLE
   54 MLEBAND  String[4] mleband
unix%  dmlist tmp1.fits cols | grep MLE
   54 MLEBAND  String[5] mleband

In the above example the string column has grown by one. Repeating the chips commands with tmp1.fits as input will cause the column to grow again.

Using numpy views of arrays can lead to data corruption. (08 Oct 2012)

numpy uses a special indexing scheme to access arrays that have been sliced (ie truncated at either end) or when for example transposing 2D arrays. The origianal data are preserved and the user only sees a view of the array.

The crates python to datamodel C interface does not handle views correctly. It may result in data being truncated or simply producing nonsensical results.

Users should make sure that any arrays sent to pycrates are in C_CONTIGUOUS order and ALGINED

chips-1> foo = np.arange(10).reshape(5,2)
chips-2> foo2 = bob.T
chips-3> foo.flags

   C_CONTIGUOUS : True
   F_CONTIGUOUS : False
   OWNDATA : False
   WRITEABLE : True
   ALIGNED : True
   UPDATEIFCOPY : False
chips-4> foo2.flags

   C_CONTIGUOUS : False
   F_CONTIGUOUS : True
   OWNDATA : False
   WRITEABLE : True
   ALIGNED : True
   UPDATEIFCOPY : False

ASCII files with non-alphanumeric column names can leads to inaccessible crates. (08 Oct 2012)

If an ASCII file is loaded into a crate that has non-alphanumeric characters, then the crate data may become inaccessible.

Column names should only contain letters A-Z (case insensitive), numbers 0-9, and underscores. Column names that contain square braces, paranthese, or any other character may cause problems.

Files with 1D linear transforms (07 Sep 2012)

pycrates does not support 1D, linear transforms. It tries to create a 2D transform which causes memory corruption and may lead to either a crash (Segmentation Violation) or could lead to incorrect results.

Users can check if their file contains a 1D transform by using dmlist to list the columns and look for any coordinates transforms that look like

unix% dmlist myfile.fits cols
...
World Coord Transforms for Columns in Table Block EVENTS
--------------------------------------------------------------------------------
 
ColNo    Name
3:    CPC(CPCX) = (+0)[mm] +(+0.0240)* (chip(chipx)-(+0.50))
         (CPCY)   (+0)      (+0.0240)  (    (chipy) (+0.50))
4:    MSC(PHI  ) = (+0)[deg] +TAN-P[(+0.000136667)* (det(detx)-(+4096.50))]
         (THETA)   (+0)             (+0.000136667)  (   (dety) (+4096.50)) 
5:    EQPOS(RA ) = (+270.1153)[deg] +TAN[(-0.000136667)* (sky(x)-(+4096.50))]
           (DEC)   (-24.0418 )           (+0.000136667)  (   (y) (+4096.50)) 
6:    pha_COORD            = pha

Here, the pha column has a 1D linear transform attached to it and would cause problems with pycrates.

Another example would be

unix% dmlist dmextract29b.out cols
 ...
--------------------------------------------------------------------------------
World Coord Transforms for Columns in Table Block HISTOGRAM
--------------------------------------------------------------------------------
 
ColNo    Name
4:    CEL_R                = +0 [arcsec] +0.4920 * (R  -0)
9:    CEL_AREA             = +0 [arcsec**2] +0.2421 * (AREA  -0)
17:   CEL_BRI              = +0 [count/arcsec**2] +4.1311 * (SUR_BRI  -0)
18:   CEL_BRI_ERR          = +0 [count/arcsec**2] +4.1311 * (SUR_BRI_ERR  -0)

All these columns are 1D linear transforms that would cause pycrates a problem.

Some example of files with linear transforms include

Radial profiles output from dmextract
ACIS Blank Sky background files with '2005' date
Event files
Aspect histogram files created by asphist

Currently there is no generic work around. If you do not need to use the column with the linear transform, you can just exclude it using a [cols -column_name] filter. If you need to use the column that has a transform attached, contact Helpdesk and we will try to help with a custom solution.

Adding a column without a name creates a column called () in the output file (03 Jan 2012)

chips> cr = TABLECrate() 
chips> cd = CrateData() 
chips> cd.values = np.arange(20,31) 
chips> add_col(cr, cd) 
chips> cr.get_colnames() 
          ['']
chips> cr.write("/tmp/temp.fits") 
chips> !dmlist /tmp/temp.fits cols
 
--------------------------------------------------------------------------------
Columns for Table Block HDU2
--------------------------------------------------------------------------------
 
ColNo  Name                 Unit        Type             Range
   1   ()                                Int4           -

Numerical subspace ranges are not propagated correctly (06 Jan 2012) (Mac OS X)

This file was read with CRATES and then written out again. The numerical subspace ranges have been replaced by "DEFAULT":

dmlist pha_copy.fits"[REGION]" subspace 
 
--------------------------------------------------------------------------------
Data subspace for block REGION: Components: 1 Descriptors: 11 
--------------------------------------------------------------------------------
 
 --- Component 1 --- 
   1 SPEC_NUM             Int2                1:16384 
   2 ROWID                String              
   3 SHAPE                String              
   4 TG_R                 Real8               DEFAULT
   5 TG_D                 Real8               DEFAULT
   6 R                    Real8               DEFAULT
   7 ROTANG               Real8               DEFAULT
   8 COMPONENT            Int2                DEFAULT
   9 INCLUDE              Int2                DEFAULT
  10 TG_SRCID             Int2                DEFAULT
  11 TG_M                 Int2                DEFAULT

Caveats

Cannot add rows to a column that is part of a virtual column. (08 Oct 2012): Users can not add rows a column which is a component of a vector using Crates (the new values are set to 0). User can extend scalar and array columns. To work around this, users must add rows the vector column itself rather than the components.