Chandra X-Ray Observatory
Skip to the navigation links
Last modified: December 2015

AHELP for CIAO 4.9


Context: crates


Reading, writing, and changing ASCII and FITS files in Python


The Crates module allows users to read and write ASCII and FITS files in Python, with support for the same features as provided by the CIAO Data Model.

chips> cr = read_file('tbl.dat[energy=500:7000][cols x,y]')
chips> x = get_colvals(cr, 'x')
chips> y = get_colvals(cr, 'y')

reads in the x and y columns from the file 'tbl.dat', after applying a filter on the energy column of 500 to 7000, and then returns the column values in the x and y variables as NumPy arrays.

Functions and methods

The Crates module provides a number of routines for common tasks, like get_colvals. For more complex, uncommon routines, there are also a large number of methods provided by the objects themselves. Try the Python dir and help commands for more information: e.g.

chips> cr = read_file('evt.fits')
chips> x = cr.get_column('x')
chips> dir(x)
chips> help(x)

Example 1

chips> cr = read_file("evt2.fits")
chips> print(get_keyval(cr, "OBJECT"))
chips> print(get_keyval(cr, "EXPOSURE") / 1000)

Here we get the values of the OBJECT and EXPOSURE keywords from the header of evt2.fits. The return value from get_keyval() uses the appropriate Python type, so OBJECT is a string and the exposure time is a number.

Example 2

>>> from pycrates import *
>>> cr = read_file("evt2.fits[ccd_id=7,energy=300:7000]")
>>> e = copy_colvals(cr, "energy")
>>> emin = e.min() / 1000
>>> emax = e.max() / 1000
>>> emean = e.mean() / 1000
>>> print("Mean E = {0} min= {1} max= {2}".format(emean, emin, emax))
Mean E = 3.10554851929 min= 0.300000335693 max= 6.99996289063

Here we read in the energy column from an event file and calculate the minimum, maximum and mean energy in keV.

Example 3

>>> import numpy as np
>>> import pycrates as *
>>> import pychips.all as pychips
>>> cr = read_file("0.5-7.0_bin2.img")
>>> img = copy_piximgvals(cr)
>>> pychips.add_image(np.log10(img))
RuntuineWarning: divide by zero encountered in log10
>>> pychips.add_image("2.3_bin2.expmap", ["wcs", "logical", "alpha",

Here we display the logarithm of the pixel values using the ChIPS add_image() command and then overlay the exposure map for the observation, using transparency to ensure that the image can still be seen.

To display the data using the EQPOS (or SKY) coordinate system you can extract the WCS transformation and sent it to add_image:

>>> pychips.clear()
>>> pychips.add_window(8, 8, "inches")
>>> tr = cr.get_transform('EQPOS')
>>> pychips.add_image(np.log10(img), tr)
RuntimeWarning: divide by zero encountered in log10
>>> pychips.add_image("2.3_bin2.expmap", ["alpha", [0.2,0.2]])

The divide-by-zero warning

The warning about divide by zero comes from NumPy and can be turned off using the seterr routine; for example

>>> olderr = np.seterr(divide='ignore')

Example 4

chips> x = np.arange(1, 10)
chips> y = np.sin(x) * np.cos(x)
chips> z = x**2 + y**2
chips> xcol = CrateData()
chips> = "x"
chips> xcol.values = x
chips> ycol = CrateData()
chips> = "y"
chips> ycol.values = y
chips> zcol = CrateData()
chips> = "zz"
chips> zcol.values = z
chips> zcol.unit = 'cm**2 s'
chips> cr = TABLECrate()
chips> add_col(cr, xcol)
chips> add_col(cr, ycol)
chips> add_col(cr, zcol)
chips> write_file(cr, "out.fits")
chips> write_file(cr, "out.dat[opt kernel=text/simple]")
chips> !cat out.dat
# x y zz
1 4.5464871341284e-01 1.206705452608
2 -3.7840124765396e-01 4.143187504226
3 -1.3970774909946e-01 9.019518255158
4 4.9467912331169e-01 16.24470743504
5 -2.7201055544468e-01 25.07398974227
6 -2.6828645900022e-01 36.07197762408
7 4.9530367784744e-01 49.24532573329
8 -1.4395165833253e-01 64.02072207994
9 -3.7549362338584e-01 81.14099546120

Here we create two files containing the contents of the x, y, and z arrays in columns called "x", "y", and "zz" respectively. The file out.fits is a FITS binary table whereas out.dat is an ASCII file (see ahelp dmascii for more information on ASCII support in the CIAO Data model).

The make_table_crate routine from the crates_contrib.utils module can be used to simplify the above.

Example 5

chips> ivals = np.arange(12).reshape(3, 4)
chips> cd = CrateData()
chips> cd.values = ivals
chips> cr = IMAGECrate()
chips> add_piximg(cr, cd)
chips> = "SIMPLE"
chips> write_file(cr, "img.fits")

Here we create an image, with dimensions of 4 (x) by 3 (y), add it to an IMAGECrate using add_piximg (note that add_image is a ChIPS call that displays an image), and write it out to the file img.fits. After these commands we have:

chips> print(cr)
   Crate Type:        <IMAGECrate>
   Crate Name:        SIMPLE

chips> !dmlist img.fits blocks
Dataset: img.fits
     Block Name                          Type         Dimensions
Block    1: SIMPLE                         Image      Int4(4x3)
chips> !dmstat img.fits centroid-
    min:	0 	      @:	( 1 1 )
    max:	11 	      @:	( 4 3 )
   mean:	5.5 
  sigma:	3.4520525295 
    sum:	66 
   good:	12 
   null:	0 

The make_image_crate routine from the crates_contrib.utils module can be used to simplify the above.

Example 6

unix% cat
#!/usr/bin/env python
import sys
import pycrates as pyc
cr = pyc.read_file(sys.argv[1])
sky = cr.get_transform("SKY")
x = float(sys.argv[2])
y = float(sys.argv[3])
lcoord = [[x, y]]
scoord = sky.apply(lcoord)
print("Logical {0} -> SKY {1}".format(lcoord[0], scoord[0]))

This simple Python script (which has no sanity checks such as ensuring the number of command-line arguments is correct) will convert the logical coordinates given on the command line to SKY coordinates using the transform taken from the first argument. An example run (which assumes the executable bit is set) is given below:

unix% ./ img.fits 1 1
Logical [1.0, 1.0] -> SKY [ 2009.5 3149.5]

Loading Crates

The Crates module is automatically imported into ChIPS and Sherpa sessions, otherwise use one of the following:

from pycrates import *


import pycrates


The Crates and transform library ahelp documentation can be listed by saying ahelp -c crates and ahelp -c transform

In particular: ahelp read_file, ahelp crate, ahelp cratedataset, ahelp cratedata, and ahelp cratekey.

There is also Python documentation available, on the module itself - e.g.

chips> import pycrates
chips> help(pycrates)

or for specific functions and methods

chips> help(read_file)
chips> help(cr.get_colnames)

Objects and metadata

The Crates library uses a set of Python objects to represent data. These objects provide access to the data and metadata - such as units, description, and other fields - stored in the files.

Object Description
TABLECrate, IMAGECrate Represents a block of a file containing a table - a set of columns - or a single n-Dimensional image.
CrateDataset, PHACrateDataset, RMFCrateDataset A set of blocks stored in a single file.
CrateData This is used to represent column and image data.
CrateKey A keyword, containing the name, value, and optional unit and description fields.
CrateSubspaceData Represent the filtering applied to a column or image.

Creating objects

Objects can be created using the normal Python idiom, by calling the constructor, or - in some cases - can be created by routines such as read_file and get_key.

Related objects

The Crates library also uses objects from the transforms and region modules.


The subspace information stored in a file can be accessed with the get_subspace_data method of a crate. There can be multiple blocks of subspace data - normally one per ACIS chip but combining filters with OR rather than AND will also lead to this - which means that you need to provide an integer and column name.

crate.get_subspace_data(cptnum, item)

Here crate refers to the return value of read_file, the cptnum argument is a positive integer, indicating the component number, and item is a string indicating the column or axis name. Note that in CIAO 4.8 there is no way to determine how many blocks there are in a file or what columns are stored in the subspace block.

>>> cr = read_file('a2142_smoothed.fits')
>>> s1 = cr.get_subspace_data(1, 'sky')
>>> print(s1)
   Name:            sky: x, y
   Range Minimum:   []
   Range Maximum:   []
   Region String:   Box(3840,3670,1460,1460)

>>> sx = cr.get_subspace_data(1, 'x')
>>> sy = cr.get_subspace_data(1, 'y')
>>> print(sx)
   Name:            x
   Range Minimum:   [ 3110.]
   Range Maximum:   [ 4570.]
   Region String:   DEFAULT

>>> print(sy)
   Name:            y
   Range Minimum:   [ 2940.]
   Range Maximum:   [ 4400.]
   Region String:   DEFAULT

>>> import region
>>> region.regPrintRegion(s1.region)
1	Box(3840,3670,1460,1460) (Flag_Coord: Unknown) (Flag_Radius: Unknown)
>>> s2 = cr.get_subspace_data(1, 'time')
>>> print(s2)
   Name:            time
   Table Name:      GTI3
   Range Minimum:   [  2.29755146e+08   2.29758594e+08   2.29795394e+08]
   Range Maximum:   [  2.29758591e+08   2.29795391e+08   2.29800307e+08]
   Region String:   DEFAULT

>>> s3 = cr.get_subspace_data(1, 'ccd_id')
>>> print(s3)
   Name:            ccd_id
   Range Minimum:   [3]
   Range Maximum:   [3]
   Region String:   DEFAULT

The return value - a CrateSubspaceData object - has the following fields:

Field Description
name The column name.
unit The units of the column, or ''.
range_min The minimum value of the data; this can contain multiple values when sets of ranges were used to filter the column.
range_max The maximum value of the data; this can contain multiple values when sets of ranges were used to filter the column.
region If a spatial filter was applied then this is a region object representing the filter, otherwise it is None.

Two subspace objects can be compared using the Python equality check; e.g.

>>> cr1.get_subspace(1, "time") == cr1.get_subspace(2, "time")

There is no way in CIAO 4.8 to add to or delete from the subspace information in a Crate.

Changes in CIAO 4.8

Adding columns

The add_column method of a table crate now works correctly when the column number is set to 0 (previously the new column was added to the end of the table, now it is the first column). The add_column method and add_col call now create an empty subspace entry for vector columns, accessible with the get_subspace_data method of the crate.

Support for variable-length arrays

Support for variable-length arrays has been improved and the CrateData object now supports the is_varlen and get_fixed_length_array methods.

Tri-state logical columns

The FITS convention for boolean columns supports a "null" value, which means that it can represent three states: true, false, and unknown. Any such null, or unknown, value will be converted to False when the data is read in to a Crate (e.g. when read by read_file), and so the information that this cell value was a null will be lost.


Crates is unable to write files that have REGION filters (subspace) stored as additional blocks in FITS files.

Specificially if the DSREF keywords reference REGION blocks.

Crates does not support np.uint32 (unsigned 32bit integers) arrays as images.

An error is raised when trying to write the crates to a file.

Trying to create a table with 0 rows with an vector column will crash when the crate is written.
  >>> cr = TABLECrate ()
  >>> col = CrateData ()
  >>> = 'POS'
  >>> col.vdim = 2
  >>> cpt_list = ['X','Y']
  >>> cpt_list = CrateDict()
  >>> for nn in ['X','Y']:
  ... cpt = CrateData()
  ... = nn
  ... cpt.parent = col
  ... cpt.vdim = 1
  ... cpt_list[nn] = cpt
  >>> col._set_cptslist(cpt_list)
  >>> cr.add_column (col)
  >>> col.values = np.empty (0, dtype=np.float)
  >>> cr.write ('tmp.fits', clobber=True)
  Segmentation fault

Crates has problems trying to populate a string column.
>>> cd = CrateData ()
>>> cd.values = np.empty (1, dtype='|S1')
>>> cd.values.fill ('b')
>>> cd.values

The array values should be filled a single letter 'b'

Crates cannot read a table that has 0 rows and contains vector array column(s).
>>> tab = read_file("acis_fov.fits[ccd_id=10]")
>>> print(tab.get_colnames())
ValueError: all the input arrays must have same number of dimensions
Crates generates a misleading "File exists but is not readble" error when asked to open a non-existent block within the file.
>>> tab = read_file("myfile[does_not_exist"])
IOError: File exists but is not readable.

Whereas the following works:

>>> tab = read_file("myfile")
Problem reading images containing unnamed WCS transforms

Crates has problems reading images that contain WCS transforms, but have chosen to omit the transform names. For example in a standard Chandra observtaion the SKY X,Y image axes have a WCS transform to RA and DEC.

Some older files may have images where the actual names of the transformed axes was omitted. This has been seen in some ROSAT weight maps (WMAPs) attached to older spectrum files.

>>> ww = read_file("orig.pi[wmap]")
>>> ii = ww.get_image()
Problem with single byte bit columns

Crates has problems when trying to write out a bit column, when the column has less than 8 bits (so fill only 1 byte).

Subspace information corrupted when images are deleted.

The subspace information for an image is corrupted when the image values are replaced (image deleted and then re-added) as one might do to change the data-type of the image.

>>> fimgcrate = read_file (infile)
>>> fimgcrate_image = fimgcrate.get_image()
>>> values = fimgcrate_image.values
>>> fimgcrate.delete_image()
>>> values = values.astype("float32")
>>> fimgcrate_image.values = values
>>> fimgcrate.add_image(fimgcrate_image)
>>> fimgcrate.write(outfile)
UserWarning: Unable to delete columns or images in update mode.
  warnings.warn("Unable to delete columns or images in update mode.")
Crates will convert all transforms into their binned form.

Transforms are assumed to be applied to image pixels, so when they are created they are adjusted to match what would be an image pixel boundary, even if the transform is created on a table column.

So, for example trying to create a transform that converts degrees F to C:

tempF = +32 + 1.8 * tempC

A linear transform with scale=1.8 and offset=32, is written out as

tempF = +32.90 [degree F] +1.80 * (tempC -0.50)

While mathematically correct, the extra half "pixel" offset is unnecessary when dealing with transforms applied to table columns.

Crates will not copy the keywords in an empty/null FITS Primary extension

The keywords in the FITS primary extension are not written by the CrateDataset's write() method

>>> from pycrates import *
>>> tab = read_file(infile)
>>> tab.get_dataset().write(outfile)

If infile has a NULL Primary extesnion (only keywords, no data), the keywords are not written to the output file.

Writing NaN to keywords causes crash

Attempting to write a NaN value to a keyword results in a crash of the python interpreter

chips-1> tab = read_file("a.fits")
chips-2> key = CrateKey()
chips-4> key.value = np.nan
chips-5> tab.add_key( key )
chips-6> write_file( tab, "b.fits")
RuntimeError: dmKeyWrite() could not write key. 'foobar'
chips-7> quit()

Exception IOError: IOError('FITS error 402 writing key foobar',) in <boundmethod CrateDataset.__del__ of Crate Dataset:
*** glibc detected *** /export/ciao-4.5/ots/bin/python: double free or
corruption (!prev): 0x0000000003026480 ***
(core dumped) ipython --profile chips -i -c "\"${cmd}\""
1D Image transforms

pycrates can only access the 1st transform associated with an image. If a 2D image has only 1 transform (eg sky=(x,y)) then there is no problem; however, if the image has two seprate 1D transforms (eg time and pi), pycrates only provides access to the 1st (x) axis.

>>> from pycrates import *
>>> cr = read_file('test.dat[bin kt=1.15:1.55:0.1,abund=0.8:1.2:0.2;factor]') 
>>> cr.get_axisnames()
>>> cr.get_axis('abund')
KeyError: 'abund not found in list of image axes.'
>>> cr.get_transform('kt')
         <pytransform.LINEARTransform; proxy of <Swig Object of type 'LINEARTransform *' at 0x103692150> > 
>>> cr.get_transform('abund')
KeyError: 'abund not found in list of image axes.' 
Region filters in the crates subspace are not writen when the file is saved.
Byte datatype subspace columns

pycrates may crash if the file contains subspace columns with byte data-type. Examples include some HRC datasets with byte type SUB_MJF.


Workaround: for the HRC case, you can remove the subspace column with

% dmcopy hrc_evt[subspace -sub_mjf] hrc_evt_mod
prior to loading the file into crates.

Write access is not robustly checked.

Users need to be careful when trying to modify a file in place as the checks on whether or not a file is writeable are not robust. Simply specifying mode="rw" will not produce an error nor an exception if the file is not actually writeable.

unix% chmod 400 myfile.fits
unix% python
>>> from pycrates import *
>>> rr = read_file("hrcf01801N006_evt2.fits", mode="rw")
>>> # do something to modify file
>>> rr.write()

The above sequence will not geneate an exception even though the file is not writeable.

Trying to write a file with a Byte data-type in the subspace fails.

HRC event files have a byte type (8 bit unsigned integer) SUB_MJF subspace column present. Trying to read in the file into a crate and writing it back out will fail.

unix% dmlist hrcf01801N006_evt2.fits subspace
Data subspace for block EVENTS: Components: 1 Descriptors: 33 
  31 ENDMNF               Int4                0:127 
  32 SUB_MJF              Byte                0:64 
  33 CLKTICKS             Int4                0:1091567608 
unix% python
>>> from pycrates import *
>>> rr = read_file("hrcf01801N006_evt2.fits")
>>> write_file(rr, "/tmp/copy")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/export/ciao-4.5/lib/python2.7/site-packages/pycrates/", line 985, in write_file
    crate.write(outfile=filename, clobber=clobber)
  File "/export/ciao-4.5/lib/python2.7/site-packages/pycrates/", line 294, in write
    backend.write( self, outfile=outfile )
  File "/export/ciao-4.5/lib/python2.7/site-packages/pycrates/io/", line 534, in write
    self.__write_block( crate, close_flag )
  File "/export/ciao-4.5/lib/python2.7/site-packages/pycrates/io/", line 578, in __write_block
    self.__write_subspace(block, crate)
  File "/export/ciao-4.5/lib/python2.7/site-packages/pycrates/io/", line 626, in __write_subspace
    ssitem.unit, ssitem.range_min, ssitem.range_max )
TypeError: dmSubspaceColCreate() mins or maxes argument is not a supported type.

The only workaround is to remove the SUB_MJF subspace before reading it into a crate.

unix% dmcopy "hrcf01801N006_evt2.fits[subspace -SUB_MJF]" copy_evt.fits
unix% python
>>> from pycrates import *
>>> rr = read_file("copy_evt.fits")
>>> write_file(rr, "copy2")
Using numpy views of arrays can lead to data corruption.

numpy uses a special indexing scheme to access arrays that have been sliced (ie truncated at either end) or when for example transposing 2D arrays. The origianal data are preserved and the user only sees a view of the array.

The crates python to datamodel C interface does not handle views correctly. It may result in data being truncated or simply producing nonsensical results.

Users should make sure that any arrays sent to pycrates are in C_CONTIGUOUS order and ALGINED

chips-1> foo = np.arange(10).reshape(5,2)
chips-2> foo2 = bob.T
chips-3> foo.flags

   OWNDATA : False
   ALIGNED : True
chips-4> foo2.flags

   OWNDATA : False
   ALIGNED : True
Files with 1D linear transforms

pycrates does not support 1D, linear transforms. It tries to create a 2D transform which causes memory corruption and may lead to either a crash (Segmentation Violation) or could lead to incorrect results.

Users can check if their file contains a 1D transform by using dmlist to list the columns and look for any coordinates transforms that look like

unix% dmlist myfile.fits cols
World Coord Transforms for Columns in Table Block EVENTS
ColNo    Name
3:    CPC(CPCX) = (+0)[mm] +(+0.0240)* (chip(chipx)-(+0.50))
         (CPCY)   (+0)      (+0.0240)  (    (chipy) (+0.50))
4:    MSC(PHI  ) = (+0)[deg] +TAN-P[(+0.000136667)* (det(detx)-(+4096.50))]
         (THETA)   (+0)             (+0.000136667)  (   (dety) (+4096.50)) 
5:    EQPOS(RA ) = (+270.1153)[deg] +TAN[(-0.000136667)* (sky(x)-(+4096.50))]
           (DEC)   (-24.0418 )           (+0.000136667)  (   (y) (+4096.50)) 
6:    pha_COORD            = pha 

Here, the pha column has a 1D linear transform attached to it and would cause problems with pycrates.

Another example would be

unix% dmlist dmextract29b.out cols
World Coord Transforms for Columns in Table Block HISTOGRAM
ColNo    Name
4:    CEL_R                = +0 [arcsec] +0.4920 * (R  -0)
9:    CEL_AREA             = +0 [arcsec**2] +0.2421 * (AREA  -0)
17:   CEL_BRI              = +0 [count/arcsec**2] +4.1311 * (SUR_BRI  -0)
18:   CEL_BRI_ERR          = +0 [count/arcsec**2] +4.1311 * (SUR_BRI_ERR  -0)

All these columns are 1D linear transforms that would cause pycrates a problem.

Some example of files with linear transforms include

  • Radial profiles output from dmextract
  • ACIS Blank Sky background files with '2005' date
  • Event files
  • Aspect histogram files created by asphist

Currently there is no generic work around. If you do not need to use the column with the linear transform, you can just exclude it using a [cols -column_name] filter. If you need to use the column that has a transform attached, contact CXC Helpdesk and we will try to help with a custom solution.

Adding a column without a name creates a column called () in the output file
chips> cr = TABLECrate() 
chips> cd = CrateData() 
chips> cd.values = np.arange(20,31) 
chips> add_col(cr, cd) 
chips> cr.get_colnames() 
chips> cr.write("/tmp/temp.fits") 
chips> !dmlist /tmp/temp.fits cols
Columns for Table Block HDU2
ColNo  Name                 Unit        Type             Range
   1   ()                                Int4           -                    
Numerical subspace ranges are not propagated correctly
(Mac OS X)

This file was read with CRATES and then written out again. The numerical subspace ranges have been replaced by "DEFAULT":

dmlist pha_copy.fits"[REGION]" subspace 
Data subspace for block REGION: Components: 1 Descriptors: 11 
 --- Component 1 --- 
   1 SPEC_NUM             Int2                1:16384 
   2 ROWID                String              
   3 SHAPE                String              
   4 TG_R                 Real8               DEFAULT
   5 TG_D                 Real8               DEFAULT
   6 R                    Real8               DEFAULT
   7 ROTANG               Real8               DEFAULT
   8 COMPONENT            Int2                DEFAULT
   9 INCLUDE              Int2                DEFAULT
  10 TG_SRCID             Int2                DEFAULT
  11 TG_M                 Int2                DEFAULT 


Crates does not support subspaces on byte columns.
OGIP compliance

crates checks the input PHA, RMF, and ARF files types for adherence to the OGIP standards: here and here.

Some older datasets, eg from the ROSAT archive, may not be entirely compliant with these standards and therefore may be unreadable by crates.

Cannot add rows to a column that is part of a virtual column.

Users can not add rows a column which is a component of a vector using Crates (the new values are set to 0). User can extend scalar and array columns. To work around this, users must add rows the vector column itself rather than the components.

See Also

add_colvals, make_image_crate, make_table_crate, scale_image_crate, smooth_image_crate, write_arrays, write_columns
add_col, add_key, add_piximg, col_exists, copy_colvals, copy_piximgvals, cratedata, cratekey, create_vector_column, create_virtual_column, delete_col, delete_key, delete_piximg, get_axis_transform, get_col, get_col_names, get_colvals, get_crate_item_type, get_crate_type, get_key, get_key_names, get_keyval, get_number_cols, get_number_rows, get_piximg, get_piximg_shape, get_piximgvals, get_transform, get_transform_matrix, is_pha, is_rmf, is_virtual, key_exists, print_crate_names, read_file, read_pha, read_rmf, set_colvals, set_key, set_keyval, set_piximgvals, write_file, write_pha, write_rmf

Last modified: December 2015
Smithsonian Institute Smithsonian Institute

The Chandra X-Ray Center (CXC) is operated for NASA by the Smithsonian Astrophysical Observatory. 60 Garden Street, Cambridge, MA 02138 USA.   Email: Smithsonian Institution, Copyright © 1998-2017. All rights reserved.