Chandra X-Ray Observatory (CXC)
Skip to the navigation links
Search the CIAO website or contact the CXC HelpDesk
Last modified: 2 April 2013

URL: http://cxc.harvard.edu/ciao/bugs/crates.html

Bugs: CRATES

Return to: Bug List Index

For detailed information and examples of running this tool, refer to the crates ahelp file.

Table of Contents

Caveats

Bugs

Caveats

Cannot add rows to a column that is part of a virtual column.  (08 Oct 2012)

Users can not add rows a column which is a component of a vector using Crates (the new values are set to 0). User can extend scalar and array columns. To work around this, users must add rows the vector column itself rather than the components.

Bugs

Write access is not robustly checked. [New] (02 Apr 2013)

Users need to be careful when trying to modify a file in place as the checks on whether or not a file is writeable are not robust. Simply specifying mode="rw" will not produce an error nor an exception if the file is not actually writeable. 

unix% chmod 400 myfile.fits
unix% python
>>> from pycrates import *
>>> rr = read_file("hrcf01801N006_evt2.fits", mode="rw")
>>> # do something to modify file
>>> rr.write()

The above sequence will not geneate an exception even though the file is not writeable.

Trying to write a file with a Byte data-type in the subspace fails. [New] (29 Mar 2013)

HRC event files have a byte type (8 bit unsigned integer) SUB_MJF subspace column present. Trying to read in the file into a crate and writing it back out will fail. 

unix% dmlist hrcf01801N006_evt2.fits subspace
 
--------------------------------------------------------------------------------
Data subspace for block EVENTS: Components: 1 Descriptors: 33 
--------------------------------------------------------------------------------
 
...
  31 ENDMNF               Int4                0:127 
  32 SUB_MJF              Byte                0:64 
  33 CLKTICKS             Int4                0:1091567608 
unix% python
>>> from pycrates import *
>>> rr = read_file("hrcf01801N006_evt2.fits")
>>> write_file(rr, "/tmp/copy")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/export/ciao-4.5/lib/python2.7/site-packages/pycrates/hlui.py", line 985, in write_file
    crate.write(outfile=filename, clobber=clobber)
  File "/export/ciao-4.5/lib/python2.7/site-packages/pycrates/tablecrate.py", line 294, in write
    backend.write( self, outfile=outfile )
  File "/export/ciao-4.5/lib/python2.7/site-packages/pycrates/io/dm_backend.py", line 534, in write
    self.__write_block( crate, close_flag )
  File "/export/ciao-4.5/lib/python2.7/site-packages/pycrates/io/dm_backend.py", line 578, in __write_block
    self.__write_subspace(block, crate)
  File "/export/ciao-4.5/lib/python2.7/site-packages/pycrates/io/dm_backend.py", line 626, in __write_subspace
    ssitem.unit, ssitem.range_min, ssitem.range_max )
TypeError: dmSubspaceColCreate() mins or maxes argument is not a supported type.

The only workaround is to remove the SUB_MJF subspace before reading it into a crate. 

unix% dmcopy "hrcf01801N006_evt2.fits[subspace -SUB_MJF]" copy_evt.fits
unix% python
>>> from pycrates import *
>>> rr = read_file("copy_evt.fits")
>>> write_file(rr, "copy2")

Writing a crate with a string column into a file increases the string length by one.  (08 Oct 2012)

Writing a crate with a string column into a file increases the string length by one. Repeatedly reading and writing a file will cause the string length to grow with each iteration. 

chips> crate = read_file('tmp.fits')
chips> crate.write('tmp1.fits')
chips> exit()

unix%  dmlist tmp.fits cols |grep MLE
   54 MLEBAND  String[4] mleband
unix%  dmlist tmp1.fits cols | grep MLE
   54 MLEBAND  String[5] mleband

In the above example the string column has grown by one. Repeating the chips commands with tmp1.fits as input will cause the column to grow again.

Using numpy views of arrays can lead to data corruption.  (08 Oct 2012)

numpy uses a special indexing scheme to access arrays that have been sliced (ie truncated at either end) or when for example transposing 2D arrays. The origianal data are preserved and the user only sees a view of the array.

The crates python to datamodel C interface does not handle views correctly. It may result in data being truncated or simply producing nonsensical results.

Users should make sure that any arrays sent to pycrates are in C_CONTIGUOUS order and ALGINED 

chips-1> foo = np.arange(10).reshape(5,2)
chips-2> foo2 = bob.T
chips-3> foo.flags

   C_CONTIGUOUS : True
   F_CONTIGUOUS : False
   OWNDATA : False
   WRITEABLE : True
   ALIGNED : True
   UPDATEIFCOPY : False
chips-4> foo2.flags

   C_CONTIGUOUS : False
   F_CONTIGUOUS : True
   OWNDATA : False
   WRITEABLE : True
   ALIGNED : True
   UPDATEIFCOPY : False

ASCII files with non-alphanumeric column names can leads to inaccessible crates.  (08 Oct 2012)

If an ASCII file is loaded into a crate that has non-alphanumeric characters, then the crate data may become inaccessible.

Column names should only contain letters A-Z (case insensitive), numbers 0-9, and underscores. Column names that contain square braces, paranthese, or any other character may cause problems.

Files with 1D linear transforms  (07 Sep 2012)

pycrates does not support 1D, linear transforms. It tries to create a 2D transform which causes memory corruption and may lead to either a crash (Segmentation Violation) or could lead to incorrect results.

Users can check if their file contains a 1D transform by using dmlist to list the columns and look for any coordinates transforms that look like 

unix% dmlist myfile.fits cols
...
World Coord Transforms for Columns in Table Block EVENTS
--------------------------------------------------------------------------------
 
ColNo    Name
3:    CPC(CPCX) = (+0)[mm] +(+0.0240)* (chip(chipx)-(+0.50))
         (CPCY)   (+0)      (+0.0240)  (    (chipy) (+0.50))
4:    MSC(PHI  ) = (+0)[deg] +TAN-P[(+0.000136667)* (det(detx)-(+4096.50))]
         (THETA)   (+0)             (+0.000136667)  (   (dety) (+4096.50)) 
5:    EQPOS(RA ) = (+270.1153)[deg] +TAN[(-0.000136667)* (sky(x)-(+4096.50))]
           (DEC)   (-24.0418 )           (+0.000136667)  (   (y) (+4096.50)) 
6:    pha_COORD            = pha

Here, the pha column has a 1D linear transform attached to it and would cause problems with pycrates.

Another example would be 

unix% dmlist dmextract29b.out cols
 ...
--------------------------------------------------------------------------------
World Coord Transforms for Columns in Table Block HISTOGRAM
--------------------------------------------------------------------------------
 
ColNo    Name
4:    CEL_R                = +0 [arcsec] +0.4920 * (R  -0)
9:    CEL_AREA             = +0 [arcsec**2] +0.2421 * (AREA  -0)
17:   CEL_BRI              = +0 [count/arcsec**2] +4.1311 * (SUR_BRI  -0)
18:   CEL_BRI_ERR          = +0 [count/arcsec**2] +4.1311 * (SUR_BRI_ERR  -0)

All these columns are 1D linear transforms that would cause pycrates a problem.

Some example of files with linear transforms include

  • Radial profiles output from dmextract
  • ACIS Blank Sky background files with '2005' date
  • Event files
  • Aspect histogram files created by asphist

Currently there is no generic work around. If you do not need to use the column with the linear transform, you can just exclude it using a [cols -column_name] filter. If you need to use the column that has a transform attached, contact Helpdesk and we will try to help with a custom solution.

Adding a column without a name creates a column called () in the output file  (03 Jan 2012)

chips> cr = TABLECrate() 
chips> cd = CrateData() 
chips> cd.values = np.arange(20,31) 
chips> add_col(cr, cd) 
chips> cr.get_colnames() 
          ['']
chips> cr.write("/tmp/temp.fits") 
chips> !dmlist /tmp/temp.fits cols
 
--------------------------------------------------------------------------------
Columns for Table Block HDU2
--------------------------------------------------------------------------------
 
ColNo  Name                 Unit        Type             Range
   1   ()                                Int4           -

Numerical subspace ranges are not propagated correctly  (06 Jan 2012)
(Mac OS X)

This file was read with CRATES and then written out again. The numerical subspace ranges have been replaced by "DEFAULT": 

dmlist pha_copy.fits"[REGION]" subspace 
 
--------------------------------------------------------------------------------
Data subspace for block REGION: Components: 1 Descriptors: 11 
--------------------------------------------------------------------------------
 
 --- Component 1 --- 
   1 SPEC_NUM             Int2                1:16384 
   2 ROWID                String              
   3 SHAPE                String              
   4 TG_R                 Real8               DEFAULT
   5 TG_D                 Real8               DEFAULT
   6 R                    Real8               DEFAULT
   7 ROTANG               Real8               DEFAULT
   8 COMPONENT            Int2                DEFAULT
   9 INCLUDE              Int2                DEFAULT
  10 TG_SRCID             Int2                DEFAULT
  11 TG_M                 Int2                DEFAULT

Bugs fixed in CIAO 4.5

The following is a list of bugs that were fixed in the CIAO 4.5 software release.

Reading in multiple files  (03 Jan 2012)

If you are writing a script or routine that reads in data using read_file or the IMAGECrate/TABLECrate objects, then the following should be used just before the crate goes out of scope, to avoid memory leaks and potential crashes: 

  cr.get_dataset().snip()

As an example: 

  def plot(filename, col1, col2):
    "Plots up col1 vs col2 from filename as circles"

    cr = read_file(filename)
    x = get_colvals(cr, col1)
    y = get_colvals(cr, col2)
    add_curve(x, y, ["line.style": "noline", "symbol.style": "circle",
                     "symbol.size": 2])

    cr.get_dataset().snip()

Writing out np.int32 columns on 64-bit builds of CIAO  (03 Jan 2012)

On 64-bit builds of CIAO, writing out a table crate that contains a column of NumPy np.int32 values with write_file or the write method of the crate results in corrupted values for that column in the output file.

You can determine if you are running a 64-bit build of CIAO by

  1. Looking at the "CIAO Installation" line of ciaover -v; 64-bit builds will display either 
      CIAO Installation: Linux64
  CIAO Installation: osx64
  2. from the ChIPS or Sherpa shells: 
      chips> sys.maxsize > 2**31
  True

Workaround:

Converting the column to the np.int64 data type using the astype method appears to avoid this corruption: e.g. if x.dtype == np.int32: 

  cd = CrateData()
  cd.name = "x"
  cd.values = x.astype(np.int64)

Data Model filter is not included in the get_filename output  (03 Jan 2012)

The get_filename command does not include any Data Model filter that was applied when reading the file: 

chips> table=read_file("asol1.fits[cols ra,dec]")
chips> table.get_filename ()
       'asol1.fits'

Problem writing files with same name as gzipped files.  (08 Oct 2012)

write may behave badly when a gzipped files exists and a user tries to write to a file with the same name without the .gz. 

chips> !ls
a.fits.gz
chips> cd = CrateData ()
chips> cd.values = np.arange (10)
chips> table = TABLECrate ()
chips> table.add_column (cd)
chips> table.write ('a.fits')
ValueError: dmSetData() operation on empty dmDescriptor pointer.

You can either pick a different file name or remove the .gz version of the file.


Last modified: 2 April 2013
CXC logo

The Chandra X-Ray Center (CXC) is operated for NASA by the Smithsonian Astrophysical Observatory. 60 Garden Street, Cambridge, MA 02138 USA.   Email: cxcweb@head.cfa.harvard.edu Smithsonian Institution, Copyright © 1998-2012. All rights reserved.