Bugs: CRATES
Table of Contents
Caveats
Bugs
Caveats
Cannot add rows to a column that is part of a virtual column.
Users can not add rows a column which is a component of a vector using Crates (the new values are set to 0). User can extend scalar and array columns. To work around this, users must add rows the vector column itself rather than the components.
Bugs
Problem with single byte bit columns
Crates has problems when trying to write out a bit column, when the column has less than 8 bits (so fill only 1 byte).
Crates will convert all transforms into their binned form.
Transforms are assumed to be applied to image pixels, so when they are created they are adjusted to match what would be an image pixel boundary, even if the transform is created on a table column.
So, for example trying to create a transform that converts degrees F to C:
tempF = +32 + 1.8 * tempC
A linear transform with scale=1.8 and offset=32, is written out as
tempF = +32.90 [degree F] +1.80 * (tempC -0.50)
While mathematically correct, the extra half "pixel" offset is unnecessary when dealing with transforms applied to table columns.
Writing NaN to keywords causes crash
Attempting to write a NaN value to a keyword results in a crash of the python interpreter
>>> tab = read_file("a.fits")
>>> key = CrateKey()
>>> key.name="foobar"
>>> key.value = np.nan
>>> tab.add_key( key )
>>> write_file( tab, "b.fits")
RuntimeError: dmKeyWrite() could not write key. 'foobar'
>>> quit()
Exception IOError: IOError('FITS error 402 writing key foobar',) in <boundmethod CrateDataset.__del__ of Crate Dataset:
...
*** glibc detected *** /export/ciao-4.5/ots/bin/python: double free or
corruption (!prev): 0x0000000003026480 ***
...
(core dumped) ipython --profile chips -i -c "\"${cmd}\""
Byte datatype subspace columns
pycrates may crash if the file contains subspace columns with byte data-type. Examples include some HRC datasets with byte type SUB_MJF.
Workaround:
Workaround: for the HRC case, you can remove the subspace column with
% dmcopy hrc_evt[subspace -sub_mjf] hrc_evt_mod
prior to loading the file into crates.
Write access is not robustly checked.
Users need to be careful when trying to modify a file in place as the checks on whether or not a file is writeable are not robust. Simply specifying mode="rw" will not produce an error nor an exception if the file is not actually writeable.
unix% chmod 400 myfile.fits
unix% python
>>> from pycrates import *
>>> rr = read_file("hrcf01801N006_evt2.fits", mode="rw")
>>> # do something to modify file
>>> rr.write()
The above sequence will not geneate an exception even though the file is not writeable.
Trying to write a file with a Byte data-type in the subspace fails.
HRC event files have a byte type (8 bit unsigned integer) SUB_MJF subspace column present. Trying to read in the file into a crate and writing it back out will fail.
unix% dmlist hrcf01801N006_evt2.fits subspace
--------------------------------------------------------------------------------
Data subspace for block EVENTS: Components: 1 Descriptors: 33
--------------------------------------------------------------------------------
...
31 ENDMNF Int4 0:127
32 SUB_MJF Byte 0:64
33 CLKTICKS Int4 0:1091567608
unix% python
>>> from pycrates import *
>>> rr = read_file("hrcf01801N006_evt2.fits")
>>> write_file(rr, "/tmp/copy")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/export/ciao-4.5/lib/python2.7/site-packages/pycrates/hlui.py", line 985, in write_file
crate.write(outfile=filename, clobber=clobber)
File "/export/ciao-4.5/lib/python2.7/site-packages/pycrates/tablecrate.py", line 294, in write
backend.write( self, outfile=outfile )
File "/export/ciao-4.5/lib/python2.7/site-packages/pycrates/io/dm_backend.py", line 534, in write
self.__write_block( crate, close_flag )
File "/export/ciao-4.5/lib/python2.7/site-packages/pycrates/io/dm_backend.py", line 578, in __write_block
self.__write_subspace(block, crate)
File "/export/ciao-4.5/lib/python2.7/site-packages/pycrates/io/dm_backend.py", line 626, in __write_subspace
ssitem.unit, ssitem.range_min, ssitem.range_max )
TypeError: dmSubspaceColCreate() mins or maxes argument is not a supported type.
The only workaround is to remove the SUB_MJF subspace before reading it into a crate.
unix% dmcopy "hrcf01801N006_evt2.fits[subspace -SUB_MJF]" copy_evt.fits
unix% python
>>> from pycrates import *
>>> rr = read_file("copy_evt.fits")
>>> write_file(rr, "copy2")
Using numpy views of arrays can lead to data corruption.
numpy uses a special indexing scheme to access arrays that have been sliced (ie truncated at either end) or when for example transposing 2D arrays. The origianal data are preserved and the user only sees a view of the array.
The crates python to datamodel C interface does not handle views correctly. It may result in data being truncated or simply producing nonsensical results.
Users should make sure that any arrays sent to pycrates are in C_CONTIGUOUS order and ALGINED
>>> foo = np.arange(10).reshape(5,2) >>> foo2 = bob.T >>> foo.flags C_CONTIGUOUS : True F_CONTIGUOUS : False OWNDATA : False WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False >>> foo2.flags C_CONTIGUOUS : False F_CONTIGUOUS : True OWNDATA : False WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False