Synopsis
The CrateData object is used to store column or image data.
Syntax
CrateData()
Description
The CRATES Library uses CrateData objects to store data values of a columm or image.
Important fields of a CrateData object
Unlike the other parts of the Crates interface, access to information in a CrateData object is restricted to the methods and fields of the Python object (i.e. there are no separate functions). The important fields are listed below.
| Field | Description |
|---|---|
| values | The data values, stored as a NumPy array. |
| name | The name of the object. |
| unit | The units of the value, if set. |
| desc | A description of the object, if set. |
Reading in a column
Here we read in the X column from the file evt2.fits and inspect the CrateData object that is returned.
>>> cr = read_file("evt2.fits")
>>> x = cr.get_column("x")
>>> x
Name: x
Shape: (151781,)
Unit: pixel
Desc: sky coordinates
Eltype: Scalar
Range:
Min: 0.5
Max: 8192.5
>>> x.values.mean()
4468.9414
>>> y = cr.get_column("y")
>>> from matplotlib import pyplot as plt
>>> plt.scatter(x.values, y.values, marker='.')
>>> plt.xlabel(f"{x.name} ({x.unit})")
>>> plt.ylabel(f"{y.name} ({y.unit})")
>>> plt.show()
Modifying a column
With the above set up, we can modify values; for instance
>>> xv = x.values >>> xv += 0.5 >>> x.values.mean() 4469.4414
Note that changing the values in the xv array change the underlying CrateData object. To ensure you are working with a copy of the data (so that changes do not get propagated back to the original Crate), use the NumPy copy() method - e.g.
>>> xv = x.values.copy()
or the copy_colvals() routine from Crates.
Creating a column
The following lines create a CrateData object storing the values from the z NumPy array and called "zcol" (the unit and description fields are optional but are added here):
>>> cd = CrateData() >>> cd.name = "zcol" >>> cd.values = z >>> cd.desc = 'The z value' >>> cd.unit = 'erg /cm**2 /s'
This can then be added to a table crate using either
>>> add_col(cr, cd)
or
>>> cr.add_column(cd)
where the optional index argument of add_column can be used to place the column at a specific location, rather than at the end (the default).
Vector columns
In the following we access the SKY vector column of a Chandra events file.
>>> sky = cr.get_column("sky")
>>> sky
Name: sky
Shape: (151781, 2)
Datatype: float32
Nsets: 151781
Unit: pixel
Desc: sky coordinates
Eltype: Vector
NumCpts: 2
Cpts: ['x', 'y']
Range:
Min: 0.5
Max: 8192.5
>>> sky.values.shape
(151781, 2)
>>> x = sky.values[:, 0]
>>> x.shape
(151781,)
>>> y = sky.values[:, 1]
>>> row0 = sky.values[0, :]
>>> print(row0)
[5071.793 5225.1724]
Virtual columns
There is no significant difference to handling virtual columns (that is, a column which is calculated by applying a transformation to an actual column in a crate):
>>> msc = cr.get_column("MSC")
>>> msc.is_virtual()
True
>>> msc
Name: MSC
Shape: (151781, 2)
Unit: deg
Desc:
Eltype: Virtual Vector
NumCpts: 2
Cpts: ['PHI', 'THETA']
How about images?
As there's no real distinction between a column and image for the CrateData() object, the read, modify, and write sections are essentially the same as above, as shown in this example
>>> cr = read_file("evt2.fits[bin sky=::8]")
>>> img = cr.get_image()
>>> img
Name: EVENTS_IMAGE
Datatype: int16
Unit:
Desc:
Eltype: Array
Ndim: 2
Dimarr: (1024, 1024)
Range:
Min: None
Max: None
>>> img.values.mean()
0.14474964141845703
>>> plt.imshow(np.log10(img.values), origin='lower')
When adding an image to a crate, use either the add_piximg command or the add_image method of the IMAGECrate; that is one of
>>> add_piximg(cr, img) >>> cr.add_image(img)
The CrateData object
There are three CrateData object types: Regular, Vector, and Virtual.
Regular Objects
Regular CrateData objects contain values from an image array or a single table column, which can be composed of either scalar or array values.
Multi-dimensional data
A CrateData object can contain multi-dimensional data, and the interpretation of whether this is an image or an array column is made by adding it to an IMAGECrate or TABLECrate respectively, with the add_col or add_piximg commands.
Vector Columns
Vector columns are two or more columns that have been grouped together under the same name, but each component column has its own name as well. For example, the vector column SKY has two components, X position and Y position. The notation for vectors in the CRATES library is
vector(cpt1, cpt2, ...)
so the sky vector is represented as
SKY(X,Y)
.
Vector CrateData objects have values which consist of two or more CrateData objects. Using the previous example, the SKY vector values point to regular columns X position and Y position.
Virtual Objects
A Virtual CrateData object has values that have been calculated via a transform from another CrateData object. For example, the virtual column RA is defined by a transform associated with the regular column X.
Vector columns can also be virtual. EQPOS is a virtual vector column comprised of two virtual column components RA and DEC. EQPOS(RA,DEC) values are determined by applying a transform to SKY(X,Y) values.
Creating a vector column
Two following routines were added to simplify creating vector columns: create_vector_column and create_virtual_column.
This example shows how to create a SKY vector component made up of X and Y arrays:
rng = np.random.default_rng() x = rng.normal(4782.3, 5, size=1000).astype(np.float32) y = rng.normal(5234.1, 5, size=1000).astype(np.float32)
creates 1000 pairs of X,Y values drawn from the normal distribution, and converted to 32-bit floats. To create a SKY vector column you would then say:
sky = create_vector_column('SKY', ['X', 'Y'])
sky.unit = 'pixel'
sky.desc = 'sky coordinates'
sky.values = np.column_stack((x,y))The sky column can then be added to a crate (a new one in this example) and written out:
cr = TABLECrate()
cr.name = 'GAUSS'
cr.add_column(sky)
cr.write('gauss.fits', clobber=True)
unix% dmlist gauss.fits blocks
--------------------------------------------------------------------------------
Dataset: gauss.fits
--------------------------------------------------------------------------------
Block Name Type Dimensions
--------------------------------------------------------------------------------
Block 1: PRIMARY Null
Block 2: GAUSS Table 1 cols x 1000 rows
unix% dmlist gauss1.fits cols
--------------------------------------------------------------------------------
Columns for Table Block GAUSS
--------------------------------------------------------------------------------
ColNo Name Unit Type Range
1 SKY(X,Y) pixel Real4 -Inf:+Inf sky coordinates
Note that the values are added to the vector column - the parent - as an array with shape (nrows, num cpts), which is what column_stack creates. Setting the unit and desc fields are not required, but can provide useful metadata.
Representing bit values
Boolean columns
A column of bollean values - that is True or False - can be created as with any other simple type. For instance, the following adds a column called ENFLAG that indicates whether the energy value of the row is between 500 and 7000:
>>> cr = read_file('evt2.fits')
>>> flag = CrateData()
>>> flag.name = 'ENFLAG'
>>> flag.desc = 'Interesting energy'
>>> envals = cr.get_column('energy').values
>>> flag.values = (envals >= 500) & (envals <= 7000)
>>> cr.add_column(flag)
>>> cr.write('flagged.evt2')which creates a file with an ENFLAG column:
>>> !dmlist "flagged.evt2[cols enflag]" cols ----------------------------------------------------------------- Columns for Table Block EVENTS ----------------------------------------------------------------- ColNo Name Unit Type Range 1 ENFLAG Logical Interesting energy
Boolean arrays
The STATUS column of a Chandra event file is a bit column, where each bit represents a different flag value. The Python representation uses an array of np.uint8 values, one for each bit, where a non-zero value indicates set and 0 is unset. For instance:
>>> cr = read_file('evt2.fits')
>>> status = cr.get_column('status')
>>> print(status)
Name: status
Shape: (107664, 32)
Datatype: uint8 | Bit[32]
Nsets: 107664
Unit:
Desc: event status bits
Eltype: Array
Ndim: 1
Dimarr: (32,)
Range:
Min: 0
Max: 255
>>> status.is_bit_array()
True
>>> len(status.values[0])
32
>>> print(status.values[0])
[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
>>> status.values[0][1] = 1
>>> status.values[0][10] = 1
>>> status.values[0][20] = 1
>>> print(status.values[0])
[0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0]
>>> cr.write('test.fits', clobber=True)
>>> !dmlist "test.fits[#row=1][cols status]" data,clean
# status[4]
01000000001000000000100000000000
Boolean arrays
The is_bit_array() method of a CrateData object can be used to determine whether a sequence of bits is being represented as a bit array. The resize_bit_array() method is used to increase or decrease the size of the bit array.
Loading Crates
The Crates module is automatically imported into Sherpa sessions, otherwise use one of the following:
from pycrates import *
or
import pycrates
Changes in CIAO 4.8
Support for variable-length arrays
Support for variable-length arrays has been improved and the CrateData object now supports the is_varlen and get_fixed_length_array methods.
Bugs
See the bug pages on the CIAO website for an up-to-date listing of known bugs.
Refer to the CIAO bug pages for an up-to-date listing of known issues.