Chandra X-Ray Observatory (CXC)
Skip to the navigation links
Search the CIAO website or contact the CXC HelpDesk
Last modified: 9 October 2012

URL: http://cxc.harvard.edu/ciao4.4/bugs/dm.html

Bugs: Data Model

Return to: Bug List Index
Related pages: Region bugs | Incorrect GTIs when time-filtering grating data with dmcopy opt=all

For detailed information and examples of running this tool, refer to the dm ahelp file.

Table of Contents

General

Filtering Data

ASCII Kernel

Binning & Rebinning Images

Python module

Bugs

General

    Incorrect values from CAR transform. [New] (08 Oct 2012)

    The WCS library that the DM uses has a problem computing coordinate transforms that involve the CAR transform.

    Omitting the cols directive may lead to a crash. [New] (08 Oct 2012)

    If a users forgets to specify the cols directive, it may lead to a segmentation violation instead of generating a useful error message. 

    % dmlist emap.hist"[bin_low, bin_high, counts]" cols
# 62735: Received error signal SIGSEGV-segmentation violation.
# 62735: An invalid memory reference was made.
# 62735: segmentation fault: DMLIST (1) is: exit_upon_error->NULL

    You may get a seg fault if you try to create a very large image. What constitutes "very large" depends on the data type, but for long and float images, 8192x8192 pixels seems to be the threshold.

    (The image doesn't have to be square; it just needs to have 8192^2 pixels.)

    This condition may be met when the "update=no" option is used. Normally, when you filter a dataset, the data subspace (which describes the boundaries of each column's data and therefore is the intersection of the initial minima and maxima with any subsequent filters) gets updated to reflect the filtering. However, when you give the "update=no" option, you instruct the DM not to update the subspace to reflect the current filter. Therefore, the full ranges for x and y are used in the binning, and you get a 8192x8192 image (and a seg fault, for the reason described above).

    # DMCOPY (CIAO): Bad data type in filter string formatting

    If a logical column exists in the file, the DM will generate this warning. The output is unaffected by this bug.

    TNULL raw header keywords are not copied to the output file  (19 Aug 2011)

    BSCALE and BZERO keywords are copied to output file  (19 Aug 2011)

    The DM doesn't treat BSCALE and BZERO as structural keywords so they get copied to the output. If you have a file which is a floating point image, the addition of these keywords will create incorrect output results.

Filtering Data

    Filtering on some WCS columns produces incorrect results [New] (08 Oct 2012)

    When filtering on WCS columns, the range is taken by converting range of the parent columns and using those as the limits of the WCS columns. When the transform is highly non-linear, eg the TAN-P transform used to go from DETX,DETY to THETA,PHI, this can leads to incorrect limits and incorrect filters. Users who want to filter on WCS columns should give explict ranges and not rely on the computed min/maxes. 

    bad%  dmcopy "evt.fits[theta=:1]" 
good% dmcopy "evt.fits[theta=0:1]"

    Incorrect results when using '0' in filters[New] (11 Sep 2012)

    The datamodel replaces '0' in filters with a small value, 1e-16. If the other values being filtered are generally smaller than this, as one might have with flux values, then the wrong set of rows or pixels will be returned. 

    unix% cat foo.dat
#data
0.0
1.0e-12
1.0e-14
1.0e-16
1.0e-18
1.0e-20

unix% dmlist foo.dat"[data>0]" data,clean
#  data
                1E-12
                1E-14

unix% dmlist foo.dat"[data=0]" data,clean
#  data
                    0
                1E-16
                1E-18
                1E-20

    For less than and greater than, users can work around this by using a small number other than 0 

    % dmlist foo.dat"[data>1e-90]" data,clean
#  data
                1E-12
                1E-14
                1E-16
                1E-18
                1E-20

    Multi-valued string filters don't work

    "col=foo" is okay, but "col=foo,bar" isn't.

    Workaround:

    Use "col=foo,col=bar" instead.

    Creating a vector on-the-fly when region filtering

    When region-filtering images, you can create a vector on the fly from any two axes by using a filter like "(#1,#3)=circle(...)". Although the image is filtered correctly with a temporary vector, the region filter isn't recorded in the subspace. Hence, tools that use the filtered file don't know that pixels outside the filter region are invalid. As a result, dmstat reports no nulls in the filtered image (unless you explicitly tell the DM to set pixels outside the filter to null by using "opt null=...").

    Applying a bit-filter expression to an integer column does not work, nor does it cause an error.

    Using incorrect syntax with the rectangle shape does not fail when filtering.

    For example, setting xmax > xmin and/or ymax > ymin. Instead it appears that the Data Model simply swaps the min and max values.

    Filtering an image on logical coordinates causes problems when the short cut of omitting a number (i.e. to indicate the default value) is used.

    The exit status of dmcopy is also incorrectly set to 0 (success): 

    unix% dmcopy "image.fits[#1=1:20,#2=:]" delme.fits
# DMCOPY (CIAO): [ftColRead]: FITS error 308 bad first element number in 
dataset image.fits Block 1 PRIMARY

unix% echo $status
0

    Workarounds:

    1. Omit the "#2=:" from the filter

    2. Specify a range for both elements: [#1=1:20,#2=1:20]

    Trying to exclude a region filter with update=no will cause the image to be filtered by the region's bounding box.

    For example: 

    unix% dmcopy "acis_img.fits[exclude sky=region(src.fits)][opt full,update=no]" filtered.fits

    The regions are correctly excluded; however, the image is also clipped at the bounding box around all the excluded shapes, so the corners of a few chips are removed.

    Workarounds:

    1. Remove update=no. In this case, the Data Model internally inverts all exclude filters to be an inclusive filter, and correctly filters the image.

      Be aware that this process is much slower if the region is large. In that case, it will also add a large region keyword to the file's header, noticeably slowing down any operation on that file.

    2. For ASCII region files, it is also possible to manually invert the filter in the file. The "field()" region syntax is used to include the entire field, then remove the undesired sources. For instance, 

      # Region file format: CIAO version 1.0
circle(1635.5,4113.5,135.11408)
circle(3975,4233,20)
circle(2565.5,4129.5,40)
circle(2129.5,4007.5,40)

      would become 

      # Region file format: CIAO version 1.0
field()
-circle(1635.5,4113.5,135.11408)
-circle(3975,4233,20)
-circle(2565.5,4129.5,40)
-circle(2129.5,4007.5,40)

      and the dmcopy filtering command would be 

      unix% dmcopy "acis_img.fits[sky=region(src.ascii)][opt full,update=no]" filtered.fits

    There is a bug in region and range filters.

    This command: 

    unix% dmcopy "input.fits[sky=circle(4096,4096,100),y=4020:4100,4250,4350]" \
      output.fits

    fails to do the y filter altogether. This bug also applies to exclude filters.

    Using "!=" with string filtering causes dmlist to fail.  (05 Mar 2007)

    For example, the following commands both fail: 

    unix% dmlist "region.fits[shape!=Annulus]" data

unix% dmlist catalog.fits"[COMMENT!='weak'][cols COMMENT]" data

    The "or" syntax ("||") doesn't work on string columns  (17 Jul 2009)

    For example, this command does not find all instances of "PMterm" in the selected columns: 

    unix% dmlist stat.fits"[src=PMterm||det=PMterm||mst=PMterm]" counts
13

    Compare to 

    unix% dmlist stat.fits"[cols det,src,mst]" data,clean | grep PMterm | wc -l
27

    The "or" syntax ("||") doesn't work inside a clause  (15 Apr 2011)

    This example command does not work: 

    unix% dmlist "evt2.fits[(ccd_id=5||ccd_id=7),pha=2500:3500]" blocks

    Workaround:

    Rewrite to include the filter conditions in each part of the conditional. 

    unix% dmlist "evt2.fits[(ccd_id=7,pha=2500:3500)||(ccd_id=5,pha=2500:3500)]" blocks

ASCII Kernel

    Virtual columns are not supported.  (11 Dec 2007)

    In the DM, you can normally do 

    unix% dmlist evt.fits"[cols ra,dec]"  data

    even though RA and Dec are just coordinate systems defined on the X and Y columns in the file; the DM applies the transform on the fly. This doesn't work yet for ASCII files.

    DTF-FIXED keyword comments may be truncated.  (15 Jan 2008)

    DTF-FIXED header lines may be up to 1024 characters long. However, if the keyword is longer than the FITS standard, the comment is truncated. 

    unix% input.txt output.dtf'[opt kernel=text/dtf-fixed]'

    In input.txt: 

    TTYPE14 = 'Class' / LV Class Exo: M = missile [B = tactical ballistic  
missile (except Redstone) apo=80:200]
R = research rocket O = orbital LV V = RTV Y = Exo weather rocket X =  
Big test rocket D= Deep space launch

    In output.dtf: 

    TTYPE14 = "Class " / LV Class Exo: M = missile [B = tactical  
ballistic missile (except Redstone) apo

Binning & Rebinning Images

    Rebinning an image with different values for the two axes causes the coordinate information to be lost

    For example: 

    unix% dmcopy acis.img"[bin x=::5,y=::6]" acis5x6.img

    Using the same value for both axes works correctly: 

    unix% dmcopy acis.img"[bin (x,y)=::5]" acis5.img

    Running a DM tool on an image where one of the axes has been filtered results in an error.  (17 Jul 2009)

    unix% dmextract
Input event file (ccd3.sky4.fits[y=3767:][bin sky=annulus(3786,3767,0:380:4)]):
Enter output file name (rprof.fits):
# dmextract (CIAO 4.0 Beta 2): WARNING: Input file, "ccd3.sky4.fits[y=3767:]", 
has no rows in it.

Bus error

Python module

    dmKeyRead on vector column returns extra, incorrect values. [New] (14 Sep 2012)

    The CXC datamodel allows for keywords and columns to be used interchangably; which allows users to extract a single column value using the dmKeyRead routine.

    For array columns this should return the 1st array element; however, currently it returns an array the length of the original column with only the first element correctly populated. 

    >>>  import cxcdm as cdm
>>>  tab = dmTableOpen("FOV.fits")
>>>  cdm.dmKeyRead( tab, "x") 
(<dmDescriptor object at 0x2b7fb5b49288>, array([  3.84334592e+003,   1.12619810e-312,   0.00000000e+000,
         0.00000000e+000,   0.00000000e+000,   0.00000000e+000,
         0.00000000e+000,   4.79243676e-322,   1.12619810e-312,
         1.12619810e-312,   0.00000000e+000,   0.00000000e+000]))

    Only the first array element above is valid; the other values are uninitialized memory.

Bugs fixed in CIAO 4.4

The following is a list of bugs that were fixed in the CIAO 4.4 software release.

Problem parsing ASCII file due to '\r' at end of line  (03 May 2011)

The ASCII kernel cannot access the last column name or value of a file if it includes the carriage return character (\r). The "\r" is included as part of the text/numerical value, which breaks the parsing.

Filtering creates incorrect GTIs if the subspace contains parentheses  (15 Dec 2010)

If a filter that contains parentheses is applied to a file, subsequent filtering will not read the subspace correctly, resulting in incorrect GTI information.

For example: 

unix% dmcopy "evt1a.fits[(tg_mlam,pi)=region(letgD1999-07-22pireg_tgmap_N0001.fits)]" bg_filtered.fits

unix% dmcopy "bg_filtered.fits[time=<time1>:<time2>]" time_filtered.fits

The first filter adds a subspace for the vector '(tg_mlam,pi)'. When reading the file to apply the second filter, the Data Model has trouble processing the subspace record for the column because of the parentheses. The DM can't read the full subspace, so the GTI information is not loaded and the time subspace is incorrect from that point onward.

Workaround:

Applying both filters in one dmcopy works correctly: 

unix% dmcopy \
      "evt1a.fits[(tg_mlam,pi)=region(letgD1999-07-22pireg_tgmap_N0001.fits),time=<time1>:<time2>]" \
      time_filtered.fits

The DM doesn't catch incorrect binning syntax  (17 Jul 2009)

unix% dmcopy foo.lis"[bin col1=0.5:1024.5:1,col2=0.5:1024:5:1]"
foo.img

The bad syntax in the col2 binning - 0.5:1024:5:1 - should produce an error. Instead it ignores the ":1" and bins col2 to a length of 205.


Last modified: 9 October 2012
CXC logo

The Chandra X-Ray Center (CXC) is operated for NASA by the Smithsonian Astrophysical Observatory. 60 Garden Street, Cambridge, MA 02138 USA.   Email: cxcweb@head.cfa.harvard.edu Smithsonian Institution, Copyright © 1998-2012. All rights reserved.