Bugs: Data Model

• Bugs: General
• Filtering Data
• ASCII Kernel
• Binning & Rebinning Images
• Bugs fixed in CIAO 4.5

Table of Contents

General

• Incorrect values from CAR transform.

• You may get a seg fault if you try to create a very large image. What constitutes "very large" depends on the data type, but for long and float images, 8192x8192 pixels seems to be the threshold.

• TNULL raw header keywords are not copied to the output file  (19 Aug 2011)

• DM support for FITS TC* keywords (02 May 2013)

Filtering Data

• Filtering on some WCS columns produces incorrect results  (08 Oct 2012)

• Multi-valued string filters don't work

• Creating a vector on-the-fly when region filtering

• Applying a bit-filter expression to an integer column does not work, nor does it cause an error.

• Using incorrect syntax with the rectangle shape does not fail when filtering.

• Filtering an image on logical coordinates causes problems when the short cut of omitting a number (i.e. to indicate the default value) is used.

• Trying to exclude a region filter with update=no will cause the image to be filtered by the region's bounding box.

• Using "!=" with string filtering causes dmlist to fail.  (05 Mar 2007)

• The "or" syntax ("||") doesn't work on string columns  (17 Jul 2009)

• The "or" syntax ("||") doesn't work inside a clause  (15 Apr 2011)

ASCII Kernel

• Virtual columns are not supported.  (11 Dec 2007)

• DTF-FIXED keyword comments may be truncated.  (15 Jan 2008)

• Problems with CSCView .tsv files with 'G' datatypes (02 May 2013)

Binning & Rebinning Images

• Rebinning an image with different values for the two axes causes the coordinate information to be lost

• Running a DM tool on an image where one of the axes has been filtered results in an error.  (17 Jul 2009)

Bugs

General

Incorrect values from CAR transform.

The WCS library that the DM uses has a problem computing coordinate transforms that involve the CAR transform.

You may get a seg fault if you try to create a very large image. What constitutes "very large" depends on the data type, but for long and float images, 8192x8192 pixels seems to be the threshold.

(The image doesn't have to be square; it just needs to have 8192^2 pixels.)

This condition may be met when the "update=no" option is used. Normally, when you filter a dataset, the data subspace (which describes the boundaries of each column's data and therefore is the intersection of the initial minima and maxima with any subsequent filters) gets updated to reflect the filtering. However, when you give the "update=no" option, you instruct the DM not to update the subspace to reflect the current filter. Therefore, the full ranges for x and y are used in the binning, and you get a 8192x8192 image (and a seg fault, for the reason described above).

TNULL raw header keywords are not copied to the output file  (19 Aug 2011)

DM support for FITS TC* keywords (02 May 2013)

There are three issues with the generic use of TC* keywords in FITS files read and written by DM tools.

The first issue: TC*n[A-Z]. The DM function that composes this keyword does not strip off the 'P' from the before adding number and letter to the end.

Second issue: TCTY* keys are not recognized and therefore are not processed.

Third issue: DM is not always retaining the T*NAM information when it stores information on the DM descriptor, resulting in output keywords T*TYP instead.

Filtering Data

Filtering on some WCS columns produces incorrect results  (08 Oct 2012)

When filtering on WCS columns, the range is taken by converting range of the parent columns and using those as the limits of the WCS columns. When the transform is highly non-linear, eg the TAN-P transform used to go from DETX,DETY to THETA,PHI, this can leads to incorrect limits and incorrect filters. Users who want to filter on WCS columns should give explict ranges and not rely on the computed min/maxes.

bad%  dmcopy "evt.fits[theta=:1]" 
good% dmcopy "evt.fits[theta=0:1]"

Multi-valued string filters don't work

"col=foo" is okay, but "col=foo,bar" isn't.

Workaround:

Use "col=foo,col=bar" instead.

Creating a vector on-the-fly when region filtering

When region-filtering images, you can create a vector on the fly from any two axes by using a filter like "(#1,#3)=circle(...)". Although the image is filtered correctly with a temporary vector, the region filter isn't recorded in the subspace. Hence, tools that use the filtered file don't know that pixels outside the filter region are invalid. As a result, dmstat reports no nulls in the filtered image (unless you explicitly tell the DM to set pixels outside the filter to null by using "opt null=...").

Applying a bit-filter expression to an integer column does not work, nor does it cause an error.

Using incorrect syntax with the rectangle shape does not fail when filtering.

For example, setting xmax > xmin and/or ymax > ymin. Instead it appears that the Data Model simply swaps the min and max values.

Filtering an image on logical coordinates causes problems when the short cut of omitting a number (i.e. to indicate the default value) is used.

The exit status of dmcopy is also incorrectly set to 0 (success):

unix% dmcopy "image.fits[#1=1:20,#2=:]" delme.fits
# DMCOPY (CIAO): [ftColRead]: FITS error 308 bad first element number in 
dataset image.fits Block 1 PRIMARY

unix% echo $status
0

Workarounds:

1. Omit the "#2=:" from the filter

2. Specify a range for both elements: [#1=1:20,#2=1:20]

Trying to exclude a region filter with update=no will cause the image to be filtered by the region's bounding box.

For example:

unix% dmcopy "acis_img.fits[exclude sky=region(src.fits)][opt full,update=no]" filtered.fits 

The regions are correctly excluded; however, the image is also clipped at the bounding box around all the excluded shapes, so the corners of a few chips are removed.

Workarounds:

1. Remove update=no. In this case, the Data Model internally inverts all exclude filters to be an inclusive filter, and correctly filters the image.

   Be aware that this process is much slower if the region is large. In that case, it will also add a large region keyword to the file's header, noticeably slowing down any operation on that file.

2. For ASCII region files, it is also possible to manually invert the filter in the file. The "field()" region syntax is used to include the entire field, then remove the undesired sources. For instance,

   # Region file format: CIAO version 1.0
   circle(1635.5,4113.5,135.11408)
   circle(3975,4233,20)
   circle(2565.5,4129.5,40)
   circle(2129.5,4007.5,40)
   

   would become

   # Region file format: CIAO version 1.0
   field()
   -circle(1635.5,4113.5,135.11408)
   -circle(3975,4233,20)
   -circle(2565.5,4129.5,40)
   -circle(2129.5,4007.5,40)
   

   and the dmcopy filtering command would be

   unix% dmcopy "acis_img.fits[sky=region(src.ascii)][opt full,update=no]" filtered.fits 
   

Using "!=" with string filtering causes dmlist to fail.  (05 Mar 2007)

For example, the following commands both fail:

unix% dmlist "region.fits[shape!=Annulus]" data

unix% dmlist catalog.fits"[COMMENT!='weak'][cols COMMENT]" data

The "or" syntax ("||") doesn't work on string columns  (17 Jul 2009)

For example, this command does not find all instances of "PMterm" in the selected columns:

unix% dmlist stat.fits"[src=PMterm||det=PMterm||mst=PMterm]" counts
13

Compare to

unix% dmlist stat.fits"[cols det,src,mst]" data,clean | grep PMterm | wc -l
27

The "or" syntax ("||") doesn't work inside a clause  (15 Apr 2011)

This example command does not work:

unix% dmlist "evt2.fits[(ccd_id=5||ccd_id=7),pha=2500:3500]" blocks 

Workaround:

Rewrite to include the filter conditions in each part of the conditional.

unix% dmlist "evt2.fits[(ccd_id=7,pha=2500:3500)||(ccd_id=5,pha=2500:3500)]" blocks 

ASCII Kernel

Virtual columns are not supported.  (11 Dec 2007)

In the DM, you can normally do

unix% dmlist evt.fits"[cols ra,dec]"  data

even though RA and Dec are just coordinate systems defined on the X and Y columns in the file; the DM applies the transform on the fly. This doesn't work yet for ASCII files.

DTF-FIXED keyword comments may be truncated.  (15 Jan 2008)

DTF-FIXED header lines may be up to 1024 characters long. However, if the keyword is longer than the FITS standard, the comment is truncated.

unix% input.txt output.dtf'[opt kernel=text/dtf-fixed]'

In input.txt:

TTYPE14 = 'Class' / LV Class Exo: M = missile [B = tactical ballistic  
missile (except Redstone) apo=80:200]
R = research rocket O = orbital LV V = RTV Y = Exo weather rocket X =  
Big test rocket D= Deep space launch

In output.dtf:

TTYPE14 = "Class " / LV Class Exo: M = missile [B = tactical  
ballistic missile (except Redstone) apo

Problems with CSCView .tsv files with 'G' datatypes (02 May 2013)

The text/tsv ASCII parser does not recognize columns with a G format used for floating-point, real valued values such as src_rate_aper_*. The exact behavior will vary based on the other columns returned, and in what order, but generally all columns after the G format will be unreadble by DM tools.

Workaround:

Users can work around the problem by editing the .tsv file and changing any (Gx.y) values to (Fx.y). For example:

Original file:

#Column src_rate_aper_b (G9.5)  Aperture-corrected net count rate in...

Modified file:

#Column src_rate_aper_b (F9.5)  Aperture-corrected net count rate in...

Binning & Rebinning Images

Rebinning an image with different values for the two axes causes the coordinate information to be lost

For example:

unix% dmcopy acis.img"[bin x=::5,y=::6]" acis5x6.img

Using the same value for both axes works correctly:

unix% dmcopy acis.img"[bin (x,y)=::5]" acis5.img

Running a DM tool on an image where one of the axes has been filtered results in an error.  (17 Jul 2009)

unix% dmextract
Input event file (ccd3.sky4.fits[y=3767:][bin sky=annulus(3786,3767,0:380:4)]):
Enter output file name (rprof.fits):
# dmextract (CIAO 4.0 Beta 2): WARNING: Input file, "ccd3.sky4.fits[y=3767:]", 
has no rows in it.

Bus error

Bugs fixed in CIAO 4.5

The following is a list of bugs that were fixed in the CIAO 4.5 software release.

% dmlist emap.hist"[bin_low, bin_high, counts]" cols
# 62735: Received error signal SIGSEGV-segmentation violation.
# 62735: An invalid memory reference was made.
# 62735: segmentation fault: DMLIST (1) is: exit_upon_error->NULL 

unix% cat foo.dat
#data
0.0
1.0e-12
1.0e-14
1.0e-16
1.0e-18
1.0e-20

unix% dmlist foo.dat"[data>0]" data,clean
#  data
                1E-12
                1E-14

unix% dmlist foo.dat"[data=0]" data,clean
#  data
                    0
                1E-16
                1E-18
                1E-20

% dmlist foo.dat"[data>1e-90]" data,clean
#  data
                1E-12
                1E-14
                1E-16
                1E-18
                1E-20

unix% dmcopy "input.fits[sky=circle(4096,4096,100),y=4020:4100,4250,4350]" \
      output.fits