Last modified: December 2022

URL: https://cxc.cfa.harvard.edu/ciao/ahelp/dmmerge.html
AHELP for CIAO 4.16

dmmerge

Context: Tools::Table

Synopsis

Merge two or more compatible tables (e.g. event files) into one.

Syntax

dmmerge  infile outfile [outBlock] [lookupTab] [columnList] [clobber]
[verbose]

Description

`dmmerge' takes a stack of tables which have the same columns and creates a single output table which concatenates the input tables, so that the number of rows in the output table is the sum of the number of rows in all the input tables. `dmmerge' is often used to merge several event files. Note that before merging event files, make sure that they have the same tangent point (see "ahelp reproject_events").

This tool supports the DM stack syntax ("ahelp stack"), meaning that the input files may be given as a comma-separated list or as a list in a text file. The default behavior is for dmmerge to combine all columns of the input event files. It is possible, however, to use the DM virtual file syntax ("ahelp dm") to select (or omit) certain columns or rows of the files. Rather than including the same list of columns on each input file, the columnList parameter applies a given column filter to each file in the input stack.

The intrinsic difficulty with this procedure is that `dmmerge' needs to combine several input headers into one header for the output file. A lookup table - specified in the lookupTab parameter - describes how header keys are to be merged.


Examples

Example 1

% punlearn dmmerge
% dmmerge "file1_evt2.fits,file2_evt2.fits" merged.fits

Merge all columns in the two event files into one output file.

Example 2

% dmmerge "in1.fits[events],in2.fits[events],in3.fits[events]" \
out.fits columnList="time,chipx,chipy"

Open three input files (specifically the 'events' block) and merge the data columns 'time,chipx,chipy'.

Example 3

% dmmerge "@stack.lis[events][columns time,chipx,chipy]" out.fits

Open the 'events' block of the files listed in 'stack.lis' and extract the data columns 'time,chipx,chipy' for merging. This is the same as the previous example, but uses different syntax.

Example 4

% dmmerge @stack.lis out.fits

Open the files listed in the stack and use all data columns for merging.


Parameters

name type ftype def min max reqd stacks
infile file input       yes yes
outfile file output       yes  
outBlock string         no  
lookupTab file         no  
columnList string         no  
clobber boolean   no     no  
verbose integer   0 0 5 no  

Detailed Parameter Descriptions

Parameter=infile (file required filetype=input stacks=yes)

Input file name(s)

CIAO stacks may be a single filename, a comma-delimited list of names, or '@' followed by the name of a text file containing one filename on each line. Each filename may use the full CIAO DM virtual file syntax.

For one or multiple input files, there should be only one single output file.

Parameter=outfile (file required filetype=output)

Output file name

Parameter=outBlock (string not required)

Output block name

Output data block name. For a null string, it will use the same block name as the first input file, which is usually what you want.

The command

dmmerge "evt.fits[events][cols time,pha][merged_events]" columnList=""
outBlock=""

is equivalent to

dmmerge "evt.fits[events]" columnList="time,pha" outBlock=merged_events

Parameter=lookupTab (file not required)

lookup table containing merging rules for header keys. The default value is a redirect to the system version, ${ASCDS_CALIB}/dmmerge_header_lookup.txt, which enforces CXC event pipeline header merging rules; see "plist dmmerge" for the full path of your system default.

Each line of the file contains a keyword and a merging rule. The rules include FORCE (always replace key), SKIP (always omit key), WarnOmit-tol (print a warning and omit if key valuess differ by more than tol), WarnFirst (print warning and use first value if keys are different), among others.

Parameter=columnList (string not required)

Column list

Apply a column filter to the input stack. "columnList=time,pha" is equivalent to appending "[cols time,pha]" to each entry in the input stack.

This parameter may also be used to omit certain columns. "columnList=-phas" is equivalent to including a "[cols -phas]" filter on the input files.

Parameter=clobber (boolean not required default=no)

Clobber output if it exists? [y/n]

Parameter=verbose (integer not required default=0 min=0 max=5)

Display informational messages


Column metadata

When combining SKY or POS vector columns - e.g. X, Y - then the range of the output column will be the union of the input values. This range is stored in the TLMIN/MAX keys of the files, and can be seen with the 'cols' or 'subspace' options of dmlist. This is to support merging event files that have been reprojected and so have X,Y ranges that lie outside the default values (eg for ACIS X and Y go from 0.5 to 8192.5).


Bugs

Caveats

Extra GTI blocks in merged event file  (16 Apr 2007)

When merging ACIS event lists with dmmerge, the "data subspace" of the two event lists is combined. Due to a technical subtlety in the way that the EXPNO (ACIS frame exposure number) column is defined, this generates unwanted extra GTI extensions in the output file if the exposure numbers are not equal.

unix% dmmerge "acisf01587N002_evt2.fits,acisf07073N001_evt2.fits" NGC3379_merge.fits

unix% dmlist NGC3379_merge.fits blocks
 
--------------------------------------------------------------------------------
Dataset: NGC3379_merge.fits
--------------------------------------------------------------------------------
 
     Block Name                          Type         Dimensions
--------------------------------------------------------------------------------
Block    1: PRIMARY                        Null        
Block    2: EVENTS                         Table        15 cols x 1392531  rows
Block    3: GTI7                           Table         2 cols x 1        rows
Block    4: GTI2                           Table         2 cols x 3        rows
Block    5: GTI5                           Table         2 cols x 1        rows
Block    6: GTI6                           Table         2 cols x 2        rows
Block    7: GTI3                           Table         2 cols x 2        rows
Block    8: GTI8                           Table         2 cols x 2        rows
Block    9: GTI7_CPT7                      Table         2 cols x 1        rows
Block   10: GTI2_CPT8                      Table         2 cols x 2        rows
Block   11: GTI5_CPT9                      Table         2 cols x 1        rows
Block   12: GTI6_CPT10                     Table         2 cols x 1        rows
Block   13: GTI3_CPT11                     Table         2 cols x 2        rows
Block   14: GTI8_CPT12                     Table         2 cols x 1        rows

This occurs because the filters could not be combined into a single, valid range for the chip. It is undesireable for further analysis, as the tools cannot use multiple GTI blocks for a single chip, resulting in incorrect exposure values.

Workaround:

The subspace-editing capabilities in CIAO 3.4 simplify the workaround for this issue. It is now possible to delete the EXPNO subspace before merging the files:

unix% dmmerge \
      "acisf01587N002_evt2.fits[subspace -expno],acisf07073N001_evt2.fits[subspace -expno]" \
      NGC3379_merge_new.fits

unix% dmlist NGC3379_merge_new.fits blocks
 
--------------------------------------------------------------------------------
Dataset: NGC3379_merge_new.fits
--------------------------------------------------------------------------------
 
     Block Name                          Type         Dimensions
--------------------------------------------------------------------------------
Block    1: PRIMARY                        Null        
Block    2: EVENTS                         Table        15 cols x 1392531  rows
Block    3: GTI7                           Table         2 cols x 2        rows
Block    4: GTI2                           Table         2 cols x 5        rows
Block    5: GTI5                           Table         2 cols x 2        rows
Block    6: GTI6                           Table         2 cols x 3        rows
Block    7: GTI3                           Table         2 cols x 4        rows
Block    8: GTI8                           Table         2 cols x 3        rows

There are now just six GTI blocks in the output file.

Note that the addition of the subspace filter means that any user who intends to create lightcurves binned on exposure number from the merged output cannot use merge_all, since that information is eliminated from the subspace. (In general, lightcurves are binned on time.)

See Also

concept
merging_rules
dm
dmfiltering, dmopt
tools::coordinates
reproject_image, reproject_image_grid
tools::core
dmappend
tools::hrc
hrc_dtfstats
tools::image
dmimgcalc, dmimgfilt, dmregrid2
tools::response
addresp
tools::table
dmgroup, dmjoin, dmpaste, dmsort, dmtabfilt, dmtcalc, dmtype2split
tools::timing
dmgti