Synopsis
Merge two or more compatible tables (e.g. event files) into one.
Syntax
dmmerge infile outfile [outBlock] [lookupTab] [columnList] [clobber] [verbose]
Description
`dmmerge' takes a stack of tables which have the same columns and creates a single output table which concatenates the input tables, so that the number of rows in the output table is the sum of the number of rows in all the input tables. `dmmerge' is often used to merge several event files. Note that before merging event files, make sure that they have the same tangent point (see "ahelp reproject_events").
This tool supports the DM stack syntax ("ahelp stack"), meaning that the input files may be given as a comma-separated list or as a list in a text file. The default behavior is for dmmerge to combine all columns of the input event files. It is possible, however, to use the DM virtual file syntax ("ahelp dm") to select (or omit) certain columns or rows of the files. Rather than including the same list of columns on each input file, the columnList parameter applies a given column filter to each file in the input stack.
The intrinsic difficulty with this procedure is that `dmmerge' needs to combine several input headers into one header for the output file. A lookup table - specified in the lookupTab parameter - describes how header keys are to be merged.
Examples
Example 1
% punlearn dmmerge % dmmerge "file1_evt2.fits,file2_evt2.fits" merged.fits
Merge all columns in the two event files into one output file.
Example 2
% dmmerge "in1.fits[events],in2.fits[events],in3.fits[events]" \ out.fits columnList="time,chipx,chipy"
Open three input files (specifically the 'events' block) and merge the data columns 'time,chipx,chipy'.
Example 3
% dmmerge "@stack.lis[events][columns time,chipx,chipy]" out.fits
Open the 'events' block of the files listed in 'stack.lis' and extract the data columns 'time,chipx,chipy' for merging. This is the same as the previous example, but uses different syntax.
Example 4
% dmmerge @stack.lis out.fits
Open the files listed in the stack and use all data columns for merging.
Parameters
| name | type | ftype | def | min | max | reqd | stacks |
|---|---|---|---|---|---|---|---|
| infile | file | input | yes | yes | |||
| outfile | file | output | yes | ||||
| outBlock | string | no | |||||
| lookupTab | file | no | |||||
| columnList | string | no | |||||
| clobber | boolean | no | no | ||||
| verbose | integer | 0 | 0 | 5 | no |
Detailed Parameter Descriptions
Parameter=infile (file required filetype=input stacks=yes)
Input file name(s)
CIAO stacks may be a single filename, a comma-delimited list of names, or '@' followed by the name of a text file containing one filename on each line. Each filename may use the full CIAO DM virtual file syntax.
For one or multiple input files, there should be only one single output file.
Parameter=outfile (file required filetype=output)
Output file name
Parameter=outBlock (string not required)
Output block name
Output data block name. For a null string, it will use the same block name as the first input file, which is usually what you want.
The command
dmmerge "evt.fits[events][cols time,pha][merged_events]" columnList="" outBlock=""
is equivalent to
dmmerge "evt.fits[events]" columnList="time,pha" outBlock=merged_events
Parameter=lookupTab (file not required)
lookup table containing merging rules for header keys. The default value is a redirect to the system version, ${ASCDS_CALIB}/dmmerge_header_lookup.txt, which enforces CXC event pipeline header merging rules; see "plist dmmerge" for the full path of your system default.
Each line of the file contains a keyword and a merging rule. The rules include FORCE (always replace key), SKIP (always omit key), WarnOmit-tol (print a warning and omit if key valuess differ by more than tol), WarnFirst (print warning and use first value if keys are different), among others.
Parameter=columnList (string not required)
Column list
Apply a column filter to the input stack. "columnList=time,pha" is equivalent to appending "[cols time,pha]" to each entry in the input stack.
This parameter may also be used to omit certain columns. "columnList=-phas" is equivalent to including a "[cols -phas]" filter on the input files.
Parameter=clobber (boolean not required default=no)
Clobber output if it exists? [y/n]
Parameter=verbose (integer not required default=0 min=0 max=5)
Display informational messages
Column metadata
When combining SKY or POS vector columns - e.g. X, Y - then the range of the output column will be the union of the input values. This range is stored in the TLMIN/MAX keys of the files, and can be seen with the 'cols' or 'subspace' options of dmlist. This is to support merging event files that have been reprojected and so have X,Y ranges that lie outside the default values (eg for ACIS X and Y go from 0.5 to 8192.5).
Bugs
Caveats
- Extra GTI blocks in merged event file (16 Apr 2007)
-
When merging ACIS event lists with dmmerge, the "data subspace" of the two event lists is combined. Due to a technical subtlety in the way that the EXPNO (ACIS frame exposure number) column is defined, this generates unwanted extra GTI extensions in the output file if the exposure numbers are not equal.
unix% dmmerge "acisf01587N002_evt2.fits,acisf07073N001_evt2.fits" NGC3379_merge.fits unix% dmlist NGC3379_merge.fits blocks -------------------------------------------------------------------------------- Dataset: NGC3379_merge.fits -------------------------------------------------------------------------------- Block Name Type Dimensions -------------------------------------------------------------------------------- Block 1: PRIMARY Null Block 2: EVENTS Table 15 cols x 1392531 rows Block 3: GTI7 Table 2 cols x 1 rows Block 4: GTI2 Table 2 cols x 3 rows Block 5: GTI5 Table 2 cols x 1 rows Block 6: GTI6 Table 2 cols x 2 rows Block 7: GTI3 Table 2 cols x 2 rows Block 8: GTI8 Table 2 cols x 2 rows Block 9: GTI7_CPT7 Table 2 cols x 1 rows Block 10: GTI2_CPT8 Table 2 cols x 2 rows Block 11: GTI5_CPT9 Table 2 cols x 1 rows Block 12: GTI6_CPT10 Table 2 cols x 1 rows Block 13: GTI3_CPT11 Table 2 cols x 2 rows Block 14: GTI8_CPT12 Table 2 cols x 1 rowsThis occurs because the filters could not be combined into a single, valid range for the chip. It is undesireable for further analysis, as the tools cannot use multiple GTI blocks for a single chip, resulting in incorrect exposure values.
-
Workaround:
The subspace-editing capabilities in CIAO 3.4 simplify the workaround for this issue. It is now possible to delete the EXPNO subspace before merging the files:
unix% dmmerge \ "acisf01587N002_evt2.fits[subspace -expno],acisf07073N001_evt2.fits[subspace -expno]" \ NGC3379_merge_new.fits unix% dmlist NGC3379_merge_new.fits blocks -------------------------------------------------------------------------------- Dataset: NGC3379_merge_new.fits -------------------------------------------------------------------------------- Block Name Type Dimensions -------------------------------------------------------------------------------- Block 1: PRIMARY Null Block 2: EVENTS Table 15 cols x 1392531 rows Block 3: GTI7 Table 2 cols x 2 rows Block 4: GTI2 Table 2 cols x 5 rows Block 5: GTI5 Table 2 cols x 2 rows Block 6: GTI6 Table 2 cols x 3 rows Block 7: GTI3 Table 2 cols x 4 rows Block 8: GTI8 Table 2 cols x 3 rowsThere are now just six GTI blocks in the output file.
Note that the addition of the subspace filter means that any user who intends to create lightcurves binned on exposure number from the merged output cannot use merge_all, since that information is eliminated from the subspace. (In general, lightcurves are binned on time.)
See Also
- concept
- merging_rules
- dm
- dmfiltering, dmopt
- tools
- addresp, dmappend, dmgroup, dmgti, dmimgcalc, dmimgfilt, dmjoin, dmmerge, dmpaste, dmregrid2, dmsort, dmtabfilt, dmtcalc, dmtype2split, hrc_dtfstats, reproject_image, reproject_image_grid