Last modified: December 2021

URL: https://cxc.cfa.harvard.edu/ciao/ahelp/merging_rules.html
AHELP for CIAO 4.16

merging_rules

Context: concept

Synopsis

A description of the merging rules used when combining header information.

Description

CIAO tools, such as dmmerge, have a number of rules for how the keywords from the header files should be combined. Some of these tools allow the rules to be configured; an example being the lookupTab parameter of dmmerge.

Default behavior

The name of the keyword is used to determine what rule to follow (see the "About the lookup table" section below for examples). When no rule exists the values is taken from the first file that is being combined. This means that

unix% dmmerge a.fits,b.fits combined-ab.fits mode=h
unix% dmmerge b.fits,a.fits combined-ba.fits mode=h
unix% dmlist combined-ab.fits,combined-ba.fits keys

can have different keyword values.

MAX

Take the maximal value among the keys that are present in the input files. If the key is not found in any of the input files, then no output and warning for the key.

Example:

File Keyword
infile1 4.
infile2 2.5
infile3 3.
outfile 4.

In this case, the output is the maximal value '4'.

MIN

Take the minimal value among the keys that are present in the input files. If the key is not found in any of the input files, then no output and warning for the key.

Example:

File Keyword
infile1 4.
infile2 2.5
infile3 3.
outfile 2.5

In this case, the output is the minimal value '2.5'.

FAIL

If the key is not present in the first file, or it is present in the first file but the value is different from others, then display a warning and don't write the key to the output. If the key is present in the first file and the value is same as others, then write the value to the output file without a warning.

Example:

File Keyword
infile1 4.
infile2 2.5
infile3 3.
outfile ( KEY not present )

In this case, KEY is present in the first file, but the value is different from others. Therefore display a warning: 'KEY values are different...FAIL...', but no output for KEY.

FAIL;Default-[value]

First, go through each input file. If the key is not present, set the key value to the default (i.e. [value]). Once each file is checked, compare the values with one another. If they are all same, write it to the output; otherwise, display a warning but no output for the key.

Example using the rule 'FAIL;Default-3':

File Keyword
infile1 ( KEY not present )
infile2 ( KEY not present )
infile3 3
outfile 3

In this case, KEY is not present in the infile1 and infile2, so the rule sets both of them to be the default value '3', which happens to be same as the one in infile3. Therefore, it writes '3' to the output without a warning.

calcForce

Calculate the value with a special hard-coded rule and write it to the output file. Time keywords with a calcFORCE rule are generally set to "sum the values from the input files". For other keys, this rule will simply copy the first value to the output file.

calcGTI

Calculate the value in the output file from the time subspace (e.g. the good time intervals [GTIs]). This rule is currently applied to ONTIME, LIVETIME, LIVTIMEn, and EXPOSURn keywords.

Note: DATE-OBS DATE-END are special keys whose merging rules are always treated like 'calcForce' in the dmmerge tool. In other words, they are always computed from the special hard-coded rules.

FORCE

First, go through each input file. If the key is not present, set the key value to the default (null, 0, blank). Once each file is checked, compare the values with one another. If they are different, write the first one to the output with a warning message; otherwise, output the value without warning.

Example:

File Keyword
infile1 3
infile2 ( KEY not present )
infile3 5
outfile 3

In this case, the KEY of infile2 is not present, therefore set it to the default (i.e. '0') first. Since KEY values are different, write the first one (i.e. '3') to the output with a warning: 'KEY has 2 different values.'

FORCE-[value]

First, go through each input file. If the key is not present, set the value to the default (i.e. [value]). Once each file is checked, compare the values with one another. If they are different, write the first one to the output with a warning message; otherwise, output the value without warning.

Example using the rule 'Force-3':

File Keyword
infile1 3
infile2 ( KEY not present )
infile3 3
outfile 3

In this case, the KEY of infile2 is not present, therefore set it to [value] (i.e. '3') first. Since KEY values are all same, write it to the output without warning.

WarnFirst

If the key is not present in the first file, give a warning and don't write it to the output. Otherwise, copy the first value to the output and display a warning message only if the first value is different from others.

Example:

File Keyword
infile1 ( KEY not present )
infile2 5
infile3 5
outfile ( KEY not present )

In this case, the KEY value of infile1 is not present, therefore no output for KEY except a warning message: "omit - keyword KEY not present in 1st file".

WarnFirst;Force-[value]

First, go through each input file. If the key is not present, set the value to the default (i.e. [value]). After each file is checked, copy the first value to the output; if it is different from others, display a warning message.

Example using the rule 'WarnFirst;Force-5':

File Keyword
infile1 ( KEY not present )
infile2 5
infile3 5
outfile 5

In this case, the KEY value of infile1 is not present, therefore force it to [value] (i.e. '5') first. Since all KEYs have same value, write it to the output without a warning.

WarnOmit-[tol]

If the key is not present in the first header, or if it is present in the first header but different from others more than [tol], then don't write the key and give a warning. Otherwise, take the first value.

Example using the rule 'WarnOmit-0.3':

File Keyword
infile1 5.5
infile2 5
infile3 5
outfile ( KEY not present )

In this case, the KEY value of infile1 is present but is different from others more than the given tolerance (i.e. 0.3), therefore no output for KEY except a warning: 'omit - KEY values different more than 0.3'.

WarnPrefer-[value]

If the key is not present in the first file, or if it is present in the first file but different from others, then write the [value] to the output and display a warning. If all the key values are same, then write it to the output with no warning.

Example using the rule 'WarnPrefer-9.0':

File Keyword
infile1 ( KEY not present )
infile2 5
infile3 5
outfile 9.0

In this case, KEY is not present in infile1, therefore set it to [value] (i.e. '9.0'). Since the KEY values are different, write the first one to the output with a warning: 'warning: KEY has 2 different values'.

Merge-[value1];Force-[value2]

First, go through each input file. If the key is not present, force it to [value2]. Once each file is checked, compare the values with one another. If different, then write [value1] to the output with a warning. If same, write the first value to the output.

Example using the rule 'MERGE-99;FORCE-10':

File Keyword
infile1 ( KEY not present )
infile2 5
infile3 5
outfile 99

In this case, the KEY is not present in infile1, therefore set it to [value2] (i.e. '10') first. Since the KEY values are different, write [value1] (i.e. '99') to the output and display a warning: 'KEY has different value..merged..'.

SKIP

Don't write the key to the output file.

Example:

File Keyword
infile1 5
infile2 5
infile3 5
outfile ( KEY not present. )

In this case, always omit KEY from the output file.

PUT_STRING-[value]

Force the output value to be [value]. It applies for the key that does not belong to the event header and has the data type of 'char'.

Example using the rule 'PUT_STRING-TEST':

File Keyword
infile1 'test1'
infile2 'test2'
infile3 'test3'
outfile 'TEST'

In this case, force the KEY value to be 'TEST'.

PUT_DOUBLE-[value]

Force the output value to be [value]. It applies for the key that does not belong to the event header and has the data type of 'DOUBLE'.

Example using the rule 'PUT_DOUBLE-99.9':

File Keyword
infile1 100.0
infile2 200.0
infile3 ( KEY not present )
outfile 99.9

In this case, force the KEY value to be 99.9.

PUT_LONG-[value]

Force the output value to be [value]. It applies for the key that does not belong to the event header and has the data type of 'LONG'.

Example using the rule 'PUT_LONG-100':

File Keyword
infile1 99
infile2 99
infile3 99
outfile 100

In this case, force the KEY value to be 100.


Special merging rules and keywords

All merging rules described in the above section can only be applied for the keys that belong to the event header except the following rules of 1.a, 1.b, and 1.c.

These three rules should be used by the keys that do not belong to the event header:

This rule can be used by any key, regardless whether or not it belongs to the event header:

The above description for this rule is only applied to an event key. For a non-event key, it will be same as the rule 'PUT_STRING-[value]':

The merging rules of the following keys are always treated like 'calcForce' in the dmmerge tool. In other words, they are always computed from the special hard-coded rules:

About the lookup table

A lookup table is required by the dmmerge tool. It assigns a merging rule to each keyword. The current table used by dmmerge can be checked by calling:

unix% punlearn dmmerge
unix% cat `pget dmmerge lookupTab`

Example of lookup table

Keyword Rule
MISSION Merge-Merged;Force-AXAF
TELESCOP Merge-Merged;Force-Unknown
OBJECT Merge-Merged;Force-Unknown
OBSERVER Merge-Merged;Force-Unknown
TITLE Merge-Merged;Force-Unknown
OBS_ID Merge-Merged;Force-Unknown
INSTRUME Merge-Merged;Force-Unknown
DETNAM Merge-Merged;Force-Unknown
GRATING Merge-Merged;Force-Unknown
OBS_MODE Merge-Merged;Force-Unknown
DATAMODE Merge-Merged;Force-Unknown
ORIGIN WarnFirst;Force-ASC
CONTENT Force
HDUNAME Force
RA_NOM WarnOmit-0.0003
DEC_NOM WarnOmit-0.0003
ROLL_NOM WarnOmit-1.0
SIM_X WarnOmit-0.001
SIM_Y WarnOmit-0.001
SIM_Z WarnOmit-0.001
FOC_LEN WarnOmit-1.0
EQUINOX WarnPrefer-2000.0
RADESYS WarnPrefer-ICRS
DATACLAS WarnPrefer-Observed
TIMVERSN WarnFirst
CLOCKAPP WarnFirst
MJDREF fail
TIERRELA Max
TIERABSO max
TIMEDEL maX
DATE calcForce
DATE-OBS calcForce
DATE-END calcForce
TSTART calcForce
TSTOP calcForce
ONTIME calcGTI
LIVETIME calcGTI
DTCOR calcForce
TIMEUNIT Fail;Default-s
TIMESYS Fail;Default-UTC
TIMEPIXR Fail;Default-0.5

Lookup Table Format

Each row consists of two columns: the keyword and the associated merging rule.

The two columns are separated by blanks or tabs.

If a default value is assigned to a merging rule, use "-" to separate them, e.g.

mergeRule-defValue

where "defValue" can be a number or a character string

If a keyword has two merging rules together, use ";" as a delimiter between the merging rules. For example:

mergeRuleOne;mergingRuleTwo
mergeRuleOne-defValueOne;mergeRuleTwo-defValueTwo

Lookup Table Restriction

For the first column:

For the second column:

See Also

tools::coordinates
reproject_image, reproject_image_grid
tools::hrc
hrc_dtfstats
tools::image
dmimgcalc, dmimgfilt, dmregrid2
tools::response
addresp
tools::table
dmmerge