merge_all

Synopsis

Combine any number of observations. If desired, create corresponding exposure maps and exposure-corrected images. *DEPRECATED*

merge_all  evtfile [asol] [dtffile] chip [refcoord] [xygrid] [energy]
[merged] [expmap] [expcorr] [intdir] [clobber]

Description

Deprecated

The merge_all script is deprecated: please use merge_obs, reproject_obs, and flux_obs (new in the October 2012 release) or fluximage instead.

In the following examples, evt2.lis is a stack file (see ahelp stack for more information) containing the names of the event files to combine. It is assumed that the observations have been reprocessed by chandra_repro and the data files remain in the repro/ directory: e.g.

unix% ls -1 */repro/*evt2.fits > evt.lis

Note that these files should not include an energy filter on them, since the energy filtering used to create the exposure-corrected images is now done by merge_obs, flux_obs, or fluximage.

Converting from merge_all to reproject_obs

The reproject_obs script is used when all you require are the reprojected event files and the merged event file. In this case all that is needed is

unix% reproject_obs @evt.lis out/

which will create a merged event file called out/merged_evt.fits, and reprojected event files out/<obsid>_reproj_evt.fits.

Note that the aspect solutions do not need to be given to reproject_obs since it can find them itself (assuming you have not moved or renamed them) using the headers of the event files.

See ahelp reproject_obs for more information.

Converting from merge_all to merge_obs

The merge_obs script is used when you want the exposure-corrected images as well as the merged event file.

unix% merge_obs @evt.lis out/

will, for ACIS data, create a broad-band (0.5 to 7.0 keV) exposure-corrected image (using a monochromatic energy of 2.3 keV for the exposure map) binned at a scale of 8 pixels, covering all the data. For HRC data the binning is by 32 pixels, there is no energy filter, and the exposure map is evaluated at 1.5 keV. For HRC-I data, an estimate of the particle background will be removed from each observation if the HRC background CALDB package is installed.

The ancillary data products - e.g. aspect solution, bad-pixel file, mask file, pbk file, DTF file - are found using the headers of the event files (assuming these files have not been renamed or moved).

You can use the bands parameter to change what energy band - or bands - are used to create the images, the binsize parameter to change the binning, or a number of other parameters to control the behavior. See ahelp merge_obs for more information.

The merge_all script

The merge_all script takes a list of level=2 event files, reprojects them to a given RA and Dec, and combines them. It can then make a combined exposure map and an exposure-corrected image. If only one event file is input, it is treated as a merged event file. One may first use the script to create a merged file and then run it a second time using this merged file as input, omitting the parameters "refcoord" and "merged".

Note that the script may be used to simply combine a set of two or more level=2 event files. In this case, the only parameters which are needed are "evtfile", "chip", "refcoord", and "merged".

If a combined aspect histogram exists from a previous run, and it is in the "intdir" and retains its original name, the script will use this instead of creating a new one.

Using Data Model filters:

It is possible to provide an energy filter. However, region filters are not supported: this is because the "xygrid" is the region filter for the exposure map, and the image size and shape must match that of the exposure map. Filters (to remove sources, etc.) can be applied to the event file prior to running the script, if desired.

Using HRC data:

The script may be used to combine HRC observations (S- or I-array), but it can only make exposure maps and exposure-corrected images for the HRC-I configuration.

Caveat: using merge file for spectral analysis

Merged files are, in general, not suitable for spectral analysis. The exposure map and exposure corrected image do not contain statistical errors, and therefore cannot be used to obtain meaningful values for source fluxes.

Caveat: lightcurves binned on exposure number

A "[subspace -expno]" filter is used in the dmmerge command as a workaround for a problem merging data with different EXPNO ranges. This means that any user who intends to create lightcurves binned on exposure number from the merged output cannot use merge_all, since that information is eliminated from the subspace. (In general, lightcurves are binned on time.)

Example 1

reproject_obs @evt.lis out/
merge_obs @evt.lis out/

Use merge_obs or reproject_obs instead

The merge_all script is deprecated and users should use the merge_obs or reproject_obs scripts instead: reproject_obs if all that is wanted is the reprojected event files (including a merged event file); merge_obs if the exposure-corrected image is also required. Please see ahelp merge_obs and ahelp reproject_obs for more information.

Example 2

merge_all evtfile="data1.fits,data2.fits,data3.fits"
asol="pcadf063874624N002_asol1.fits,pcadf063875522N002_asol1.fits,pcadf0
63902942N002_asol1.fits" chip="0,1,2,3,6,7" refcoord=data3.fits
xygrid="3056.5:4056.5:#1000,3024.5:4044.5:#1020" energy=spectrum.dat
merged=data123.fits expmap=expmap123.fits expcorr=expcorr123.fits

Combine three level=2 event files using a time-sorted list of asol files. Reproject the data to match the tangent point in data3.fits. Create a full-resolution exposure map as well as an exposure-corrected image of a portion of the data using a spectral weights file.

Example 3

merge_all evtfile=hrcf00461N005_evt2.fits
asol=pcadf064938693N003_asol1.fits dtffile=hrcf00461_001N004_dtf1.fits
chip=hrc-i xygrid="1:16384:#128,1:16384:#128" energy=1.5
expmap=hrc_expmp.fits expcorr=hrc_expcorr.fits intdir=.

Make an exposure map and exposure-corrected image for a single HRC-I observation. Write temporary files to the current directory.

Parameters

Detailed Parameter Descriptions

Parameter=evtfile (file required filetype=input stacks=yes)

Input level 2 file(s)

Comma-separated list of input level=2 event files. Region filters should not be used. Can use a merged file made from a previous run of the script.

Parameter=asol (file filetype=input stacks=yes)

Input aspect solution file(s)

Comma-separated list of aspect solution (asol) files. These are only required if you are creating an exposure map.

The script assumes that the aspect files are arranged in chronological order. If you have not altered the original filenames, a simple "ls" will put them in order, as the time is listed in the filename.

Parameter=dtffile (file filetype=input stacks=yes)

Input DTF files

Comma-separated list of DTF file name(s). Only necessary when creating an exposure map for HRC-I observations.

Parameter=chip (string required)

ACIS CCD numbers or "HRC-I" or "HRC-S"

Either ccd_id(s) for the ACIS CCDs you want or "HRC-I" or "HRC-S". If entering ccd_ids, can use either a range (seperated by ":" or "-") or a comma-separated list.

Parameter=refcoord (string filetype=input)

Tangent point to which data is reprojected

Either "RA, Dec" (comma-separated) or a level=2 event file to which the tangent point of observations will be reprojected. This parameter is not used if the "evtfile" file is a single observation or a previously-merged file.

Parameter=xygrid (string)

Input grid specification

Input grid for creating the exposure map and exposure-corrected image; should be of the same form expected by the tool mkexpmap: "xmin:xmax:#(no. of x bins),ymin:ymax:#(no. of y bins)"; for example, "0.5:8192.5:#1024,0.5:8192.5:#1024". Do not need if only using the script to merge data.

Parameter=energy (string filetype=input units=keV)

Energy value or spectral weights file for mkinstmap

Either an energy value (in keV) or a normalized spectral weights file. This will be used to calculate the instrument map, see "ahelp mkinstmap" for details on the format of the spectral weights file.

The script uses a decimal test to see if the value is a number or a file, so spectral weight filenames cannot consist entirely of numbers; "2" is not a valid filename, but "2.wgt" is.

Parameter=merged (file filetype=output)

Output merged event file name

Not required if inputting an already-merged file or single level=2 event file.

Parameter=expmap (file filetype=output)

Output exposure map

Not required if only merging data.

Parameter=expcorr (file filetype=output)

Output exposure corrected image.

Leave blank if output is not desired.

Parameter=intdir (string default=/tmp)

Directory for intermediate products

Directory where intermediate products are placed. The temporary files have a timestamp value as the root. Intermediate products are deleted from /tmp when the script finishes. If the tmpdir is set to anything other than /tmp, the intermediate data products are not deleted.

Parameter=clobber (boolean default=no)

Overwrite output files if they already exist?

Changes in the October 2012 release

The merge_all script is deprecated. The merge_obs, reproject_obs, flux_obs, and fluximage scripts should be used instead.

About Contributed Software

This script is not an official part of the CIAO release but is made available as "contributed" software via the CIAO scripts page. Please see the installation instructions page for help on installing the package.

Bugs

Incorrect results when combining observations done with an offset  (25 Jun 2009)

This bug affects a small percentage of observations. For instance, roughly 5% of observations done with Chandra in 2008 used a pointing offset.

When merge_all runs dmmerge to combine the event files, it creates a custom lookup table containing merging rules for header keys. Instead of omitting the DETNAM, ROLL_NOM, and SIM_Z header values that differ by more than the tolerance in dmmerge, it always uses the value from the first file.

The dz values in the aspect solution are relative to the SIM_Z in the event header. If the observation was done with an offset, the aspect solution files for the second observation is not corrected for the difference in SIM_Z between the files. The result is that the combined exposure map and fluxed image will be wrong, as seen in this image:

The white box has the same WCS coordinates in both frames, illustrating how the exposure map is offset from the event data.

Check if the observation was done with an offset: search for the ObsId in WebChaser, choose the "View Observation Information" link on the results page, and read the "Details" section to see if an offset was used. For instance, ObsId 5826 has a Y offset of -0.5 arcmin.

Workaround:

The merged data created by the workaround below is appropriate for getting an approximation of the flux. It should not be used for source detection or other science analyses. The output from reproject_image is not an integer image and therefore breaks the Poisson assumptions in wavdetect, csmooth, and other CIAO tools.

To create a merged, fluxed image:

1. Create a full-resolution image and matching exposure map for each observation by following the appropriate CIAO Imaging thread.

   Let's call these files image1, image2, expmap1, and expmap2.

2. Reproject image2 to match image1. Reproject expmap2 to match expmap1:

   reproject_image image2 image1 image2_proj method=sum
   reproject_image expmap2 expmap1 expmap2_proj method=average
   

3. Sum the two event images together. Sum the two exposure map images together.

   dmimgcalc image1 image2_proj image add
   dmimgcalc expmap1 expmap2_proj expmap add
   

4. Divide the merged event image by the merged exposure map to create a fluxed image:

   dmimgcalc image expmap flux div
   

If you have any questions or problems with the workaround, please contact the Helpdesk for assistance.

Caveats

dmmerge issue: extra GTI blocks in merged event file  (16 Apr 2007)

The dmmerge tool is called by the merge_all script. Details on the problem are available on the dmmerge bug page.

Workaround:

A "[subspace -expno]" filter was added to the dmmerge command in merge_all v3.6 as a workaround for this problem. Users no longer need to filter the data before using it as input to merge_all.

Note that the addition of the subspace filter means that any user who intends to create lightcurves binned on exposure number from the merged output cannot use merge_all, since that information is eliminated from the subspace. (In general, lightcurves are binned on time.)