merge_obs

Synopsis

Reproject and combine multiple observations to create a merged event file and exposure-corrected images.

merge_obs  infiles outroot [bands] [xygrid] [maxsize] [binsize]
[asolfiles] [badpixfiles] [maskfiles] [pbkfiles] [dtffiles] [refcoord]
[units] [expmapthresh] [background] [parallel] [nproc] [tmpdir]
[cleanup] [clobber] [verbose]

Description

The merge_obs script takes a stack of event files and reprojects them to the same tangent point, merges them, creates exposure maps for each observation, and divides the resulting images to produce a coadded, exposure-corrected image. This process can be done for one or more energy bands.

For recently-processed data the script can be run with only two parameters: for example

unix% merge_obs "*/repro/" out/

will store the merged data products in the directory out/, creating this directory if necessary. The output files include:

• out/merged_evt.fits, the merged events file, and
• out/broad_flux.img, the exposure-corrected broad-band image.

We took advantage of the stack support to process all the directories that match "*/repro/" - the quotes are needed to ensure that the shell does not do the matching, which would result in a parameter file error - and then looking for event files in these directories (see the description of the infile parameter for a full discussion of what file names are searched for). The examples below show other ways to use the stack support for specifying files.

How is merge_obs better than merge_all?

The merge_all script combines the reprojected event files, then creates a single multi-chip exposure map for this file. The merge_obs script creates exposure maps for each of the individual (reprojected) observations and then sums the resulting counts and exposure map images. The method merge_all uses does not correctly handle the case when observations were not taken with the same settings, such as differences in the SIM_X and SIM_Z values. This can lead to discrepencies between where the software calculates that chip edges and bad columns should be (when creating response functions like the exposure map or ARF) and where they actually are. The merge_all script does not use the observation-specific bad-pixel, mask, and PBK or DTF files, which means that certain instrumental effects (e.g. bad pixels, ACIS subarrays, cosmic-ray dead-time correction) are not included in the exposure map it creates. It also can not include variations in the response over time - e.g. when combining observations separated by multiple years, time dependent quantities, such as the amount of contamination on the ACIS optical-blocking filters, are not handled correctly.

The extra steps taken by merge_obs means it can take longer to run, but the script can do many of these in parallel if your machine has multiple processors, which can reduce this overhead.

If you only wish to reproject the event files and merge them - i.e. you do not want the images or exposure maps - then you can use the reproject_obs script.

The merge_obs, reproject_obs and flux_obs scripts

The merge_obs script combines the reproject_obs and flux_obs script, so if you need more control, or wish to create exposure-corrected images with multiple bin sizes or grids, then it is suggested that you use reproject_obs and flux_obs instead.

WCS alignment

The individual event files are reprojected to the same tangent point, but no attempt is made to align the observations. If you find that the WCS of the individual observations are not aligned - e.g. you can see "double" sources in the reprojected, merged data - then you need to update the WCS before running merge_obs. An example is the Correcting Absolute Astrometry thread.

Example 1

unix% download_chandra_obsid 5826,5827
unix% chandra_repro 5826,5827 outdir=""
unix% punlearn merge_obs
unix% merge_obs 5826/,5827/ m87

The two ACIS observations in the directories 5826/ and 5827/ are reprojected and combined to create m87_broad_flux.img, which is the broad-band (0.5 to 7 keV) image divided by an exposure map created at 2.3 keV. As these are ACIS observations the pixel size is set to 8 (i.e. 3.936 arc seconds).

The location of the necessary support files - e.g. aspect solution, bad-pixel file, mask file, and PBK file - are taken from the headers of the event files.

The processing steps are run in parallel when possible to reduce the run time of the script; this can be turned off by setting the parallel flag to "no", or the number of processors used changed with the nproc parameter.

What event files are used?

When given a directory name, the script looks for files that match one of the following (the first pattern to match is used):

• dirname/repro/*evt*
• dirname/primary/*evt*
• dirname/*evt*

Example 2

unix% punlearn merge_obs
unix% merge_obs
5826/repro/acisf05826_repro_evt2.fits,5827/repro/acisf05827_repro_evt2.f
its m87

Here we explicitly list the event files to use as a comma-separated list of names.

Example 3

unix% merge_obs "*/repro/*evt2.fits" merged/

Here we use the stack support which lets CIAO tools expand out patterns to list the input event files; in this case all files that end in evt2.fits and are in the repro/ sub-directory are used.

Since the outroot is set to a directory, the exposure-corrected image will be called "merged/broad_flux.img". The directory will be created if it does not already exist.

If the quotes are not used around the pattern then the shell will find all the matching files and send them as a space-separated list to the script, which will lead to an error from the CIAO parameter library.

Example 4

unix% find . -name \*repro_evt2.fits > evt2.lis
unix% punlearn merge_obs
unix% merge_obs @evt2.lis merged/ clobber+

Here we take advantage of the stack support (see ahelp stack) to combine all the files ending in "repro_evt2.fits" that are in the current directory or any of its children (this is the pattern used by chandra_repro to name its output).

Example 5

unix% merge_obs "@evt2.lis[ccd_id=0:3]" merged/acisi

A DataModel filter is used to restrict the combination to only those chips with a ccd_id of 0 to 3 (i.e. the ACIS-I array). This filter is applied to each line of the stack file (evt2.lis). Those event file which contain no data after the filter will be ignored by the script.

The outroot has been changed so that the files do not clash with those from the previous run. If you wish to overwrite them then you would set outroot to merged/ and add clobber=yes to the command line call.

Example 6

unix% merge_obs @evt2.lis merged/ bands=csc

The csc band is a short form for the soft, medium, and hard bands defined by the Chandra Source Catalog. This means that three exposure-corrected images will be created: merged/soft_flux.img, merged/medium_flux.img, and merged/hard_flux.img.

Example 7

unix% merge_obs @evt2.lis hires bin=2

The binsize parameter has been changed from its default value of 8 to 2. This may lead to large images if there is not significant overlap of all the observations.

Example 8

unix% merge_obs @evt2.lis hires maxsize=2048

In this example we force the longest dimension of the output images to have 2048 pixels.

Example 9

unix% merge_obs @hrc.lis merged/

In this example a set of HRC-I observations are being merged, which means that the wide band is used by default. The exposure-corrected image will be called merged/wide_flux.img.

The default setting for the background parameter is "default", which means that an estimate of the HRC-I particle background will be subtracted from each observation if the HRC CALDB package is installed and a match is found (as determined by the hrc_bkgrnd_lookup script). Observations for which no matching background file can be found will be skipped; to include these observations set background=none.

Example 10

unix% merge_obs @hrc.lis merged/ background=none

In this case the HRC-I data is processed without any background subtraction (which will be significantly quicker). Please see the caveat about "rings" seen in exposure-corrected HRC-I data for more information on why the particle background is removed by default.

Example 11

unix% punlearn merge_obs
unix% pset merge_obs xygrid=2000.5:6000.5:8,2048.5:6128.5:8
unix% merge_obs @evt2.lis hires

Here we give the explicit grid for the output image (a 500 by 510 pixel image with a bin size of 8). All observations are reprojected, but only those that cover the grid will be used to create the combined image.

Example 12

unix% merge_obs @evt2.lis hires xygrid=merged.img

The grid for the images is taken to match that used to create the image merged.img (i.e. it is the output of get_sky_limits when run on merged.img).

Example 13

unix% merge_obs @evt2.lis combined/ bands=broad,wgt.dat
refcoord="123.45 -12.47"

The reprojection is done so that the center - i.e. (4096.5,4096.5) for ACIS data - is positioned at a Right Ascension of 123.45 degrees and Declination of -12.23 degrees. Note that there is no check that this location is close to any of the observations.

The position can also be given in sexagesimal notation using colon-separated values - e.g.

refcoord="8:13:48 -12:28:12"

Two energy ranges are used: the broad band and the range defined in the spectral weights file "wgt.dat" (the output for this is labelled band1). The weights file.dat can be created manually or with the make_instmap_weights script.

Example 14

unix% merge_obs @evt2.lis combined2/ refcoord=combined/merged_evt.fits

Here we reproject the data using the same reference location as the file combined/merged_evt.files (e.g. created by a previous run of merge_obs).

Example 15

unix% merge_obs @evt2.lis combined/ bands=plwgt.dat,bbwgt.dat nproc=-1

There will be two exposure-corrected images created in combined/, both using spectral weights files: band1_flux.img for plwgt.dat and band2_flux.img for bbwgt.dat.

The default parameter settings for parallel (yes) and nproc (INDEF) mean that all processors on your machine are used. In this example we change nproc to -1, which means that it uses all-but-one of your processors; so on a four-core machine the script will use three cores.

Example 16

unix% punlearn merge_obs
unix% pset merge_obs asolfiles=obs1.asol,obs2a.asol,obs2b.asol
unix% pset merge_obs badpixfiles=obs1.bpix,obs2.bpix
unix% pset merge_obs maskfiles=NONE
unix% pset merge_obs pbkfile=obs1.pbk,pbs2.pbk
unix% merge_obs obs1.evt2,obs2.evt2 out/ bands=0.5:4:2

Here the two event files to be combined are called obs1.evt2 and obs2.evt2 and the ancillary files - aspect solution, badpixel, and PBK files - are explicitly given. No mask files will be used since maskfiles is set to NONE. In this example the second observation has two asol files - here called obs2a.asol and obs2b.asol.

Rather than use one of the pre-canned energy bands we use the 0.5 to 4 keV band with the exposure map evaluated as 2 keV.

Stack files

We could have also used stack files - e.g.

unix% ls -1 *asol > asol.lis
unix% ls -1 *bpix > bpix.lis
unix% ls -1 *pbk > pbk.lis
unix% ls -1 *evt2 > evt2.lis
unix% merge_obs @evt2.lis out/ bands=0.5:4:2 asol=@asol.lis
badpix=@bpix.lis pbk=@pbk.lis mask=NONE

Example 17

unix% merge_obs @evt2.lis merged/ units=time

Here the exposure maps have units of seconds, rather than cm^2 s count / photon, and so the exposure-corrected images have units of count / s. The energy value of the band parameter - in this case 2.3 keV - is ignored and only the low and high energy limits are used.

Parameters

Detailed Parameter Descriptions

Parameter=infiles (file required stacks=yes)

Input event files

Multiple event files are provided as a comma-separated list or as a stack; see "ahelp stack" for more information. The observations do not need to be given in time order.

Listing just a directory name

Rather than list the full path to an event file, you can just give a directory name; in such a case the script searches for any files that match the pattern

• dirname/repro/*evt*
• dirname/primary/*evt*
• dirname/*evt*

where the search is done in the order given above, and stops as soon as a match (or matches) is found.

Skipping files

Any observations that do not match the instrument of the first observation are skipped, as are CC mode observations or those with no data (e.g. because of a ccd_id filter). For HRC data, the first observation determines whether HRC-I or HRC-S are being combined, with observations from the other instrument being skipped. Any file with an OBS_ID keyword of "Merged" is skipped.

If multiple files with the same OBS_ID keyword are found, then all but the first are skipped, unless they are part of an ACIS interleaved-mode observation (i.e. they have CYCLE=P or CYCLE=S).

The input files must contain at least the following columns, otherwise they are skipped:

File headers

The header of the event file(s) is used to determine the names of the supporting data files - aspect solution, bad pixel, parameter block, mask, and dead-time factor files - if they are not explicitly set in the parameter file.

Parameter=outroot (string required)

Root of output files

If the parameter ends in "/" then the output files are placed in this directory (which will be created if necessary), otherwise it is prepended to the file names, with a "_" added. So, with bands=broad and outroot=fi/ the exposure-corrected image is called fi/broad_flux.img whereas with outroot=fi the file is fi_broad_flux.img.

An empty outroot value, or one set to ".", will cause the output files to be written to the current working directory with no prefix.

Parameter=bands (string default=default)

What energy band, or bands, should be used?

The script will create an exposure-corrected image for each energy band. A band can be given using a name (which will use the appropriate definitions from the Chandra Source Catalog), by explicit limits, or by giving the name of a file containing the spectral weights (see ahelp make_instmap_weights).

For the first two options, the names for the output files are created using the minimum, maximum and effective-energy values for each band. When using a weights file, the output files are labelled using "band1" to "bandn", where n is the number of weights file given to the parameter.

The default setting

The value of 'default' is converted to 'broad' for ACIS data and 'wide' for HRC data.

Band names

The following names - based on the definitions from the Chandra Source Catalog - can be used; energies are given in keV and the effective energy is the monochromatic energy used to calculate the exposure map for the band:

CSC

The value "csc" can be used as a short form for the combination of "soft,medium,hard".

Explicit ranges

If you want to use different values to those of the CSC then you can use the format lo:hi:eff - where lo, hi and eff give the minimum, maximum and effective energies to use for the band (values are in keV). The low and or high values can be left blank, in which case all data is used, but it is suggested that this is only used for HRC data.

Spectral weights

If a filename is given then it is assumed to be an ASCII file containing spectral weights; two colums giving the energy in keV and the weight value for that energy. This file can be created using the make_instmap_weights script, the save_instmap_weights() routine from the sherpa_contrib.utils module, or manually; see the description of the spectrumfile parameter of mkinstmap for more information on the format of this file.

The energy limits to use for the images are calculated from this file. If the file contains two lines (before the data) with the format

# ENERG_LO = elo
# ENERG_HI = ehi

then the elo and ehi values are taken for the limits (they are assumed to be in keV). This format is created by the make_instmap_weights script and the save_instmap_weights() routine. If these lines do not exist, or the elo/ehi values are not valid numbers, then the limits will be calculated from the first column using the formulae (x1 is the first value, xn is the last, and xm the last-but-one value):

elo = (3 * x1 - x2) / 2

ehi = (3 * xn - xm) / 2

Multiple bands

Multiple bands can be given either by using the name "csc" or by using a comma-separated list. The different schemes can be combined, so the following are all valid (where wgt1.dat and wgt2.dat are weights files):

bands="csc,broad"
bands="0.5:7:2.5,ultrasoft"
bands="0.5:7:2.5,csc,wgt1.dat,wgt2.dat"

HRC data

Since the HRC has no real energy resolution when in imaging mode, the low and high energy values should not be used; that is you should use either the wide band or use a value like "::1.5". Multiple bands are not supported with HRC data.

Parameter=xygrid (string default=)

xygrid for output or filename

When set, this parameter overrides both the maxsize and bin parameters, and defines the SKY coordinate range and binning used to create the output count images, exposure maps, and exposure-corrected images.

xygrid is set

When set, it can either be:

• a filename,
• a grid specification of the form "xlo:xhi:#nx or dx,ylo:yhi:#ny or dy".

If a filename is given then its sky range is used to determine the output; this is useful if you want to create an exposure-corrected image that matches an existing image. Otherwise the grid range is given explicitly; examples include "3824.5:5112.5:#161,4008.5:5296.5:#161" and "3824.5:5112.5:8,4008.5:5296.5:8".

xygrid is not set

When left at the default (empty) value, the script calculates the xygrid for the output image based on the observational data; the skyfov tool is used to determine the maximum area of the sky that is covered by the observations. In this mode the bin size is determined by either the maxsize or binsize parameters.

Parameter=maxsize (real default=INDEF min=1)

Maximum image width or height in pixels

This parameter is only used if the xygrid is empty (set to "").

If set (i.e. changed from INDEF) then the maximum dimension of the output image is set to this value. This can be useful if you quickly want to look at data and want to make sure the output image is not too large. Note that due to the way the ranges are calculated the actual image dimension is only guaranteed to be within a few pixels of the requested value.

The SKY ranges for the images are determined automatically, based on the ranges of the FOV files for each observation (see "ahelp skyfov").

Parameter=binsize (real default=INDEF min=0)

Image binning factor

This parameter is only used if the xygrid is empty (set to "") and maxsize is set to INDEF.

When set to INDEF a default value is used: 8 for ACIS data and 32 for HRC data. The binning factor has units of the SKY pixel (0.492" for ACIS and 0.1318" for HRC), so binsize=4 for an ACIS observation will create images with 1.968 arcsecond pixels. The value need not be an integer, and can be less than 1 if you want to look at small-scale structure near the aim point of the observation; in this case it is strongly suggested that a spatial filter is applied to ensure manageable file sizes.

The SKY ranges for the images are determined automatically, based on the ranges of the FOV files for each observation (see "ahelp skyfov").

Parameter=asolfiles (file stacks=yes)

Input aspect solutions

If the value is empty then the ASOLFILE keyword from the events files will be used to find the files to use. Unlike the other ancillary products, an aspect solution is required for the script to run.

The aspect solution files have names like pcadf*_asol1.fits and are included in the output directory of the chandra_repro script. To explicitly specify the asol files use the stack syntax (e.g. a comma, or space, separated string or an external file as described in "ahelp stack" for more information). So to use asol1.fits, asol2.fits, and asol3.fits you could say

asolfile="asol1.fits,asol2.fits,asol3.fits"

or

asolfile="asol1.fits asol2.fits asol3.fits"

or

asolfile=@asol.lis

where asol.lis contains the names of each file, one per line. The files do not have to be given in time order.

Since there may be multiple asol files for an observation there may be more entries in this parameter than there are in the infiles parameter.

Aspect histograms - the output of the asphist tool - can not be used here.

Parameter=badpixfiles (file stacks=yes)

Input bad pixel files

If the value is empty then the BPIXFILE keyword from the events files will be used to find the files to use.

A value of "CALDB" will use the CalDB bad-pixel file and "NONE" will use no bad-pixel file (the value is case insensitive). The bad-pixel file is not required to run the script, but care should be taken when analyzing the output if it is not used.

The bad-pixel file has a name like acisf*bpix1.fits and is included in the output directory of the chandra_repro script. If you have used or created a custom bad-pixel file for the observations then these should be used.

Use the stack syntax - see 'ahelp stack' - to specify multiple files either as a comma (or space) separated list of file names or read from a file using the '@' syntax.

The order of the bad-pixel files do not need to match that of the infiles parameter.

Parameter=maskfiles (file stacks=yes)

Input mask files

If the parameter is empty then the MASKFILE keyword in the events files will be used. A value of "NONE" will use no mask file in the analysis; in this case care should be taken when analyzing the results since the exposure map will not be correct if the data was taken using a sub array (i.e. not all the chips or ccds were read out).

The mask file has a name like acisf*msk1.fits and is included in the output directory of the chandra_repro script.

Use the stack syntax - see 'ahelp stack' - to specify multiple files either as a comma (or space) separated list of file names or read from a file using the '@' syntax.

The order of the mask files do not need to match that of the infiles parameter.

Parameter=pbkfiles (file stacks=yes)

Input pbk files for ACIS observations

This parameter is only used with ACIS observations.

If the parameter is empty then the PBKFILE keyword in the events files will be used. A value of "NONE" will use no parameter block in the analysis, which means that the cosmic-ray induced dead-area correction will not be included in the exposure map.

The parameter block has a name like acisf*pbk0.fits and is included in the output directory of the chandra_repro script.

Use the stack syntax - see 'ahelp stack' - to specify multiple files either as a comma (or space) separated list of file names or read from a file using the '@' syntax.

Unlike most of the other ancillary files the order of the PBK files must match that of the event files given in the infiles parameter.

Parameter=dtffiles (file stacks=yes)

Input dtf files for HRC observations

This parameter is only used with HRC observations.

If the parameter is empty then the DTFFILE keyword in the events files will be used. A value of "NONE" will mean that only a mean DTF value will be used.

The parameter block has a name like hrcf*dtf1.fits and is included in the output directory of the chandra_repro script.

Use the stack syntax - see 'ahelp stack' - to specify multiple files either as a comma (or space) separated list of file names or read from a file using the '@' syntax.

Unlike most of the other ancillary files the order of the DTF files must match that of the event files given in the infiles parameter.

Parameter=refcoord (string default=)

Reference coordinates or evt2 file

This parameter defines the central coordinate of the reprojected data; that is the Right Ascension and Declination that maps to SKY=(4096.5,4096.5) for ACIS data, (16384.5,16384.5) for HRC-I data, or (32768.5,32768.5) for HRC-S data.

The default setting ("") means that this value is calculated from the tangent points of the observations. When set it can be either:

• the name of a file, so that the tangent position of the file is used;
• or a space-separated value taken to be the RA and Declination to use (ICRS) in decimal degrees or colon-separated sexagesimal formats.

Examples include:

refcoord=fluxed/broad_flux.img
refcoord="123.45 -12.34"
refcoord="8:13:48 -12:20:24"

Parameter=units (string default=default)

Units for the exposure map

This parameter determines the units of the instrument and exposure maps.

This parameter combines the normalize parameter of mkexpmap and the modifiers for the detsubsys and mirror parameters of mkinstmap (see 'ahelp ardlib').

What is units=time equivalent to?

When units=time is used, instrument maps are created with the additional settings:

• ";IDEAL" is appended to the detsubsys parameter to remove any correction for the detector quantum efficiency (QE), quantum efficiency uniformities (QEU), or the ACIS contamination model;
• the mirror parameter is set to "HRMA;area=1" rather than "HRMA" to ignore the effective area of the Chandra mirrors;
• and the dafile parameter is set to "NONE" to remove the ACIS "dead area" correction, which accounts for the (small) decrease in detector efficiency due to the background cosmic ray flux.

The result is that the instrument map pixels have values of 0 or 1.

Parameter=expmapthresh (string default=1.5%)

Remove low-exposure regions?

If set to "" then no clipping of the counts images or exposure maps is performed before creating the exposure-corrected image.

When set, the parameter determines how the per-observation counts image and exposure maps are clipped before being combined together to create the exposure-corrected image. This can be used to exclude those pixels at the edge of a chip which, due to the strong gradient in the exposure map, end up dominating the final image.

For a full list of supported options, see the description of the cut parameter of dmimgthresh. Here we discuss the two most common options:

• expmapthresh="2%", which excludes those pixels where the exposure map falls below 2% of its peak value,
• and expmapthresh="100", which excludes those pixels where the exposure map value is less than 100 (in this case the value of the units parameter is important).

Note that both the counts image and the exposure map for each observation are thresholded before being combined together rather than the thresholding being applied to the final, coadded, image and exposure map.

Parameter=background (string default=default)

Method for background removal (HRC-I)

This parameter is only used for HRC-I data.

If set to "none" then the data is processed with no background subtraction. When set to "default" - and the HRC-I background CALDB package has been installed - then the particle background is subtracted using the HRC-I background event files. The background-subtraction step should help to remove the "rings" seen in exposure-corrected HRC-I data, but does significantly increase the time taken to run the script.

When background=default, observations for which no HRC-I background file can be found are skipped. If no background files can be found - either because they are not installed or there are none for the observations being processed - then the background parameter is set to "none".

Parameter=parallel (boolean default=yes)

Run processes in parallel?

When run on a multi-processor system, many of the tasks can be run in parallel to reduce the execution time of the script. Since the tasks are likely to be memory or I/O-bound, the reduction in run time will be less than the number of cores on a machine. When parallel=yes, the default behaviour is to use all the CPU processors, but this can be changed with the nproc parameter.

This option can be ignored when run on a single-processor system.

Parameter=nproc (integer default=INDEF)

Number of processors to use

This parameter is only used when parallel=yes. It determines the number of processors to use. If maxproc is the actual number of processors on your machine, then a value of INDEF - the default value - means that all maxproc processors will be used. A positive value means to use that number of processors (any value larger than maxproc will be set to maxproc). A negative value is added to maxproc (and any value less than one is set to one).

Parameter=tmpdir (string default=${ASCDS_WORK_PATH})

Directory for temporary files.

Directory for storing temporary files that require further processing before becoming useful. If the directory does not exist then it will be created for use by the script, and then deleted.

Parameter=cleanup (boolean default=yes)

Cleanup intermediary files on exit.

If set to "yes", the intermediate data products are deleted when the script ends, leaving only" the reprojected event files; merged event file; per-observation images, exposure maps, and exposure-corrected images; and the combined images, exposure maps, and exposure-corrected images.

Parameter=clobber (boolean default=no)

Overwrite existing files?

Parameter=verbose (integer default=1 min=0 max=5)

Output verbosity.

The default verbosity value of 1 prints status messages as the script runs. Higher verbosity settings print the commands that are being run. Setting verbose=0 turns off most of the screen output (some output is currently unavoidable).

Energy bands

The energy bands - when given by name - are taken from the Chandra Source Catalog definitions. The default bands setting (i.e. "default") means to use the "broad" band setting for ACIS data and the "wide" band setting for HRC data. Another common choice (for ACIS data) is "csc", which creates the "soft", "medium", and "hard" bands.

The examples and description of the band parameter describe how explicit energy ranges and effective energies can be specified if required. A spectral weights file can also be used if required.

Examples of specifying the bands are:

unix% merge_obs "*evt2.fits" out/
unix% merge_obs "*evt2.fits" out/ bands=csc
unix% merge_obs @evt2.lis out/ bands=0.5:2:1.7
unix% merge_obs @evt2.lis out/ bands=csc,broad
unix% merge_obs @evt2.lis out/ bands=wgt.dat

which use

• the default band (so broad for ACIS, wide for HRC),
• the CSC setting (soft, medium, and hard bands),
• an explicit range of 0.5 to 2 keV with an exposure map evaluated at 1.7 keV,
• four bands (soft, medium, hard, and broad),
• and a spectral-weights file (wgt.dat), created either manually or with the make_instmap_weights script.

Parallel processing

The execution time can be reduced by running some steps in parallel. This is controlled by the parallel flag, which defaults to yes. The nproc parameter controls the number of processors that will be used; the default is to use all of then. If run on a single-processor machine then both parameters are ignored.

Output files

The primary output files are named using the following schemes:

Combined data

Per-observation data

where elabel is either the band name (e.g. soft, broad, ...), "elo-ehi" where elo and ehi are the limits of the energy range used in keV, or band1 to bandn (when using spectral weight files). The above is when outroot does not end in a "/"; if it does then remove "outroot_" and the files will be created in the directory named by outroot. Examples of the exposure-corrected image name for a combination of bands and outroot settings are given in the following table:

Names of the exposure-corrected coadded images

For HRC data the label is either "wide" or - if bands=::eval is used - then the label "all" is used.

Interleaved mode

For interleaved-mode data, "e1" or "e2" is appended to the ObsId value when creating file names.

Output image area and size

The default behavior - when xygrid="" - is to calculate the output image size so that it covers all the data: a FOV file is created for the observation and the limits of this file are used to determine the X and Y range of the image. In this mode the binning is set by either the maxsize or binzise parameters. The maxsize parameter - if set - determines the maximum number of pixels used along the longest edge of the output, otherwise the binsize parameter is used. This parameter is in units of the native pixel size (i.e. 0.492" for ACIS and 0.1318" for HRC), and can be set to sub-pixel values (e.g. 0.5).

The tool can also be given an explicit grid for the output image size using the xygrid parameter (in this mode the binsize parameter is ignored). The grid can either be:

• a filename,
• or a grid specification, of the form "xlo:xhi:#nx or dx,ylo:yhi:#ny or dy".

When given a file, the sky range of the file is used for the output (e.g. the get_sky_limits tool is run on the file and its xygrid parameter value is used). Otherwise, the grid is given using the same format as the xygrid parameter of mkexpmap; examples include:

3824.5:5112.5:8,4008.5:5296.5:8

and

3824.5:5112.5:#161,4008.5:5296.5:#161

If the requested grid results in the last row or column being partially filled - in other words the width or height divided by the bin size is not a whole number - then the exposure map values in the column or row will be over estimated (since they are calculated assuming that the pixel is "full").

Handling filter or grids that do not overlap the observation

The tool will error out when there is no overlap between the observation and the xygrid (if given) or any spatial filter applied to the input file.

Required data files

For observations that has been reprocessed by chandra_repro, then all that is needed is to give the events file since the required ancillary files can be found using the header of the events file (the chandra_repro script ensures that all the files needed for this script are copied to the same directory as the events file). These ancillary files can be located in one of the following directories, relative to the events file:

• .
• repro/
• primary/
• secondary/
• ../repro/
• ../primary/
• ../secondary/

If the files are not in one of these directories, they have been renamed, or new versions have been created during your analysis (e.g. a new bad-pixel file or a reprojected aspect solution created by reproject_aspect) then they can be explicitly set using the following parameters:

Ancillary data products

The only required information is that provided by the aspect solution file(s); any other file can be ignored, (either explicitly, by setting the parameter to NONE, or implicitly because it can not be found), in which case the appropriate correction or information is ignored.

Processing steps

An overview of how merge_obs works:

• The event files are checked to ensure they are all for the same instrument (ACIS-I and ACIS-S, HRC-I, or HRC-S). Event files with no data - perhaps because of a ccd_id filter - or that are for CC-mode observations are ignored.
• The ancillary files for these observations are found, if not given explicitly given, and matched up with the observations.
• The reference position is either calculated from the data or taken from the refcoord parameter.
• A summary of the observations is displayed; e.g. time taken, observation length, and several other useful details such as the focal-plane temperature (for ACIS), SIM position, and separation from the reference position.
• The event files are reprojected to the new reference position, or copied if no reprojection is needed.
• The new events files are merged together to create a single events file, where only those columns common to all the input files are used. This file can be used for further analysis but in most cases it should not be used for spectral analysis; the individual files should be used instead.
• For each new event file images, exposure maps, and exposure-corrected images are created for the required band(s). An optional thresholding step is applied to both the image and exposure map when the expmapthresh parameter is set. This step is similar to running fluximage on each observation.
• For each band, the counts images and exposure maps are summed, and used to create exposure-corrected images, one per band.

How do merge_obs, reproject_obs, and flux_obs work together

The merge_obs script can be considered to be a run of reproject_obs followed by flux_obs on all the reprojected events files. That is

unix% merge_obs "*/repro/" out/

is equivalent to

unix% reproject_obs "*/repro/" out/
unix% flux_obs out/ out/

although the latter also creates links (or copies over) the ancillary files to out/. The advantage to using reproject_obs and flux_obs rather than merge_obs is that you have more control: e.g. if you do not want the exposure-corrected images you can just use reproject_obs, or you can do the reprojection once and then run flux_obs multiple times with different bin sizes or grids.

Converting merge_all to merge_obs

Most of the parameters used in merge_all do not need to be specified when running merge_obs, since it calculates sensible values for them given the input data. For instance the command

unix% merge_all @evt.all.lis @asol.lis chip=2,3,5,6,7 energy=2.3 \
       refcoord="123.45 -23.78" xygrid="2000.5:6728.5:8,3168.5:5824.5:8" \
       merged=out/merged.evt expmap=out/merged.emap expcorr=out/merged.fluxed

- where the entries in evt.all.lis include the energy filter "[energy=500:7000]" - can be replaced by

unix% merge_obs @evt.lis out/

or

unix% reproject_obs @evt.lis out/

depending on whether or not you want the exposure map and exposure-corrected image. Note that the files in evt.lis should not include any energy filter (the default band for ACIS data is 0.5 to 7.0 keV). If you wish to over-ride the script - e.g. to specify a tangent point or output grid - then you can.

The following table lists the merge_all parameters and their equivalents in merge_obs. It does not describe the new parameters.

Comparison of parameters

Changes in the scripts 4.5.2 (April 2013) release

Handling HRC data

When merging HRC files, the sub-spaces of several columns - including CLKTICKS, AV1, and AU1 - have been removed to avoid the possibility of creating multiple GTI blocks. See the HRC subspace caveat for more information.

The script will now warn you if the HRC observations contain old and new data, in which the defnition of the PI column has changed. If you see this warning we strongly advise you to reprocess the old dataset(s) with chandra_repro and re-do your analysis.

Event file selection

The script now skips event files which have an OBS_ID keyword of "Merged".

Interleaved mode data

The script will now include both the primary and secondary cycles (i.e. the "e1" and "e2" forms) of ACIS interleaved mode data. Previously it would just select one of the files (generally the "e1", which is the shorter of the two). If you only want to include one of the cycles then use an explicit list of observations. When processing interleaved-mode data, file names are labelled as "<obsid>e1" or "<obsid>e2".

Aspect solutions

The program now checks that the aspect-solution files are not aspect histograms.

Improved warnings and errors

The handling of missing ancillary files has been improved; if none of a particular type - such as PBK or mask - can be found then the program runs as if you has set that parameter to 'NONE', otherwise the error message now tells you what observation was missing a file.

Changes in the scripts 4.5.1 (December 2012) release

The script has been updated to work in CIAO 4.5.

About Contributed Software

This script is not an official part of the CIAO release but is made available as "contributed" software via the CIAO scripts page. Please see this page for installation instructions - such as how to ensure that the parameter file is available.

Bugs

There are no known bugs for this tool.

See Also

calibration

ardlib

psf

psf

tools

acis_bkgrnd_lookup, acis_fef_lookup, acis_set_ardlib, add_grating_orders, add_grating_spectra, addresp, aprates, asphist, combine_spectra, dither_region, dmarfadd, dmextract, eff2evt, flux_obs, fluximage, fullgarf, hrc_bkgrnd_lookup, make_instmap_weights, mean_energy_map, mkacisrmf, mkarf, mkexpmap, mkgarf, mkgrmf, mkinstmap, mkpsfmap, mkrmf, mkwarf, psextract, psf_project_ray, reproject_obs, rmfimg, sky2tdet, specextract