reproject_obs

Synopsis

Reproject a set of observations to a common tangent point and create a merged event file.

reproject_obs  infiles outroot [asolfiles] [merge] [refcoord]
[parallel] [nproc] [linkfiles] [tmpdir] [clobber] [verbose]

Description

Given a set of observations (event files and, optionally, aspect solutions), reproject the event files to a common tangent point and merge the event files. The default behavior is to calculate the new reference position from the observations, but an explicit value can also be given.

To ensure that the reprojected event files can be used with other CIAO tools and scripts, the matching aspect solution, bad-pixel, mask, and dtf files (where applicable) are linked (or copied) to the output location, and the header keywords updated to match the new files.

By default the event files will also be merged together to form a single event file which can be used in ds9 or to create images, time series, or spectra. Care should be taken when using this merged product with other CIAO tools since some information may have been lost (e.g. if observations have different response files in the CALDB); the preference is to use the individual event files to create separate data products and then either analyze them simultaneously or combine them.

This tool can be combined with flux_obs to replicate merge_obs: for instance

unix% reproject_obs "*/repro/*evt2.fits" reproj/
unix% flux_obs reproj/ reproj/

where the observations have been re-processed by chandra_repro, hence the search for the original files in the repro/ sub-directory.

WCS alignment

The individual event files are reprojected to the same tangent point, but no attempt is made to align the observations. If you find that the WCS of the individual observations are not aligned - e.g. you can see "double" sources in the reprojected, merged data - then you need to update the WCS before running reproject_obs. An example is the Correcting Absolute Astrometry thread.

Examples

Example 1

unix% reproject_obs 5826/,5827/ reproj/
unix% ds9 reproj/merged_evt.fits -bin factor 8 -scale log -region
reproj/merged.fov

When given a directory name, the script searches for event files in the directory using the following patterns:

• dirname/repro/*evt*
• dirname/primary/*evt*
• dirname/*evt*

where the search stops once a match is made.

So this example reprojects the observations stored in the directories 5826/ and 5827/, and creates a merged events file called reproj/merged_evt.fits.

The aspect solutions are assumed to be in the same directory as the event files and their names are taken from the ASOLFILE keyword.

Note that the event files do not need to be listed in observation order; the script will re-orders them if necessary.

Example 2

unix% reproject_obs "*/repro/*evt2*" reproj/
unix% ds9 reproj/merged_evt.fits

All the "evt2" files in the repro sub-directories are reprojected to create files called "reproj/<obsid>_reproj_evt.fits".

Example 3

unix% pset reproject_obs outroot=reproj/
unix% reproject_obs
5826/repro/acisf05826_repro_evt2.fits,5827/repro/acisf05827_repro_evt2.f
its

In this example the event files are listed explicitly, as a comma-separated string.

Example 4

unix% find . -name \*repro_evt2.fits > evt2.lis
unix% reproject_obs @evt2.lis reproj/ clobber+

This is similar to the first example except that a stack file is used to list the files to be processed (see "ahelp stack" for more information). The UNIX find command is used to list all files that end in the text repro_evt2.fits that are in the current directory or any subdirectories, and this list is passed to reproject_obs using the "@" syntax.

Example 5

unix% reproject_obs @evt2.lis"[ccd_id=0:3]" acisi/

Here we apply a DataModel filter to the stack so that only the ACIS-I chips (ie those with ccd_id between 0 and 3 inclusive), will be reprojected and merged.

Example 6

unix% reproject_obs @evt2.lis reproj/ ref="123.45 -2.2"

In this case the reference coordinate is given explicitly, as a space separated pair of decimal degrees, rather than being calculated for us.

Example 7

unix% reproject_obs @evt2.lis reproj/ ref="8:13:48 -2:12:0"

This is the same as the previous example but the reference position is given in sexagesimal notation.

Example 8

unix% reproject_obs "@evt2.lis[ccd_id=7]" reproj6/
ref=reproj/merged_evt.fits

Here we reproject and merge only the ACIS-S3 chips from the obervations (if an event file in evt2.lis has no data from this chip then it will be skipped). The reprojection is done to match the tangent point from the file reproj/merged_evt.fits.

Example 9

unix% reproject_obs @evt2.lis reproj/ asolfiles=@asol.lis

In this example the aspect solutions are explicitly listed, rather than taken from the ASOLFILE keyword in the header of each event file. The order of the files in asol.lis does not need to match that of evt2.lis since the script matches up aspect solutions to event files.

Parameters

Detailed Parameter Descriptions

Parameter=infiles (file required stacks=yes)

Input event files

Multiple event files may be provided as a comma-separated list or as a stack; see "ahelp stack" for more information. The event files do not need to be given in time order.

Listing just a directory name

Rather than list the full path to an event file, you can just give a directory name; in such a case the script searches for any files that match the pattern

• dirname/repro/*evt*
• dirname/primary/*evt*
• dirname/*evt*

where the search is done in the order given above, and stops as soon as a match (or matches) is found.

Skipping files

Any observations that do not match the instrument of the first observation are skipped, as are CC mode observations or those with no data (e.g. because of a ccd_id filter). For HRC data, the first observation determines whether HRC-I or HRC-S are being combined, with observations from the other instrument being skipped. Any file with an OBS_ID keyword of "Merged" is skipped.

If multiple files with the same OBS_ID keyword are found, then all but the first are skipped, unless they are part of an ACIS interleaved-mode observation (i.e. they have CYCLE=P or CYCLE=S).

The input files must contain at least the following columns. otherwise they are skipped:

File headers

If the asolfiles parameter is left empty then the ASOLFILE keyword of each event file is used to find the relevant aspect solution files.

Parameter=outroot (string required default=)

Root name for the output merged files.

Any directories included in the root name will be created if they do not exist. If the parameter ends in "/" then the output files will be named

<obsid>_reproj_evt.fits

in the specified directory. Otherwise, the filename will be

<outroot>_<obsid>_reproj_evt.fits

where <obsid> is the value of the OBS_ID header keyword for each events file.

Merged event file

If the merge parameter is set to "yes" then the merged event file will be named

<outroot>/merged_evt.fits

or

<outroot>_merged_evt.fits

depending on whether the outroot parameter ends in "/" or not.

Interleaved mode

ACIS interleaved mode files may have "e1" or "e2" appended to the ObsId when creating the output name (it depends on whether the file is unique to the primary or secondary cycle, such as the event or bad-pixel files, or is the same for both cycles, such as the aspect solution).

Parameter=asolfiles (file stacks=yes)

Input aspect solutions

If the value is empty then the ASOLFILE keyword from the events files will be used to find the files to use.

The aspect solution files have names like pcadf*_asol1.fits and are included in the output directory of the chandra_repro script. To explicitly specify the asol files use the stack syntax (e.g. a comma, or space, separated string or an external file as described in "ahelp stack" for more information). So to use asol1.fits, asol2.fits, and asol3.fits you could say

asolfile="asol1.fits,asol2.fits,asol3.fits"

or

asolfile="asol1.fits asol2.fits asol3.fits"

or

asolfile=@asol.lis

where asol.lis contains the names of each file, one per line. The files do not have to be given in time order.

Since there may be multiple asol files for an observation there may be more entries in this parameter than there are in the infiles parameter.

Aspect histograms - the output of the asphist tool - can not be used here.

Parameter=merge (boolean default=yes)

Merge event files?

Should the reprojected event files be merged together? If so, the merged event file will be called outroot/merged_evt.fits or outroot_merged_evt.fits. This file can be treated as a normal event file but great care should be taken when using it for anything other than simple imaging or timing analysis. In general it is safer to use the individual reprojected event files and either combine the results or do a joint analysis, treating each observation separately.

Only those columns that are common to all the event files are copied to the merged event file. The "expno" subspace in the merged event file does not reflect any filtering applied to the original event files, which means that great care should be taken if you wish to do timing analysis based on the EXPNO column.

Parameter=refcoord (string default=)

Reference coordinates or evt2 file

This parameter defines the central coordinate of the reprojected data; that is the Right Ascension and Declination that maps to SKY=(4096.5,4096.5) for ACIS data, (16384.5,16384.5) for HRC-I data, or (32768.5,32768.5) for HRC-S data.

The default setting ("") means that this value is calculated from the tangent points of the observations. When set it can be either:

• the name of a file, so that the tangent position of the file is used;
• or a space-separated value taken to be the RA and Declination to use (ICRS) in decimal degrees or colon-separated sexagesimal formats.

Examples include:

refcoord=fluxed/broad_flux.img
refcoord="123.45 -12.34"
refcoord="8:13:48 -12:20:24"

Parameter=parallel (boolean default=yes)

Run processes in parallel?

When run on a multi-processor system, many of the tasks can be run in parallel to reduce the execution time of the script. Since the tasks are likely to be memory or I/O-bound, the reduction in run time will be less than the number of cores on a machine. When parallel=yes, the default behaviour is to use all the CPU processors, but this can be changed with the nproc parameter.

This option can be ignored when run on a single-processor system.

Parameter=nproc (integer default=INDEF)

Number of processors to use

This parameter is only used when parallel=yes. It determines the number of processors to use. If maxproc is the actual number of processors on your machine, then a value of INDEF - the default value - means that all maxproc processors will be used. A positive value means to use that number of processors (any value larger than maxproc will be set to maxproc). A negative value is added to maxproc (and any value less than one is set to one).

Parameter=linkfiles (boolean default=yes)

Link (rather than copy) files?

Should the ancillary files (e.g. aspect solution, bad-pixel file) be linked or copied to the output location (given by the outroot parameter)? If link is set to yes then a soft link is used, which saves disk space, otherwise the ancillary files are copied.

This parameter is set to "no" for file systems that do not support soft links.

Parameter=tmpdir (string default=${ASCDS_WORK_PATH})

Directory for temporary files.

Directory for storing temporary files that require further processing before becoming useful. If the directory does not exist then it will be created for use by the script, and then deleted.

Parameter=clobber (boolean default=no)

Overwrite existing files?

Parameter=verbose (integer default=1)

Output verbosity.

The default verbosity value of 1 prints status messages as the script runs. Higher verbosity settings print the commands that are being run. Setting verbose=0 turns off the screen output.

Ancillary files

The reprojected event file for an observation is either

outroot/obsid_reproj_evt.fits

or

outroot_obsid_reproj_evt.fits

depending on whether the outroot parameter ends in a '/' or not. Along with this file, the following ancillary files (where appropriate) are also created:

If the linkfiles parameter is set to yes then these files are soft links to the files in the input directory structure (i.e. the location of the input event files). If the file system does not support soft links, or linkfiles is no, then a copy of the file will be used instead.

The headers of the reprojected event files are updated so that they reference the new file names (apart from FOVFILE). This allows tools like flux_obs and fluximage to pick the files up automatically when given just the name of the reprojected event file.

Interleaved mode

When handling interleaved-mode observations, also known as alternating-exposure mode, the bad-pixel and mask files are labelled with the cycle (i.e. "e1" or "e2" is appended to the ObsId), but the aspect solution files are shared between the two cycles, so do not get this extra label.

Multi-OBI datasets

A small number of Chandra data sets contain multiple-OBI blocks. For these cases, the splitobs script will create multiple directories, one for each OBI, and then they can be analysed separately. To identify these observations, the ObsId number will have the OBI number appended to it as a three-digit integer; for example: 3057_001 and 3057_002.

Processing steps

An overview of how reproject_obs works:

• The event files are checked to ensure they are all for the same instrument (ACIS-I and ACIS-S, HRC-I, or HRC-S). Event files with no data - perhaps because of a ccd_id filter - or that are for CC-mode observations are ignored.
• The ancillary files for these observations are found, if not given explicitly given, and matched up with the observations.
• The reference position is either calculated from the data or taken from the refcoord parameter.
• A summary of the observations is displayed; e.g. time taken, observation length, and several other useful details such as the focal-plane temperature (for ACIS), SIM position, and separation from the reference position.
• The event files are reprojected to the new reference position, or copied if no reprojection is needed.
• For each new event file the ancillary files are either linked, or copied, to the output location. The header of the event file is updated so that it points to these new files. At this point the per-observation FOV files are created using the reprojected event files. A combined FOV file (merged.fov) is also created which includes the individual FOV regions (the CCD_ID information is not included in this file).
• The new events files are merged together to create a single events file, where only those columns common to all the input files are used. The resulting file can be used for further analysis but in most cases it should not be used for spectral analysis; the individual files should be used instead.

Changes in the scripts 4.16.0 (December 2023) release

File validation checks at the initialization of the script had an overlooked dependency on the presence of mask files, which are neither a parameter input nor used by the script. The script has been updated to break this unnecessary dependency.

Changes in the scripts 4.14.0 (December 2021) release

The script now generates the FOV file for each reprojected event file and a combined region file (merged.fov) that includes all the individual observation FOV files.

Changes in the scripts 4.13.0 (December 2020) release

The warning message about not using the merged event file for calculating instrument responses now restricts the message so that the value does not include excess precision, but is limited to the limit value. The tangent point in the screen output has also seen its accuracy reduced (this only affects the screen output).

Changes in the scripts 4.9.2 (April 2017) release

The script now works when one (or more) of the ObsIds to be reprojected has multiple aspect-solution files and the Python 3.5 version of CIAO 4.9 is being used. The previous behavior was to error out with the message "ERROR name 'xrange' is not defined".

Changes in the scripts 4.8.1 (December 2015) release

The code has been updated to avoid warning messages from NumPy version 1.9. There is no difference to how the script behaves.

Changes in the scripts 4.7.3 (June 2015) release

Improved support for multi-obi datasets

The script is now able to work with multi-obi datasets that have been processed by splitobs and then chandra_repro.

Changes in the scripts 4.6.6 (September 2014) release

Reprojecting observations

Observations will only be reprojected if the tangent point shift is larger than 0.05 arcseconds, otherwise the original event file is copied.

Changes in the scripts 4.6.1 (December 2013) release

Removed support for the Parameter Block File

With the deprecation of the pbkfile parameter in CIAO 4.6 tools, this script no longer updates the PBKFILE header keyword or copies over/links the pbk files.

Changes in the scripts 4.5.2 (April 2013) release

Handling HRC data

When merging HRC files, the sub-spaces of several columns - including CLKTICKS, AV1, and AU1 - have been removed to avoid the possibility of creating multiple GTI blocks. See the HRC subspace caveat for more information.

The script will now warn you if the HRC observations contain old and new data, in which the defnition of the PI column has changed. If you see this warning we strongly advise you to reprocess the old dataset(s) with chandra_repro and re-do your analysis.

Event file selection

The script now skips event files which have an OBS_ID keyword of "Merged".

Interleaved mode data

The script will now include both the primary and secondary cycles (i.e. the "e1" and "e2" forms) of ACIS interleaved mode data. Previously it would just select one of the files (generally the "e1", which is the shorter of the two). If you only want to include one of the cycles then use an explicit list of observations. When processing interleaved-mode data, file names are labelled as "<obsid>e1" or "<obsid>e2".

Aspect solutions

The program now checks that the aspect-solution files are not aspect histograms.

Improved warnings and errors

Prior to this release, reproject_obs would error out with the less-than helpful message "'NoneType' object is not subscriptable" when an ancillary file (such as the mask file) was missing. The program now tells you that the file can not be found but carries on with the reprojection.

Changes in the scripts 4.5.1 (December 2012) release

The script has been updated to work in CIAO 4.5.

About Contributed Software

This script is not an official part of the CIAO release but is made available as "contributed" software via the CIAO scripts page. Please see this page for installation instructions - such as how to ensure that the parameter file is available.

Bugs

There are no known bugs for this tool.

See Also

tools::composite

flux_obs, fluximage, merge_obs

tools::coordinates

reproject_aspect, reproject_events, sso_freeze, wcs_match, wcs_update