Merging Data from Multiple Imaging Observations
![[CXC Logo]](../../imgs/cxc-logo.gif){kind=logo}
CIAO 4.17 Science Threads
Overview
Synopsis:
When merging event files from observations with similar, but not necessarily identical, target coordinates, the event sky coordinates (X,Y in the event list) must be recomputed. This is often referred to as "regridding" or "reprojecting events to a different tangent point." It is necessary because the sky coordinates are computed relative to the nominal R.A. and Dec. for each individual observation, which will likely differ between observations of the same target.
Purpose:
To combine ACIS or HRC imaging data from multiple observations, e.g. to search for faint sources or low-surface-brightness extended features.
The merged event list should not be used for spectral analysis, since it does not contain sufficient information to generate correct response files. The recommended technique for the spectral analysis case is to generate separate PHA, RMF, and ARF files for each observation and to analyze them simultaneously.
Related Links:
- Overview: Reprojecting Files
-
Why combine data? and the How & Why (or Why Not) to Combine Data.
Correcting Absolute Astrometry thread.
Last Update: 26 Jul 2023 - Corrected and renamed the Correcting Astrometry Prior to Merging section. Specifically added the missing step to update the event file and to remove the inaccurate information about preserving the EDSER subpixel event position.
Contents
- Getting Started
- Merging about a Common Tangent Point
- Correcting Astrometry Prior to Merging
- History
- Images
Getting Started
Download the sample data: 1842 (ACIS-I, G21.5-09); 1843 (ACIS-I, G21.5-09)
unix% download_chandra_obsid 1842,1843
unix% chandra_repro 1842,1843 outdir=
Merging about a Common Tangent Point
Each event list contains all the World Coordinate System (WCS) information required to transform between sky coordinates and celestial coordinates in its header. If the WCSs are accurate, one may merge event lists simply by converting sky coordinates in one event list to celestial, using that list's WCS, and then converting those celestial coordinates to sky coordinates in the second event list using the second WCS (see the ds9 thread coordinate definitions for more information).
The level=2 event files may be reprojected to a common tangent point in one of two ways:
- Using WCS Information to Merge: all the files are reprojected to match one file.
- Merging about a Specific R.A. and Dec.: the files are reprojected to a new tangent point that is different than any of the input observations.
Using WCS Information to Merge
In this case we choose to map ObsId 1842 onto 1843, so we set the refcoord parameter of reproject_obs to 1843/repro/acisf01843_repro_evt2.fits:
unix% punlearn reproject_obs unix% reproject_obs 1842,1843 merged1/ refcoord=1843/repro/acisf01843_repro_evt2.fits Running reproject_obs Version: 05 November 2020 Found 1842/repro/acisf01842_repro_evt2.fits Found 1843/repro/acisf01843_repro_evt2.fits Verifying 2 observations. Tangent point is taken from the file 1843/repro/acisf01843_repro_evt2.fits New tangent point: RA=18h 32m 11.140s Dec=-10d 34' 9.85" Observations to be reprojected: Obsid Obs Date Exp DETNAM SIM_Z FP Sepn PA (ks) (mm) (K) (') (deg) --------------------------------------------------------------- 1 1843 2000-09-02 7.9 ACIS-012367 -245.857 153.4 0.0 +0 2 1842 2000-09-02 7.4 ACIS-012367 -231.239 153.4 5.1 +101 Running tasks in parallel with 4 processors. Reprojecting 2 event files to a common tangent point. Merging reprojected events files to merged1/merged_evt.fits The following files were created: The reprojected event files: merged1/1843_reproj_evt.fits merged1/1842_reproj_evt.fits The merged event file: merged1/merged_evt.fits Warning: the merged event file merged1/merged_evt.fits should not be used to create ARF/RMF/exposure maps because the RA_NOM keyword varies by 0.0851 (limit is 0.0003) the DEC_NOM keyword varies by 0.0159 (limit is 0.0003) the ROLL_NOM keyword varies by 2.5 (limit is 1.0) the SIM_Z keyword varies by 14.6 (limit is 0.1)
The contents of the parameter file may be checked using dmhistory:
unix% dmhistory merged1/merged_evt.fits reproject_obs reproject_obs infiles="1842,1843" outroot="merged1/" asolfiles="" merge="yes" refcoord="1843/repro/acisf01843_repro_evt2.fits" parallel="yes" nproc="INDEF" linkfiles="yes" tmpdir="./" clobber="yes" verbose="1"
and the script created the following files:
unix% ls -1 merged1/ 1842.asol@ 1842.bpix@ 1842.mask@ 1842_reproj_evt.fits 1843.asol@ 1843.bpix@ 1843.mask@ 1843_reproj_evt.fits merged_evt.fits
The merged event file can be displayed in ds9:
unix% ds9 -cmap b merged1/merged_evt.fits -bin filter 'energy=500:7000' -bin about 3815 3975 -zoom 8
as shown in Figure 1.
Figure 1: Data merged by WCS information
![[Thumbnail image: The source photon distribution in the merged event list appears as slightly double-peaked.]](wcs.png)
[Version: full-size]
Figure 1: Data merged by WCS information
The appearance of the merged source - which has two peaks - suggests that there is a small systematic offset in the two WCSs.
The source photon distribution in the merged event list appears as slightly double-peaked, indicating that there is in fact a small (2-3 pixels) systematic offset in the two WCSs. This may be minimized by first correcting the aspect, as described in the Correcting Aspect Prior to Merging section.
Merging about a Specific R.A. and Dec.
It is also possible to reproject the events to a new tangent point that is different than any of the input observations. In this example, both event lists are reprojected to the arbitrarily chosen point (R.A.,Dec) = (278.086,-10.58625) prior to merging.
unix% punlearn reproject_obs unix% reproject_obs 1842,1843 merged2/ refcoord="278.086,-10.58625" Running reproject_obs Version: 05 November 2020 Found 1842/repro/acisf01842_repro_evt2.fits Found 1843/repro/acisf01843_repro_evt2.fits Verifying 2 observations. New tangent point: RA=18h 32m 20.640s Dec=-10d 35' 10.50" Observations to be reprojected: Obsid Obs Date Exp DETNAM SIM_Z FP Sepn PA (ks) (mm) (K) (') (deg) --------------------------------------------------------------- 1 1843 2000-09-02 7.9 ACIS-012367 -245.857 153.4 2.5 -67 2 1842 2000-09-02 7.4 ACIS-012367 -231.239 153.4 2.7 +89 Running tasks in parallel with 4 processors. Reprojecting 2 event files to a common tangent point. Merging reprojected events files to merged2/merged_evt.fits The following files were created: The reprojected event files: merged2/1843_reproj_evt.fits merged2/1842_reproj_evt.fits The merged event file: merged2/merged_evt.fits Warning: the merged event file merged2/merged_evt.fits should not be used to create ARF/RMF/exposure maps because the RA_NOM keyword varies by 0.0851 (limit is 0.0003) the DEC_NOM keyword varies by 0.0159 (limit is 0.0003) the ROLL_NOM keyword varies by 2.5 (limit is 1.0) the SIM_Z keyword varies by 14.6 (limit is 0.1)
The contents of the parameter file may be checked using dmhistory:
unix% dmhistory merged2/merged_evt.fits reproject_obs reproject_obs infiles="1842,1843" outroot="merged2/" asolfiles="" merge="yes" refcoord="278.086,-10.58625" parallel="yes" nproc="INDEF" linkfiles="yes" tmpdir="./" clobber="yes" verbose="1"
The data looks the same as Figure 1 apart from a different SKY to Ra,Dec mapping; the choice of tangent point means that the source is close to 4090,4095 rather than 3815,3975 above:
unix% ds9 -cmap b merged2/merged_evt.fits -bin filter 'energy=500:7000' -zoom 8
Correcting Astrometry Prior to Merging
In order to improve on the results of the previous examples, we must address the problem of the systematic shift in the coordinate systems of the two event lists. To do this, we will correct the astrometry of the data to make them consistent with each the other.
![[WARNING]](../../imgs/warning.png)
If you choose to use this method, the shifted aspect solution should then be used throughout the rest of the analysis in order to obtain correct results. This is why we update the ASOLFILE keyword in the reprojected event file.
For the details of how to compute and apply astrometric corrections, users should read the Correcting Absolute Astrometry thread. Below is an abbreviated and lightly annotated example of one approach to do this.
First we determine what shift to apply to the data. In this case we use dmstat to calculate the centroid of the source in the two observations, but other approaches, such as using the positions calculated by wavdetect are also possible.
There is a bug in the data model where the virtual EQPOS column cannot be accessed in the same command where other regular column filters are applied. Therefore we filter the file first with dmcopy to the region of interest (and in this example, the energy band, though this is not necessary for correcting astrometry), then use dmstat to compute the mean RA and DEC coordinates. In this example, these two tools are run in one line to skip the creation of an extraneous intermediate file; with this syntax, the output of dmcopy ingested by dmstat is not saved to the directory, and only the output of dmstat is returned.
unix% punlearn dmstat
unix% dmcopy "1842/repro/acisf01842_repro_evt2.fits[energy=500:7000][sky=circle(4418,4087,5)]" - | dmstat "-[cols eqpos]"
EQPOS(RA, DEC)[deg]
min: ( 278.08616151 -10.587086206 ) @: ( 63 70 )
max: ( 278.08718637 -10.586138674 ) @: ( 34 12 )
mean: ( 278.08680354 -10.586491986 )
sigma: ( 0.00022748737552 0.00016481664255 )
sum: ( 22246.944283 -846.91935884 )
good: ( 80 80 )
null: ( 0 0 )
unix% dmcopy "1843/repro/acisf01843_repro_evt2.fits[energy=500:7000][sky=circle(3805,3974,5)]" - | dmstat "-[cols eqpos]"
EQPOS(RA, DEC)[deg]
min: ( 278.08649298 -10.58658084 ) @: ( 14 42 )
max: ( 278.08732554 -10.585526826 ) @: ( 26 97 )
mean: ( 278.08692147 -10.586098349 )
sigma: ( 0.000140929028 0.00020055219192 )
sum: ( 31423.822127 -1196.2291135 )
good: ( 113 113 )
null: ( 0 0 )
We decide to shift ObsId 1842 onto 1843 since it is the shorter observation (although in this case the difference is not large). ObsID 1843 will be the reference dataset. To find the shift:
-
calculate the difference between the two centroids:
unix% python >>> ra1 = 278.08680354 >>> ra_ref = 278.08692147 >>> dec1 = -10.586491986 >>> dec_ref = -10.586098349 >>> dra = ra_ref - ra1 >>> ddec = dec_ref - dec1 >>> dra 0.00011792999998760934 >>> ddec 0.00039363700000016877 -
correct the delta RA by "cos(Dec)" to get a reasonable approximation of the distance between the two points.
>>> mean_dec = ( dec1 + dec_ref )/2.0 >>> dra_ref = dra * np.cos(mean_dec * np.pi / 180.0) >>> dra_ref 0.00011592275133369907 -
convert the values to sky pixels. The delta x conversion is done with -1.3667E-4 deg/pixel because the x value is opposite in sign from RA. Dec and y have the same sign, so 1.3667E-4 deg/pixel is used for that calculation. (note that 0.492/3600 = 1.3667e-4, where 0.492 is the ACIS pixel size)
>>> conv = 0.492 / 3600 >>> dx = dra_ref * -1 / conv >>> dy = ddec / conv >>> dx -0.8482152536612126 >>> dy 2.880270731708552
The tool wcs_update is used to apply the offsets. The shifts are input to the deltax and deltay parameters in units of sky pixels. Additionally, the WCS information from the event file for the match observation (1843/repro/acisf01843_repro_evt2.fits) is supplied in the wcsfile parameter. Note that the transformfile parameter is left empty.
unix% punlearn wcs_update unix% pset wcs_update infile=1842/repro/pcadf01842_000N001_asol1.fits unix% pset wcs_update outfile=1842/repro/new_asol1.fits unix% pset wcs_update wcsfile=1843/repro/acisf01843_repro_evt2.fits unix% pset wcs_update deltax=-0.85 deltay=2.88 unix% wcs_update Input coordinate transform file (): Either input asol file, or file with WCS to be updated (1842/repro/pcadf01842_000N001_asol1.fits): Output asol file (1842/repro/new_asol1.fits):
We then also need to update the event file. Since the event file is modified in place we first make a copy and set the wcs_update outfile to blank, "".
unix% dmcopy 1842/repro/acisf01842_repro_evt2.fits 1842/repro/new_evt2.fits clob+ unix% wcs_update infile=1842/repro/new_evt2.fits outfile="" mode=h
![[NOTE]](../../imgs/note.png)
Earlier versions of this thread had a discussion about preserving the EDSER subpixel event position information and how it can get lost when using reproject_events. However, the information that was presented was not entirely accurate and the suggestion to run acis_process_events was incorrect.
EDSER subpixel event position is preserved when running reproject_events without specifying the aspect solution file, ie setting aspect="" (blank).
Since the fine astrometric correction is applied to both the event file and the aspect solution file the physical sky coordinates are unchanged.
Finally we update the ASOLFILE keyword in the new events file to point to the new aspect solution:
unix% dmhedit 1842/repro/new_evt2.fits filelist= op=add key=ASOLFILE value=new_asol1.fits
Manually reprojecting the data
We can now reproject ObsId 1842 onto 1843 before merging them (making sure to list the inputs to dmmerge in time order):
unix% punlearn reproject_events unix% reproject_events 1842/repro/new_evt2.fits 1842/repro/reproj_evt2.fits 1843/repro/acisf01843_repro_evt2.fits unix% punlearn dmmerge unix% dmmerge 1843/repro/acisf01843_repro_evt2.fits,1842/repro/reproj_evt2.fits merged.fits omit - DEC_PNT values different more than 0.000300 warning: DS_IDENT has different value...Merged... warning: OBJECT has different value...Merged... warning: OBS_ID has different value...Merged... omit - RA_PNT values different more than 0.000300 omit - ROLL_NOM values different more than 1.000000 omit - ROLL_PNT values different more than 1.000000 warning: SEQ_NUM has different value...Merged... omit - SIM_Z values different more than 0.100000
The merged event file can be displayed in ds9:
unix% ds9 -cmap b merged.fits -bin filter 'energy=500:7000' -bin about 3815 3975 -zoom 8
(as shown in Figure 2).
Figure 2: Aspect solution corrected before reprojection
![[Thumbnail image: The source now has a single peak, rather than the double peak visible in previous versions.]](corrasol.png)
[Version: full-size]
Figure 2: Aspect solution corrected before reprojection
The shift in the peak emission visible in earlier versions - such as Figure 1 - is no longer seen.
Using reproject_obs
When combining data from multiple observations - or you wish to reproject the data to a different position - it is easier to run reproject_obs on all the observations, rather than running reproject_events and dmmerge manually. For example, we could have said
unix% punlearn reproject_obs
unix% reproject_obs 1842/repro/new_evt2.fits,1843/repro/acisf01843_repro_evt2.fits merged3/
Running reproject_obs
Version: 05 November 2020
Verifying 2 observations.
Calculating new tangent point.
New tangent point: RA=18h 32m 21.347s Dec=-10d 34' 38.44"
Observations to be reprojected:
Obsid Obs Date Exp DETNAM SIM_Z FP Sepn PA
(ks) (mm) (K) (') (deg)
---------------------------------------------------------------
1 1843 2000-09-02 7.9 ACIS-012367 -245.857 153.4 2.6 -79
2 1842 2000-09-02 7.4 ACIS-012367 -231.239 153.4 2.6 +101
Running tasks in parallel with 4 processors.
Reprojecting 2 event files to a common tangent point.
Merging reprojected events files to merged3/merged_evt.fits
The following files were created:
The reprojected event files:
merged3/1843_reproj_evt.fits
merged3/1842_reproj_evt.fits
The merged event file:
merged3/merged_evt.fits
Warning: the merged event file merged3/merged_evt.fits
should not be used to create ARF/RMF/exposure maps because
the RA_NOM keyword varies by 0.0851 (limit is 0.0003)
the DEC_NOM keyword varies by 0.0159 (limit is 0.0003)
the ROLL_NOM keyword varies by 2.5 (limit is 1.0)
the SIM_Z keyword varies by 14.6 (limit is 0.1)
unix% ds9 -cmap b merged3/merged_evt.fits -bin filter 'energy=500:7000' -bin about 4121 4038 -zoom 8
which creates an image similar to Figure 2, although with different mapping between SKY and WORLD coordinates, because of the different tangent point.
History
| 10 Jan 2012 | reviewed for CIAO 4.4: clarified the order of the steps in the Correcting Aspect Prior to Merging section |
| 15 Oct 2012 | Updated to use reproject_obs rather than merge_all, which is now deprecated. The Correcting Aspect Prior to Merging section has been updated. |
| 03 Dec 2012 | Review for CIAO 4.5 |
| 15 Mar 2013 | Fixed links in manually merging section. |
| 27 Jun 2013 | Tweaked information in the distance equations. |
| 02 Dec 2013 | Review for CIAO 4.6. Noted that the PBK file for ACIS observations is no-longer copied/linked by reproject_obs. |
| 16 Dec 2014 | Reviewed for CIAO 4.7. Added a link to the merged datasets spectrum caveat. |
| 27 Nov 2017 | Added link to bug in DM that affects [cols eqpos] filtering. |
| 13 May 2019 | Updated to use python to compute offsets. |
| 18 Oct 2021 | Updated for Repro-5 changes and CIAO 4.13/CALDB4.9.6. |
| 13 Jan 2022 | Reviewed for CIAO 4.14. No changes. |
| 26 Jul 2023 | Corrected and renamed the Correcting Astrometry Prior to Merging section. Specifically added the missing step to update the event file and to remove the inaccurate information about preserving the EDSER subpixel event position. |