Why combine data?
Due to scheduling constraints - that are getting more severe as Chandra ages - a long observation is likely to be performed as several, smaller, observations (each with a separate Observation Id). Many targets have also been observed by multiple observers, whether to go deeper, monitor sources, or to cover a larger area than possible with a single observation. In both cases, it would make life easier for you, as a user, to be able to combine all the observations into one, and use this for your analysis. Unfortunately, as described below and elsewhere on this site - e.g. the How & Why (or Why Not) to Combine Data document - this is often not the best solution, and can lead to incorrect results.
The time-varying response
The reason that data can not - in general - be simply combined with the dmmerge tool is that the response for a particular chip coordinate (i.e. where the photon strikes the detector) depends on time:
-
within an observation, the telescope dither pattern means that the area of sky seen by each detector (pixels for ACIS, grid/channels for the HRC) varies with time;
-
between observations, the telescope and detector properties can change - for instance the ACIS time-dependent gain and contamination model, or the HRC-I and HRC-S gain settings;
-
the detector properties may even vary during an observation (due to scheduling constraints, the focal-plane temperature may vary significantly within an observation, which can change the ACIS detector response);
-
the background may vary (e.g. due to changes in the intensity or spectrum of the particle background along Chandra's orbit).
For more information on instrumental issues please review the Chandra Data Caveats page.
Since the detector response - that is, terms Am(E), Rm(E,h), and Bm(h) in equation 1 of How & Why (or Why Not) to Combine Data - varies with time, then this variation would need to be accounted for when combining the responses (c.f. equation 3 Ibid.). Although tools like mkwarf and addresp can combine responses (whether implicitly or explicitly), they should not be used to create a response from a combined event file, as some of the information needed to track down all the time dependencies has been lost.
In the following we show the difference between "combining and then calculating the response" - as used in merge_all (tool withdrawn) - and "calculate the individual responses and then combine" - as used by merge_obs - when calculating exposure-corrected ACIS images. For simplicity we focus on the CSC broad band, that is 0.5 to 7.0 keV band with the exposure map evaluated at 2.3 keV.
The test data
In the following we shall use the 5 Chandra observations of 3C186, all ACIS-S, since
-
the observations span a large time range - May 2002 to December 2007 - so the response of the ACIS detector changes significantly between them (in particular the change in the contamination layer);
-
the second set, taken in December 2007, represent a 200 ks observation that had to be split up;
-
for the combined 200 ks data, the first three observations - 9407, 9774, and 9775 - have the same SIM position, but the last one (9408) is slightly shifted, so there may be issues with combining data from all four observations;
-
and the first observation (3098) was taken using only part of ACIS-S3 (as it was taken using the sub-array mode).
The analysis presented below was performed with CIAO 4.4 Tuesday, June 5, 2012, that is the CIAO 4.4.1 release, using CALDB version 4.5.3.
The data can be retrieved and reprocessed by saying:
unix% find_chandra_obsid 3c186 # obsid sepn inst grat time obsdate piname target 3098 0.0 ACIS-S NONE 37.9 2002-05-16 Siemiginowska Q0740+380 9407 0.0 ACIS-S NONE 67.2 2007-12-03 Siemiginowska "3C 186" 9408 0.0 ACIS-S NONE 40.1 2007-12-11 Siemiginowska "3C 186" 9774 0.0 ACIS-S NONE 76.1 2007-12-06 Siemiginowska "3C 186" 9775 0.0 ACIS-S NONE 16.1 2007-12-08 Siemiginowska "3C 186" unix% find_chandra_obsid 3c186 download=all ... a lot of output ... unix% chandra_repro 3098,9407,9408,9774,9775 Output directory (default = $indir/repro) (): # acis_process_events (CIAO 4.4): The following error occurred 13 times: dsAPEPULSEHEIGHTERR -- WARNING: pulse height is less than split threshold when performing serial CTI adjustment. Output from acis_process_events: # acis_process_events (CIAO 4.4): The following error occurred 4 times: dsAPEPULSEHEIGHTERR -- WARNING: pulse height is less than split threshold when performing serial CTI adjustment. Output from acis_process_events: # acis_process_events (CIAO 4.4): The following error occurred 13 times: dsAPEPULSEHEIGHTERR -- WARNING: pulse height is less than split threshold when performing serial CTI adjustment. Output from acis_process_events: # acis_process_events (CIAO 4.4): The following error occurred 3 times: dsAPEPULSEHEIGHTERR -- WARNING: pulse height is less than split threshold when performing serial CTI adjustment. unix% punlearn reproject_obs unix% reproject_obs '*/repro/*evt*' reproj/ Running reproject_obs Version: 17 October 2012 Verifying 5 observations. Calculating new tangent point. New tangent point: RA=7h 44m 16.584s Dec=37d 54' 9.612" Observations to be reprojected: Obsid Obs Date Exp DETNAM SIM_Z FP Sepn PA (ks) (mm) (K) (') (deg) --------------------------------------------------------------- 1 3098 2002-05-16 34.4 ACIS-7 -190.143 153.3 1.5 +177 2 9407 2007-12-03 66.3 ACIS-23567 -190.140 154.1 0.4 -2 3 9774 2007-12-06 75.1 ACIS-23567 -190.140 154.4 0.4 -2 4 9775 2007-12-08 15.9 ACIS-23567 -190.140 153.6 0.4 -3 5 9408 2007-12-11 39.6 ACIS-23567 -190.143 153.8 0.4 -6 WARNING - EXPTIME values differ: Obsid 3098 has EXPTIME=0.4 and the rest have 3.1 Running tasks in parallel with 4 processors. Reprojecting 5 event files to a common tangent point. Merging reprojected events files to: reproj/merged_evt.fits The following files were created: The reprojected event files: reproj/3098_reproj_evt.fits reproj/9407_reproj_evt.fits reproj/9774_reproj_evt.fits reproj/9775_reproj_evt.fits reproj/9408_reproj_evt.fits The merged event file: reproj/merged_evt.fits Warning: the merged event file reproj/merged_evt.fits should not be used to create ARF/RMF/exposure maps because the RA_NOM keyword varies by 0.00256923286001 (limit is 0.0003) the DEC_NOM keyword varies by 0.030215177097 (limit is 0.0003) the ROLL_NOM keyword varies by 167.915572499 (limit is 1.0) the EXPTIME keyword contains: 0.4 3.1 which means that the DTCOR value, and hence LIVETIME/EXPOSURE keywords are wrong the aim points fall on CCDs: 7 which means that the ONTIME/LIVETIME/EXPOSURE keywords do not reflect the full observation length.
Not shown directly above - although it is mentioned in the output of reproject_obs - is the variation in the roll between the observations. As shown below, 3098 is rotated by almost 180 degrees from the other observations; 9407, 9774, and 9775 are very close; and 9408 is slightly different.
unix% foreach o ( 3098 9407 9774 9775 9408 )
foreach? echo "$o `dmkeypar $o/repro/acisf0${o}_repro_evt2.fits roll_nom echo+`"
foreach? end
3098 270.57447599566
9407 102.65890349618
9774 102.65890960359
9775 102.65892363247
9408 103.70777695244
We use a combination of reproject_obs and flux_obs in the following - rather than just using merge_obs - to avoid repeated work, namely the re-projection of the event files.
The merged event file looks like the following, which shows the 0.5 to 7.0 keV energy range, and the sub-array used for ObsId 3098 is highlighted by the green polygon:
![[The merged observations for 3C186, with a rectangle showing the location of ObsId 3098]](merged_event.png)
unix% ds9 reproj/merged_evt.fits -smooth -scale log -bin factor 4 \
-bin about 3900 4800 -zoom 0.5 -cmap b \
-bin filter 'energy=500:7000' \
-region 3098/primary/acisf03098_001N003_fov1.fits
Although the merged event file can be used for visualizing the data, it should not be used for data extraction (e.g. creating spectra) when knowledge of the telescope response is required.
Combining "identical" observations
We start by trying to combine the three observations that are "identical", namely 9407, 9774, and 9775. We start by using flux_obs to take the reprojected event files created by reproject_obs above, and create a combined exposure-corrected image:
unix% punlearn flux_obs unix% pset flux_obs bin=4 unix% flux_obs 'reproj/9407_reproj_evt.fits,reproj/97*_reproj_evt.fits' simple/ Running flux_obs Version: 17 October 2012 Verifying 3 observations. Using CSC ACIS broad science energy band. Calculating the output grid The merged images will have 774 by 825 pixels, a pixel size of 1.968 arcsec, and cover x=2020.5:5116.5:4, y=3196.5:6496.5:4. Creating the fluxed images for 3 observations in parallel. Creating aspect histograms for obsid 9775 Creating aspect histograms for obsid 9774 Creating aspect histograms for obsid 9407 Creating 5 instrument maps for obsid 9775 Creating 5 instrument maps for obsid 9407 Creating 5 instrument maps for obsid 9774 Creating 5 exposure maps for obsid 9775 Exposure map limits: 0.000000e+00, 4.661458e+06 Writing exposure map to simple/9775_5_broad.expmap Exposure map limits: 0.000000e+00, 3.929060e+06 Writing exposure map to simple/9775_3_broad.expmap Exposure map limits: 0.000000e+00, 4.524344e+06 Writing exposure map to simple/9775_6_broad.expmap Exposure map limits: 0.000000e+00, 3.939074e+06 Writing exposure map to simple/9775_2_broad.expmap Exposure map limits: 0.000000e+00, 5.937810e+06 Writing exposure map to simple/9775_7_broad.expmap Creating 5 exposure maps for obsid 9407 Exposure map limits: 0.000000e+00, 2.469366e+07 Writing exposure map to simple/9407_7_broad.expmap Exposure map limits: 0.000000e+00, 1.881372e+07 Writing exposure map to simple/9407_6_broad.expmap Exposure map limits: 0.000000e+00, 1.938741e+07 Writing exposure map to simple/9407_5_broad.expmap Exposure map limits: 0.000000e+00, 1.637222e+07 Writing exposure map to simple/9407_2_broad.expmap Exposure map limits: 0.000000e+00, 1.634182e+07 Writing exposure map to simple/9407_3_broad.expmap Exposure map limits: 0.000000e+00, 2.800297e+07 Writing exposure map to simple/9774_7_broad.expmap Exposure map limits: 0.000000e+00, 2.198827e+07 Writing exposure map to simple/9774_5_broad.expmap Creating 5 exposure maps for obsid 9774 Exposure map limits: 0.000000e+00, 1.852836e+07 Writing exposure map to simple/9774_3_broad.expmap Combining 5 exposure maps for obsid 9775 Combining 5 exposure maps for obsid 9407 Thresholding data for obsid 9775 Thresholding data for obsid 9407 Exposure-correcting image for obsid 9775 Exposure-correcting image for obsid 9407 Exposure map limits: 0.000000e+00, 1.857118e+07 Writing exposure map to simple/9774_2_broad.expmap Exposure map limits: 0.000000e+00, 2.133700e+07 Writing exposure map to simple/9774_6_broad.expmap Combining 5 exposure maps for obsid 9774 Thresholding data for obsid 9774 Exposure-correcting image for obsid 9774 Combining 3 observations. The following files were created: The co-added clipped counts image is: simple/broad_thresh.img The co-added clipped exposure map is: simple/broad_thresh.expmap The co-added exposure-corrected image is: simple/broad_flux.img
and compare it to the merge_all results (which we make match the positioning and size of the flux_obs output, and for which we decide to use the bad-pixel table from the 9774 observation, since it is the longest one):
unix% cat simple_all.evt2 9407/repro/acisf09407_repro_evt2.fits[energy=500:7000] 9774/repro/acisf09774_repro_evt2.fits[energy=500:7000] 9775/repro/acisf09775_repro_evt2.fits[energy=500:7000] unix% cat simple_all.asol1 9407/repro/pcadf313081138N002_asol1.fits 9774/repro/pcadf313307273N002_asol1.fits 9775/repro/pcadf313532704N002_asol1.fits unix% acis_set_ardlib 9774/repro/acisf09774_repro_bpix1.fits Updated ardlib parameter file: /data/param/ardlib.par AXAF_ACIS0_BADPIX_FILE -> CALDB AXAF_ACIS1_BADPIX_FILE -> CALDB AXAF_ACIS2_BADPIX_FILE -> /data/9774/repro/acisf09774_repro_bpix1.fits[BADPIX2] AXAF_ACIS3_BADPIX_FILE -> /data/9774/repro/acisf09774_repro_bpix1.fits[BADPIX3] AXAF_ACIS4_BADPIX_FILE -> CALDB AXAF_ACIS5_BADPIX_FILE -> /data/9774/repro/acisf09774_repro_bpix1.fits[BADPIX5] AXAF_ACIS6_BADPIX_FILE -> /data/9774/repro/acisf09774_repro_bpix1.fits[BADPIX6] AXAF_ACIS7_BADPIX_FILE -> /data/9774/repro/acisf09774_repro_bpix1.fits[BADPIX7] AXAF_ACIS8_BADPIX_FILE -> CALDB AXAF_ACIS9_BADPIX_FILE -> CALDB unix% get_sky_limits simple/broad_flux.img verbose=0 unix% pget get_sky_limits xygrid 2020.5:5116.5:#774,3196.5:6496.5:#825 unix% mkdir simple_all unix% punlearn merge_all unix% pset merge_all chip=2,3,5,6,7 energy=2.3 unix% pset merge_all refcoord=reproj/merged_evt.fits unix% pset merge_all xygrid=2020.5:5116.5:#774,3196.5:6496.5:#825 unix% pset merge_all merged=simple_all/merged_event.fits unix% pset merge_all expmap=simple_all/broad.expmap unix% pset merge_all expcorr=simple_all/broad_flux.img unix% merge_all @simple_all.evt2 @simple_all.asol1 dtffile= mode=h Reprojecting 9407/repro/acisf09407_repro_evt2.fits to reproj/merged_evt.fits Reprojecting 9774/repro/acisf09774_repro_evt2.fits to reproj/merged_evt.fits Reprojecting 9775/repro/acisf09775_repro_evt2.fits to reproj/merged_evt.fits Merging data... SIM_Z of event file: -190.14006604987 SIM_Z of 9407/repro/pcadf313081138N002_asol1.fits: -190.14006604987 Difference is 0 mm: SIM_Z differs by less than ~1/4 pixel. SIM_Z of 9774/repro/pcadf313307273N002_asol1.fits: -190.14006604987 Difference is 0 mm: SIM_Z differs by less than ~1/4 pixel. SIM_Z of 9775/repro/pcadf313532704N002_asol1.fits: -190.14006604987 Difference is 0 mm: SIM_Z differs by less than ~1/4 pixel. Creating aspect histogram for CCD 2... Creating aspect histogram for CCD 3... Creating aspect histogram for CCD 5... Creating aspect histogram for CCD 6... Creating aspect histogram for CCD 7... Making instrument map for CCD 2... Making instrument map for CCD 3... Making instrument map for CCD 5... Making instrument map for CCD 6... Making instrument map for CCD 7... Making exposure map for CCD 2... Exposure map limits: 0.000000e+00, 4.044487e+07 Making exposure map for CCD 3... Exposure map limits: 0.000000e+00, 4.028789e+07 Making exposure map for CCD 5... Exposure map limits: 0.000000e+00, 4.601305e+07 Making exposure map for CCD 6... Exposure map limits: 0.000000e+00, 4.613702e+07 Making exposure map for CCD 7... Exposure map limits: 0.000000e+00, 5.862443e+07 Merging exposure maps Creating image Normalizing image All done!
As a reminder, the exposure-corrected image was created by:
-
creating an image and exposure map for each observation, summing the values, then dividing them (flux_obs and merge_obs);
-
or merging the event and aspect-solution files, using the combined dataset to create a single image and exposure map (well, one per ccd), and then dividing them (merge_all).
Below we compare the exposure-corrected images created by both approaches, after applying light smoothing, logarithmic scaling, and using the same display range:
![[Other than bright pixels at the edge of the merge_all version, the images look the same]](compare_simple.png)
Figure 2: The bright pixels at the edges of the chips in the merge_all output are where the exposure varies strongly within a pixel; they do not show up in the flux_obs output since they have been filtered out by the expmapthresh parameter. Other than this difference, the two images look identical, but there are some systematic differences, as shown in Figure 3.
We can also look at the percentage difference between the two, calculated as 100 * (flux_obs - merge_all) / flux_obs so that a positive value means that the flux_obs flux is larger than the merge_all one:
![[Overall the results are similar, but there are systematic differences]](difference_simple.png)
Figure 3: The results look similar, for instance most of ACIS-S3, which contains the cluster, is close to no difference, but you can see some systematic features, including:
-
the front-illuminated chips have a ~ 3% larger value in the flux_obs version, since this script includes the reduction in the ACIS-FI effective area due to cosmic rays whereas merge_all does not (note that the back-illuminated chips are not affected by this, which is why ACIS-S3 and ACIS-S1 are close to no difference);
-
and bad pixel and column differences, which appear as small squares or faint columns, since flux_obs can use the per-observation bad-pixel files but merge_all only uses the list from one observation (in this case we chose 9774).
If acis_set_ardlib (or manually setting the ardlib parameter file) is not used before calling merge_all, then the bad-pixel table used by merge_all will be taken from the CALDB, and the difference in results is shown in Figure 4:
![[The major difference now is the difference in the bad columns]](difference_simple.caldb.png)
Figure 4: The differences to Figure 3 are due to the fact that the CALDB bad-pixel table does not match the per-observation one. The most obvious difference is the bad columns, where there are more in the per-observation table, leading to an increased flux in the flux_obs version, but there are also individual bad pixels missing in the CALDB version, which give rise to the small squares that can be seen on all chips.
Combining similar observations
Here we extend the analysis to include the 9408 observation, which was taken only three days after 9775 but has a slightly-different SIM_Z value.
unix% punlearn flux_obs
unix% flux_obs 'reproj/9*_evt.fits' simz/ bin=4
Running flux_obs
Version: 17 October 2012
Verifying 4 observations.
Using CSC ACIS broad science energy band.
Calculating the output grid
The merged images will have 786 by 826 pixels, a pixel size of 1.968 arcsec,
and cover x=2012.5:5156.5:4, y=3192.5:6496.5:4.
Creating the fluxed images for 4 observations in parallel.
Creating aspect histograms for obsid 9408
... a lot more of the same ...
Exposure-correcting image for obsid 9775
Combining 4 observations.
The following files were created:
The co-added clipped counts image is:
simz/broad_thresh.img
The co-added clipped exposure map is:
simz/broad_thresh.expmap
The co-added exposure-corrected image is:
simz/broad_flux.img
and (again using the bad-pixel table from ObsId 9774):
unix% cat simz_all.evt2 9407/repro/acisf09407_repro_evt2.fits[energy=500:7000] 9774/repro/acisf09774_repro_evt2.fits[energy=500:7000] 9775/repro/acisf09775_repro_evt2.fits[energy=500:7000] 9408/repro/acisf09408_repro_evt2.fits[energy=500:7000] unix% cat simz_all.asol1 9407/repro/pcadf313081138N002_asol1.fits 9774/repro/pcadf313307273N002_asol1.fits 9775/repro/pcadf313532704N002_asol1.fits 9408/repro/pcadf313765973N002_asol1.fits unix% acis_set_ardlib 9774/repro/acisf09774_repro_bpix1.fits Updated ardlib parameter file: /data/param/ardlib.par AXAF_ACIS0_BADPIX_FILE -> CALDB AXAF_ACIS1_BADPIX_FILE -> CALDB AXAF_ACIS2_BADPIX_FILE -> /data/9774/repro/acisf09774_repro_bpix1.fits[BADPIX2] AXAF_ACIS3_BADPIX_FILE -> /data/9774/repro/acisf09774_repro_bpix1.fits[BADPIX3] AXAF_ACIS4_BADPIX_FILE -> CALDB AXAF_ACIS5_BADPIX_FILE -> /data/9774/repro/acisf09774_repro_bpix1.fits[BADPIX5] AXAF_ACIS6_BADPIX_FILE -> /data/9774/repro/acisf09774_repro_bpix1.fits[BADPIX6] AXAF_ACIS7_BADPIX_FILE -> /data/9774/repro/acisf09774_repro_bpix1.fits[BADPIX7] AXAF_ACIS8_BADPIX_FILE -> CALDB AXAF_ACIS9_BADPIX_FILE -> CALDB unix% get_sky_limits simz/broad_flux.img verbose=0 unix% pget get_sky_limits xygrid 2012.5:5156.5:#786,3192.5:6496.5:#826 unix% mkdir simz_all unix% punlearn merge_all unix% pset merge_all chip=2,3,5,6,7 energy=2.3 unix% pset merge_all refcoord=reproj/merged_evt.fits unix% pset merge_all xygrid=2012.5:5156.5:#786,3192.5:6496.5:#826 unix% pset merge_all merged=simz_all/merged_event.fits unix% pset merge_all expmap=simz_all/broad.expmap unix% pset merge_all expcorr=simz_all/broad_flux.img unix% merge_all @simz_all.evt2 @simz_all.asol1 dtffile= mode=h Reprojecting 9407/repro/acisf09407_repro_evt2.fits to reproj/merged_evt.fits Reprojecting 9774/repro/acisf09774_repro_evt2.fits to reproj/merged_evt.fits Reprojecting 9775/repro/acisf09775_repro_evt2.fits to reproj/merged_evt.fits Reprojecting 9408/repro/acisf09408_repro_evt2.fits to reproj/merged_evt.fits Merging data... SIM_Z of event file: -190.14006604987 SIM_Z of 9407/repro/pcadf313081138N002_asol1.fits: -190.14006604987 Difference is 0 mm: SIM_Z differs by less than ~1/4 pixel. SIM_Z of 9774/repro/pcadf313307273N002_asol1.fits: -190.14006604987 Difference is 0 mm: SIM_Z differs by less than ~1/4 pixel. SIM_Z of 9775/repro/pcadf313532704N002_asol1.fits: -190.14006604987 Difference is 0 mm: SIM_Z differs by less than ~1/4 pixel. SIM_Z of 9408/repro/pcadf313765973N002_asol1.fits: -190.14258036517 Difference is -0.00251431530000445 mm: SIM_Z differs by less than ~1/4 pixel. Creating aspect histogram for CCD 2... # asphist (CIAO 4.4): ERROR: *_PNT keywords are required in file 'simz_all/merged_event.fits[ccd_id=2]'. Will assume PNT == NOM which may produce bad results for reprojected datasets and multi-obi obsids. Creating aspect histogram for CCD 3... # asphist (CIAO 4.4): ERROR: *_PNT keywords are required in file 'simz_all/merged_event.fits[ccd_id=3]'. Will assume PNT == NOM which may produce bad results for reprojected datasets and multi-obi obsids. Creating aspect histogram for CCD 5... # asphist (CIAO 4.4): ERROR: *_PNT keywords are required in file 'simz_all/merged_event.fits[ccd_id=5]'. Will assume PNT == NOM which may produce bad results for reprojected datasets and multi-obi obsids. Creating aspect histogram for CCD 6... # asphist (CIAO 4.4): ERROR: *_PNT keywords are required in file 'simz_all/merged_event.fits[ccd_id=6]'. Will assume PNT == NOM which may produce bad results for reprojected datasets and multi-obi obsids. Creating aspect histogram for CCD 7... # asphist (CIAO 4.4): ERROR: *_PNT keywords are required in file 'simz_all/merged_event.fits[ccd_id=7]'. Will assume PNT == NOM which may produce bad results for reprojected datasets and multi-obi obsids. Making instrument map for CCD 2... Making instrument map for CCD 3... Making instrument map for CCD 5... Making instrument map for CCD 6... Making instrument map for CCD 7... Making exposure map for CCD 2... Exposure map limits: 0.000000e+00, 5.058647e+07 Making exposure map for CCD 3... Exposure map limits: 0.000000e+00, 5.043441e+07 Making exposure map for CCD 5... Exposure map limits: 0.000000e+00, 5.753366e+07 Making exposure map for CCD 6... Exposure map limits: 0.000000e+00, 5.775658e+07 Making exposure map for CCD 7... Exposure map limits: 0.000000e+00, 7.338729e+07 Merging exposure maps Creating image Normalizing image All done!
which produces Figures 5 and 6 below.
![[This looks similar to Figure 2]](compare_simz.png)
Figure 5: The image is very similar to Figure 2.
![[There are differences to Figure 3 at the chip edges]](difference_simz.png)
Figure 6: Overall this looks similar to Figure 3, as we would expect, but you can see some small, systematic, differences at the edges of the chips, caused by the small difference in SIM position and roll angle between ObsId 9408 and the other observations, as noted in the screen output created by merge_all.
So, when combining data which is "similar" - in that it was taken close enough together in time that the same calibration products can be used and the instrument details are very similar - e.g. the SIM position - then you can combine data before creating the response products, as merge_all does, with only small-scale, but systematic, differences. How much of an issue this is depends on your science goals.
Combining all the observations
Now we add in the 3098 observation, which we expect to cause problems for merge_all since it is significantly different to the other observations. In Figure 7 we show the merged event file, focusing just on ACIS-S3, which clearly shows the sub-array used in the first observation:
![[The merged observations for 3C186, concentrating on just the ACIS-S3 chip]](merged_event7.png)
unix% ds9 reproj/merged_evt.fits -smooth -scale log -bin factor 4 \
-bin about 4100 4000 -cmap b \
-bin filter 'energy=500:7000,ccd_id=7'
unix% punlearn flux_obs
unix% flux_obs 'reproj/*_reproj_evt.fits' combined/ bin=4
Running flux_obs
Version: 17 October 2012
Verifying 5 observations.
Using CSC ACIS broad science energy band.
Calculating the output grid
The merged images will have 786 by 826 pixels, a pixel size of 1.968 arcsec,
and cover x=2012.5:5156.5:4, y=3192.5:6496.5:4.
Creating the fluxed images for 5 observations in parallel.
Creating aspect histograms for obsid 9408
... more output ...
Exposure-correcting image for obsid 9774
Combining 5 observations.
The following files were created:
The co-added clipped counts image is:
combined/broad_thresh.img
The co-added clipped exposure map is:
combined/broad_thresh.expmap
The co-added exposure-corrected image is:
combined/broad_flux.img
and
unix% cat combined_all.evt2 3098/repro/acisf03098_repro_evt2.fits[energy=500:7000] 9407/repro/acisf09407_repro_evt2.fits[energy=500:7000] 9774/repro/acisf09774_repro_evt2.fits[energy=500:7000] 9775/repro/acisf09775_repro_evt2.fits[energy=500:7000] 9408/repro/acisf09408_repro_evt2.fits[energy=500:7000] unix% cat combined_all.asol1 3098/repro/pcadf137943492N003_asol1.fits 9407/repro/pcadf313081138N002_asol1.fits 9774/repro/pcadf313307273N002_asol1.fits 9775/repro/pcadf313532704N002_asol1.fits 9408/repro/pcadf313765973N002_asol1.fits unix% acis_set_ardlib 9774/repro/acisf09774_repro_bpix1.fits Updated ardlib parameter file: /data/param/ardlib.par AXAF_ACIS0_BADPIX_FILE -> CALDB AXAF_ACIS1_BADPIX_FILE -> CALDB AXAF_ACIS2_BADPIX_FILE -> /data/9774/repro/acisf09774_repro_bpix1.fits[BADPIX2] AXAF_ACIS3_BADPIX_FILE -> /data/9774/repro/acisf09774_repro_bpix1.fits[BADPIX3] AXAF_ACIS4_BADPIX_FILE -> CALDB AXAF_ACIS5_BADPIX_FILE -> /data/9774/repro/acisf09774_repro_bpix1.fits[BADPIX5] AXAF_ACIS6_BADPIX_FILE -> /data/9774/repro/acisf09774_repro_bpix1.fits[BADPIX6] AXAF_ACIS7_BADPIX_FILE -> /data/9774/repro/acisf09774_repro_bpix1.fits[BADPIX7] AXAF_ACIS8_BADPIX_FILE -> CALDB AXAF_ACIS9_BADPIX_FILE -> CALDB unix% get_sky_limits combined/broad_flux.img verbose=0 unix% pget get_sky_limits xygrid 2012.5:5156.5:#786,3192.5:6496.5:#826 unix% mkdir combined_all unix% punlearn merge_all unix% pset merge_all chip=2,3,5,6,7 energy=2.3 unix% pset merge_all refcoord=reproj/merged_evt.fits unix% pset merge_all xygrid=2012.5:5156.5:#786,3192.5:6496.5:#826 unix% pset merge_all merged=combined_all/merged_event.fits unix% pset merge_all expmap=combined_all/broad.expmap unix% pset merge_all expcorr=combined_all/broad_flux.img unix% merge_all @combined_all.evt2 @combined_all.asol1 dtffile= mode=h Reprojecting 3098/repro/acisf03098_repro_evt2.fits to reproj/merged_evt.fits Reprojecting 9407/repro/acisf09407_repro_evt2.fits to reproj/merged_evt.fits Reprojecting 9774/repro/acisf09774_repro_evt2.fits to reproj/merged_evt.fits Reprojecting 9775/repro/acisf09775_repro_evt2.fits to reproj/merged_evt.fits Reprojecting 9408/repro/acisf09408_repro_evt2.fits to reproj/merged_evt.fits Merging data... SIM_Z of event file: -190.14258036517 SIM_Z of 3098/repro/pcadf137943492N003_asol1.fits: -190.14258036517 Difference is 0 mm: SIM_Z differs by less than ~1/4 pixel. SIM_Z of 9407/repro/pcadf313081138N002_asol1.fits: -190.14006604987 Difference is 0.00251431530000445 mm: SIM_Z differs by less than ~1/4 pixel. SIM_Z of 9774/repro/pcadf313307273N002_asol1.fits: -190.14006604987 Difference is 0.00251431530000445 mm: SIM_Z differs by less than ~1/4 pixel. SIM_Z of 9775/repro/pcadf313532704N002_asol1.fits: -190.14006604987 Difference is 0.00251431530000445 mm: SIM_Z differs by less than ~1/4 pixel. SIM_Z of 9408/repro/pcadf313765973N002_asol1.fits: -190.14258036517 Difference is 0 mm: SIM_Z differs by less than ~1/4 pixel. Creating aspect histogram for CCD 2... # asphist (CIAO 4.4): ERROR: *_PNT keywords are required in file 'combined_all/merged_event.fits[ccd_id=2]'. Will assume PNT == NOM which may produce bad results for reprojected datasets and multi-obi obsids. # asphist (CIAO 4.4): WARNING: '3098/repro/pcadf137943492N003_asol1.fits' has no records fall in event gti's. Creating aspect histogram for CCD 3... # asphist (CIAO 4.4): ERROR: *_PNT keywords are required in file 'combined_all/merged_event.fits[ccd_id=3]'. Will assume PNT == NOM which may produce bad results for reprojected datasets and multi-obi obsids. # asphist (CIAO 4.4): WARNING: '3098/repro/pcadf137943492N003_asol1.fits' has no records fall in event gti's. Creating aspect histogram for CCD 5... # asphist (CIAO 4.4): ERROR: *_PNT keywords are required in file 'combined_all/merged_event.fits[ccd_id=5]'. Will assume PNT == NOM which may produce bad results for reprojected datasets and multi-obi obsids. # asphist (CIAO 4.4): WARNING: '3098/repro/pcadf137943492N003_asol1.fits' has no records fall in event gti's. Creating aspect histogram for CCD 6... # asphist (CIAO 4.4): ERROR: *_PNT keywords are required in file 'combined_all/merged_event.fits[ccd_id=6]'. Will assume PNT == NOM which may produce bad results for reprojected datasets and multi-obi obsids. # asphist (CIAO 4.4): WARNING: '3098/repro/pcadf137943492N003_asol1.fits' has no records fall in event gti's. Creating aspect histogram for CCD 7... # asphist (CIAO 4.4): ERROR: *_PNT keywords are required in file 'combined_all/merged_event.fits[ccd_id=7]'. Will assume PNT == NOM which may produce bad results for reprojected datasets and multi-obi obsids. Making instrument map for CCD 2... Making instrument map for CCD 3... Making instrument map for CCD 5... Making instrument map for CCD 6... Making instrument map for CCD 7... Making exposure map for CCD 2... Exposure map limits: 0.000000e+00, 4.700833e+07 Making exposure map for CCD 3... Exposure map limits: 0.000000e+00, 4.688738e+07 Making exposure map for CCD 5... Exposure map limits: 0.000000e+00, 5.334176e+07 Making exposure map for CCD 6... Exposure map limits: 0.000000e+00, 5.343070e+07 Making exposure map for CCD 7... Exposure map limits: 0.000000e+00, 8.069480e+07 Merging exposure maps Creating image Normalizing image All done!
![[Significant differences are now visible]](compare_combined.png)
Figure 8: Unlike previous versions (e.g. Figure 2 and 5), the merge_all version shows significant differences to the flux_obs one, namely:
-
a spatial offset which makes the exposure map not match up to the chip boundaries, causing the obvious banding between chips and at the edge of chips;
-
the exposure map for 3098 includes the whole chip, rather than just the sub-array, in the merge_all version, which causes the area around the cluster to have an artificially lowered flux (in this figure this is most noticeable as the change in the brighness of the chip boundary in the ACIS-S2 and ACIS-S3 chips to the East of the cluster).
![[There are now significant differences]](difference_combined.png)
Figure 9: Here we see the problems discussed in Figure 8 - the spatial shift and the drastic change in the flux close to the cluster because of ObsId 3098 - but you can also see that there is an overall difference in the normalisation of the merge_all result, resulting in it producing fluxes 5 to 10 percent less than merge_obs. This is because the exposure maps created by merge_all use the response information for the first observation (3098) for the whole data set, and so does not include the reduced effective area due to the ACIS contamination layer for the 2007 observations, which form the bulk of the observing time.
This example shows some of the problems that can occur when trying to create response products - in this case the instrument and exposure maps - from a combined dataset (as was done by merge_all). The responses should be created per observation and then - if desired and possible - combined, as done by merge_obs and flux_obs.