Reprocessing III of Chandra Data

What is Reprocessing III?

Reprocessing III - or "Repro III" for short - is a complete reprocessing of all observations in the Chandra Data Archive. It began with data from January 2005 and will run through January 2006; then it will go back to December 2004 and run backwards to the start of the mission. The quality of the reprocessed data will be improved over the current data in the archive due to updates in software and calibration.

In addition to improving the data products, Repro III makes it possible to apply other, non-processing CALDB changes directly to archived data. For instance, the new ACIS gain files make it possible to create -120 C reponses with mkacisrmf in place of mkrmf; see the Creating ACIS RMFs with mkacisrmf thread. More users will be able to do analysis on archive data "out of the box" without having to reprocess for calibration purposes.

The Summary of Changes for Reprocessing III of Chandra Data contains additional information.

Should I redo my analysis with reprocessed data from the Archive?

Most users will not need to redo their analysis. If you have followed the CIAO Science threads and reprocessed your data as they advise, you have a similar data product to what Repro III will create since it uses calibration and software updates that are currently available via the CIAO software.

To see what specifically has changed since the last time your data was processed, check the ASCDSVER header keyword in your data file. The analysis tools modify the ASCDSVER to reflect the CIAO version, so you will either see the standard data processing version or the CIAO version used to create the file.

DS version:

unix% dmkeypar acisf03838N001_evt2.fits ASCDSVER echo+
6.12.0

Compare this version to the information in the Software and CALDB changes in Repro III section of this page to see what improvements have been made.

CIAO version:

unix%  dmkeypar acis_459_new_evt2.fits ASCDSVER echo+
CIAO3.3

Users who are working with CIAO 3.3 or higher should not have to redo the analysis. The recommended minimum version of the CALDB is 3.2.1 for ACIS data and 3.2.0 for HRC data. (Note that the CALDBVER header keyword is not updated by the CIAO tools. Even if you have reprocessed your data manually in CIAO to apply new calibration, the value always reflects the version of the CALDB applied in SDP.)

If the CIAO version is lower than 3.3, Repro III has significant improvements to the data processing.

How will I know when an observation has gone through Repro III?

Repro III started with DS 7.6.7; if the release version for your ObsID is at least 7.6.7, the data have been reprocessed.

Principal Investigators will receive email notification of data processed within one year of the public release date of the observation.

Anyone may use the Processing Status webpage to see if a particular file has been reprocessed. Enter the ObsId and choose "Long Report", then click the "Go" button. The search results will include a "Release" version in the summary at the beginning of the page.

All public data are available through ChaSeR or WebChaSeR.

Software and CALDB changes in Repro III

• ACIS
• HRC-I
• HRC-S
• Aspect Solution
• PIXLIB
• Miscellaneous

Here is a summary of calibration and software changes that are in Repro III; this list includes any significant updates to SDP since Reprocessing II was completed. The list is organized by detector/data product; issues that refer to more than one category are repeated in all relevant sections. A link to the appropriate CALDB release notes is included for all calibration file changes; refer to those release notes for technical details.

ACIS

The CTI and gain improvements allow the ACIS energy scale to be accurate to within 0.3% over most of the area of the chips.

Bad pixel map

A correction has been made to change the STATUS bits on the node boundary columns for all ten chips. This will reduce the number of events excluded due to proximity to the node boundary, because these will be treated differently than ordinary bad pixels. [CALDB 3.2.0]

Time-dependent gain corrections to the PHA

New TGAIN calibration files contain higher-spatial density corrections for all ACIS chips. [CALDB 3.2.0]

The gain for the back-illuminated chips - ACIS-S1 and S3 - has been upgraded. The old S1 gains in particular are very bad for order sorting; significant improvement is seen with the new file. [CALDB 3.2.1]

CTI corrections for front-illuminated ACIS chips

New calibration files to correct for the charge transfer inefficiency in front-illuminated ACIS chips. [CALDB 3.2.0]

CTI-corrected GAIN file for use with mkacisrmf

This file contains the appropriate gain adjustments for the Phase 2 Responses to be generated by mkacisrmf, for the chips I0, I1, I2, I3, S1, S2, and S3. The gain modifications to the FI chips and S3 are moderate. There is an improvement in the gain function near the Si K-edge. [CALDB 3.0.0]

ACIS-S1 gain function

Calibration for the ACIS-S1 (ccd_id=5) back-illuminated chip has been greatly improved, which benefits ACIS-S/grating observations. The gain modifications to the front-illuminated chips are slight. [CALDB 3.0.0]

ACIS-S2 gain function

The detector gain has been updated for the S2 chip (ccd_id=6) only. Changes with respect to CALDB 3.1.0 vary from -3.2% to +0.5% with the largest shifts at the lowest PHA values. In general changes are about -0.5% for most PHAs. [CALDB 3.2.0]

Destreak

The destreak tool is updated to fix problems using all events instead of only using good events. This only affects data taken on the ACIS S4 chip (ccd_id=8). [ASCDSVER 7.6.4]

Continuous-clocking Mode

For ACIS-CC-mode, the CTI correction is applied and the split-threshold file is correctly utilized. These affect the PHA, PI, and ENERGY values in an event file. [ASCDSVER 7.6.3 and 7.6.4]

File headers

The headers in ACIS and HRC event files are upgraded to reference other data products, including CALDB files (e.g., ASOLFILE records the aspect solution, THRFILE records the ACIS split threshold). [ASCDSVER 7.6.3]

Additional minor changes

There were some additional small changes affecting both ACIS and HRC data, including a bug fix in the axis of rotation in the aspect solution and updating a GTI limit for the HRC Next-In-Line time gap. [ASCDSVER 7.6.4]

HRC-I

Gap removal calibration

The new data are from better flat-field data analysis and include the tap-ringing correction algorithm. This will reduce the visible spread of on-axis point sources to essentially within about 1.2 arc seconds. [CALDB 2.2]

Hyperbolic test (FPTEST) coefficients file

The event hyperbolic test is the key method for removing "ghost" images which result when certain event positions are incorrectly determined in the HRC electronics. [CALDB 2.4]

Bad pixel map

A new hot pixel appeared near the south-west edge of the HRC-I. It first appeared in OBSID 1964 and it persists. A new row of data has been entered into the bad pixel file, including the new pixel location and its effective date and time. [CALDB 2.13]

Gain file

For the first time, gain correction maps are available for HRC-I data for use in hardness ratio estimates. For more information, refer to the Gain Response of the HRC webpage. [CALDB 3.2.0]

File headers

The headers in ACIS and HRC event files are upgraded to reference other data products, including CALDB files (e.g., ASOLFILE records the aspect solution, THRFILE records the ACIS split threshold). [ASCDSVER 7.6.3]

Additional minor changes

There were some additional small changes affecting both ACIS and HRC data, including a bug fix in the axis of rotation in the aspect solution and updating a GTI limit for the HRC Next-In-Line time gap. [ASCDSVER 7.6.4]

HRC-S

LETG/HRC-S PI region filter file

By filtering on the pulse-height value PI as a function of dispersed-photon wavelength, the background rate in the HRC-S, when used in conjunction with the LETG, can be reduced by a factor of two to five (depending on wavelength and the level of filtering); see the LETG/HRC-S Background Filtering with CIAO calibration webpage for details. [CALDB 2.3]

Hyperbolic test (FPTEST) coefficients file

The event hyperbolic test is the key method for removing "ghost" images which result when certain event positions are incorrectly determined in the HRC electronics. [CALDB 2.4]

Bad pixel map

The new file has a time-dependent addition for a rectangular hot spot on plate 1 of HRC-S, which became enlarged. [CALDB 2.28]

Gap removal calibration

A first iteration of corrections for the known non-linearity in the HRC-S/LETG spatial wavelength scale is now available. It will improve the performance of the LETGS, but does not account fully for all the systematic errors in the dispersion relation. The RMS deviation of the wavelength differences drop from 0.014 A (0.010 A over just the central plate) prior to the correction, to 0.010 A (0.006 A over the central plate). [CALDB 3.2.0]

File headers

The headers in ACIS and HRC event files are upgraded to reference other data products, including CALDB files (e.g., ASOLFILE records the aspect solution, THRFILE records the ACIS split threshold). [ASCDSVER 7.6.3]

Additional minor changes

There were some additional small changes affecting both ACIS and HRC data, including a bug fix in the axis of rotation in the aspect solution and updating a GTI limit for the HRC Next-In-Line time gap. [ASCDSVER 7.6.4]

Aspect Solution

Astrometry corrections to within 0.2 arcseconds on axis

Corrections for the ACIS-S fid light positions to allow corrections in the ACIS-S astrometry at the level of one arcsecond or less. This is extreme fine-tuning of the ACIS-S sky positions, effective for the entire observational period of Chandra since launch. [CALDB 2.25]

AXAF Guide and Acquisition Star Catalog (AGASC) v1.6

This slightly improves aspect quality and absolute position accuracy; see the AGASC webpage for more information. [ASCDSVER 7.5.0.1]

VV aspect check limits

Minor changes in aspect quality limits (e.g., image centroiding error, offset) are applied in determining a bad guide star or fid light to exclude in calculating the aspect solution. [ASCDSVER 7.5.3]

Proper motion of a guide star

In calculating the aspect solution, the aspect pipeline correctly applies the proper motion of a guide star. This will slightly improve the accuracy of the aspect solution. Previously, a guide star with a large proper motion was excluded in calculating the aspect solution. [ASCDSVER 7.6]

PIXLIB

Off-axis astrometry for outlying ACIS chips

The new file includes a more accurate representation of the telescope system geometry, including separate Rowland spacings for the HETG and LETG, as well as the corrected focal length for the HRMA. [CALDB 2.9]

Improved chip corner positions to correct the ACIS-S wavelength/energy scale

The HETG calibration and science planning teams released corrected ACIS-S chip corner positions for chips ACIS-45689. (ACIS-7 is unchanged, as are all other corner positions.) These corrections eliminate wavelength errors of the order .002 Anstroms in the HETG and LETG spectra with ACIS-S. In addition, a small correction to the MEG grating period has been included, which makes the HEG and MEG spectra more mutually consistent. [CALDB 3.0.1]

Miscellaneous

Large file support

Added support for large files (> 2GB). Previously, when an event file was bigger than 2 GB (for long observations of very bright sources), pipeline software could not handle it and had to apply manual intervention either by removal of non-vital columns or by splitting into multiple smaller pieces. [ASCDSVER 7.6]