About Chandra Archive Proposer Instruments & Calibration Newsletters Data Analysis HelpDesk Calibration Database NASA Archives & Centers Chandra Science Links


Reprocessing of Chandra Data: Series III

The third reprocessing series consists of one large reprocessing project and a few smaller ones:
  1. Reprocessing III: Uniform calibration of almost all data in the archive
    This run consists of a complete reprocessing of all data obtained between 29 January 2000 and 15 November 2005. The aim of this run was to provide a complete set of archived data, processed with a uniform calibration database from the moment the frontend temperature went to -120° C, in January 2000. Details of the changes may be found below. This bulk of this processing run was performed between February 2006 and June 2007.

  2. Reprocessing IIIa: Aspect correction for early 2007
    During the August 2007 we reprocessed observations made between December 2006 and May 2007, using an updated aspect calibration. That correction is only needed for this particular set of observations.
    More specifically: It affects all data obtained after 2006-12-10 and processed on or before 2007-05-22. These data were subject to a systematic aspect offset of approximately 0.4 arcsec, believed to be the result of a changed thermal environment following the procedure to cool the ACA CCD from -15° C to -19° C that took place between late Nov-2006 and early Jan-2007. This error has been corrected in the newly processed version of the data.
    More information is available at this location .

  3. Reprocessing IIIb: Override Obspar
    Details
    Under construction

  4. Reprocessing IIIc: Calibration update of all HRC-I observations since 29 January 2000
    Details
    The following concerns HRC-I observations only and is of interest to you if one or more of the following apply:
    • you have performed hardness ratio or quantile color-color analysis
    • your analysis was particularly sensitive to the Enclosed Count Fraction (ECF)
    • your sources were close to on-axis in observations witha large SIM-z offset


1. Reprocessing III: Uniform calibration of almost all data in the archive

Summary of Changes for Reprocessing III of Chandra Data

The main component of the third reprocessing run of Chandra data was started in February 2006 and covers most of the mission. It began with data from January 2005 and ran through November 2005; then it went back to December 2004 and is running backwards to the moment when the frontend temperature went down to -120° C, in January 2000; the very early part of the mission may be reprocessed later. The quality of the reprocessed data will be better because of improvements in software and, especially, calibration since the current default data in the archive were created.

Whether the new data version would affect any scientific conclusion drawn on the basis of the older versions depends on many factors: the nature of the conclusions, the type of analysis, the signal-to-noise ratio, etc. Ultimately, the user may need to make a comparison in order to judge whether the changes are significant for her or his purposes, but to aid in that assessment we have summarized in the following tables the most significant CalDB and software changes. There are three keywords in the headers of all CXC FITS data products that indicate what version those products were processed with:

  • DATE: The date and time of the file's creation (in UTC)
  • ASCDSVER: The CXC-DS software release version used
  • CALDBVER: The CalDB version used for creating the data product
The values of these keywords should be looked up in older versions of data products and used when consulting the tables below in order to determine whether reprocessed products may differ significantly.

Note also, that the new products will be fully compatible with the current version of CIAO, something that is not necessarily the case for older data versions.


Major CalDB Changes

 The following is a short summary of Chandra calibration (CalDB) upgrades that have affected various aspects of pipeline processing since the completion of the previous (second) round of CXCDS archival reprocessing. The specific relevant portions of the CalDB are ACIS, HRC, PIXLIB (coordinate library), and PCAD (aspect).


ACIS

The first four improvements allow the acis energy scale to be accurate to within 0.3% over most of the area of the chips.
Date CalDB version Component Description
2005-12-15T17:30:00 3.2.1 BADPIX -120° C Version N0003: Status bits changed on node boundary chips so they are not counted as normal bad columns. Hence adjacent columns will not be automatically included in the *bpix1.fits primary data files.
2005-11-16T16:00:00 3.2.0 CTI Version N0005 Epochs 1-22 (to May 2005)
Due to the CTI time-dependence added in CTI N0005 files below, T-GAIN is now approximately re-zeroed at the start of Epoch 11, which is August 1, 2002, for the FI chips ONLY!
BI chips T_GAIN corrections have not yet been affected by CTI correction, because no CTI correction has been released for them.
Epochs 21 and 22 are added; however all files have been changed to include a higher-density spatial grid, which eliminates a calibration problem, to wit a gain-sag most notably at high energies near the central 50 columns of each chip.
2005-11-16T16:00:00 3.2.0 DETGAIN CTI files version N0005, time-dependence added to FI chips to account for decay in their resolution with time. Two files have been included, for T_GAIN epochs 1-10 and 11-current. Only the S2 data for epochs 1-10 are updated with a better model. (BI chip CTI corrections are not yet included.)
2005-11-15T15:30:00 3.2.0 BADPIX An improved bad pixel map
2005-06-23T15:17:00 3.1.0 TGAIN Time-dependent gain corrections to the PHA in Level 1
2005-06-23T15:17:00 3.1.0 CTI Phase 2 response gains for Chips 4, 8, and 9 (S0,4,5), making the set complete for use with mkacisrmf.
2004-12-14T18:00:00 3.0.0 DETGAIN S1-3, I0-3 gain function modifications (improved gain model) near the Si-K edge
2004-12-14T18:00:00 3.0.0 DETGAIN S1 gain function improvements for imaging and grating (order-sorting) observations


HRC

Date CalDB version Component Description
2005-11-16T16:00:00 3.2.0 GAINMAP Gain maps included in processing for HRC-I for the first time, version N0001. Second file applies to all data after HV was reduced in the HRC-I, on 1999-10-04.
2005-11-16T16:00:00 3.2.0 DEGAP Improved gap removal calibrations for HRC-S. For a complete description, see this page.
GAPLOOKUP version N0002: This file is the first iteration of corrections for the known non-linearity in the HRC-S/LETG spatial wavelength scale that has been frequently alluded to in the literature.
2004-08-11T19:30:00 2.28 BADPIX Upgrades to the bad pixel map for HRC-S.
This new file has a time-dependent addition for a rectangular hot spot on plate 1 of HRC-S, which recently became enlarged.
2002-04-04T15:05:00 2.13 BADPIX Upgrades to the bad pixel map for HRC-I
2002-11-08T00:00:00 2.4 EVENT FILTER Upgrades to event tests to eliminate ghost images and some background events, such as event flatness, tapringing, amplifier saturation, and f-p test
2001-03-07T14:50:00 2.3 EVENT FILTER HRC-S GAIN and PI region filters for background elimination
2001-01-25T18:00:00 2.2 DEGAP Improved gap removal calibrations for HRC-I


PCAD

Date CalDB version Component Description
2003-12-30T14:00:00 2.25 ASTROMETRY Astrometry corrections to within 0.2 arc seconds on axis, in particular for ACIS


PIXLIB

Date CalDB version Component Description
2005-02-15T16:45:00 3.0.1 AIMPOINTS Version N0002, HRC-S imaging default aimpoint added.
2005-02-15T16:45:00 3.0.1 GEOM Improved chip corner positions to correct the HETG/ACIS-S wavelength/energy scale to within prelaunch specifications.
Version N0005 includes ACIS-S chip corner positions modifications for chips 4, 5, 6, 8, and 9. (ACIS-I0-3 and ACIS-7 are not changed.) These corrections, while having negligible affect on the imaging astrometry for ACIS-S, eliminate wavelength errors of the order .002 Anstroms in the HETG and LETG spectra with ACIS-S. Further an adjustment to the MEG period has been included, which makes the HEG and MEG spectra more mutually consistent.
2001-10-22T16:00:00 2.9 GEOM Off-axis astrometry improvements for outlying ACIS chips. spectra more mutually consistent.


Other areas of the CalDB have been significantly upgraded since the saecond reprocessing run, e.g., the response input data for ACIS (CalDB 3.2.0), the spatially-dependent ACIS OBF contamination correction (CalDB 3.0.0), the LSF parameter input data for grating RMFs (CalDB 2.27, 2.28), the upgrades in the detector quantum efficiencies and uniformities (CalDB 2.9 for HRC-I, 2.28 for ACIS, 3.0.1 for HRC-S), and the HRMA effective area modifications (CalDB 3.2.1). However, these do not directly affect processing through Level 2 which is the work of the current reprocessing run. Nevertheless, the reprocessing of the older data does allow these new analysis-level calibrations to be applied more appropriately and directly to archival data (i.e. without user reprocessing or custom processing). Hence reprocessing will make the latest and best Chandra calibrations more immediately applicable for the bulk of the archive.


Major Software Changes

Updates will that will change and/or improve standard pipeline data products

Date DS software version Description
2006-02-15 7.6.7 No change
2006-01-10 7.6.6 No change
2006-01-01 7.6.4.2 No change
2005-12-05 7.6.4.1 No change
2005-11-15 7.6.4
  1. In ACIS CC-mode, the values of PHA, PI, ENERGY in an event file change because of a CALDB interface change. (see also an update in DS7.6.3).
  2. The ACIS destreak tool is updated to fix problems using all events instead of only using good events. This only affects data taken in the ACIS S4 chip (or ccdid=8).
  3. Other minor changes affecting both ACIS and HRC data, including a bug fix in the axis of rotation in the aspect solution and updating a GTI limit for the HRC Next-In-Line time gap.
2005-09-01 7.6.3
  1. The headers in ACIS and HRC event files are upgraded to reference other data products, including CALDB files (e.g., ASOLFILE = aspect solution, THRFILE = ACIS split threshold).
  2. For ACIS-CC-mode, the CTI correction is applied and the split-threshold file is correctly utilized. These affect PHA/PI/ENERGY values in an event file (see also an update in DS7.6.4).
2005-07-26 7.6.2 No change
2005-06-15 7.6
  1. In calculating the aspect solution, the aspect pipeline properly applies the proper motion of a guide star. This will slightly improve the accuracy of the aspect solution. Previously, a guide star with a large proper motion is excluded in calculating the aspect solution.
  2. Add support for large files (> 2GB).
    Previously, when an event file is bigger than 2 GB (for long observations of very bright sources), pipeline s/w could not handle it and had to apply manual intervention either by removal of non-vital columns or by splitting into multiple smaller pieces.
2005-05-02 7.5.4 No change
2005-03-23 7.5.3
  1. Changes in VV aspect check limits.
    Minor changes in aspect quality limits (e.g., image centroiding error, offset) are applied in determining a bad guide star or fid light to exclude in calculating the aspect solution.
2005-02-18 7.5.2 No change
2005-02-15 7.5.1 No change
2005-01-28 7.5.0.1
  1. Update AGASC (AXAF Guide and Acquisition Star Catalog) to v1.6.
    This slightly improves aspect quality and absolute position accuracy. See this page for more information.
2004-12-14 7.5.0 No change
2004-11-30 7.4.0
  1. ACIS hot pixel tools are added to the ACIS L1 pipeline to effectively flag both hot and after-glow (transient) pixels. The tool acis_detect_afterglow was removed from the pipeline.
  2. The ACIS EXPNO column has a real data-subspace with a specified range of exposure numbers in each GTI per chip.
  3. Fixed ACIS TGAIN computation that affects very high PHA values (above ~4100 ADU). PHA, PI, and ENERGY values will be different.
  4. For ACIS-S + LETG, the spectral extraction width changes from 1.33e-03 deg to 2.00e-3 deg to compensate for a residual rotation of dispersed spectra.
  5. Fixed contents of ACIS mask files when SAMP_CYC (sampling cycle) is not 0 or 1.
2004-07-13 7.3.2 No change
2004-06-25 7.3.1 No change (most data are still labeled as 7.3.0)
2004-06-22 7.3.0
  1. ACIS time dependent gain correction added to the ACIS L1 pipeline. PHA, PI, and ENERGY values in the event file will change.
  2. ACIS destreaking is applied to the S4 chip in the ACIS L1 pipeline. (Note. It is later improved in DS 7.6 started on 2005-11-15.)
  3. ACIS+CC mode data processing (e.g., algorithm to determine chip-y coordinate) is enhanced. Note that more enhancements for the CC mode are later added in DS7.6.3 (2005-09-01) and DS7.6.4 (2005-11-15).
  4. Several minor changes to ACIS mask files. One row specifies that a chip is on or off. The columns PHAMIN and PHAMAX were updated. The columns WINDOW and OVERLAP were dropped. The keywords PHAMIN, PHARANGE, FIRSTROW and LASTROW were updated.
  5. HRC AMPSFCORR test is turned on to properly deal with tap-ringing induced errors.
  6. HRC dead time correction factors is corrected for a double-counting problem in the rate-scalars.
  7. HRC chip offsets are changed based on the LETG analysis. Chip positions and all coordinates will change.
  8. Pixel randomization is turned off for all HRC and all grating datasets when going from chip to angular coordinates.
  9. Fixed a sign error in the grating angle for LETG data for both ACIS and HRC. This will change both resolved events and grating masks.
  10. A new fits header keyword as a dataset identifier, DS_IDENT is added. Its value is for example "ADS/Sa.CXO#obs/01234"
  11. The Level2 merged products (only for multiple obi data) retain more per-obi header keywords (e.g., FP_TEMP, SIM_*, *BEP, BTIM*, *_PNT) based on relaxed merging rules.
2004-04-26 7.2.2 No change (most data are still labeled as 7.2.1)
2004-03-25 7.2.1 No change
2004-03-09 7.2.0
  1. Fixed a systematic timing error by adding 285 microseconds (to strip files by PI3)
2003-12-30 7.1.1 No change
2003-10-06 7.0.3 No change (some data labeled as 7.0.2)
2003-10-02 7.0.2 No change
2003-09-26 7.0.1 No change
2003-09-23 7.0.0
  1. Significantly enhanced infra-structure including CIAO 3.0 tools integrated into the pipeline processing and upgraded telemetry processing (pi3 replacing an old pi2).
  2. Revised a chip y-coordinate (CHIPY) in a ACIS continuous clocking (CC) mode, when finding the correct position-dependent gain for an event. An estimate of the actual CHIPY location of the source is used instead of the recorded value of CHIPY, which represents time instead of position.
  3. Fixed a problem with the way the keywords CTI_CORR and CTIFILE are populated. CTI_CORR is set to "T" if and only if the CTI adjustment is applied. Likewise, CTIFILE is populated with the name of a CTI adjustment file if and only if the CTI adjustment is applied.
  4. Fixed a problem in using a CTI-corrected gain file (implemented in DS6.11) for graded mode data (no PHAS in the grade mode to correct CTI).
  5. The HRC tap ringing correction algorithm was implemented. All the coordinates of HRC data will change slightly.
  6. Adjusted HEG and MEG region width scaling factors (20 to 35) and background outer limits (from 23.9 to 21.6) in the grating L1.5 pipeline. (see more in this memo)
  7. The observation parameter (obspar) files was upgraded to version=6. Because the data type of the ObsId is changed from integer to string, all old obspar files can not be used after 7.0.0. Other changes include sim_z_offset updated in the actual obspar based on the SIM telemetry.
  8. Upgraded a SIM stripper which produces a single strip file for both A and B, but SIM-A and SIM-B L0 products remain separated. There should be no change to the L0 products, however, the bogus SIM-B L0 products will no longer be created.
2003-07-17 6.13.4
  1. Added support of the 2nd Inertial Reference Unit (IRU2) switchover to the ASPECT L1 pipeline. The aspect solution changed only slightly (~0.01 arcsec).
2003-05-06 6.13.0 No change
2003-05-06 6.12.0 (6.11.0 deployed at the same time)
  1. The observation parameter (obspar) files was upgraded to version=5 to support the enhanced roll and time window constraints started to use in an observing cycle AO5.
2003-05-06 6.11.0 (deployed at the same time as 6.12.0)
  1. ACIS CTI (Charge Transfer Inefficiency) correction was applied for ccdid = 0-3 and 6 (or I0-I3 and S2) taken at a ccd temperature of -120° C. It will change PHA, GRADE, FLTGRADE, ENERGY and PI values in ACIS event files (acis...evt1.fits and acis...evt2.fits).
  2. Fixed incorrect sky field-of-views (in acis...fov1.fits and in acis...full_img2.fits) for observations with multiple observational intervals (OBI).
  3. Fixed a s/w bug in resolving events of grating CC mode data for using incorrect chip coordinates and angular coordinates.
  4. Applied a correct trigonometric function instead of a small angle approximation (sin(a) ~= a) for the coordinate transforms in the grating pipeline (tg_resolve_events).
  5. Fixed a problem in the ASPECT L1 pipeline when the first input file has no data.
2002-10-16 6.9.2
  1. Upgraded to the AXAF Guide and Acquisition Star Catalog (AGASC) from 1.3 to 1.5 (see more in AGASC interface)
  2. Fixed the computation of DTCOR for interleaved-mode observations.


2. Reprocessing IIIa: Aspect correction for early 2007

Follow this link.


3. Reprocessing IIIb: Override Obspar

Under construction


4. Reprocessing IIIc: Calibration update of all HRC-I observations since 29 January 2000

During the third Chandra Reprocessing run (January 2006 - September 2007) all observations since 29 January 2000, including those made by HRC-I, were processed with a new common set of calibrations. However, since the start of that run there have been significant improvements in two calibration areas for HRC-I: the gain map (GMAP) and DEGAP (GAPLOOKUP). We have decided to perform a supplemental reprocessing of all HRC-I observations since Feb 2000 in order to update all of them to the latest gain and degap calibrations; This is designated as Reprocessing Run 3C.

The GMAP is used to generate PI values from PHA values in the event list. The PI values may be used to estimate hardness ratios, and to perform quantile color-color analyses of HRC-I observed sources. If your analyses have included hardness ratios or quantile color-color analysis, or you would like to perform such analyses of your sources, you may wish to review the Repro 3C processing of your observations.

The upgrades to the GAPLOOKUP tables (see section III.B of the CalDB 3.4.1 Release Notes) for HRC-I have made minor, but still significant improvements to the encircled energy fraction for on-axes sources located at or near the nominal aimpoint position of HRC-I. The latest upgrade to GAPLOOKUP, which has been added as of September 14, 2007, makes significant improvements in the degap solution for on-axis source observations with a particular large SIM_Z offset, so that the source appears in CRSU = 12-13 region, and small improvements elsewhere. If your analyses are particularly sensitive to the ECF, you may want to review your results with the Repro 3C processing of your observations.



This page is maintained by the Chandra Data Archive Operations group

Last modified: 2008-02-25



Chandra Science | Chandra Home | Astronomy links | iCXC (CXC only) | Search


The Chandra X-Ray Center (CXC) is operated for NASA by the Smithsonian Astrophysical Observatory.
60 Garden Street, Cambridge, MA 02138 USA.    Email: cxcweb@head.cfa.harvard.edu
Smithsonian Institution, Copyright © 1998-2004. All rights reserved.