Chandra X-Ray Observatory
	(CXC)
Skip to the navigation links
Last modified: 2 Dec 2014

URL: http://cxc.harvard.edu/ciao/threads/createL2/

Reprocessing Data to Create a New Level=2 Event File

CIAO 4.7 Science Threads


Overview

Synopsis:

The structure of this thread is presented as a series of questions and answers, the latter of which are links that take you further down the analysis tree. Following this thread from the top down will not produce correct results.

Alternately, use the chandra_repro script to reprocess your data and be confident it has the newest calibration applied. If you run chandra_repro, you have completed all steps in this thread and you are done.

Purpose:

To generate a new level=2 event file (evt2.fits) for any Chandra observation.

Related Links:

Last Update: 2 Dec 2014 - Updated for removal of acis_process_events calc_cc_times parameter from CIAO 4.7.


Contents


Use the chandra_repro script (Recommended)

The CXC strongly recommends you recalibrate your event data so that it is consistent with the other calibrations you will need. The easiest way is to use the chandra_repro reprocessing script automates the recommended data processing steps presented in the CIAO analysis threads. The script reads data from the standard data distribution (e.g. primary and secondary directories) and creates a new bad pixel file, a new level=2 event file, and a new level=2 Type II PHA file (grating data only) along with the appropriate per-order and per-arm ARF and RMF files.

To run the script:

unix% cd <obsid>

unix% ls
primary/  secondary/

unix% chandra_repro

After chandra_repro is finished, this thread is complete and you may continue with your analysis. You should not follow the rest of this thread.

If you want to learn more about the individual processing steps, the options available, and the custom calibrations that can be applied you may continue reading this thread.


Do I have to run this thread?

Our recommendation is that all users reprocess their data in CIAO to ensure that the newest software and consistent calibration updates are applied to the dataset before you begin your analysis. Not all data needs to be reprocessed, but any data can be reprocessed without any negative effects.

Proceed to the next section to get started.

Software and Calibration Updates

Updates to ACIS TGAIN calibration for -120 C data

The ACIS time-dependent gain (TGAIN) calibration files for -120 C focal plane temperature are frequently updated in the calibration database (CALDB). If your data were taken recently - even if the ASCDSVER is DS 7.6.7 or higher - a newer TGAIN file may be available for your dataset.

Newly acquired data processed in standard data processing uses a predictive TGAIN file -- a flat extrapolation from the previous epoch. It may be several months until the final TGAIN calibration is available. Typically the differences between ephocs are small; however, if your science requies high fideltiy energy calibrations (large number of counts and a line dominated spectrum), the you will want to use the definitive TGAIN file when it is available.

The Improvements in Calibration section of the TGAIN why topic lists the updates by date and CALDB version. The observation start date and focal plane temperature (in K) are recorded in the header of the event file:

unix% dmkeypar acis_evt1.fits DATE-OBS echo+
2008-01-19T20:41:39

unix% dmkeypar acis_evt1.fits FP_TEMP echo+
153.60722351

Most data in the Chandra Archive has been updated as part of the bulk reprocessing effort known as Repro-4 effort. However, calibration and software changes have been introduced since the start of reprocessing. Users are therefore still encouraged to reprocess their data to ensure they have a consistent dataset.

Proceed to the next section to begin.


Get Started

Since this thread is meant to represent the most generic case, it is not written with a specific ObsID. Files are referred to by the detector and type - acisf01843_000N001_evt1.fits becomes acis_evt1.fits, hrcf01557_000N001_std_flt1.fits becomes hrc_std_flt1.fits, and so on.

Furthermore, it is assumed that all relevant files are in the same working directory.

Proceed to the next section.


ACIS or HRC?

Question: which detector was used for your observation? (It does not matter yet whether or not a transmission grating was also used.)


ACIS Observations

Question: was the observation taken in continuous-clocking mode (i.e. the read mode is "continuous")?

unix% dmkeypar acis_evt1.fits readmode echo+
CONTINUOUS

Check Accuracy of Source Coordinates (CC-mode data only)

It is important to verify that the coordinates of the source are precise to less than 0.5 arcsec (i.e. one pixel). The source coordinates are used to determine the absolute times of arrival for continuous-clocking mode data; if they are imprecise, the output times of arrival will also be inaccurate.

The coordinates that should be used are the best-known right ascension (RA_TARG) and declination (DEC_TARG) coordinates for the source:

unix% dmlist acis_evt1.fits header | egrep '(RA|DEC)_TARG'
0057 RA_TARG                     83.6333330              Real8        Observer's specified target RA
0058 DEC_TARG                    22.0144720              Real8        Observer's specified target Dec

These coordinates in this file have the correct level of precision for the time-of-arrival correction.

If the accuracy of the coordinates in the header needs to be improved, here are some suggestions on how to determine the values:

  • use previously-published source coordinate values.
  • if the coordinates are not well known, try obtaining them from a Chandra observation of the source which was taken in TE mode.
  • if neither of the above methods are available, it may be possible to use two, separate continuous-clocking mode observations to find the source coordinates, provided that the observations did not have the same roll angle.

It is not possible to fix the coordinates in the dataset if the position is not known from some other dataset or publication (i.e. you cannot determine accurate source coordinates from the dataset you are attempting to correct).

Once the new source coordinates are determined, modify the file header with dmhedit:

unix% dmhedit acis_evt1.fits filelist="" operation=add key=RA_TARG value='newvalue'
unix% dmhedit acis_evt1.fits filelist="" operation=add key=DEC_TARG value='newvalue'

Note that the uncertainties in the times of arrival include the uncertainties in the coordinates RA_TARG and DEC_TARG, the PSF, and the aspect reconstruction. If RA_TARG and DEC_TARG are accurate to much less than a pixel, then the times of arrival should be accurate to about 4 ms for most observations (i.e. those without any unusual aspect problems).

Proceed to the next step.


Reset ACIS status bits

Status bits 1-5, 14-20 and 23 in the input event file are reset so that they can be updated by destreak and acis_process_events. This reset includes removing the effects of acis_detect_afterglow and (previously) acis_run_hotpix. More information is available in the cosmic-ray afterglow why topic. Details on what each status bit represents are available from the "STATUS bits in event files" memo.

The acis_clear_status_bits script clears the status bits in place; it does not create an output file. The file must have write permission, and you may wish to make a copy before modifying it.

unix% acis_clear_status_bits acis_evt1.fits

The CLSTBITS header keyword is added to record which bits were cleared.

unix% dmkeypar acis_evt1.fits  CLSTBITS echo+
11111111011000000011111111000001

Proceed to the next step.


Remove streak events (Optional)

Running destreak on the ACIS-S4 chip (ccd_id=8) can improve the quality of your data. Removing the streaks before creating a new bad pixel file improves the streak detection efficiency and prevents misidentifying pixels with many streak events as hot pixels. For details on how the tool works, see the Destreaking ACIS Data why topic and "ahelp destreak".

Note that there is some evidence that for bright, continuum-dominated sources observed with the gratings, applying destreak removes more source events than streak events; see the destreak why topic for details. Furthermore, spectral order sorting alone will greatly reduce the amount of streak data present in any extracted spectrum. Therefore destreaking should be considered optional for this case.

The mask parameter is set to "None" so that all events in the input file are evaluated as possible streak events. The parameter setting filter=no flags the streak events in the output file, but does not remove them.

unix% punlearn destreak
unix% pset destreak infile=acis_evt1.fits
unix% pset destreak outfile=acis_dstrk_evt1.fits
unix% pset destreak mask=None filter=no
unix% destreak
Input dataset/block specification (acis_evt2.fits): 
Output dataset/block specification (acis_dstrk_evt2.fits): 

Question: was the observation taken in timed mode (i.e. the read mode is "TIMED", not "CONTINIOUS")?

unix% dmkeypar acis_dstrk_evt1.fits readmode echo+
TIMED

Create a new ACIS bad pixel file

This section of the thread does not apply when the input data were taken in ACIS continuous-clocking mode, as the ACIS afterglow tools cannot be used with this data mode.

There are three steps in creating a new ACIS bad pixel file:

  1. identify known bad pixels and columns from the calibration database and pixels with bad bias values (acis_build_badpix with a custom bitflag )
  2. search for afterglows and hot pixels (acis_find_afterglow)
  3. mark pixels adjacent to afterglows and hot pixels and sort the list of bad pixels and afterglows (a second run of acis_build_badpix)

In order to run acis_build_badpix, you need an obs.par file for the observation, created by running dmmakepar

unix% dmmakepar acis_dstrk_evt1.fits acis_obs.par

You will also need a list of the bias files.

unix% cat bias.lis
acis_2_bias0.fits
acis_4_bias0.fits
acis_5_bias0.fits
acis_3_bias0.fits
acis_1_bias0.fits

Now run the tools:

unix% punlearn acis_build_badpix 
unix% pset acis_build_badpix obsfile=acis_obs.par
unix% pset acis_build_badpix pbkfile=acis_pbk0.fits  
unix% pset acis_build_badpix biasfile=@bias.lis 
unix% pset acis_build_badpix outfile=acis_abb1_bpix1.fits 
unix% pset acis_build_badpix bitflag=00000000000000020021100020022222
unix% pset acis_build_badpix calibfile=CALDB 
unix% acis_build_badpix mode=h verbose=0

unix% punlearn acis_find_afterglow
unix% pset acis_find_afterglow infile=acis_dstrk_evt1.fits
unix% pset acis_find_afterglow outfile=acis_aglow_bpix1.fits
unix% pset acis_find_afterglow badpixfile=acis_abb1_bpix1.fits 
unix% pset acis_find_afterglow maskfile=acis_msk1.fits
unix% pset acis_find_afterglow statfile=acis_stat1.fits
unix% acis_find_afterglow mode=h verbose=0

unix% punlearn acis_build_badpix 
unix% pset acis_build_badpix obsfile=acis_obs.par
unix% pset acis_build_badpix pbkfile=acis_pbk0.fits  
unix% pset acis_build_badpix biasfile=None
unix% pset acis_build_badpix outfile=acis_repro_bpix.fits
unix% pset acis_build_badpix calibfile=acis_aglow_bpix1.fits
unix% pset acis_build_badpix procbias=no
unix% acis_build_badpix mode=h verbose=0

The new bad pixel file is called acis_repro_bpix.fits.

Proceed to the next step.


Run acis_process_events to make a new level=1 event file

Running acis_process_events produces a new level=1 event file that has the latest calibration applied:

Determine the eventdef

The eventdef parameter specifies the names and data types of the columns in the output event data file. Four definitions are included in the parameter file for acis_process_events:

READMODE DATAMODE event mode eventdef string
TIMED (V)FAINT timed exposure (very) faint stdlev1
TIMED GRADED timed exposure graded grdlev1
CONTINUOUS CC(33)_FAINT continuous clocking (3x3) faint cclev1
CONTINUOUS CC(33)_GRADED continuous clocking (3x3) graded ccgrdlev1

The event mode of an observation can be found in the READMODE and DATAMODE values stored in the file header:

unix% dmkeypar acis_evt1.fits READMODE echo+
TIMED

unix% dmkeypar acis_evt1.fits DATAMODE echo+
FAINT

This is a timed exposure faint observation, so the proper eventdef parameter is "stdlev1."

Set the parameters

  • If you created a new bad pixel file when running the Reprocessing Data to Create a New Level=2 Event File thread or the Customizing an ACIS Bad Pixel File thread, use that file in this analysis. Otherwise, use the bpix1.fits file from the Archive.

  • In some rare cases, there will be more than one aspect solution file (pcad_asol1.fits) for an observation. All the files must be input to the acaofffile parameter in chronological order; the time is in the filename, so "ls" lists them in order. The files may be provided as a comma-separated list or as a stack. Here a stack is used:

    unix% cat pcad_asol1.lis 
    pcadf063874624N002_asol1.fits
    pcadf063875522N002_asol1.fits
    pcadf063902942N002_asol1.fits
    
  • The parameters for VFAINT background cleaning (check_vf_pha) is then set. The tool will ignore this parameter if your observation was not taken in this mode.

    The check_vf_pha flag should be used very carefully as it can removed valid source events as discussed in the why topic. The default for chandra_repro is check_vf_pha=no.

unix% punlearn acis_process_events
unix% pset acis_process_events infile=acis_dstrk_evt1.fits
unix% pset acis_process_events outfile=acis_new_evt1.fits
unix% pset acis_process_events badpixfile=acis_new_bpix1.fits
unix% pset acis_process_events acaofffile=@pcad_asol1.lis
unix% pset acis_process_events mtlfile=acis_mtl1.fits
unix% pset acis_process_events eventdef=")stdlev1"
unix% pset acis_process_events check_vf_pha=yes

Run the tool

unix% acis_process_events
Input event file or stack (acis_evt1.fits): 
Output event file name (acis_new_evt1.fits): 
aspect offset file ( NONE | none | <filename>) (pcad_asol1.fits):

A number of possible warning messages are explained in the Common acis_process_events Warnings section.


Update Header Keywords

For some data, taken early in the Chandra mission, it will be necessary to update the event file headers with some keywords that were added during the last Chandra reprocessing (known as "Repro-4").

The r4_header_update script is used to add several keywords. If the keywords already exist then the tool will simply ignore them; so it is always safe to run it. The file is modified in place.

% dmkeypar acis_new_evt1.fits SUM_2X2 echo+
# dmkeypar (CIAO): ERROR: Keyword 'SUM_2X2' was not found in file 'acis_new_evt1.fits'.

unix% dmkeypar acis_new_evt1.fits DY_AVG echo+
# dmkeypar (CIAO): ERROR: Keyword 'DY_AVG' was not found in file 'acis_new_evt1.fits'.

unix% r4_header_update acis_new_evt1.fits
Missing keywords 'OCLKPAIR,ORC_MODE,SUM_2X2,FEP_CCD' from event file 'acis_new_evt1.fits' header.
Missing keywords 'DY_AVG,DZ_AVG,DTH_AVG' from event file 'acis_new_evt1.fits' header.

unix% dmkeypar acis_new_evt1.fits SUM_2X2 echo+
0
unix% dmkeypar acis_new_evt1.fits DY_AVG echo+
0.045452421825

If very old data is being reprocessed, it may be necessary to also specify the pbkfile and the asolfile parameters.


Question: is your observation imaging or grating data?


Imaging Observations

Filter on Grade, Status, and Good Time

There are two filtering steps for ACIS imaging observations:

  1. Filter for bad grades and for a "clean" status column (i.e. all bits set to 0):

    unix% punlearn dmcopy
    unix% dmcopy "acis_new_evt1.fits[EVENTS][grade=0,2,3,4,6,status=0]" \
          acis_flt_evt1.fits
    
  2. The Good Time Intervals (GTIs) supplied by the pipeline now need to be applied; these are stored in the acis_flt1.fits file. Simultaneously, unnecessary columns are eliminated from the output:

    unix% punlearn dmcopy
    unix% dmcopy "acis_flt_evt1.fits[EVENTS][@acis_flt1.fits][cols -phas]" \
          acis_repro_evt2.fits
    

    Be sure to include the @ symbol in the filter expression; the command will not be executed properly if it is omitted.

The thread is now complete. acis_repro_evt2.fits is a level=2 event file with the latest calibration products applied.


Grating Observations

There are CIAO threads that deal specifically with generating a level=2 event file for an ACIS grating observation. Follow the appropriate one, depending on which transmission grating was used:

Single source:

Multiple sources:

Once you leave this thread, you do not need to return to complete any further steps.


Common acis_process_events Warnings

It is important to remember that the following messages are warnings, not errors. The tool may print one of these messages, but will still create a valid event file.

WARNING: problem reading ctifile, cti adjustment will not be applied.
# acis_process_events (CIAO 4.5): WARNING: problem reading ctifile, cti adjustment will not be applied. 
                                  Changing apply_cti=yes to apply_cti=no.

This warning may also be triggered by product=T_GAIN.

The most common cause for this warning is that the observation does not contain -120 C data. Since the CTI and TGAIN are only calibrated for this focal plane temperature, it is not possible to use it on other observations.

Since no file is found, acis_process_events continues running as if apply_cti=no or apply_tgain=no and produces a valid output file that has not been CTI-corrected or had the TGAIN applied.

The parameter check_vf_pha=yes, but the DATAMODE key does not indicate a very-faint mode observation.
# acis_process_events (CIAO 4.5): WARNING: The parameter check_vf_pha=yes, but the DATAMODE key does not indicate
                                  a very-faint mode observation.  Changing to check_vf_pha=no.

ACIS very-faint background cleaning can only be performed on observations that were taken in VFAINT mode. If the check_vf_pha parameter is set to "yes" but the data has a DATAMODE other than VFAINT, acis_process_events continues to reprocess the data as if check_vf_pha=no and produces a valid output file that does not have the background cleaning applied.


HRC Observations

Observations using the HRC-S (imaging and grating) might be affected by an asymmetry and broadening of the PSF. Chandra aim point drift over time has placed the aim point onto an area of the detector that has not previously been subject to detailed PSF calibration. Read the HRC-S Event Position Errors Near the Aim Point webpage for details.

All HRC-S (imaging and grating) observations taken after 01 January 2012, which were processed with a CalDB version released prior to the 4.5.1 release (24 July 2012), should be reprocessed to apply the updated HRC-S time-dependent gain map calibration released with CalDB 4.5.1. The High Voltage settings of the HRC-S microchannel plates were increased in March 2012, resulting in a corresponding gain increase and required change to HRC-S T_GAIN calibration.

Running hrc_process_events produces a new level=1 event file that has the latest calibration applied:

Determine the "instrume" parameter

The instrume parameter, which specifies the instrument with which the data was collected, first needs to be determined:

unix% dmkeypar hrc_A_evt1.fits detnam echo+
HRC-I

unix% dmkeypar hrc_B_evt1.fits detnam echo+
HRC-S

The instrume value is hrc-i for file A and hrc-s for file B.

HRC-S: Determine the 'gainfile' parameter

On 29 March 2012, the High Voltage (HV) settings of the HRC-S microchannel plates were increased, causing a corresponding rise in the detector gain, and a required update to the HRC-S time-dependent gain map (T_GMAP) calibration. As a result, all observations taken after 01 January 2012 must be reprocessed using one of the two new calibration T_GMAP files available in CalDB 4.5.1:

	  hrcsD1999-07-22t_gmapN0002.fits
          hrcsD2012-03-29t_gmapN0002.fits
        

If you are analyzing one of the ObsIDs listed below, you must manually set the 'gainfile' parameter of hrc_process_events to the appropriate CalDB path and name, as shown below.

ObsID HV setting Object Grating Start Date
14324 old Mkn421 LETG 2012-07-04 04:09:45
14396 old Mkn421 LETG 2012-07-03 19:09:24
14397 old Mkn421 LETG 2012-07-04 13:10:06
14238 new HZ43 LETG 2012-03-18 05:16:00
Old High-Voltage Setting:
unix% pset hrc_process_events gainfile ${CALDB}/data/chandra/hrc/t_gmap/hrcsD1999-07-22t_gmapN0002.fits

New High-Voltage Setting:
unix% pset hrc_process_events gainfile ${CALDB}/data/chandra/hrc/t_gmap/hrcsD2012-03-29t_gmapN0002.fits

Otherwise, you may run hrc_process_events with 'gainfile' set to the default 'CALDB', and the correct time-dependent gain map will automatically be applied.

Set the parameters

  • In some rare cases, there will be more than one aspect solution file (pcad_asol1.fits) for an observation. All the files must be input to the acaofffile parameter in chronological order; the time is in the filename, so "ls" lists them in order. The files may be provided as a comma-separated list or as a stack. Here a stack is used:

    unix% cat pcad_asol1.lis 
    pcadf062419415N003_asol1.fits
    pcadf062461940N003_asol1.fits
    pcadf062492182N003_asol1.fits
    
  • Set the instrume parameter (here, hrc-s) to the correct value for the dataset.

unix% punlearn hrc_process_events
unix% pset hrc_process_events infile=hrc_evt1.fits
unix% pset hrc_process_events outfile=hrc_new_evt1.fits
unix% pset hrc_process_events badpixfile=hrc_bpix1.fits
unix% pset hrc_process_events acaofffile=@pcad_asol1.lis
unix% pset hrc_process_events badfile=NONE
unix% pset hrc_process_events instrume=hrc-s
unix% pset hrc_process_events do_amp_sf_cor=yes

Run the tool

unix% hrc_process_events
input level 0 event file/stack (hrc_evt1.fits): 
output level 1 file (hrc_new_evt1.fits): 
bad pixel file ( NONE | none | <filename>) (hrc_bpix1.fits): 

A number of possible warning messages are explained in the Common hrc_process_events Warnings section.


Update Header Keywords

For some data, taken early in the Chandra mission, it will be necessary to update the event file headers with some keywords that were added during the last Chandra reprocessing (known as "Repro-4").

The r4_header_update script is used to add several keywords. If the keywords already exist then the tool will simply ignore them; so it is always safe to run it. The file is modified in place.

unix% dmkeypar hrc_new_evt1.fits DY_AVG echo+
# dmkeypar (CIAO): ERROR: Keyword 'DY_AVG' was not found in file 'hrc_new_evt1.fits'.

unix% r4_header_update hrc_new_evt1.fits
Missing keywords 'DY_AVG,DZ_AVG,DTH_AVG' from event file 'hrc_new_evt1.fits' header.

unix% dmkeypar hrc_new_evt1.fits DY_AVG echo+
0.045452421825

If very old data is being reprocessed, it may be necessary to also specify the asolfile parameter.

Note: The set of keywords that are updated are different for ACIS and HRC data.


Question: is your observation HRC-S imaging, HRC-I imaging, or grating data?


HRC-S Imaging Observations

There are two filtering steps for HRC-S imaging observations:

  1. Apply the status filter that is specific to HRC-S observations; a value of 0 demands that the bit be flagged as "good", a value of x indicates that either status (0/1) is acceptable. The PI filter removes about 20% of the background with no X-ray losses:

    unix% punlearn dmcopy
    unix% dmcopy \
          "hrc_new_evt1.fits[status=xxxxxx00xxxx0xxx0000x000x00000xx]" \
          hrc_flt_evt1.fits
    
  2. The Good Time Intervals (GTIs) supplied by the pipeline now need to be applied; these are stored in the hrc_std_flt1.fits file. Simultaneously, unnecessary columns are eliminated from the output:

    unix% punlearn dmcopy
    unix% dmcopy \
          "hrc_flt_evt1.fits[EVENTS][@hrc_std_flt1.fits][cols -crsu,-crsv,-amp_sf,-av1,-av2,-av3,-au1,-au2,-au3,-raw,-sumamps]" \
          hrc_repro_evt2.fits
    

    Be sure to include the @ symbol in the filter expression; the command will not be executed properly if it is omitted.

Computing Average HRC Dead Time Corrections

The average HRC deadtime correction is computed using the tool hrc_dtfstats to ensure the accuracy of the deadtime statistics. (The deadtime should be recomputed after any time filtering is applied to an HRC dataset, e.g. applying GTIs or filtering background flares.)

unix% punlearn hrc_dtfstats
unix% pset hrc_dtfstats infile=hrc_dtf1.fits
unix% pset hrc_dtfstats outfile=hrc_dtfstats_new.fits
unix% pset hrc_dtfstats gtifile=hrc_repro_evt2.fits"[gti]"
unix% hrc_dtfstats
Input file (hrc_dtf1.fits): 
Output file (hrc_dtfstats_new.fits): 
File containing GTI to filter on (<filename>|NONE) (hrc_repro_evt2.fits[gti]): 

In this example, the new dtfstats file has a value of DTCOR=0.35535:

unix% dmlist hrc_dtfstats_new.fits"[cols DTCOR]" data,clean
#  DTCOR
     0.35535140943668

The ONTIME header value is multiplied by the new DTCOR value, and the LIVETIME and EXPOSURE header keyword values are updated with dmhedit. The commands in general are:

unix% dmkeypar hrc_repro_evt2.fits ONTIME echo+

ONTIME * NEWDTCOR = NEWTIME

unix% dmhedit hrc_repro_evt2.fits filelist="" op=add key=LIVETIME value=NEWTIME
unix% dmhedit hrc_repro_evt2.fits filelist="" op=add key=EXPOSURE value=NEWTIME
unix% dmhedit hrc_repro_evt2.fits filelist="" op=add key=DTCOR value=NEWDTCOR

The following commands use example values to illustrate:

unix% dmkeypar hrc_repro_evt2.fits ONTIME echo+
2189.40009910

2189.40009910*0.35535140943668 = 778.00641103599186717498

unix% dmhedit hrc_repro_evt2.fits filelist="" op=add key=LIVETIME value=778.0064110
unix% dmhedit hrc_repro_evt2.fits filelist="" op=add key=EXPOSURE value=778.0064110
unix% dmhedit hrc_repro_evt2.fits filelist="" op=add key=DTCOR value=0.35535140943668

The thread is now complete; hrc_repro_evt2.fits is a level=2 event file with the latest calibration products applied.

User may optionally want to apply a 'pi' based filter to their data as discussed in the HRC PI filtering thread.


HRC-I Imaging Observations

There are two filtering steps for HRC-I imaging observations:

  1. Apply the status filter that is specific to HRC-I observations; a value of 0 demands that the bit be flagged as "good", a value of x indicates that either status (0/1) is acceptable:

    unix% punlearn dmcopy
    unix% dmcopy \
          "hrc_new_evt1.fits[status=xxxxxx00xxxx0xxx00000000x0000000]"  \
          hrc_flt_evt1.fits
    
  2. The Good Time Intervals (GTIs) supplied by the pipeline now need to be applied; these are stored in the hrc_std_flt1.fits file. Simultaneously, unnecessary columns are eliminated from the output:

    unix% punlearn dmcopy
    unix% dmcopy \
          "hrc_flt_evt1.fits[EVENTS][@hrc_std_flt1.fits][cols -crsu,-crsv,-amp_sf,-av1,-av2,-av3,-au1,-au2,-au3,-raw,-sumamps]" \
          hrc_repro_evt2.fits
    

    Be sure to include the @ symbol in the filter expression; the command will not be executed properly if it is omitted.

Computing Average HRC Dead Time Corrections

The average HRC deadtime correction is computed using the tool hrc_dtfstats to ensure the accuracy of the deadtime statistics. (The deadtime should be recomputed after any time filtering is applied to an HRC dataset, e.g. applying GTIs or filtering background flares.)

unix% punlearn hrc_dtfstats
unix% pset hrc_dtfstats infile=hrc_dtf1.fits
unix% pset hrc_dtfstats outfile=hrc_dtfstats_new.fits
unix% pset hrc_dtfstats gtifile=hrc_repro_evt2.fits"[gti]"
unix% hrc_dtfstats
Input file (hrc_dtf1.fits): 
Output file (hrc_dtfstats_new.fits): 
File containing GTI to filter on (<filename>|NONE) (hrc_repro_evt2.fits[gti]): 

In this example, the new dtfstats file has a value of DTCOR=0.35535:

unix% dmlist hrc_dtfstats_new.fits"[cols DTCOR]" data,clean
#  DTCOR
     0.35535140943668

The ONTIME header value is multiplied by the new DTCOR value, and the LIVETIME and EXPOSURE header keyword values are updated with dmhedit. The commands in general are:

unix% dmkeypar hrc_repro_evt2.fits ONTIME echo+

ONTIME * NEWDTCOR = NEWTIME

unix% dmhedit hrc_repro_evt2.fits filelist="" op=add key=LIVETIME value=NEWTIME
unix% dmhedit hrc_repro_evt2.fits filelist="" op=add key=EXPOSURE value=NEWTIME
unix% dmhedit hrc_repro_evt2.fits filelist="" op=add key=DTCOR value=NEWDTCOR

The following commands use example values to illustrate:

unix% dmkeypar hrc_repro_evt2.fits ONTIME echo+
2189.40009910

2189.40009910*0.35535140943668 = 778.00641103599186717498

unix% dmhedit hrc_repro_evt2.fits filelist="" op=add key=LIVETIME value=778.0064110
unix% dmhedit hrc_repro_evt2.fits filelist="" op=add key=EXPOSURE value=778.0064110
unix% dmhedit hrc_repro_evt2.fits filelist="" op=add key=DTCOR value=0.35535140943668

The thread is now complete; hrc_repro_evt2.fits is a level=2 event file with the latest calibration products applied.

User may optionally want to apply a 'pi' based filter to their data as discussed in the HRC PI filtering thread.


Grating Observations

There are CIAO threads that deal specifically with generating a level=2 event file for an HRC grating observation. Follow the appropriate one, depending on which detector was used:

Single source:

Multiple sources:

Once you leave this thread, you do not need to return to complete any further steps.


Common hrc_process_events Warnings

It is important to remember that the following messages are warnings, not errors. The tool may print one of these messages, but will still create a valid event file.

mjd_obs keyword is missing in infile. value from computation.
# hrc_process_events (CIAO 4.5): WARNING: mjd_obs keyword is missing in infile. value from computation.

Some older event files do not have the MJD_OBS header keyword. In this case, the tool uses the DATE-OBS and MJDREF values to compute the correct MJD_OBS.

Out of sequence events discovered in hrc_evt1.fits.
# hrc_process_events (CIAO 4.5): The following error occurred 2482 times: dsHPEEVENTSEQERR - 
                                 WARNING: Out of sequence events discovered in hrc_evt1.fits.

There are occasional events that appear in the telemetry stream with time tags that make them seem to have occurred out-of-sequence. One special case where this occurs has been documented (see the Out-of-sequence HRC Time Tags memo); other occurrences are most likely to be caused by hiccups in the HRC or by double-bit errors in telemetry (single-bit errors are corrected).

This warning may be ignored in most cases, as long as the number of events flagged as out-of-sequence is a small fraction of the total number of events in the file; see the hrc_process_events bug page for more information.


History

22 Dec 2004 updated for CIAO 3.2: minor changes to parameter files; use ACIS bad pixel file (badpixfile parameter)
12 Jan 2005 grating sections now include links to multiple-source analysis threads
01 Feb 2005 added note about "Event island contains 1 or more bad pixels" warning to acis_process_events section
20 Jun 2005 CIAO 3.2.2 patch: minor acis_process_events parameter change (default value of threshfile is CALDB instead of NONE)
12 Dec 2005 updated for CIAO 3.3: the CTI and time-dependent gain corrections may be applied to CC-mode data (see the Continuous Clocking Mode why topic for details); parameter file updates for destreak
01 Dec 2006 updated for CIAO 3.4: CIAO version in errors and warnings
08 Feb 2008 updated for CIAO 4.0: links point to why topics and dictionary entries instead of other ACIS and HRC threads, which have been deprecated
31 Mar 2008 updated for CALDB 3.4.3: new -120 C TGAIN files and first -110 C TGAIN file: both changes made in Do I have to run this thread? section
16 Jun 2008 added CTI correction for back-illuminated chips to Do I have to run this thread? section
07 Jan 2009 updated for CIAO 4.1: acis_process_events no longer prints the "Event island contains 1 or more bad pixels." warning when verbose=0; other warnings were updated
18 Dec 2009 updated for CIAO 4.2: new ACIS TGAIN, HRC-I TGAIN, and HRC-S TGAIN released in CALDB 4.2.0
28 Dec 2009 additional updates for CIAO 4.2: HRC imaging - "pha=0:254" filter is replaced by "pi=0:300"
20 Jan 2010 linked to Customizing an ACIS Bad Pixel File thread; include "[cols -status]" when running acis_process_events
18 Feb 2010 added the MJD_OBS hrc_process_events warning
01 Jul 2010 the chandra_repro reprocessing script automates data processing for ACIS imaging data
02 Sep 2010 incorporated ACIS very-faint background cleaning into this thread; destreak mask parameter updated for VFAINT data
27 Sep 2010 added "Computing Average Dead Time Corrections" section for HRC data
27 Oct 2010 added information on DS 8.3.3 in "do I have to run this thread?" section
15 Dec 2010 updated for CIAO 4.3: new ACIS sub-pixel event repositioning; new ACIS calibration; new HRC-I gain map; removed DS 8.3.3 warning from "do I have to run this thread?" section since the calibration now matches CIAO 4.3 and CALDB 4.4.1.
25 Feb 2011 the 25 Feb release of the chandra_repro reprocessing script supports data processing for ACIS and HRC grating data
04 Mar 2011 added Check Accuracy of Source Coordinates section for continuous-clocking mode data
26 Jul 2011 required software updates are listed in Synopsis
15 Dec 2011 reviewed for CIAO 4.4: reset status bits before running destreak, run destreak before creating a new bad pixel file, use acis_find_afterglow in place of acis_run_hotpix
27 Feb 2012 added link to HRC-S Event Position Errors Near the Aim Point webpage in the Synopsis.
24 Jul 2012 added notice of HRC-S T_GAIN calibration changes to the Synopsis
13 Dec 2012 Review for CIAO 4.5. Many of the details in software and calibration changes were related to repro-3; repro-4 is now underway with additional changes. Added additional clarification that check_vf_pha=no is the default in chandra_repro due to the possiblity of excluding true source events. Added note about predictive TGAIN in SDP.
24 Apr 2013 Added info about scripts release 4.5.2 that includes creation of ARF and RMF for grating datasets. Also make clear that after chandra_repro is run, the thread is complete.
27 Jun 2013 Removed the pi=0:300 filter from HRC-S/No Grating section based on the recommendation from the calibration team in connection with the CALDB 4.5.7 release.
14 Jul 2014 Reviewed for CIAO 4.6; added r4_header_update steps, if needed after acis_process_event or hrc_process_events.
13 Nov 2014 Clarified HRC-S t-gain file pick for observation's high-voltage setting.
02 Dec 2014 Updated for removal of acis_process_events calc_cc_times parameter from CIAO 4.7.


Last modified: 2 Dec 2014
Smithsonian Institute Smithsonian Institute

The Chandra X-Ray Center (CXC) is operated for NASA by the Smithsonian Astrophysical Observatory. 60 Garden Street, Cambridge, MA 02138 USA.   Email:   cxcweb@head.cfa.harvard.edu Smithsonian Institution, Copyright © 1998-2014. All rights reserved.