Maxim Markevitch <maxim % head-cfa.harvard.edu>, last change 15 Jun 06

This file describes what to do with the ACIS background datasets --
specifically, how to use lc_clean to exclude the background flare intervals,
how to use make_acisbg to fill the sky coordinate columns in the background
files, and how to process the data the same way as the background files. For
general info about ACIS background, please see the calibration web page
(http://cxc.harvard.edu/cal/, click "ACIS", then "Background", then "General
discussion"). For a detailed description of how the event files were made,
see the README file accompanying the files.

Note for CfA HEAD users: the tools described here, with their parameter
files, an SM macro, and this COOKBOOK file are also in the directory
/home/maxim/, and the event files are in /home/maxim/DAT/AX/ACIS/BG/.

---------------------------------------------------------------------------
			SOFT GALACTIC FLUX AND NH

All observations included in the B,C,D background datasets have high
Galactic b, moderate to low NH (1-5e20), and low soft X-ray brightness in
the RASS R4-R5 band images (see Snowden et al. 1997,
http://www.xray.mpe.mpg.de/rosat/survey/sxrb/12/fits.html). (Note that prior
to the 190702 version, the aciss_C files included pointings to the North
Polar Spur, now excluded -- thanks to several people who pointed that out.)

For period A, there isn't much choice and half of the total exposure is from
observations with high NH. The notes below apply to the B,C,D datasets only.

The Galactic contribution doesn't matter at E>2 keV or so, where the
particle background dominates. If the soft band (E<1 kev or so) is critical
for your analysis, you need to check the RASS flux at your sky position (see
the above web page; or at CfA, see /home/maxim/DAT/ROS/SXRB) and compare it
to that for observations included in the background dataset.  The R4-R5 RASS
fluxes are given in the README file accompanying the datasets.  If they are
very different from your observation, these files may not be useful.

One can also use the outlying regions of the FOV for modeling the Galactic
excess/deficiency in a given observation w.r.t. these background files (as
we did, e.g., in ApJ 563, 95). For this, be sure to use the background
datasets assembled using the same observations for different chips.

A more radical way is to use the "stowed" background (which does not include
any sky signal) and model the entire soft sky component.

---------------------------------------------------------------------------
			LC_CLEAN

Since the background files are made using only the quiescent periods,
background flares should be excluded from the science data also. It is
important that it is done using exactly the same criteria (described below)
-- almost all background problems reported so far were due to the incomplete
removal of flare periods. This can be done by looking at the light curve of
the observation. The tool lc_clean makes a light curve for the selected
chips (or regions) and checks it for the presence of increased rate
intervals, as well as intervals with no data (sometimes not excluded from
the standard GTI, usually the very beginning or the end of the observation).
On output, it makes a GTI file with bad intervals excluded. This GTI file
should be applied to the data when extracting the spectra. The tool also
prints the clean exposure length. The program does not do anything to the
input event file(s).

The program reads from a parameter file; a documented example is in this
directory (lc_clean.par). The name of the par file can be given as a
command-line argument to lc_clean (if no filename is given, it is assumed to
be ./lc_clean.par in the current directory):

lc_clean par.file

The parameter gti_file should be set to the standard GTI file extracted from
the event file:

fextract evt2.fits"[gti]" gti.fits

Or you can simply specify the extension name (or, for the multi-GTI Chandra
event files, the ext. number you want) in the standard FITSIO manner in
lc_clean.par, e.g.:

gti_file=event.fits[7]

Make further edits in the par file and run lc_clean. It extracts the light
curve, bins it and calculates the average count rate (after whatever FITSIO
filtering you specified in the event filenames) over the standard GTI, using
N-sigma clipping (N is set in the par file). Then it excludes the bins above
or below that mean by a specified confidence level (in sigmas) or by a
specified factor, depending on what you set in the par file.

On output, in addition to the cleaned GTI file, the program creates ASCII
files with light curves before and after cleaning (in counts/s vs. time) to
be checked using your favorite viewing tool. I include an SM macro
lc_clean_sm; it plots the uncleaned light curve in red and then overplots
the cleaned curve in white.

For observations with moderate flares, the automated algorithm usually works
OK. However, if the observation has strong or prolonged flares, you will
likely find that the mean rate calculated by the program is not what you
want (what you want is the rather predictable quiescent rate, see the README
file with nominal rates in different bands) and the program excluded wrong
intervals. What I usually do in such cases is look at the light curve,
select a relatively quiescent period, and restrict the time in the input
event files to that period, for example:

file1=good_evt.fits[events][ccd_id==7&&time>51234500]

The program calculates the mean quiescent rate over that interval. I then
give this value in the parameter mean (unsetting the parameter clip to force
the program to use the predefined rate), remove the time restriction and
rerun the program. 

An even better way is to fix the mean rate to the nominal rate from the
corresponding background dataset (excluding from it the same source
regions), instead of letting lc_clean calculate the quiescent rate.

If there are strong sources in the field, you can supply a spatial filter to
the event file in the standard FITSIO manner, as shown above for time and
ccd_id filtering.

For best results, use the same cleaning setup as that used for making the
background files. The background observations were cleaned separately for BI
and FI chips, excluding ASCA grades 1,5,7, excluding hot pixels (see
below). The energy band to use is 0.3-12 kev for FI chips and 2.5-7 kev for
S3 chip (2.5-6 kev for S1). I used

binsize=259.28    for several FI chips and 0.3-12 kev band, or
binsize=1037.12   for S3 and 2.5-7 kev band, and

max_factor=1.2

The binsize is selected such that Poisson scatter is less than max_factor;
if you use a much smaller region for cleaning, increase the binsize. Note
that, unfortunately, we cannot use the convenient source-free energy band
above 10 keV for flare cleaning (as, e.g., XMM does), because the most
frequent flare species is much weaker at high energies (for S3, they are
most visible in the 3-6 keV interval due to the shapes of the quiescent and
flare spectra, while flares in FI chips are harder).

From the experience, it seems that the quiescent background rate is rather
predictable and constant apart from the slow secular change (see the above
background web page). Thus, if the final count rate in the source-free
regions of your image differs by more than 15% from that in the
corresponding region of the background file (appropriately corrected for the
secular change) then something is wrong. More than likely, the flares are
not cleaned correctly.  Set `mean' in lc_clean.par to the expected rate to
see if the observation suffers from a long faint flare. Because the flare
spectra are very different from the quiescent spectra, expect spectral
fitting problems if such flares are not excluded (e.g., astro-ph/0205333).

Flares in S1 and S3 have very close (almost identical) spectra.  So if your
source is in S3 and you cannot exclude the flare periods completely, and if
you have both S1 and S3, you can try modeling the residual flare flux using
chip S1 (use 'cutoffpl/b' XSPEC model) and use it for S3.  Be sure to use
the *_57_* datasets which are made from the same observations.  More details
on flare modeling are coming soon.


** New cleaning method as of ver. 061005: as described in astro-ph/0512542,
one can filter flares more aggressively by excluding time bins in which the
ratio of rates in the 2.5-7 and 9-12 kev is anomalous. If there is a 
parameter fileref* for each file*, e.g.,

file1=evt.fits[events][energy>2500&&energy<7000]
fileref1=evt.fits[events][energy>9000&&energy<12000]

then the ratio in these bands will be used instead of a rate in one band.
The quiescent bg spectral shape is constant (to 2-3%) even when the
normalization changes significantly, so even if observations of the same
field are months apart, one can use the same nominal value of the ratio for
all of them. Of course, for stat. reasons, this method requires much bigger
bins, 10-20 ks.

---------------------------------------------------------------------------
			MAKE_ACISBG

Note that there is now a CIAO tool 'reproject_events' that can do the same
thing (I haven't tried it yet but you are encouraged to), so eventually,
make_acisbg may disappear.

make_acisbg takes a background event file and fills its sky coordinate
columns X,Y with values relevant for a given observation. It does so by
assigning a random time value (from the GTI file) for each background photon
and applying the aspect offsets (from the given aspect file *aoff1*)
corresponding to this time.

To run this program, you need to have the standard CIAO setup. If you get an
error "PIXLIB cannot do something", make sure you are using the standard
released CIAO version and the standard setup.

make_acisbg reads from a parameter file; an example (make_acisbg.par) is in
this directory (or up 1 dir). The name of the par file can be given as a
command-line argument (if no filename is given, it is assumed to be
./make_acisbg.par in the current working directory):

make_acisbg par.file

In the par file, you need to specify the name of the event file from your
observation (the program only reads several header keywords from it), the
*aoff1* filename, a GTI file that you have used for data filtering, and a
copy of the background event file.  This latter file will be modified. The
background event files are found in the subdirectory ./data/ in this
directory.

#### If you have read this instruction before, please note that starting
from version 090201, you do not need to run asp_apply_sim on the *aoff1*
file anymore, because make_acisbg now does the equivalent calculations
internally. This fixes some coordinate problems that appeared after
switching to CIAO2. If make_acisbg detects that asp_apply_sim was run, it
will give an error message.

---------------------------------------------------------------------------
			BACKGROUND NORMALIZATION

After running make_acisbg, you may extract background spectra and images in
sky coordinates. These spectra and images can be normalized approximately by
the ratio of the exposures -- for the science data, the exposure calculated
by lc_clean, and for the background, the one written in the fits
headers. Note that the background exposures are different for different
chips, except for the multi-chip files.

However, the best way is to normalize by the ratio of the fluxes in some
high energy band free of sky emission (energy=9-12 keV, or PHA=2500-3000
ADU, for example). The background files for period A have an upper energy
cut at 10 keV, and the other files have upper PHA (rather than energy)
cutoff at 3250. See details in the README files in the data directory.

Once flares are properly removed, the background rate above 3-5 kev seems
predictable to about +-2% rms if normalized by the high-energy rate, while
the soft background is more variable due to the position- (and time!)
dependent sky component.

Starting from 2004, the particle background rate has been increasing (by
x1.3 during 2004), but its spectral shape has remained constant.

---------------------------------------------------------------------------
		RELEVANT DATA PROCESSING AND FILTERING ISSUES

		1. Gain, PI and energy columns

The PI and energy columns in the background files are calculated using the
following gain tables:

acisD1999-08-13gainN0002 for the -100C data (period A),
acisD1999-09-16gainN0005 for the -110C data (period B),
acisD2000-01-29gain_ctiN0003    for the -120C data (periods C,D),

If your data have been processed with an older gain file (check the keyword
GAINFILE in your fits headers), you may update the file using
acis_process_events:

acis_process_events infile=your_obsid_evt.fits outfile=out_evt.fits \
 acaofffile=NONE stop="none" doevtgrade=no apply_cti=no calculate_pi=yes\
 gainfile=your_caldb_dir/acisD1999-09-16gainN0005.fits

Similarly, you can update the background event files should a new gain file
be released (note that the file energy cuts will change slightly as a
result). After this, check that the GTI extension is intact as
acis_process_events behavior depends on the CIAO version.

               1a. CTI correction

Period C and D files for chips 01236 are CTI-corrected using
acis_process_events with options apply_cti=yes, doevtgrade=yes,
calculate_pi=yes, ctifile=acisD2000-01-29ctiN0002.fits (insert the full
directory paths). The older, uncorrected versions are no longer included.

For periods B,C,D chips 012367, there are alternative versions of the
datasets processed by the PSU CTI corrector (Townsley et al. 2000, ApJ 534,
L139) instead of acis_process_events.  They can be used for observations
processed similarly, together with the detector responses supplied with the
corrector. Be sure not to run acis_process_events or any other tools with
options that can change event PIs, energies or grades (see also notes in
README). In other respects, these event files are similar to others.

	       1b. Time-dependent gain

Starting from Aug 03, the background files are corrected for the
time-dependent gain (see
http://cxc.harvard.edu/contrib/alexey/tgain/tgain.html) using the
corresponding *t_gainN0002* CALDB files.  This correction is implemented in
CIAO as of version 3.2 (acis_process_events).

The PSU CTI corrector effectively corrects for this time-dependence as well
(although the results for chip I2 should be different).

		2. Aspect 

Because of the calibration updates, your observation may have been processed
with older (or newer) cal files than those read by make_acisbg from your
CIAO installation. If so, when you make an image from a science file and a
background file processed by make_acisbg, they may have a relative offset in
sky coords. To fix it, reprocess your science file using
acis_process_events:

acis_process_events infile=your_obsid_evt.fits outfile=output_evt.fits \
  acaofffile=your_obsid_asol1.fits \
  doevtgrade=no docentroid=no calculate_pi=no

(either *asol1* or *aoff1* files may be given to it; note that only *aoff1*
file may be given to make_acisbg). You can combine the above two
acis_process_events runs (the PI and the sky coords corrections) in one run.

If you have any problems with coordinates, make sure you have the latest
CALDB and CIAO (and make_acisbg v090201 or later) and reprocess the data. As
of this writing, no known Chandra coordinate problems remain.


		3. Bad pixels and columns

For periods C and D, the bad pixel tables used to create the background
datasets are identical to the tables to be included in the oncoming CALDB
release. For periods AB, I excluded more bad pixels and columns. The lists
of excluded pixels are in the ASCII ./data/badpix_[ABCD]_* files (see
badpix_README in that dir for a description). The AB list includes all
pixels and columns that I have ever noticed to be bad, disregarding the fact
that some have been bad only temporarily. The badpix_[CD]_* lists are made
in a more systematic way and include only the persistent bad (bright or
dark) pixels/columns for the respective periods. The user should check for
possible temporary hot pixels (due to occasional bias map defects) and
exclude them from the data in addition to the persistent bad pixels. It can
be done either by hand or by creating a small ASCII file with the new pixels
using a format similar to the badpix* file, and running a program
badpixfilter as shown below.

Note that I did not exclude pixels and columns *adjacent* to the genuine bad
pixels and columns, as is done for the archival Level 2 event files (note
that in VF mode evt2 files, TWO adjacent columns on each side of each bad
column have been excluded for some time, but no longer). You may exclude the
adjacent columns manually from the background files (keeping in mind that
event chip coords were randomized in these files, so just filtering by
status bit 5 won't work -- all status bits are simply reset to 0.) 

I recommend that you don't use evt2 files and instead create your own clean
file starting from evt1, keeping the adjacent pixels/columns in and not
filtering by status=0.  The only useful thing in evt2 files is the GTI
extension, which can be used as input for lc_clean.

##### NEW:
You can keep those adjacent pixels when following the CIAO thread on how to
create a new evt2 file. Instead of the status=0 filter, use the following
expression (or its analog for dmcopy) to retain events with the relevant
status bit (bit 5, counting from 0) set to 1:

fcopy jnk1"[events][status==b00000000000000000000000000x00000]" jnk2

Make sure to use the same bad pixel exclusion for CIAO exposure maps.  Use
option bpmask, as below (thanks to John Davis):

mkinstmap detsubsys="ACIS-4;bpmask=0x1F8FF" ....
#####

The total area of the detector covered by hot pixels is very small, so
different pixel filtering is not a concern, unless a bright hot pixel is
left in the user's data. Bad columns are more important and ideally, the
same bad pixel and column list should be applied to the data and the
background, and also used for making the exposure maps. I hope that files
badpix*.fits work with the standard exposure map generation tools. To apply
the ASCII badpix* lists to the event file, there is a tool badpixfilter (by
A. Vikhlinin):

Note that badpix_*_041104 tables cut off the chip edges (chipy=1-9) which
are affected by framestore vignetting. It is recommended that these tables
are applied to the science and background evt files and for the exposure map
calculation. However, a decision has been made to not filter out these
events by default, so the C,D background files are made with the *noedge
version of these tables, for consistency with CALDB. Thus, an extra
filtering step is required for these bg files.

badpixfilter in_evt.fits out_evt.fits ./badpix_C_041104

The binary is in the directories bin.*/ and the source in src/.


		4. Cosmic ray afterglows

Another species of "bad pixels", flickering pixels or cosmic ray afterglows
(generating up to 10-20 consecutive events in a single pixel), are excluded
from the B,C,D files (using a CIAO tool acis_detect_afterglow with default
settings). They add a negligible fraction to the total background
rate. These events can be excluded from the data by filling the respective
status bits and then filtering by those bits:

acis_detect_afterglow infile=in_evt.fits outfile=jnk
fcopy jnk"[events][status==bxxxxxxxxxxxx0000xxxxxxxxxxxxxxxx]" out_evt.fits

#### As of version 150606, I still have not switched to the new recommended
CIAO method of omitting acis_detect_afterglow and using acis_run_hotpix; but
it makes negligible difference as long as hot pixels are removed in both the
data and the background.


	        5. Streaks in chip S4 (and occasionally in other FI chips)

I ran 'destreak' (originally by John Houck, now part of CIAO) with default
settings on *ALL* FI chip data (as of ver. 200803 datasets) to remove those
bright soft streaks along chipx.  It removes most but not all of the streaks
in S4, so use S4 data below E<2 keV with care. Note that it should be run
with status set to 0, because the `destreak' result depends on the input
file and it does not consider events with status!=0 (e.g., if the VF status
bit was set). I simply rename the status column and create a fake status=0
column, run destreak, then rename the column back.

### NEW IN CIAO 3.3: you can use option mask="" instead, which tells
destreak to use events with all status values.


		6. VFAINT mode

For observations performed in VF mode, one can reduce the background by
applying the A. Vikhlinin's filter (see the ACIS cal page, section
"Background", then "VF mode cleaning"). acis_process_events now has an
option 'check_vf_pha=yes' emulating the Vikhlinin's algorithm and filling in
a certain status bit. The period D files are composed of VF mode
observations and have this bit filled. One can run acis_process_events with
this option on their data and then filter both the data and the background
files:

fcopy in_evt.fits"[events][status==bxxxxxxxx0xxxxxxxxxxxxxxxxxxxxxxx]" out_evt.fits

As of ver. 271103, there are no other nonzero status bits left in the bg
event files, so one can just filter by status==b0.

