Maxim Markevitch , last change 8 Jul 07 This file describes what to do with the ACIS background datasets -- specifically, how to use lc_clean to exclude the background flare intervals, how to use make_acisbg to fill the sky coordinate columns in the background files, and how to process the data the same way as the background files. For general info about ACIS background, please see the calibration web page (http://cxc.harvard.edu/cal/, click "ACIS", then "Background", then "General discussion"). For a detailed description of how the event files were made, see the README file accompanying the files. Note for CfA HEAD users: the tools described here, with their parameter files, an SM macro, and this COOKBOOK file are also in the directory /home/maxim/, and the event files are in /home/maxim/DAT/AX/ACIS/BG/. --------------------------------------------------------------------------- SOFT GALACTIC FLUX AND NH All observations included in the B,C,D background datasets have high Galactic b, moderate to low NH (1-5e20), and low soft X-ray brightness in the RASS R4-R5 band images (see Snowden et al. 1997, http://www.xray.mpe.mpg.de/rosat/survey/sxrb/12/fits.html). (Note that prior to the 190702 version, the aciss_C files included pointings to the North Polar Spur, now excluded -- thanks to several people who pointed that out.) For period A, there isn't much choice and half of the total exposure is from observations with high NH. The notes below apply to the B,C,D datasets only. The Galactic contribution doesn't matter at E>2 keV or so, where the particle background dominates. If the soft band (E<1 kev or so) is critical for your analysis, you need to check the RASS flux at your sky position (see the above web page; or at CfA, see /home/maxim/DAT/ROS/SXRB) and compare it to that for observations included in the background dataset. The R4-R5 RASS fluxes are given in the README file accompanying the datasets. If they are very different from your observation, these files may not be useful. One can also use the outlying regions of the FOV for modeling the Galactic excess/deficiency in a given observation w.r.t. these background files (as we did, e.g., in ApJ 563, 95). For this, be sure to use the background datasets assembled using the same observations for different chips. A more radical way is to use the "stowed" background (which does not include any sky signal) and model the entire soft sky component. --------------------------------------------------------------------------- LC_CLEAN Since the background files are made using only the quiescent periods, background flares should be excluded from the science data also. It is important that it is done using exactly the same criteria (described below) -- almost all background problems reported so far were due to the incomplete removal of flare periods. This can be done by looking at the light curve of the observation. The tool lc_clean makes a light curve for the selected chips (or regions) and checks it for the presence of increased rate intervals, as well as intervals with no data (sometimes not excluded from the standard GTI, usually the very beginning or the end of the observation). On output, it makes a GTI file with bad intervals excluded. This GTI file should be applied to the data when extracting the spectra. The tool also prints the clean exposure length. The program does not do anything to the input event file(s). The program reads from a parameter file; a documented example is in this directory (lc_clean.par). The name of the par file can be given as a command-line argument to lc_clean (if no filename is given, it is assumed to be ./lc_clean.par in the current directory): lc_clean par.file The parameter gti_file should be set to the standard GTI file extracted from the event file: fextract evt2.fits"[gti]" gti.fits Or you can simply specify the extension name (or, for the multi-GTI Chandra event files, the ext. number you want) in the standard FITSIO manner in lc_clean.par, e.g.: gti_file=event.fits[7] Make further edits in the par file and run lc_clean. It extracts the light curve, bins it and calculates the average count rate (after whatever FITSIO filtering you specified in the event filenames) over the standard GTI, using N-sigma clipping (N is set in the par file). Then it excludes the bins above or below that mean by a specified confidence level (in sigmas) or by a specified factor, depending on what you set in the par file. On output, in addition to the cleaned GTI file, the program creates ASCII files with light curves before and after cleaning (in counts/s vs. time) to be checked using your favorite viewing tool. I include an SM macro lc_clean_sm; it plots the uncleaned light curve in red and then overplots the cleaned curve in white. For observations with moderate flares, the automated algorithm usually works OK. However, if the observation has strong or prolonged flares, you will likely find that the mean rate calculated by the program is not what you want (what you want is the rather predictable quiescent rate, see the README file with nominal rates in different bands) and the program excluded wrong intervals. What I usually do in such cases is look at the light curve, select a relatively quiescent period, and restrict the time in the input event files to that period, for example: file1=good_evt.fits[events][ccd_id==7&&time>51234500] The program calculates the mean quiescent rate over that interval. I then give this value in the parameter mean (unsetting the parameter clip to force the program to use the predefined rate), remove the time restriction and rerun the program. An even better way is to fix the mean rate to the nominal rate from the corresponding background dataset (excluding from it the same source regions), instead of letting lc_clean calculate the quiescent rate. If there are strong sources in the field, you can supply a spatial filter to the event file in the standard FITSIO manner, as shown above for time and ccd_id filtering. For best results, use the same cleaning setup as that used for making the background files. The background observations were cleaned separately for BI and FI chips, excluding ASCA grades 1,5,7, excluding hot pixels (see below). The energy band to use is 0.3-12 kev for FI chips and 2.5-7 kev for S3 chip (2.5-6 kev for S1). I used binsize=259.28 for several FI chips and 0.3-12 kev band, or binsize=1037.12 for S3 and 2.5-7 kev band, and max_factor=1.2 The binsize is selected such that Poisson scatter is less than max_factor; if you use a much smaller region for cleaning, increase the binsize. Note that, unfortunately, we cannot use the convenient source-free energy band above 10 keV for flare cleaning (as, e.g., XMM does), because the most frequent flare species is much weaker at high energies (for S3, they are most visible in the 3-6 keV interval due to the shapes of the quiescent and flare spectra, while flares in FI chips are harder). From the experience, it seems that the quiescent background rate is rather predictable and constant apart from the slow secular change (see the above background web page). Thus, if the final count rate in the source-free regions of your image differs by more than 15% from that in the corresponding region of the background file (appropriately corrected for the secular change) then something is wrong. More than likely, the flares are not cleaned correctly. Set `mean' in lc_clean.par to the expected rate to see if the observation suffers from a long faint flare. Because the flare spectra are very different from the quiescent spectra, expect spectral fitting problems if such flares are not excluded (e.g., astro-ph/0205333). Flares in S1 and S3 have very close (almost identical) spectra. So if your source is in S3 and you cannot exclude the flare periods completely, and if you have both S1 and S3, you can try modeling the residual flare flux using chip S1 (use 'cutoffpl/b' XSPEC model) and use it for S3. Be sure to use the *_57_* datasets which are made from the same observations. More details on flare modeling are coming soon. ** New cleaning method as of ver. 061005: as described in astro-ph/0512542, one can filter flares more aggressively by excluding time bins in which the ratio of rates in the 2.5-7 and 9-12 kev is anomalous. If there is a parameter fileref* for each file*, e.g., file1=evt.fits[events][energy>2500&&energy<7000] fileref1=evt.fits[events][energy>9000&&energy<12000] then the ratio in these bands will be used instead of a rate in one band. The quiescent bg spectral shape is constant (to 2-3%) even when the normalization changes significantly, so even if observations of the same field are months apart, one can use the same nominal value of the ratio for all of them. Of course, for stat. reasons, this method requires much bigger bins, 10-20 ks. ** New development (Jul 07): it appears that from mid-2005, the S3 spectrum has started to change shape, while FI chip shape still stays the same. So the above method is only applicable to FI chips. --------------------------------------------------------------------------- MAKE_ACISBG Note that there is now a CIAO tool 'reproject_events' (see http://cxc.harvard.edu/ciao/threads/acisbackground/) that can do the same thing (although slower at present), so eventually, make_acisbg may disappear. make_acisbg takes a background event file and fills its sky coordinate columns X,Y with values relevant for a given observation. It does so by assigning a random time value (from the GTI file) for each background photon and applying the aspect offsets (from the given aspect file *aoff1*) corresponding to this time. To run this program, you need to have the standard CIAO setup. If you get an error "PIXLIB cannot do something", make sure you are using the standard released CIAO version and the standard setup. make_acisbg reads from a parameter file; an example (make_acisbg.par) is in this directory (or up 1 dir). The name of the par file can be given as a command-line argument (if no filename is given, it is assumed to be ./make_acisbg.par in the current working directory): make_acisbg par.file In the par file, you need to specify the name of the event file from your observation (the program only reads several header keywords from it), the *aoff1* filename, a GTI file that you have used for data filtering, and a copy of the background event file. This latter file will be modified. The background event files are found in the subdirectory ./data/ in this directory. #### If you have read this instruction before, please note that starting from version 090201, you do not need to run asp_apply_sim on the *aoff1* file anymore, because make_acisbg now does the equivalent calculations internally. This fixes some coordinate problems that appeared after switching to CIAO2. If make_acisbg detects that asp_apply_sim was run, it will give an error message. --------------------------------------------------------------------------- BACKGROUND NORMALIZATION After running make_acisbg, you may extract background spectra and images in sky coordinates. These spectra and images can be normalized approximately by the ratio of the exposures -- for the science data, the exposure calculated by lc_clean, and for the background, the one written in the fits headers. Note that the background exposures are different for different chips, except for the multi-chip files. However, the best way is to normalize by the ratio of the fluxes in some high energy band free of sky emission (energy=9-12 keV, or PHA=2500-3000 ADU, for example). The background files for period A have an upper energy cut at 10 keV, and the other files have upper PHA (rather than energy) cutoff at 3250. See details in the README files in the data directory. Once flares are properly removed, the background rate above 3-5 kev seems predictable to about +-2% rms if normalized by the high-energy rate, while the soft background is more variable due to the position- (and time!) dependent sky component. Starting from 2004, the particle background rate has been increasing (by x1.3 during 2004), but its spectral shape has remained constant. *** NEW DEVELOPMENT (Jul 07): apparently, the spectral shape of the S3 background has started to change from mid-2005. As a result, by normalizing the current period D blank-sky background (which includes observations up to late 2004) for later observations using the 9-12 kev rate, one can get an error of up to 10% at the useful energies, in the sense that the background will be undersubtracted. This is under investigation (eventually, a new period E blank-sky dataset will be created). The FI background spectral shape is still constant to 2-3%. --------------------------------------------------------------------------- RELEVANT DATA PROCESSING AND FILTERING ISSUES 1. Gain, PI and energy columns The PI and energy columns in the background files are calculated using the following gain tables: acisD1999-08-13gainN0002 for the -100C data (period A), acisD1999-09-16gainN0005 for the -110C data (period B), acisD2000-01-29gain_ctiN0006 for the -120C data (periods C,D), If your data have been processed with an older gain file (check the keyword GAINFILE in your fits headers), you may update the file using acis_process_events: acis_process_events infile=your_obsid_evt.fits outfile=out_evt.fits \ acaofffile=NONE stop="none" doevtgrade=no apply_cti=no calculate_pi=yes\ gainfile=your_caldb_dir/acisD1999-09-16gainN0005.fits Similarly, you can update the background event files should a new gain file be released (note that the file energy cuts will change slightly as a result). After this, check that the GTI extension is intact as acis_process_events behavior depends on the CIAO version. 1a. CTI correction Period C and D files for chips 01236 are CTI-corrected using acis_process_events with options apply_cti=yes, doevtgrade=yes, calculate_pi=yes, ctifile=acisD2000-01-29ctiN0006.fits or 2002-08-01ctiN0006 (insert the full directory paths). This includes correction for both the FI and BI chips. The CTI-uncorrected versions for ACIS-S C,D datasets are also included (may be useful for Graded mode analysis). For periods B,C,D chips 012367, there are alternative versions of the datasets processed by the PSU CTI corrector (Townsley et al. 2000, ApJ 534, L139) instead of acis_process_events. They can be used for observations processed similarly, together with the detector responses supplied with the corrector. Be sure not to run acis_process_events or any other tools with options that can change event PIs, energies or grades (see also notes in README). In other respects, these event files are similar to others. 1b. Time-dependent gain The background files are corrected for the time-dependent gain (see http://cxc.harvard.edu/contrib/alexey/tgain/tgain.html) using the corresponding *t_gainN0006* CALDB files. This correction is implemented in CIAO as of version 3.2 (acis_process_events). The PSU CTI corrector effectively corrects for this time-dependence as well (although the results for chip I2 should be different). 2. Aspect Because of the calibration updates, your observation may have been processed with older (or newer) cal files than those read by make_acisbg from your CIAO installation. If so, when you make an image from a science file and a background file processed by make_acisbg, they may have a relative offset in sky coords. To fix it, reprocess your science file using acis_process_events: acis_process_events infile=your_obsid_evt.fits outfile=output_evt.fits \ acaofffile=your_obsid_asol1.fits \ doevtgrade=no docentroid=no calculate_pi=no (either *asol1* or *aoff1* files may be given to it; note that only *aoff1* file may be given to make_acisbg). You can combine the above two acis_process_events runs (the PI and the sky coords corrections) in one run. If you have any problems with coordinates, make sure you have the latest CALDB and CIAO (and make_acisbg v090201 or later) and reprocess the data. As of this writing, no known Chandra coordinate problems remain. 3. Bad pixels and columns For periods C and D, the bad pixel tables used to create the background datasets are identical to the tables to be included in the oncoming CALDB release. For periods AB, I excluded more bad pixels and columns. The lists of excluded pixels are in the ASCII ./data/badpix_[ABCD]_* files (see badpix_README in that dir for a description). The AB list includes all pixels and columns that I have ever noticed to be bad, disregarding the fact that some have been bad only temporarily. The badpix_[CD]_* lists are made in a more systematic way and include only the persistent bad (bright or dark) pixels/columns for the respective periods. The user should check for possible temporary hot pixels (due to occasional bias map defects) and exclude them from the data in addition to the persistent bad pixels. It can be done either by hand or by creating a small ASCII file with the new pixels using a format similar to the badpix* file, and running a program badpixfilter as shown below. Note that I did not exclude pixels and columns *adjacent* to the genuine bad pixels and columns, as is done for the archival Level 2 event files (note that in VF mode evt2 files, TWO adjacent columns on each side of each bad column have been excluded for some time, but no longer). You may exclude the adjacent columns manually from the background files (keeping in mind that event chip coords were randomized in these files, so just filtering by status bit 5 won't work -- all status bits are simply reset to 0.) I recommend that you don't use evt2 files and instead create your own clean file starting from evt1, keeping the adjacent pixels/columns in and not filteting by status=0. The only useful thing in evt2 files is the GTI extension, which can be used as input for lc_clean. ##### NEW: You can keep those adjacent pixels when following the CIAO thread on how to create a new evt2 file. Instead of the status=0 filter, use the following expression (or its analog for dmcopy) to retain events with the relevant status bit (bit 5, counting from 0) set to 1: fcopy jnk1"[events][status==b00000000000000000000000000x00000]" jnk2 Make sure to use the same bad pixel exclusion for CIAO exposure maps. Use option bpmask, as below (thanks to John Davis): mkinstmap detsubsys="ACIS-4;bpmask=0x1F8FF" .... ##### The total area of the detector covered by hot pixels is very small, so different pixel filtering is not a concern, unless a bright hot pixel is left in the user's data. Bad columns are more important and ideally, the same bad pixel and column list should be applied to the data and the background, and also used for making the exposure maps. I hope that files badpix*.fits work with the standard exposure map generation tools. To apply the ASCII badpix* lists to the event file, there is a tool badpixfilter (by A. Vikhlinin): Note that badpix_*_041104 tables cut off the chip edges (chipy=1-9) which are affected by framestore vignetting. It is recommended that these tables are applied to the science and background evt files and for the exposure map calculation. However, a decision has been made to not filter out these events by default, so the C,D background files are made with the *noedge version of these tables, for consistency with CALDB. Thus, an extra filtering step is required for these bg files. badpixfilter in_evt.fits out_evt.fits ./badpix_C_041104 The binary is in the directories bin.*/ and the source in src/. 4. Cosmic ray afterglows Another species of "bad pixels", flickering pixels or cosmic ray afterglows (generating up to 10-20 consecutive events in a single pixel), are excluded from the B,C,D files (using a CIAO tool acis_detect_afterglow with default settings). They add a negligible fraction to the total background rate. These events can be excluded from the data by filling the respective status bits and then filtering by those bits: acis_detect_afterglow infile=in_evt.fits outfile=jnk fcopy jnk"[events][status==bxxxxxxxxxxxx0000xxxxxxxxxxxxxxxx]" out_evt.fits #### As of version 150606, I still have not switched to the new recommended CIAO method of omitting acis_detect_afterglow and using acis_run_hotpix; but it makes negligible difference as long as hot pixels are removed in both the data and the background. 5. Streaks in chip S4 (and occasionally in other FI chips) I ran 'destreak' (oroginally by John Houck, now part of CIAO) with default settings on *ALL* FI chip data (as of ver. 200803 datasets) to remove those bright soft streaks along chipx. It removes most but not all of the streaks in S4, so use S4 data below E<2 keV with care. Note that it should be run with status set to 0, because the `destreak' result depends on the input file and it does not consider events with status!=0 (e.g., if the VF status bit was set). I simply rename the status column and create a fake status=0 column, run destreak, then rename the column back. ### NEW IN CIAO 3.3: you can use option mask="" instead, which tells destreak to use events with all status values. 6. VFAINT mode For observations performed in VF mode, one can reduce the background by applying the A. Vikhlinin's filter (see the ACIS cal page, section "Background", then "VF mode cleaning"). acis_process_events now has an option 'check_vf_pha=yes' emulating the Vikhlinin's algorithm and filling in a certain status bit. The period D files are composed of VF mode observations and have this bit filled. One can run acis_process_events with this option on their data and then filter both the data and the background files: fcopy in_evt.fits"[events][status==bxxxxxxxx0xxxxxxxxxxxxxxxxxxxxxxx]" out_evt.fits As of ver. 271103, there are no other nonzero status bits left in the bg event files, so one can just filter by status==b0.