Filtering Lightcurves
![[CXC Logo]](../../imgs/cxc-logo.gif){kind=logo}
CIAO 4.0 Science Threads
OverviewLast Update: 23 Jun 2008 - updated image display to place figures inline with text Synopsis: It may be necessary to filter your data to remove periods of anomalous background levels, such as the flares seen in ACIS observations. The CIAO package includes the dmextract tool to calculate the lightcurve of a dataset (or region within a dataset). Purpose: To analyze the output of dmextract in order to clean your dataset of periods of anomalous background rates. Read this thread if: you are working with an ACIS imaging observation and would like to check for unusual background rates. The algorithm used to detect flares is simple, so it may not be sufficient in all cases. If you intend to use the ACIS blank-sky background files, the cleaning described in this thread is not conservative enough. Instead, use the alternative method described in the ACIS "Blank-Sky" Background Files thread. Related Links:
|
Contents
- Get Started
- Remove bright/variable sources from the dataset
- Create the lightcurve (dmextract)
- Analyze the lightcurve (analyze_ltcrv.sl)
- Parameter files:
- History
- Images
Get Started
Sample ObsID used: 1712 (ACIS-S, 3C 273)
File types needed: evt2
This thread uses the analyze_ltcrv.sl script. The plotting capabilities of analyze_ltcrv.sl have not been put back in CIAO 4.0 yet. Run run this thread in CIAO 3.4 if you require plotting.
unix% grep Id $ASCDS_CONTRIB/share/slsh/local-packages/analyze_ltcrv.sl % Id: analyze_ltcrv.sl,v 1.6 2007/04/11 11:32:15 egalle Exp $
Please check that you are using the most recent version before continuing. If you do not have the script installed or need to update to a newer version, refer to the Scripts page.
For this thread we shall restrict attention to the ACIS-S3 chip, and the 0.5 to 7 keV energy range:
unix% punlearn dmcopy unix% dmcopy "acisf01712N003_evt2.fits[energy=500:7000,ccd_id=7]" evt2_c7.fits
An image of the data is shown in Figure 1.
![[Thumbnail image: bitmap image of ds9 display of ACIS-S3 observation of a field]](field.png)
[Version: full-size]
Figure 1: ACIS-S3 observation of a field
ACIS-S3 observation in the 0.5 to 7 keV energy range.
Remove bright/variable sources from the dataset
To avoid any background variations due to sources in the field we first filter out regions of high/variable emission from the dataset. There are a number of ways to create a list of regions that should be excluded - for instance you may wish to use the output from one of the source detection programs. In this thread we shall use visual estimation, as an example.
Using ds9, we chose several regions and saved them in CIAO format as exclude.reg, which looks like:
unix% cat exclude.reg # Region file format: CIAO version 1.0 rotbox(4200.3328,4137.9892,1129.5056,74.07019,24.22333) circle(4076.5,4088.5,316) circle(4296.5,5024.5,48)
The chosen regions are shown in Figure 2 (note that we ignored several obvious point sources in this example).
![[Thumbnail image: bitmap image of ds9 display of image with "source" regions]](sources.png)
[Version: full-size]
Figure 2: "Source" regions
For this example, the regions were chosen to contain most of the source emission.
Create the lightcurve (dmextract)
We can now create a lightcurve of this background region using the CIAO dmextract tool. The best way to bin the lightcurve depends on the dataset and your scientific objectives; for this example we use a bin length of 200 seconds:
unix% punlearn dmextract unix% pset dmextract "evt2_c7.fits[exclude sky=region(exclude.reg)][bin time=::200]" unix% pset dmextract outfile=lc_c7.fits unix% pset dmextract opt=ltc1 unix% dmextract Input event file (evt2_c7.fits[exclude sky=region(exclude.reg)][bin time=::200]): Enter output file name (lc_c7.fits):
The contents of the parameter file may be checked with plist dmextract.
Analyze the lightcurve (analyze_ltcrv.sl)
The analyze_ltcrv.sl script performs an iterative sigma-clipping algorithm, removing those points that fall outside +/-3 sigma from the mean at each iteration until all data points are within +/-3 sigma. This algorithm is robust but not perfect; it can easily "overclean" a noisy lightcurve and should not be used blindly. The output is a list of the accepted time periods and their lengths.
The script must be loaded into slsh before it can be run (this step is only necessary once per session):
unix% slsh
slsh version 0.8.2-0; S-Lang version: 2.1.3
Copyright (C) 2005-2007 John E. Davis <jed@jedsoft.org>
This is free software with ABSOLUTELY NO WARRANTY.
slsh> () = evalfile( "analyze_ltcrv.sl" );
slsh> analyze_ltcrv("lc_c7.fits");
((time > 77377570.949928) && (time < 77399570.949928)) ; 22.00 ksec
((time > 77404770.949928) && (time < 77406770.949928)) ; 2.00 ksec
slsh> quit;
The output time periods can be used to filter the event list, either by using dmgti to create a GTI file, or directly within a DM filter expression:
1. Using dmgti
The time limits can be used by dmgti to create a GTI file, as shown below:
unix% punlearn dmgti unix% pset dmgti infile=lc_c7.fits unix% pset dmgti outfile=lc_c7.gti unix% pset dmgti userlimit = \ "((time > 77377570.949928) && (time < 77399570.949928))||((time > 77404770.949928) && (time < 77406770.949928))" unix% dmgti Input MTL file (lc_c7.fits): Output GTI file (lc_c7.gti): User defined limit string (((time > 77377570.949928) && (time < 77399570.949928))||((time > 77404770.949928) && (time < 77406770.949928))
The contents of the parameter file may be checked with plist dmgti.
The output of dmgti, here lc_c7.gti, can then be used to filter an event list:
unix% dmcopy "evt2_c7.fits[@lc_c7.gti]" evt2_c7_clean1.fits unix% dmlist evt2_c7.fits subspace -------------------------------------------------------------------------------- Data subspace for block EVENTS: Components: 1 Descriptors: 15 -------------------------------------------------------------------------------- --- Component 1 --- 1 time Real8 TABLE GTI7 77379634.1499300152: 77407432.5499559045 2 ccd_id Int2 7:7 ... unix% dmlist evt2_c7_clean1.fits subspace -------------------------------------------------------------------------------- Data subspace for block EVENTS: Components: 1 Descriptors: 15 -------------------------------------------------------------------------------- --- Component 1 --- 1 time Real8 TABLE GTI7 77379634.1499300152: 77399470.9499280006 77404670.9499280006: 77406670.9499280006 2 ccd_id Int2 7:7 ...
2. Using a DM filter
Alternatively, you can use filter the event list directly:
unix% dmcopy \
"evt2_c7.fits[time=77377570.949648:77399570.949648,77404770.949648:77406770.949648]" \
evt2_c7_clean2.fits
unix% dmlist evt2_c7_clean2.fits subspace
--------------------------------------------------------------------------------
Data subspace for block EVENTS: Components: 1 Descriptors: 15
--------------------------------------------------------------------------------
--- Component 1 ---
1 time Real8 TABLE GTI7
77379634.1499300152: 77399570.9496479928
77404770.9496479928: 77406770.9496479928
2 ccd_id Int2 7:7
...
The two filtering methods produce slightly different results, which can be seen by looking at the cleaned exposure times:
unix% dmkeypar evt2_c7_clean1.fits EXPOSURE echo+ 21560.289287868 unix% dmkeypar evt2_c7_clean2.fits EXPOSURE echo+ 21659.022751194
The difference is due to the way the range endpoints are handled in the two cases. The dmgti call explicitly gives open intervals, e.g. (time > a) && (time < b), so the endpoints aren't included. Data Model range filters on real-valued columns include the lower bound but exclude the upper one, so time=a:b is interpreted as (time >= a) && (time < b). If you change the dmgti expression to include the lower bounds:
unix% pset dmgti \
userlimit="((time >= 77377570.949648) && (time < 77399570.949648))||((time >= 77404770.949648) && (time < 77406770.949648))"
then the results will be closer.
Parameters for /home/username/cxcds_param/dmextract.par
#--------------------------------------------------------------------
#
# DMEXTRACT -- extract columns or counts from an event list
#
#--------------------------------------------------------------------
infile = evt2_c7.fits[exclude sky=region(exclude.reg)][bin time=::200] Input event file
outfile = lc_c7.fits Enter output file name
(bkg = ) Background region file or fixed background (counts/pixel/s) subtraction
(error = gaussian) Method for error determination(poisson|gaussian|<variance file>)
(bkgerror = gaussian) Method for background error determination(poisson|gaussian|<variance file>)
(bkgnorm = 1.0) Background normalization
(exp = ) Exposure map image file
(bkgexp = ) Background exposure map image file
(sys_err = 0) Fixed systematic error value for SYS_ERR keyword
(opt = ltc1) Output file type: pha1
(defaults = ${ASCDS_CALIB}/cxo.mdb -> /soft/ciao/data/cxo.mdb) Instrument defaults file
(wmap = ) WMAP filter/binning (e.g. det=8 or default)
(clobber = no) OK to overwrite existing output file(s)?
(verbose = 0) Verbosity level
(mode = ql)
|
Parameters for /home/username/cxcds_param/dmgti.par
infile = lc_c7.fits Input MTL file
outfile = lc_c7.gti Output GTI file
userlimit = ((time > 77377570.949928) && (time <
77399570.949928))||((time > 77404770.949928) && (time <
77406770.949928)) User defined limit string
(mtlfile = none) Optional output smoothed/filtered MTL file
(lkupfile = none) Lookup table defining which MTL columns to check against (NONE|none|<filename>)
(smooth = yes) Smooth the input MTL data?
(clobber = no) Clobber output file if it exists?
(verbose = 0) Debug level
(mode = ql)
|
History
| 14 Dec 2004 | updated for CIAO 3.2: path for script |
| 08 Dec 2005 | updated for CIAO 3.3: default value of dmextract error and bkgerror parameters is "gaussian" |
| 01 Dec 2006 | updated for CIAO 3.4: dmgti uses the value of the TIMEPIXR header keyword to adjust start and stop times (users may see a small shift in the time filter when compared to CIAO 3.3 because of this); kernel parameter removed from dmgti; ChIPS version |
| 18 Jan 2008 | updated for CIAO 4.0: analyze_ltcrv.sl v1.6 (plotting routines have been removed from the script; plotting will be replaced once it is updated for the new ChIPS syntax); filename and screen output updated for reprocessed data (version N003 event file); slsh version 0.8.2-0/S-Lang version 2.1.3 |
| 06 May 2008 | changed energy filter to 0.5 to 7 keV (500:7000) |
| 23 Jun 2008 | updated image display to place figures inline with text |
