CIAO 4.3 Release Notes
![[New]](../imgs/new.gif)
CALDB
4.4.6.1 was released on 02 December 2011.
Notable changes and improvements in CIAO 4.3:
-
The EDSER sub-pixel algorithm has been implemented as the default in acis_process_events, replacing the +/-0.5 pixel randomization. A temperature-dependent CTI calibration is available for observations taken at the -120 C focal plane temperature, and parallel CTI adjustment has been added for the front-illuminated CCDs of GRADED mode observations. New parameters have been added to acis_process_events and chandra_repro to support these features.
-
Two new tools for use when creating ARF files: arfcorr (correct an ARF for the finite extraction region) and sky2tdet (improved weighting algorithm for spatial variations in the ARF). Both are discussed in the How CIAO 4.3 and CALDB 4.4.1 Affect Your Analysis section of these release notes.
-
A new tool, addresp, adds together multiple imaging ARFs and RMFs to create a pair of merged ARF and RMF files. The tool is used by the combine_spectra script which replaces the deprecated acisspec script. acisspec has been removed from the contributed tarfile.
-
The Sherpa modeling and fitting package has a number of enhancements, such as support for model expressions with different type of instruments and combinations of convolved and non-convolved model components, caching of model parameters, improved support for multi-core processing, new iterative fitting methods and many new high level user interface functions.
-
The ChIPS plotting application includes support for creating axes with WCS meta data associated with them. There are new commands for panning and zooming in plots, as well as improved image support and many enhancements and bug fixes.
-
The Data Model supports TSV format ASCII files, including the extended header detail provided by the Chandra Source Catalog (CSC) output format.
-
Support for the S-Lang scripting language has been removed from ChIPS and the CIAO tools and modules. S-Lang was removed from Sherpa in the CIAO 4.2 Sherpa v2 release (July 2010).
- Software Patch: version 2 of the tools package
- How CALDB 4.4.6.1 Affects Your Analysis
- How CALDB 4.4.6 Affects Your Analysis
- How CALDB 4.4.5 Affects Your Analysis
- How CALDB 4.4.3 Affects Your Analysis
- How CALDB 4.4.2 Affects Your Analysis
- How CIAO 4.3 and CALDB 4.4.1 Affect Your Analysis
- Installation
- Tools
- Parameter Files
- Data Model
- ChIPS
- Sherpa
- GUIs (Graphical User Interfaces)
- Libraries
- Environment
- Analysis Scripts
- Documentation
Software Patch: version 2 of the tools package
mkrmf bug fix
-
Version 2 of the tools package for CIAO 4.3 patches the tool mkrmf to correct the bug which resulted in corrupted CHANNEL data in the EBOUNDS extension. Information on which observations were affected is available in the bug report.
No other CIAO tools have changed in this release.
Users who ran mkrmf in CIAO 4.3 prior to this patch should remake the response file to get the correct CHANNEL data. Use dmhistory to extract the command from the file header:
unix% dmhistory 3c273.rmf mkrmf mkrmf infile="/soft/ciao-4.3/CALDB/data/chandra/acis/fef_pha/acisD1999-09-16fef_phaN0002.fits [FUNCTION][ccd_id=7,chipx=257:288,chipy=353:384]" outfile="3c273.rmf" axis1="energy=0.1:11.0:0.01" axis2="pi=1:1024:1" logfile="./.psp.mlog" weights="" thresh="1e-05" outfmt="legacy" clobber="no" verbose="2" axis3="none" axis4="none" axis5="none"
How CALDB 4.4.6.1 Affects Your Analysis
ACIS Grating Data
-
HETG Grating Efficiency (version N0007)
The cross-calibration of the first-order HEG and MEG portions of the HETG in the grating efficiency file (GREFF) has been adjusted via analysis of 30 Blazar observations. The new file (N0007) will be used by default when creating HETG response files (ARFs). The CIAO tools and scripts which use this file are:
For instructions on how to continue using the vN0006 file, read the HETG Grating Efficiency why topic.
How CALDB 4.4.6 Affects Your Analysis
ACIS Imaging and Grating Data
-
Time-dependent ACIS Gain (TGAIN) Files for -120 C Data
There are new time-dependent ACIS Gain (TGAIN) files for 01 May 2011 - 31 July 2011 (Epoch 46):
acisD2011-02-01t_gainN0005.fits acisD2011-02-01t_gainN0006.fits acisD2011-05-01t_gainN0002.fits acisD2011-05-01t_gain_biN0002.fits
The files are applicable to -120 C focal plane temperature only. The acisD2011-05-01t_gain_biN0002.fits file is the new default processing file in the CALDB.
Users working with ACIS data taken during this period may wish to reprocess to improve the TGAIN calibration in their data. The DATE-OBS header keyword records the observation start date.
Note that unless you are fitting a spectra with oxygen emission lines, the gain refinement is unlikely to have an effect on the spectrum larger than the uncertainties in determining the gain.
-
Temperature Range Change for -120 C Data
This change only affects observations with a focal plane temperature between 158.2 K and 159.16 K.
The temperature range in the detector gain (DET_GAIN), the mkacisrmf calibration (P2_RESP), the QEU, mkrmf calibration (FEF_PHA), and the order-sorting (OSIP) files has been updated to match the range specified in the temperature-dependent CTI calibration files. The updated range is 151.16 K (-122 C) to 159.16 K (-114 C).
For consistency, the relevant DET_GAIN, OSIP, and FEF_PHA files for NON-CTI-corrected data have also been updated.
Prior to this update, any tool trying to retrieve one of the named calibration files would fail due to the focal plane temperature being outside of the allowed CALDB range.
How CALDB 4.4.5 Affects Your Analysis
ACIS Imaging Data
-
Time-dependent ACIS Gain (TGAIN) Files for -120 C Data
There are new time-dependent ACIS Gain (TGAIN) files for 01 February 2011 - 30 April 2011 (Epoch 45):
acisD2010-11-01t_gainN0005.fits acisD2010-11-01t_gainN0006.fits acisD2011-02-01t_gainN0002.fits acisD2011-02-01t_gain_biN0002.fits
The files are applicable to -120 C focal plane temperature only. The acisD2011-02-01t_gain_biN0002.fits file is the new default processing file in the CALDB.
Users working with ACIS data taken during this period may wish to reprocess to improve the TGAIN calibration in their data. The DATE-OBS header keyword records the observation start date.
Note that unless you are fitting a spectra with oxygen emission lines, the gain refinement is unlikely to have an effect on the spectrum larger than the uncertainties in determining the gain.
HRC-S Imaging and Grating Data
-
QEU Upgraded to vN0006
Version N0006 of the HRC-S QEU takes into account the very slow QE degradation with time. The new file will be used by default when creating exposure maps or grating ARFs in CIAO.
The new QEU fixes the time-dependence in the HRC-S for all observational periods from 1999 through the present. Since the changes are normalized to be zero at 2008, the gARFs and exposure maps will not change for observations with DATE-OBS between 2008-01-01T00:00:00 and 2009-01-01T00:00:00.
All other response will be slightly modified, by about half a percent per year difference from mid-year 2008.
How CALDB 4.4.3 Affects Your Analysis
ACIS Imaging Data
-
Time-dependent ACIS Gain (TGAIN) Files for -120 C Data
There are new time-dependent ACIS Gain (TGAIN) files for 01 November 2010 - 31 January 2011 (Epoch 44):
acisD2010-08-01t_gainN0005.fits acisD2010-08-01t_gainN0006.fits acisD2010-11-01t_gainN0002.fits acisD2010-11-01t_gain_biN0002.fits
The files are applicable to -120 C focal plane temperature only. The acisD2010-11-01t_gain_biN0002.fits file is the new default processing file in the CALDB.
Users working with ACIS data taken during this period may wish to reprocess to improve the TGAIN calibration in their data. The DATE-OBS header keyword records the observation start date.
Note that unless you are fitting a spectra with oxygen emission lines, the gain refinement is unlikely to have an effect on the spectrum larger than the uncertainties in determining the gain.
-
Correction to Epoch 42 TGAIN Files
There was an error in the time-dependent gain correction for Epoch 42 which affected observations taken from 01 May 2010 - 31 July 2010. If you reprocessed one of these datasets in CIAO 4.3 with CALDB 4.4.1 or 4.4.2, the TGAIN correction was incorrect. The observation date is recorded in the DATE-OBS header keyword.
While the error is less than 0.5% for most photon energies, it is a systematic error that gives an absolute miscalculation of the PHA for a given position.
Reprocess the data with CALDB 4.4.3 to apply the correct TGAIN correction to the data.
ACIS Grating Data
-
Time-dependent ACIS Gain (TGAIN) Files for -120 C Data
There are new time-dependent ACIS Gain (TGAIN) files for 01 November 2010 - 31 January 2011 (Epoch 44):
acisD2010-08-01t_gainN0005.fits acisD2010-08-01t_gainN0006.fits acisD2010-11-01t_gainN0002.fits acisD2010-11-01t_gain_biN0002.fits
The files are applicable to -120 C focal plane temperature only. The acisD2010-11-01t_gain_biN0002.fits file is the new default processing file in the CALDB.
Users working with ACIS data taken during this period may wish to reprocess to improve the TGAIN calibration in their data. The DATE-OBS header keyword records the observation start date.
Note that unless you are fitting a spectra with oxygen emission lines, the gain refinement is unlikely to have an effect on the spectrum larger than the uncertainties in determining the gain.
-
Correction to Epoch 42 TGAIN Files
There was an error in the time-dependent gain correction for Epoch 42 which affected observations taken from 01 May 2010 - 31 July 2010. If you reprocessed one of these datasets in CIAO 4.3 with CALDB 4.4.1 or 4.4.2, the TGAIN correction was incorrect. The observation date is recorded in the DATE-OBS header keyword.
While the error is less than 0.5% for most photon energies, it is a systematic error that gives an absolute miscalculation of the PHA for a given position.
Reprocess the data with CALDB 4.4.3 to apply the correct TGAIN correction to the data.
-
LETG Grating Efficiency (version N0007)
The LETG grating efficiency file (GREFF) was updated to be version N0007. The new file will be used by default when creating LETG response files (ARFs). The CIAO tools and scripts which use this file are:
There is no change in the responses for +/1 order. The revised effective area curves (ARF files) for orders 2-7 will be lower than those created with the previous LETG GREFF calibration. Refer to the LETG Higher Order Efficiencies calibration page for technical details.
HRC-S/LETG Grating Data
-
LETG Grating Efficiency (version N0007)
The LETG grating efficiency file (GREFF) was updated to be version N0007. The new file will be used by default when creating LETG response files (ARFs). The CIAO tools and scripts which use this file are:
There is no change in the responses for +/1 order. The revised effective area curves (ARF files) for orders 2-7 will be lower than those created with the previous LETG GREFF calibration. Refer to the LETG Higher Order Efficiencies calibration page for technical details.
How CALDB 4.4.2 Affects Your Analysis
ACIS Imaging & Grating Data
-
A correction was made to the header of the vN0006 ACIS QE contamination model, which was released in CALDB 4.4.1. Prior to this release, CIAO would fail when trying to look up the contamination model correction for chips ACIS-8 (S4) and ACIS-9 (S5).
How CIAO 4.3 and CALDB 4.4.1 Affect Your Analysis
Some tool changes and calibration files released in CIAO 4.3 and CALDB 4.4.1 affect analyses in progress. This section of the release notes describes those changes and which tasks are affected.
Compatibility
-
CALDB 4.4.1 is required in order for CIAO 4.3 to work correctly.
ACIS Imaging Data
-
ACIS Sub-pixel Event Resolution
For sources near the optical axis of the telescope, the size of the point spread function is smaller than the size of the ACIS pixels. Li et al. (2004, ApJ, 610, 1204) describe various sub-pixel event-repositioning algorithms that can be used to improve the image quality of ACIS data for such sources.
As of CIAO 4.3, the default pixel adjustment in acis_process_events is the sub-pixel event-repositioning algorithm "EDSER" described in Li et al. Refer to the description of the parameter pix_adj for more details.
Most users will not notice a difference in the data with the EDSER sub-pixel resolution applied. The exception is users working with high-resolution (< 1 arcsec) data on-axis. The ACIS Sub-Pixel Event Repositioning why topic has more details.
-
Time-dependent ACIS Gain (TGAIN) Files for -120 C Data
There are new time-dependent ACIS Gain (TGAIN) files for 01 August 2010 - 31 October 2010 (Epoch 43):
acisD2010-05-01t_gainN0005.fits acisD2010-05-01t_gainN0006.fits acisD2010-08-01t_gain_biN0002.fits acisD2010-08-01t_gainN0002.fits
The files are applicable to -120 C focal plane temperature only. The acisD2010-08-01t_gainN0002.fits file is the new default processing file in the CALDB.
Users working with ACIS data taken during this period may wish to reprocess to improve the TGAIN calibration in their data. The DATE-OBS header keyword records the observation start date.
Note that unless you are fitting a spectra with oxygen emission lines, the gain refinement is unlikely to have an effect on the spectrum larger than the uncertainties in determining the gain.
-
Temperature-dependent ACIS CTI for -120 C Data
A temperature-dependence has been added to the ACIS CTI correction for observations taken at the -120 C focal plane temperature. In order to apply the temperature-dependent correction, the mission timeline (MTL) file must be supplied to acis_process_events in the new mtlfile parameter.
Introduction of a temperature dependence to the CTI adjustment involves the inclusion of scaling factors for each trap-density map. As the temperature increases (relative to -119.7 deg C), the number of traps is expected to increase. The percentage increase varies from one CCD to another. Typical values are about 2.6% per deg C for the parallel CTI on front-illuminated (FI) CCDs and about 1% per deg C for the serial CTI on back-illuminated (BI) CCDs.
For OBS_ID 57036, which has a mean temperature of -114.0 deg C (i.e. a change of +5.7 deg C), the values of PHA, PI, and ENERGY increase by an energy-dependent amount that is no larger than about 15% at energies below 2 keV and that is no larger than about 5% at the highest energies. Note that this is one of the warmest Chandra observations; other observations will see a smaller percentage change.
Users working with ACIS data taken at this focal plane temperature may wish to reprocess to improve the TGAIN calibration in their data. The FP_TEMP header keyword records the observation focal plane temperature.
Note: there is an interval from September 16, 2005 to October 16, 2005 during which the temperatures in the MTL files are 1.3 deg C higher than the real temperature. As a result, the temperature-dependent CTI adjustment would not be accurate. A workaround has been implemented in the CTI CALDB file so that the reference temperature is 154.75 K instead of the usual 153.45 K. In the future, the MTL files will be corrected instead of applying this workaround.
-
GRADED mode CTI
It is now possible to apply a parallel CTI adjustment to events on front-illuminated CCDs for GRADED mode observations. No adjustments are performed for GRADED mode events on back-illuminated CCDs.
Users working with ACIS data taken in GRADED mode to reprocess to improve the CTI calibration in their data. The DATAMODE header keyword records the observation mode.
-
Creating ARF Files
-
Running mkarf: pointlike sources
arfcorr calculates the approximate fraction of the point spread function (PSF) enclosed by a region. The tool then applies an energy-dependent correction to the ARF file.
When should arfcorr be used? An aperture correction should be applied to the ARF unless the source is extended compared to the local PSF; however if the source extraction region is much larger than the the local PSF, then the correction is unity at all energies and may be omitted. That is:
pointlike source on-axis: create the response with mkarf, then correct it with arfcorr
extended source: create the response with mkwarf, do not apply arfcorr. Note that even for a small extraction region covering a small fraction of an extended source, an aperture correction is unsuitable because the flux lost from the region due to PSF spillover is compensated by flux spilling into the region from the rest of the source.
pointlike source off-axis: create the response with mkwarf, then correct it with arfcorr (you may wish to experiment as the correction may be omitted as negligible if the extraction region is sufficiently large). Note that use of mkwarf rather than mkarf is recommended to average the detector response since the PSF is spread over many pixels.
arfcorr should not be applied to background ARFs, since the background is an extended component.
-
Running mkwarf: weighted responses
The new tool, sky2tdet, should be used to create the WMAP for input to mkwarf. The mkwarf tool has been updated to look for the "FRACCTS" keyword in the WMAP (written by sky2tdet), which gives the overall normalization for fraction of the counts that land on chip. If the keyword is not present, it uses a value of "1.0".
The WMAP created by sky2tdet properly weights the ARF based on how much of the source flux fell onto the bad pixels, columns, or a node boundary and which bad pixels are actually exposed. Without accounting for these effects, the ARF is significantly over-estimated.
-
ACIS QE Contamination Model vN0006
The ACIS QE contamination model has been upgraded to vN0006. This version of the file contains improvements to the ACIS-I contamination correction. The ACIS-S portion of the file has been interpolated to match the time range of the new ACIS-I model, but is otherwise unchanged.
These CIAO response tools automatically apply the contamination file when creating ACIS response files:
As well as the scripts which use them:
- specextract (calls mkwarf)
- psextract (calls mkarf)
- fullgarf (calls mkgarf)
- fluximage (calls mkinstmap)
- merge_all
-
-
Running the specextract Script
There were several enhancements to the specextract script, which resulted in a few changes in syntax.
![[Updated]](../imgs/updated.gif)
(22 Dec 2010) A number of
bug fixes were released for specextract; download
the updated version from the Scripts page.
Extended sources (weighted response files)
-
The default value of the new weight parameter - yes - tells the script to use mkwarf to make the ARF file(s). This is appropriate for extended sources.
-
The sky2tdet tool, new in CIAO 4.3, has been added to specextract to create a more accurate WMAP file for use with mkwarf; it's automatically run when weight = yes. This WMAP includes which bad pixels are within the regions and how much of the source flux falls onto those bad pixels.
In order to run sky2tdet, specextract requires an aspect histogram file to be input to the asp parameter (created by the asphist tool). A future release of specextract will take the aspect solution files directly and make the aspect histogram for you.
-
There is also a new mskfile parameter; the mask file is used as input to mkwarf.
A comparison of the CIAO 4.2 and CIAO 4.3 syntax:
CIAO 4.2 specextract infile="evt2.fits[sky=region(src.reg)]" bkgfile="evt2.fits[sky=region(bg.reg)]" \ outroot=extended pbkfile=pbk0.fits grouptype=NUM_CTS binspec=15 CIAO 4.3 specextract infile="evt2.fits[sky=region(src.reg)]" bkgfile="evt2.fits[sky=region(bg.reg)]" \ outroot=extended pbkfile=pbk0.fits asp=extended.asphist mskfile=msk1.fits \ grouptype=NUM_CTS binspec=15
Pointlike sources
The specextract script has been enhanced to be able to created response files for pointlike sources, i.e. to use the tool mkarf to create the ARF file. specextract may now be used in place of the psextract, which will be deprecated in the future.
-
If the new weight parameter is set to no, specextract uses mkarf to make the ARF file(s). This is appropriate for pointlike sources.
In order to run mkarf, specextract requires an aspect histogram file to be input to the asp parameter (created by the asphist tool). A future release of specextract will take the aspect solution files directly and make the aspect histogram for you.
The mskfile parameter is also used as input to mkarf.
-
The arfcorr tool, new in CIAO 4.3, has been added to specextract. If correct=yes, arfcorr is run to apply an energy-dependent point-source aperture correction to the ARF.
A comparison of the psextract and specextract syntax:
psextract events="evt2.fits[sky=region(src.reg)]" bgevents="evt2.fits[sky=region(bg.reg)]" \ root=point asol=@pcad_asol1.lis pbkfile=pbk0.fits \ gtype=NUM_CTS gspec=15 specextract infile="evt2.fits[sky=region(src.reg)]" bkgfile="evt2.fits[sky=region(bg.reg)]" \ outroot=point asp=point.asphist pbkfile=pbk0.fits mskfile=msk1.fits \ weight=no correct=yes \ grouptype=NUM_CTS binspec=15
Merging spectra
An option for combining stacks of output spectra and associated responses was added to the script. Setting combine=yes runs the combine_spectra script after the individual spectra and responses are created.
-
-
Merging ACIS Imaging Spectra
The combine_spectra script sums multiple imaging source PHA spectra, and the associated background PHA spectra and ARF files. combine_spectra should be used in combination with the specextract script in place of acisspec. Refer to the combine_spectra help file for examples of use.
The acisspec script has been removed from the CIAO 4.3 contributed tarfile. Contact CXC Helpdesk if you need assistance transitioning from acisspec syntax to combine_spectra.
ACIS Grating Data
-
ACIS Sub-pixel Event Resolution
For sources near the optical axis of the telescope, the size of the point spread function is smaller than the size of the ACIS pixels. Li et al. (2004, ApJ, 610, 1204) describe various sub-pixel event-repositioning algorithms that can be used to improve the image quality of ACIS data for such sources.
As of CIAO 4.3, the default pixel adjustment in acis_process_events is the sub-pixel event-repositioning algorithm "EDSER" described in Li et al. Refer to the description of the parameter pix_adj for more details.
Most users will not notice a difference in the data with the EDSER sub-pixel resolution applied. The exception is users working with high-resolution (< 1 arcsec) data on-axis. The ACIS Sub-Pixel Event Repositioning why topic has more details.
-
Time-dependent ACIS Gain (TGAIN) Files for -120 C Data
There are new time-dependent ACIS Gain (TGAIN) files for 01 August 2010 - 31 October 2010 (Epoch 43):
acisD2010-05-01t_gainN0005.fits acisD2010-05-01t_gainN0006.fits acisD2010-08-01t_gain_biN0002.fits acisD2010-08-01t_gainN0002.fits
The files are applicable to -120 C focal plane temperature only. The acisD2010-08-01t_gainN0002.fits file is the new default processing file in the CALDB.
Users working with ACIS data taken during this period may wish to reprocess to improve the TGAIN calibration in their data. The DATE-OBS header keyword records the observation start date.
Note that unless you are fitting a spectra with oxygen emission lines, the gain refinement is unlikely to have an effect on the spectrum larger than the uncertainties in determining the gain.
-
Temperature-dependent ACIS CTI for -120 C Data
A temperature-dependence has been added to the ACIS CTI correction for observations taken at the -120 C focal plane temperature. In order to apply the temperature-dependent correction, the mission timeline (MTL) file must be supplied to acis_process_events in the new mtlfile parameter.
Introduction of a temperature dependence to the CTI adjustment involves the inclusion of scaling factors for each trap-density map. As the temperature increases (relative to -119.7 deg C), the number of traps is expected to increase. The percentage increase varies from one CCD to another. Typical values are about 2.6% per deg C for the parallel CTI on front-illuminated (FI) CCDs and about 1% per deg C for the serial CTI on back-illuminated (BI) CCDs.
For OBS_ID 57036, which has a mean temperature of -114.0 deg C (i.e. a change of +5.7 deg C), the values of PHA, PI, and ENERGY increase by an energy-dependent amount that is no larger than about 15% at energies below 2 keV and that is no larger than about 5% at the highest energies. Note that this is one of the warmest Chandra observations; other observations will see a smaller percentage change.
Users working with ACIS data taken at this focal plane temperature may wish to reprocess to improve the TGAIN calibration in their data. The FP_TEMP header keyword records the observation focal plane temperature.
Note: there is an interval from September 16, 2005 to October 16, 2005 during which the temperatures in the MTL files are 1.3 deg C higher than the real temperature. As a result, the temperature-dependent CTI adjustment would not be accurate. A workaround has been implemented in the CTI CALDB file so that the reference temperature is 154.75 K instead of the usual 153.45 K. In the future, the MTL files will be corrected instead of applying this workaround.
-
GRADED mode CTI
It is now possible to apply a parallel CTI adjustment to events on front-illuminated CCDs for GRADED mode observations. No adjustments are performed for GRADED mode events on back-illuminated CCDs.
Users working with ACIS data taken in GRADED mode to reprocess to improve the CTI calibration in their data. The DATAMODE header keyword records the observation mode.
-
ACIS QE Contamination Model vN0006
The ACIS QE contamination model has been upgraded to vN0006. This version of the file contains improvements to the ACIS-I contamination correction. The ACIS-S portion of the file has been interpolated to match the time range of the new ACIS-I model, but is otherwise unchanged.
These CIAO response tools automatically apply the contamination file when creating ACIS response files:
As well as the scripts which use them:
- specextract (calls mkwarf)
- psextract (calls mkarf)
- fullgarf (calls mkgarf)
- fluximage (calls mkinstmap)
- merge_all
-
Correction to the ACIS-S/LETG Grating Angle & New LSFPARM File
A new PIXLIB geometry file has been released to resolve a long-standing problem with the geometry of the LETG/ACIS-S configuration relating to the grating dispersion angle. A related change was made in the PIXLIB sky file, which defines the grating dispersion plane. The solution is not a physical correction of the configuration, but an adaptation of the current setup to allow a more accurate derivation of LETG/ACIS-S wavelengths in the dispersion plane.
In conjunction with the change in PIXLIB, the default tg_d limits in tgextract have been narrowed to:
tgextract.min_tg_d = -6.6e-4 deg tgextract.max_tg_d = 6.6e-4 deg
and new LSFPARM files have been created for the new, narrower extraction region, allowing for the creation of more precise ACIS-S/LETG grating RMF files.
acissleg-1D1999-07-22lsfparmN0004.fits acissleg1D1999-07-22lsfparmN0004.fits
The new files will be automatically applied for this detector and grating arm combination when running mkgrmf, as described in the ACIS-S Grating RMFs thread.
HRC-S Imaging Data
-
SAMP-based RMF vN0001
This new HRC-S RMF file, hrcsD1999-07-22samprmfN0001.fits, is required for use with event data that have PI values derived from the SAMP HRC-S gain maps.
The spectral resolution of the HRC-S in imaging mode is poor compared to that of the ACIS detectors, but the HRC-S does exhibit a non-linear response to photon energies that can be advantageous in characterizing spectral hardness. This RMF can be used to calibrate hardness ratios or quantile color-color diagrams (QCCD) to distinguish between gross differences in the spectra.
We do not advocate using this RMF in spectral fits; the spectral response is not sufficiently constraining to achieve a good fit with reasonable errors.
HRC-I Imaging & Grating Data
-
HRC-I SAMP GMAP
A new time-dependent gain map for HRC-I observations taken since 2009-09-25 has been released. It is a SUMAMPS-based gain map, which is the standard for HRC. For both the HRC-I and HRC-S, the scaled sum of amplifier signals (SUMAMPS) is a better proxy for spectral response than the PHA.
Technical details on the new files are available in the SUMAMPS-based Gain Maps for the HRC-I memo (PDF).
Users working with HRC-I data taken during this period may wish to reprocess to improve the TGAIN calibration in their data. The DATE-OBS header keyword records the observation start date.
-
HRC-I QE version vN0008
An updated model for the HRC-I quantum efficiency has been released. The new file, hrciD1999-07-22qeN0008.fit, is based on cross-calibration with the HRC-S and ACIS and includes the effects of updated response models for those instruments.
Details on the new file are available on the An Update to the HRC-I MCP QE Model calibration page. This table, reproduced from that page, compares the predicted counts rates with the old N0007 file and the new N0008 file for three sources.
Source Observed Count Rate Predicted Count Rate with N0007 with N0008 G21.5-0.9 0.541+/- 0.008 0.601 +/- 0.018 (+11.09%) 0.541 +/- 0.016 (0%) PKS 2155-304 1.537 +/- 0.014 1.639 +/- 0.052 (+6.64%) 1.535 +/- 0.052 (-0.10%) HZ 43 3.88 +/- 0.07 3.46 +/- 0.21 (-10.82%) 3.91 +/- 0.24 (+0.86%)
Installation
Users should be aware of these installation items before installing CIAO 4.3. Additional problems which are seen less frequently are listed on the Installation & Smoke Tests bug page.
Mac OS X 10.5: incompatible library
-
There is a conflict between CIAO and a Mac OS X system library on 10.5 (Leopard) only. The workaround, which involves changing one file link in the CIAO directory, is on the Installation & Smoke Tests bug page.
The issue specifically affects users who have installed the Apple security update #2010-007, released on 10 Nov 2010. If you run the smoke tests without completing the workaround, obsvis-smoke004 will fail with:
[4/41] Running test obsvis-smoke004 ... FAIL Review /tmp/smoke.username/obsvis-smoke004/diff.log for details 75116 Trace/BPT trap $OBSVIS_DS9 -source obsvis_test.tcl >> ${OUTFILE} 2>&1 unix% cat /tmp/smoke.username/obsvis-smoke004/diff.log < dyld: Library not loaded: /usr/X11/lib/libfreetype.6.dylib < Referenced from: /usr/X11/lib/libXft.2.dylib < Reason: Incompatible library version: libXft.2.dylib requires version 13.0.0 or later, but libfreetype.6.dylib provides version 10.0.0
Solaris: CALDB download fails with a checksum error
-
The full tarfile for CALDB 4.4.1 is larger than 2 GB. Solaris systems running older versions of wget and md5 will not be able to download the full file because of a 2GB file limit in those utilities; the download will fail due to a checksum error.
If need assistance downloading CALDB, contact CXC Helpdesk.
Remove the .fontconfig cache
-
When switching from a 32-bit to 64-bit build (or vice versa), incompatible .fontconfig files are cached on the user's system. To ensure that the graphical applications such as ChIPS run correctly, remove the .fontconfig cache:
unix% rm ~/.fontconfig/*
Remove old parameter files and .rc files
-
With every new CIAO release, some parameter files have changed: new parameters may be added and occasionally old ones removed or renamed.
Similarly, there have been changes to the .chips.rc and .sherpa.rc configuration files to coincide with the removal of S-Lang support. Remove or rename existing .chips.rc and .sherpa.rc files in order to run those programs. Note that any other customizations will have to be copied to the new ~/.chips.rc.
unix% rm ~/cxcds_param4/* unix% rm ~/.chips.rc unix% rm ~/.sherpa.rc
Tools
acis_process_events
-
It is now possible to apply the ENERGY- and FLTGRADE-dependent sub-pixel event-repositioning algorithm EDSER. See the new parameter pix_adj.
-
As part of the implementation of the new sub-pixel algorithm, the CENTROID (i.e. docentroid=yes) and pixel randomization (e.g. rand_pix_size=0.5) algorithms are now included as choices for the parameter pix_adj. Therefore the parameters docentroid and rand_pix_size have been removed.
-
A new CTI adjustment algorithm for GRADED mode observations has been implemented.
-
The CTI adjustments for FAINT, FAINT_BIAS, GRADED, and VFAINT mode observations now include a temperature dependence. See the new parameter mtlfile.
addresp
-
addresp adds together multiple
imaging ARFs and RMFs to create a pair of merged ARF and RMF
files. The tool may be run with just ARF files as input to
create a merged ARF.
aprates
-
The quantity (rn-m)/(rf-g), which is the solution for net counts S in the set of simultanaeous linear equations given in the ahelp file, is used to determine the transition between Bayesian and Gaussian approaches for the max_counts parameter.
arestore
-
Bug fix: the tool correctly handles the case where the PSF and image are the same size. Previously, the kernel would not be shifted when its origin was at L/2, where L was the length of the data axes.
arfcorr
-
arfcorr calculates the
approximate fraction of the point spread function (PSF)
enclosed by a region. The tool then applies an
energy-dependent correction to the ARF file.
axbary
-
Updated to recognize FITS 3.0 files with RADESYS instead of RADECSYS and EQUINOX.
calvalid, calmerge, & calindex
-
These tools have been modified to remove the CXC parameter interface. The input parameters now must be specified in the correct order on the command line; optional parameters are input via a "--parameter=value" syntax.
celldetect
-
Bug fix: if the input file was missing a WCS, there would be bad detection properties in the region file (e.g. NaNs for the axis length).
create_bkg_map
-
Enable the @@ parameter file syntax.
dither_region
-
If a DTF file is supplied, the dtf_err weighted value is used.
dmcoords
-
The tool is able to parse a wider variety of sexagesimal coordinate specifications:
String entered Translation -------------- ----------- 23 59 59.9999 -89 59 59.999 23 59 59.9999 -89 59 59.999 23:59:59.9999 -89:59:59.999 23 59 59.9999 -89 59 59.999 23.7654, +0.6857 01 35 03.6960 +00 41 08.520 09h 16m 54.28s 32d 15' 6.1" 09 16 54.2800 +32 15 06.100 10.9876h, +14 10 59 15.3600 +14 00 00.000 0, 0 00 00 00.0000 +00 00 00.000 14 12, 16 10 14 12 00.0000 +16 10 00.000 14 12 +16 10 14 12 00.0000 +16 10 00.000 14 12 16d 10 14 12 00.0000 +16 10 00.000 14 12 16 10 AMBIGUOUS: treated as 14 12 16.0000 +10 00 00.000
dmellipse
-
The output file is a FITS region file instead of an ASCII region file.
-
The fraction achieved and state of convergence are recorded as column values.
dmextract
-
All modes: the following changes refer to all modes of dmextract (i.e. any value of the "opt" parameter).
- If the EXPOSURE time is missing, print a warning and use the LIVETIME instead.
- If the binned column has a coordinate transform associated with it, the transform is copied to the output. The exception is when the column is one component of a vector column; then output will not have copy of the coordinate system.
- Bug fix: If the region specified in the bin command began with an annulus(), the rest of the string was ignored.
- New warnings have been added for less-common extraction cases. In pha mode, warn if not binning on pha or pi. In lightcurve mode, warn if not binning on time or expno. In generic mode, warn if binning on pha, pi, time, or exposure number (expno).
-
Generic mode: the following changes refer to when dmextract is run with opt=generic or opt=generic2.
- Image pixels are checked against the BLANK/NULL value; if the value matches, the image pixel is excluded from the counts and the area.
- The "CEL_R" coordinate system has been added to the "R" array column to give distance in arcsec.
-
Lightcurve mode: the following changes refer to when dmextract is run with opt=ltc1 or opt=ltc2.
- AREA, NORM_BG_COUNTS, NORM_BG_ERR columns are included in the output file.
- A coordinate system is included on the bin column to give offset since the first record: DT, DT_MIN, and DT_MAX.
dmgti
-
Limited the range of values (0-5) for the verbose parameter.
dmimg2jpg
-
The "cool" and "staircase" color lookup tables have been added.
dmimgfilt
-
The values of the ONTIME, LIVETIME, LIVTIMEn, and EXPOSURn keywords are recalculated in the output file from the time subspace (e.g. GTIs), instead of being summed from the input files. This is done by a new merging rule - calcGTI - in the lookup table file.
dmimgcalc
-
Improved error message for case where output array requires more memory.
-
The values of the ONTIME, LIVETIME, LIVTIMEn, and EXPOSURn keywords are recalculated in the output file from the time subspace (e.g. GTIs), instead of being summed from the input files. This is done by a new merging rule - calcGTI - in the lookup table file.
dmimgpm
-
Limited the range of values (0-5) for the verbose parameter.
-
Changed the datatype of the clobber parameter from string to boolean.
dmmaskbin
-
Bug fix: the tool always returned a non-zero exit status value if the snrfile parameter was blank (the default) or set to "none".
dmmerge
-
The values of the ONTIME, LIVETIME, LIVTIMEn, and EXPOSURn keywords are recalculated in the output file from the time subspace (e.g. GTIs), instead of being summed from the input files. This is done by a new merging rule - calcGTI - in the lookup table file.
dmregrid2
-
The values of the ONTIME, LIVETIME, LIVTIMEn, and EXPOSURn keywords are recalculated in the output file from the time subspace (e.g. GTIs), instead of being summed from the input files. This is done by a new merging rule - calcGTI - in the lookup table file.
dmstat
-
Bug fix: integer zero (0) is a valid NULL value.
-
Limited the range of values (0-5) for the verbose parameter.
dmtype2split
-
The "rows" parameter has been removed. Use a Data Model filter to select the desired rows, as shown in the examples in the help file.
eff2evt
-
If grating data is input, the tool issues a warning and uses the zeroth-order grating efficiencies for the calculations.
-
Two new parameters: rmfimg and rmfcut. eff2evt uses the RMF image to determine a range of highly probable energies at which to compute the flux; the rmfcut parameter determines what probability cut-off to use.
This is an experimental feature in CIAO 4.3. The hope is that this information can eventually be used to construct meaningful error bars on a per-event basis that can be propagated into further analysis tools.
evalpos
-
Improved verbosity at values greater than 0.
hrc_dtfstats
-
The values of the ONTIME, LIVETIME, LIVTIMEn, and EXPOSURn keywords are recalculated in the output file from the time subspace (e.g. GTIs), instead of being summed from the input files. This is done by a new merging rule - calcGTI - in the lookup table file.
hrc_process_events
-
Bug fix: the tool recalculates the PI range of the data subspace instead of copying it from the input file. The SUMAMPS scale (used since CIAO 4.2) extends from 0-1023, while the previously-used PHA scale extended only from 0-255.
mkpsfmap
-
Units are included in the output image: options are "arcsec", "logical" pixels, and "physical" pixels.
-
Bug fix: limited range of the ecf parameter value to between 0 and 1; the tool will not run with a value outside this range.
-
The data subspace is copied from the input image to the output file.
mkrmf
-
The CHANNEL value datatype in the EBOUNDS extension has been changed to INTEGER for OGIP compatibility.
mkwarf
-
The new tool, sky2tdet, should be used to create the WMAP for input to mkwarf. The mkwarf tool has been updated to look for the "FRACCTS" keyword in the WMAP (written by sky2tdet), which gives the overall normalization for fraction of the counts that land on chip. If the keyword is not present, it uses a value of "1.0".
-
The tool no longer ignores bad pixels with non-zero flux when calculating the weights.
-
Off-chips warnings are suppressed when verbose=0.
modelflux
-
Bug fix: modelflux correctly calculates values in units of counts s^-1, photon cm^-2 s^-1 and erg cm^-2 s^-1.
pileup_map
-
The output units are recorded in the file header.
pimms
-
PIMMS can be run in a web browser or locally using the command-line interface. As of Chandra Cycle 13, PIMMS is not part of the CIAO Proposal toolkit and users wishing to use the command-line version need to download PIMMS from HEASARC.
psf_project_ray
-
New parameter: "blur_coord" specifies whether the blurring parameters (xblur, yblur, and ablur) should be applied to the "sky" or the "det" coordinate systems.
-
Limited the range of values (0-5) for the verbose parameter.
-
The ARDLIB settings are recorded in the HISTORY of the file header.
reproject_events
-
Bug fixes: the tool would segmentation fault if the infile didn't exist. It would also fail if the output existed and clobber=no.
reproject_image
-
The values of the ONTIME, LIVETIME, LIVTIMEn, and EXPOSURn keywords are recalculated in the output file from the time subspace (e.g. GTIs), instead of being summed from the input files. This is done by a new merging rule - calcGTI - in the lookup table file.
reproject_image_grid
-
The values of the ONTIME, LIVETIME, LIVTIMEn, and EXPOSURn keywords are recalculated in the output file from the time subspace (e.g. GTIs), instead of being summed from the input files. This is done by a new merging rule - calcGTI - in the lookup table file.
rmfimg
-
Added a warning about inaccurate WCS when the input RMF is not uniformly gridded.
sky2tdet
-
sky2tdet projects an image for a
region of interest in sky(x,y) coordinates to TDET
coordinates. The TDET image is appropriate for use as a weight
map (WMAP) to be input to the mkwarf
tool.
sso_freeze
-
The values of the ONTIME, LIVETIME, LIVTIMEn, and EXPOSURn keywords are recalculated in the output file from the time subspace (e.g. GTIs), instead of being summed from the input files. This is done by a new merging rule - calcGTI - in the lookup table file.
srcextent
tgextract
-
The default LETG extraction width was changed to match the calibration adjustment in the grating angle for ACIS.
wavdetect
-
Bug fix: improved handling of input files. The input file no longer needs to include a "." (e.g. ".fits"). It is also possible to have more than one set of brackets in the file specification, e.g. "[energy=500:8000][bin X=4275:4925:1,Y=3775:4425:1]"
-
The default value of the "interdir" parameter is ${ASCDS_WORK_PATH} instead of the working directory (".").
-
Bug fix: if the input filename did not contain a "." (e.g. ".fits"), the tool deleted the input file and exited with an error.
wcs_match
-
Bug fix: the exit status is set to be non-zero when a fatal error occurs.
wcs_update
-
Bug fix: the exit status is set to be non-zero when a fatal error occurs.
wrecon
-
Bug fix: if the input filename did not contain a "." (e.g. ".fits"), the tool deleted the input file and exited with an error.
wtransform
-
Bug fix: if the input filename did not contain a "." (e.g. ".fits"), the tool deleted the input file and exited with an error.
Parameter Files
A summary of parameter files changes is provided in this section. Refer to the Tools section of these release notes for complete details.
We recommend deleting all the old parameter files or renaming the parameter-file directory before running any new version of CIAO to avoid conflicts. More information is available in the FAQ.
acis_process_events
-
It is now possible to apply the ENERGY- and FLTGRADE-dependent sub-pixel event-repositioning algorithm EDSER. See the new parameter pix_adj.
-
As part of the implementation of the new sub-pixel algorithm, the CENTROID (i.e. docentroid=yes) and pixel randomization (e.g. rand_pix_size=0.5) algorithms are now included as choices for the parameter pix_adj. Therefore the parameters docentroid and rand_pix_size have been removed.
-
The CTI adjustments for FAINT, FAINT_BIAS, GRADED, and VFAINT mode observations now include a temperature dependence. See the new parameter mtlfile.
calvalid, calmerge, & calindex
-
These tools have been modified to remove the CXC parameter interface. The input parameters now must be specified in the correct order on the command line; optional parameters are input via a "--parameter=value" syntax.
dmgti
-
Limited the range of values (0-5) for the verbose parameter.
dmimgpm
-
Limited the range of values (0-5) for the verbose parameter.
-
Changed the datatype of the clobber parameter from string to boolean.
dmstat
-
Limited the range of values (0-5) for the verbose parameter.
dmtype2split
-
The "rows" parameter has been removed. Use a Data Model filter to select the desired rows, as shown in the examples in the help file.
eff2evt
-
Two new parameters: rmfimg and rmfcut. eff2evt uses the RMF image to determine a range of highly probable energies at which to compute the flux; the rmfcut parameter determines what probability cut-off to use.
This is an experimental feature in CIAO 4.3. The hope is that this information can eventually be used to construct meaningful error bars on a per-event basis that can be propagated into further analysis tools.
mkpsfmap
-
Bug fix: limited range of the ecf parameter value to between 0 and 1; the tool will not run with a value outside this range.
psf_project_ray
-
New parameter: "blur_coord" specifies whether the blurring parameters (xblur, yblur, and ablur) should be applied to the "sky" or the "det" coordinate systems.
-
Limited the range of values (0-5) for the verbose parameter.
srcextent
wavdetect
-
The default value of the "interdir" parameter is ${ASCDS_WORK_PATH} instead of the working directory (".").
Data Model
FITS V3.0 and WCS standards
-
The DataModel has a number of changes for FITS V3.0 standards and WCS handling. This results in some keyword changes in the output files created by the Data Model.
RADESYS has replaced RADECSYS to define the reference frame for the primary WCS (the actual keyword definition is RADESYSa, where 'a' is blank for the primary WCS or a character A to Z for alternate WCS's). RADECSYS is now deprecated. See Table 22 of the FITS standard and the associated notes.
Note that EQUINOXa is not allowed when the corresponding RADESYSa is ICRS, which is the case for Chandra data. See section 8.3 of the FITS standard.
A summary of the keyword changes:
- recognize keywords TCNA# and iCNA# in addition to the current TCNAM# and iCNAM#; output keyword TCNA# instead of TCNAM#.
- handle spaces in CNAME/TCNA/iCNA keyword values by replacing these spaces with underscore char. There is an exception for the case of a single space value which is considered blank and ignored.
- recogize WCS pairings of the form abLN,abLT
- add default vector names for GAL, EC, HEC, and SGAL
- recognize both RADECSYS and RADESYS
- add column-specific EQUI#/RADE#/LATP# to reserved keyword list so they will not be automatically transferred on copy.
- adjust logic for assigning EQUINOX value to coordinate
descriptors.
- apply global EQUINOX and override with column specific EQUI#.
- apply RADESYS/EQUINOX to Equatorial and Ecliptic coords only.
- do not output EQUI# keywords if RADESYS is "ICRS".
- improve usage of RADESYS and EQUINOX information when setting up wcslib structure.
- output "NAXIS=0" instead of "NAXIS=1,NAXIS1=0" for null HDU.
Enhancements & Bug Fixes
-
When using the subspace edit option, remove any associated TABLE definition as well as clearing the range list. For example, "[subspace -time]" should remove the time subspace and the associated GTI block.
-
When filtering on an array column and a non-array column at the same time, the "cells" modifier is only applied to the array. In this way, one can display a scalar column as well as a cell range of an array column.
ASCII Kernel
-
Added support for TSV format ASCII files. Generic TSV format files are recognized, as well as the extended header detail provided by the Chandra Source Catalog (CSC) output format.
Caveats:
- TSV flavor can not be auto-determined, it must either be specified in the DM kernel syntax ([opt kernel=text/tsv]) or by putting the TSV flavor specification line at the top of the file (#TEXT/TAB-SEPARATED-VALUES)
- Some of the additional header info (e.g. UCD) from the CSC format will be lost, since the DM does not yet support these concepts.
-
The full file is read to determine the correct datatypes and string column lengths, not just the header row.
Crates Library
-
A blockname attribute has been added to the Crate object. Crates retains the block name from the input file and uses that as the default block name on output. If the block name is not specified, a default value is used, e.g. "Table" when writing out a TABLECrate.
-
IMAGECrate will now recognize multiple coordinate systems on the axes and set them up in Crates.
-
New helper methods in the base Crates class:
- get_path() - returns the directory path of the file
- get_filter() - returns any filter being applied to the file
ChIPS
Removal of S-Lang Support
-
The CXC no longer supports the S-Lang scripting language as of CIAO 4.3. As a consequence, the ChIPS S-Lang interface has been removed and there is no means of importing ChIPS as a S-Lang module. Users may contact CXC Helpdesk for assistance in transitioning to Python.
Startup Options and .chips.rc
-
Since S-Lang support has been removed, the "-l" startup switch has been deprecated and related changes have been made to the .chips.rc file.
Remove or rename an existing ~/.chips.rc file in order to run ChIPS. Note that any other customizations will have to be copied to the new ~/.chips.rc.
-
The option "chips.shell" is no longer needed in the user's .chips.rc file. If this is set to "slang", the application will exit with an error:
chips ERROR: chips no longer offers a S-Lang interface. The 'shell' option in /home/username/.chips.rc is no longer used.
WCS Support
-
WCS tangent-plane projections with a rotation are now reprojected so that North is up (parallel to the Y axis). This is equivalent to the "Align to WCS" option that DS9 provides.
-
Users may create axes with WCS meta data associated with them. The axes tickmarks and grid lines will augment appropriately so that they accurately represent the WCS. The metadata can be brought in by displaying an image, contour or curve with WCS information or by passing a transform to the axes directly.
-
CIAO 4.2 state files which have plots with wcs contours or images and multiple axes may not load into CIAO 4.3 correctly. We suggest using make_script in CIAO 4.2 to save the state to a script, then use the script to load it into ChIPS in CIAO 4.3.
-
Caveats:
- when using make_script on a plot that has WCS coordinates, the resultant python script may not recreate the correct limits (in particular if there are multiple axes in the plot).
- undo will not fully work on a set of axes that have been created in cartesian and then converted to wcs, e.g. by adding an image or contour. Undoing the added object will leave the axes as WCS axes.
- creating a single axis in cartesian coordinates and then adding an image with wcs coordinates to it may cause the border axes to not line up correctly.
- Minor tickmarks near the poles may not always be spaced nicely.
Panning and Zoom
-
A new zoom function has been added
which will zoom in or out of a given data system. The zoom
function takes in an optional scale factor. Values greater
than one will zoom into the plot; fractional values less than
one will zoom out of the plot.
-
Two new functions, pan and panto,
allow the user to adjust the data
limits to a particular point. The panto function takes
in a point and will adjust the plot limits so that point is in
the center of the plot. The pan function is an
interactive function which allows a user to drag the data
system.
Picking Coordinates
-
pick and get_pick now retrieve coordinates with axis formats applied. For example, formats like RA and Dec will return values in sexagesimal notation as well as degrees.
-
Bug fix: pick_limits works with WCS projections, included rotated data sets.
Exporting Fonts
-
The ability to export pfa and pfb fonts to postscript and pdf files has been added. If export is turned off, the font name is still preserved in the postscript file.
-
Caveat: true type fonts (ttf) cannot be exported to postscript. If this type of font is used, the font will appear correctly on the screen but will use helvetica in the postscript.
Error Messages
-
Users can now get more detailed error message in ChIPS by adjusting the traceback limits. By setting sys.tracebacklimit, users can control how much information is returned on error. For example, sys.tracebacklimit=3 will display 3 levels up in the traceback stack.
Additional Enhancements & Bug Fixes
-
The advanced module function load_font will look in the working directory for a custom *.ttf font if a path is not specified.
-
Bug fix: delete_colorbardelete_colorbar("name") works correctly.
-
Bug fix: plot objects are drawn correctly when axis IDs have underscores.
-
Bug fix: set_preferences(["curve.*.color","red"]) would fail with UnboundLocalError.
-
Bug fix: set_preference("histogram.err.*", ..) sets err.up/down properties in a single command.
-
set_colorbar accepts chips_horizontal and chips_vertical as an argument for orientation.
-
aspect ratio information is now saved in the state file.
-
set_plot("margins") no longer breaks undo.
-
make_script saves the symbol size of the curves.
Sherpa
Removal of S-Lang Support
-
The CXC no longer supports the S-Lang scripting language as of CIAO 4.3. As a consequence, the Sherpa S-Lang interface has been removed and there is no means of importing Sherpa as a S-Lang module (e.g. 'require("sherpa");'). Users may contact CXC Helpdesk for assistance in transitioning to Python.
Sherpa no longer reads the environment variables CIAO_SCRIPT_LANG or SHERPA_SCRIPT_LANG, and will issue error messages if these variables are set to "slang".
Startup Options and .sherpa.rc
-
Since S-Lang support has been removed, the "-l" startup switch has been deprecated and related changes have been made to the .sherpa.rc file.
Remove or rename an existing ~/.sherpa.rc file in order to run Sherpa. Note that any other customizations will have to be copied to the new ~/.sherpa.rc.
-
The option "shell" is no longer needed in the user's .sherpa.rc file. If this is set to "slang", the application will exit with an error:
Sherpa ERROR: Sherpa no longer offers a S-Lang interface. The 'shell' option in /home/username/.sherpa.rc is no longer used.
-
A new section [parallel] that includes a switch "numcores"; more information is availablein the "Parallelization" section, below. The default value for "numcores" is None. To specify the number of cores to use for all Sherpa concurrent functions, change the default value:
[parallel] numcores : 3
Parallelization
-
Parallel evaluation of proj() and conf() is now automatically turned off on single core machines.
-
A new option, "numcores" has been added to the following
functions to specify the number of cores to use in
parallelization. The default is to use all cores available.
-
Bug fix: when Ctrl-C is issued while proj() or conf() are running, Sherpa correctly kills off slave processes from any parallel evaluation.
Loading and Viewing Data
-
New function: get_bkg_scale(). This returns the background
scale associated with a PHA data set, that is used when
subtracting background data, or when simultaneously fitting
source and background models in one fit to PHA data.
The background scale is the value associated with the OGIP PHA
header keyword BACKSCAL.
The background scale is also shown by the show_data() and show_all() functions.
-
show_psf() now hides the header information if the PSF is from file.
-
Bug fix: show_all() displays the correct response information for PHA data sets.
Filtering Data
-
Updates to load_filter and set_filter include the ability to read FITS images that hold filter information. There is a new keyword argument "ignore" that indicates whether the filter information should be used to notice or ignore data points. "ignore" is False by default.
sherpa> load_filter(id=1, filename, bkg_id=None, ignore=False, ncols=2) sherpa> set_filter(id=1, val, bkg_id=None, ignore=False)
Instrument Responses and PSFs
-
New function:
get_response()
returns the associated PHA instrument (RMF + ARF) or any
combination or iteration. The response object may be used in a
model expression, and background responses are supported by
using the bkg_id argument.
sherpa> rsp = get_response() sherpa> set_full_model(rsp(xsphabs.abs1*powlaw1d.p1))
-
The functions get_arf() and get_rmf() return "instrument" versions of the ARF or RMF dataset that include a callable functionality. This allows the user to define a response in a model expression. The multiplication with the exposure time is implicit.
arf = get_arf() rmf = get_rmf() set_full_model(rmf(arf(xsphabs.abs1*powlaw1d.p1)))
-
Bug fix to correctly handle rectangular PSFs from file. Rectangular PSF images are also correctly displayed from file in DS9.
-
Bug fix: allow simultaneous fitting when a different PSF model is assigned to each data set in a fit.
Models and Parameters
-
Sherpa now allows the user to define model expressions that apply response matrices, or PSFs, to some models, while not applying the response or the PSF to the rest of the model expression. An example of this kind of model is an expression where a spectral model is defined in energy space, and folded through a response matrix; then, a background model defined in *counts*, which is not folded through the response, is added to the model expression.
The new functions set_full_model() and set_bkg_full_model() allow users to explicitly define instruments and convolutions that are applied to specified model components.
Legacy functionality is still supported with set_source() and set_model(); scripts using these functions will continue to work in the current Sherpa.
Automatic Manual Definition ============= ================= set_source() set_full_model() set_model() set_bkg_source() set_bkg_full_model() set_bkg_model()
-
Sherpa models now have a "cache" attribute for users to control behavior. The default value for "cache" is a non-zero value indicating that caching will be turned on only if all parameters in the model are frozen. Indicating zero will turn off caching even if all model parameters are frozen.
Model caching is now available for XSPEC and 1D analytic models; caching is turned on by default. 2D Sherpa analytical models are not cached by default due to the potential impact on memory usage. Compared to fit results in previous releases, there should be no change to any calculated value. The only difference that may be seen is in reduced program execution time.
sherpa> set_source( xsphabs.abs1 * powlaw1d.p1 ) sherpa> thaw(abs1) sherpa> abs1.cache=0
-
New 1D integration helper function, "integrate1d". Users can
use a helper function to define 1D numerical integration on a
particular arbitrary Sherpa model expression. This function is
used inside a Sherpa model definition as an explicit indication
that Sherpa should numerically integrate the expression in
parenthesis. The helper function numerically integrates the
expression as a whole, correctly, instead of each component
individually. Note that the model expression should not include
any XSPEC additive models, as those models perform integration on
themselves.
sherpa> set_model(integrate1d.int1(beta1d.b1*gauss1d.g1)) sherpa> print int1 integrate1d.int1 Param Type Value Min Max Units ----- ---- ----- --- --- ----- int1.epsabs frozen 2.22045e-16 -3.40282e+38 3.40282e+38 int1.epsrel frozen 0 -3.40282e+38 3.40282e+38 int1.maxeval frozen 10000 -3.40282e+38 3.40282e+38This helper function is used in lieu of the "integrate" flag found on Sherpa models.
-
New Sherpa models:
- scale1d - 1D constant amplitude model
- scale2d - 2D constant amplitude model
- logparabola - a log-parabolic function
The logparabola model has the form
f(x) = A (x/x_ref)^[-gamma - beta*log10(x/x_ref)]
-
The Sherpa table model has been updated to support interpolation of data points on the data set grid from the grid supplied from file. The grids need not be of constant or comparable bin size. If the table model grid is not sorted, Sherpa will sort it in ascending order.
-
New arguments have been added to the list_models() command. Valid arguments are "all", "xspec", "1d", and "2d".
-
There are new plotting functions to plot individual model components. Users can pass one or more Sherpa model components to the plotting function to quickly view the contribution to the model. Similarly, users can view 2D model components in DS9.
-
The plot_source() command now supports the "factor" setting of set_analysis(). Calling plot_source() with a setting of factor=1 corresponds to the XSPEC plot eufspec, while a setting of factor=2 represents eeufspec.
eufspec: E f(E) set_analysis("energy", factor=1) plot_source() eeufspec: E^2 f(E) set_analysis("energy", factor=2) plot_source() eufspec: \lambda f(\lambda) set_analysis("wave", factor=1) plot_source() eeufspec: \lambda^2 f(\lambda) set_analysis("wave", factor=2) plot_source() -
New function:
add_model() that
assigns a user-defined model class as a Sherpa model
type. User-defined model classes that inherit from the Sherpa
Arithmetic model class or other Sherpa models are accepted.
from sherpa.models import PowLaw1D class MyPowLaw(PowLaw1D): pass add_model(MyPowLaw) set_model(mypowlaw.p1)
-
Sherpa includes the "additive" and "multiplicative" models of XSpec version 12.6.0l. (Note that the ahelp files packaged with CIAO incorrectly state the version as "12.6.0.h".)
New XSpec models included in Sherpa are:
- xssirf
-
XSPEC-style table models are now supported using load_table_model. Additive and multiplicative table models are supported (Atable and Mtable).
-
The XSPEC function set_xsabund() is updated to accept user-defined files containing solar abundances.
-
New XSPEC functions:
set_xsxset()
and
get_xsxset().
These functions are associated with the XSPEC function XSET
for defining model environment variables.
sherpa> set_xsxset("NEIVERS", "2.0") sherpa> set_xsxset("NEIAPECROOT", "/home/brefsdal/apec") sherpa> set_model(xsvnei.nn) sherpa> get_xsxset("NEIAPECROOT") sherpa> '/home/user/apec' -
Bug fix: flag parameters are always frozen for XSpec models.
Fitting
-
Two iterative fitting methods have been added to Sherpa: Primini's methods, and sigma-rejection. The essence of an iterative fitting method is that the fit method can be called several times, until some criterion is met.
Primini's method is to re-calculate statistical errors, using the best-fit model parameters from the *previous* fit, until the fit can no longer be improved.
Sigma-rejection is based on the IRAF SFIT function. In successive fits, data points for which ((data - model) / error) exceeds some threshold are added to the filter, and automatically excluded from the next fit.
Several new functions have been added to allow users to set the iterative fitting method, to find out what the current iterative fitting method is, and to get and set options for this method. These functions are:
- set_iter_method() - set the iterative fitting method
- get_iter_method_name() - print the name of the current iterative fitting method
- list_iter_methods() - list all possible iterative fitting methods
- get_iter_method_opt() - get the value of the named option for the current iterative fitting method. If no argument, list all options.
- set_iter_method_opt(, value) - set the named option to a new value for the current iterative fitting method
If the iterative fitting method is "none" (the default value), then no iterative fitting is done. When fit() is called, the optimization method is called once, and Sherpa otherwise operates as expected.
The statistic and optimization methods are selected independently of the iterative fitting method, e.g.
sherpa> set_stat("chi2datavar") sherpa> set_method("neldermead") sherpa> set_iter_method("primini") sherpa> fit() # Primini's method is called sherpa> set_iter_method("none") sherpa> fit() # Nelder-Mead is called oncePrimini's method and sigma-rejection can only be called when the statistic is a chi-squared statistic. They cannot be used with least-squares, Cash or C-statistic.
-
New functions:
calc_stat_info() and
get_stat_info().
These implements functionality similar to the GOODNESS command
from Sherpa 3.4. Goodness-of-fit information usually found in
a Sherpa fit result is now also calculated separately. It is
no longer necessary to perform a new fit just to get
goodness-of-fit information.
Plotting
-
New functions set x and y axis scales for Sherpa plots:
The functions arguments similar to plot(): "data", "model", "source", "delchi", etc.
-
The plot functions int_proj() and int_unc() include dashed lines indicating the best fit statistic and best fit parameter value.
-
The plot_bkg_unconvolved command has been removed; use the identical plot_bkg_source command instead. (The old command is still referenced in a few help files.)
-
Bug fix: plot error bars are correctly displayed when statistical error is calculated by the statistic and systematic error is set by the user.
Saving
-
The save() function now includes XSPEC module settings in the saved binary file. These settings are the abundance, chatter level, cosmology, electron cross-section, and model variables set via the XSPEC "xsxset()" function.
The restore() function reads these settings back in and calls the appropriate XSPEC module functions to reset the values.
-
The save_all() function, which saves attributes of a Sherpa session to a human-readable text file, has been modified. The text file created by save_all() now includes:
- The iterative fitting method, if that has been set;
- Source model assignment via either set_source() or set_full_model(), as appropriate;
- XSPEC module settings for abundances, chatter level, cosmology, electron cross-section, and model settings via the xsxset() function.
Text files created with the CIAO 4.2 version of save_all() can still be used in the CIAO 4.3 version of Sherpa.
Additions to the utils module
-
A new utility function: parallel_map().This function is a parallelized version of the native Python map which evaluates a function over a list of arguments.
sherpa> from sherpa.utils import parallel_map sherpa> parallel_map(lambda x: x*x, [1,2,3]) -> [1,4,9]
-
The follow statistical PDF functions for multivariate normal and multivariate student T distributions have been added to Sherpa utils module.
sherpa> dof = get_fit_results().dof sherpa> sigma = get_covar_results().extra_output sherpa> mu = get_model().thawedpars sherpa> x = numpy.random.multivariate_normal(mu, sigma) sherpa> from sherpa.utils import multinormal_pdf, multit_pdf sherpa> t_pdf = multit_pdf(x, mu, sigma, dof) sherpa> normal_pdf = multinormal_pdf(x, mu, sigma)
-
A vectorized function: "interpolate".
sherpa> from sherpa.utils import interpolate sherpa> yout = interpolate(xout, xin, yin, method='linear'|'nearest')
Additional Enhancements & Bug Fixes
-
The energy and photon flux is calculated more efficiently for PHA data sets with multiple responses.
-
Sherpa now uses a private DS9 window for communication, with id 'sherpa' for session imaging.
-
Bug fix: numerical issues meant that powlaw1d results were incorrect when gamma was very close to 1.
-
Bug fix: show_bkg_source() would show all the components for an id even if the bkg_id keyword was set.
-
Bug fix: the atten model should ignore the integrate setting.
-
Re-linking a model parameter implicitly deletes any existing link.
GUIs (Graphical User Interfaces)
ds9 and dax
-
ds9 v6.2 (Mac & Linux) is packaged with CIAO on Mac OS X and Linux. ds9 v6.1.2 is packaged on Solaris, as v6.2 was not yet available for that platform.
-
Plotting from dax: a ChIPS server is started in the background when ds9 is launched; this ChIPS session is then used for any plotting. If ChIPS is not available, dax uses BLT to plot instead; the user will see an informational message in the terminal in this case.
To modify a ChIPS plot, first connect to the ChIPS server with the connect command. Then any ChIPS commands may be used to change the appearance of the plot.
Once connected to the server, the plot may be saved with the make_script command.
prism
-
Prism will reopen a file using virtual filter syntax to filter out undesired columns from the display. The filtering is currently only supported for tables.
-
Support has been added for boolean columns data. The columns display TRUE or FALSE and allows the user to edit the entries as well.
-
Plots and histograms created from prism are wrapped in a ChIPS undo block so that the plot may be undone/redone by a single undo/redo command.
-
Support for string array types has been added.
Libraries
cxcparam
-
Changed paramopen so that if @@<something> is the first argument, it will override the tool name. This allows pquery and some other tools to use the @@ parameter syntax.
-
Bug fix: the last character of a shell redirect "))something" is removed only if it is a newline character
-
The "value" and the "worth" of a parameter redirection are added to the readline history.
-
Improved support for special characters in readline.
dslib
-
Updated the autonaming routine to remove any filter from the filename, starting at '['.
hdrlib
-
Added new merging rule: calcGTI that only applies to ONTIME, LIVETIME, and EXPOSURE keywords. The values of the ONTIME, LIVETIME, LIVTIMEn, and EXPOSURn keywords are recalculated in the output file from the time subspace (e.g. GTIs), instead of being summed from the input files.
-
Bug fix: corrected problem when more than 1e6 history records are written.
pixlib
-
Added support for new instrument-dependent grating angle GEOM files.
regionlib
-
Added modulo 360 when doing regExtent for pie shapes to improve speed with angles such as "350:366".
scconvlib
-
A library that parses a wide variety of sexagesimal
coordinate specifications:
String entered Translation -------------- ----------- 23 59 59.9999 -89 59 59.999 23 59 59.9999 -89 59 59.999 23:59:59.9999 -89:59:59.999 23 59 59.9999 -89 59 59.999 23.7654, +0.6857 01 35 03.6960 +00 41 08.520 09h 16m 54.28s 32d 15' 6.1" 09 16 54.2800 +32 15 06.100 10.9876h, +14 10 59 15.3600 +14 00 00.000 0, 0 00 00 00.0000 +00 00 00.000 14 12, 16 10 14 12 00.0000 +16 10 00.000 14 12 +16 10 14 12 00.0000 +16 10 00.000 14 12 16d 10 14 12 00.0000 +16 10 00.000 14 12 16 10 AMBIGUOUS: treated as 14 12 16.0000 +10 00 00.000
stacklib
-
Improved error handling when the stack file doesn't exist.
tcdlib
-
Bug fix: correct a segv when missing closing ")".
Environment
Automatic Update of Configuration Files for Python Users
-
The ChIPS and Sherpa applications use the $HOME/.ipython-ciao/ directory to store configuration information. The system used to update this information was changed in CIAO 4.2, which means that the first time ChIPS or Sherpa is started you will see a message asking whether you want to update your ipythonrc-chips or ipythonrc-sherpa file.
Unless you have manually changed your $HOME/.ipython-ciao/ipythonrc-chips (or -sherpa) file, it is safe to answer "Y" here.
The outdated file is renamed with a timestamp to preserve it.
IPython
-
CIAO includes IPython in the CIAO OTS directory which is used by Sherpa and ChIPS to provide command-line user interfaces. These programs create an IPython profile in the directory $HOME/.ipython-ciao.
If IPython users want any personal customizations to be available when running CIAO, they will have to copy them from $HOME/.ipython to $HOME/.ipython-ciao.
Off-the-Shelf (OTS) Package Versions
-
The following OTS packages are included with CIAO 4.3. For more information on how the OTS packages are built for use with CIAO, refer to the INSTALL_SOURCE file distributed with the software.
- cfitsio v3.25
- CCfits v2.2
- DS9 v6.2 (Mac & Linux); v6.1.2 (Solaris)
- gsl v1.14
- GTK v2.16.6 (Mac & Linux); v2.12 (Solaris)
- Python v2.6.6
- readline v6.1
- S-Lang v2.1.4
- wcssubs v3.8.1
- xerces v3.1.1
- xpa v2.1.12
- xspec v12.6.0l (models only)
Analysis Scripts
Infrastructure Changes
-
The CXC no longer supports the S-Lang scripting language as of CIAO 4.3. Therefore, the S-Lang versions of the contributed modules have been removed.
Importing Routines from the Contributed Modules
-
To order to import all routines from each of the contributed modules, use the syntax:
from <name>_contrib.all import *
instead of
from <name>_contrib import *
This applies to chips_contrib, ciao_contrib, crates_contrib, and sherpa_contrib.
acisspec
-
The acisspec script has been removed from the contributed tarfile. combine_spectra should be used in combination with the specextract script in place of acisspec. Refer to the combine_spectra help file for examples of use.
Contact CXC Helpdesk if you need assistance transitioning from acisspec syntax to combine_spectra.
chandra_repro
-
The script has been updated for the CIAO 4.3 changes to acis_process_events:
- The "edser" subpixel algorithm has been added as choice for the pix_adj parameter. This option only applies to ACIS data. Leaving pix_adj="default" also applies EDSER, as it is the default in acis_process_events in CIAO 4.3.
- The script requires the mtl1.fits file (mission timeline) as an input when reprocessing ACIS data. An error will be printed if the file is not found in the secondary data directory.
combine_spectra
-
The new addresp tool is used to combine the ARF and RMF response files. Prior to this release, combine_spectra combined ARFs via the dmarfadd tool - with results equivalent to the output of addresp - and did not combine RMFs.
SherpaCL
-
The sherpacl application has not been updated to work in CIAO 4.3. Please contact the CXC Helpdesk if you would like to use SherpaCL in CIAO 4.3.
specextract
-
The specextract script has been moved to the contributed tarfile. Once the contributed tarfile is installed, users will not notice any difference in running this script.
-
There were several enhancements to the specextract script, which resulted in a few changes in syntax.
(22 Dec 2010) A number of
bug fixes were released for specextract; download
the updated version from the Scripts page.
Extended sources (weighted response files)
-
The default value of the new weight parameter - yes - tells the script to use mkwarf to make the ARF file(s). This is appropriate for extended sources.
-
The sky2tdet tool, new in CIAO 4.3, has been added to specextract to create a more accurate WMAP file for use with mkwarf; it's automatically run when weight = yes. This WMAP includes which bad pixels are within the regions and how much of the source flux falls onto those bad pixels.
In order to run sky2tdet, specextract requires an aspect histogram file to be input to the asp parameter (created by the asphist tool). A future release of specextract will take the aspect solution files directly and make the aspect histogram for you.
-
There is also a new mskfile parameter; the mask file is used as input to mkwarf.
A comparison of the CIAO 4.2 and CIAO 4.3 syntax:
CIAO 4.2 specextract infile="evt2.fits[sky=region(src.reg)]" bkgfile="evt2.fits[sky=region(bg.reg)]" \ outroot=extended pbkfile=pbk0.fits grouptype=NUM_CTS binspec=15 CIAO 4.3 specextract infile="evt2.fits[sky=region(src.reg)]" bkgfile="evt2.fits[sky=region(bg.reg)]" \ outroot=extended pbkfile=pbk0.fits asp=extended.asphist mskfile=msk1.fits \ grouptype=NUM_CTS binspec=15
Pointlike sources
The specextract script has been enhanced to be able to created response files for pointlike sources, i.e. to use the tool mkarf to create the ARF file. specextract may now be used in place of the psextract, which will be deprecated in the future.
-
If the new weight parameter is set to no, specextract uses mkarf to make the ARF file(s). This is appropriate for pointlike sources.
In order to run mkarf, specextract requires an aspect histogram file to be input to the asp parameter (created by the asphist tool). A future release of specextract will take the aspect solution files directly and make the aspect histogram for you.
The mskfile parameter is also used as input to mkarf.
-
The arfcorr tool, new in CIAO 4.3, has been added to specextract. If correct=yes, arfcorr is run to apply an energy-dependent point-source aperture correction to the ARF.
A comparison of the psextract and specextract syntax:
psextract events="evt2.fits[sky=region(src.reg)]" bgevents="evt2.fits[sky=region(bg.reg)]" \ root=point asol=@pcad_asol1.lis pbkfile=pbk0.fits \ gtype=NUM_CTS gspec=15 specextract infile="evt2.fits[sky=region(src.reg)]" bkgfile="evt2.fits[sky=region(bg.reg)]" \ outroot=point asp=point.asphist pbkfile=pbk0.fits mskfile=msk1.fits \ weight=no correct=yes \ grouptype=NUM_CTS binspec=15
Merging spectra
An option for combining stacks of output spectra and associated responses was added to the script. Setting combine=yes runs the combine_spectra script after the individual spectra and responses are created.
-
Documentation
CIAO News
-
The news announcements will no longer be posted at the bottom navigation bar. Refer to the CIAO News page for new announcements; the page is also linked from the Introduction section of the navigation bar.
Analysis Threads
-
The CIAO, Sherpa, and ChIPS Threads are in the process of being tested and updated for the CIAO 4.3 release. Please check the "Last Update" information at the top of each thread to see if it has been reviewed yet.