/ ciao / releasenotes / ciao_4.5_release.html

CIAO 4.5 Release Notes

Version History

Notable changes and improvements in CIAO 4.5:

The mkpsf tool has been withdrawn, as advertised in the CIAO 4.4 release notes. Users should use either ChaRT or MARX to simulate the point spread function.
A new tool, tg_findzo, has been released to provide an accurate 0-th order location for those datasets with heavy pileup or with 0-th order blocked.
The modelflux tool has updates to produce unabsorbed fluxes.
Updates to reproject_events and dmmerge allow for mosaics to be created that exceed the typical image sizes.
wcs_match now has the option of computing a simple translation only solution as well as the existing translate, rotate, and scale method.
Updates to the ChIPS GUI to allow annotations (lines, points, etc) to be added directly via the GUI.
Updates to sherpa include a comprehensive review of how grouping and filtering interact when both source and background datasets are in use. Also, a new sample_flux routine is available to allow users to compute the flux due to a sub-component of a Sherpa model.

How CALDB 4.5.9 Affects Your Analysis
How CALDB 4.5.8 Affects Your Analysis
How CALDB 4.5.7 Affects Your Analysis
How CALDB 4.5.6 Affects Your Analysis
How CALDB 4.5.5 Affects Your Analysis
Installation
Tools
Parameter Files
Data Model
ChIPS
Sherpa
GUIs (Graphical User Interfaces)
Analysis Scripts
Libraries
Environment
Documentation

How CALDB 4.5.9 Affects Your Analysis

CALDB 4.5.9 Release Notes (release 19 Nov 2013)

ACIS Imaging and Grating Data

ACIS QE Contamination Model vN0008

The ACIS QE contamination model has been upgraded to N0008:
```
acisD1999-08-13contamN0008.fits 
```
This version of the file is necessary due to recent changes in the optical depth and composition of the contaminant deposited on the optical blocking filter over time, resulting in a new ACIS contamination model.

Note that these changes do not have any effects at energies above ~1 keV (see changes in effective areas in the ACIS QE contamination model Why document)

The new model provides significantly lower ACIS effective areas (EAs) compared to the N0007 model for present and future observations; albeit, early- and mid-mission (1999-2009) EAs are not particularly affected by the new model, except for near the carbon, oxygen, and fluorine absorption K-edges. It should be noted that changes near these edges do not significantly affect the fitting results for mid-mission (2004-2009) observations.

The new contamination model is more realistic than previous models and provides a more accurate representation and prediction of current and future changes of effective ACIS quantum efficiency due to the contamination layer.

The rate of contaminant build-up has increased significantly since mid-2009, and recently, has presented itself as changing more linearly with time than the previous model estimated (released as CONTAM file model version N0007), which had predicted a leveling-off of the contamination in the future. The concentration of several elemental components in the contamination layer has also been changing, requiring time-varying changes to the optical depths for the C- and F-K absorption edges with respects to the O-K edge, and changes to the spatial-dependent ACIS model, which is also included here.

Updating the new contamination model for the full mission means that the N0008 CONTAM model is not identically backwards-compatible with the previous N0007 model, but the new model fits the mid-mission contaminant optical depths as the previous model had, indicating that the new model does not significantly affect the early- and mid-mission results.

ACIS-S3 high CHIPY position

When using a high CHIPY location on ACIS-S3 for imaging spectroscopy, there is some uncertainty in the new model on the depth of the contaminant layer, which may compromise fitting results at this location.

More information can be found in the CALDB 4.5.9 Release Notes caveat.

These CIAO response tools automatically apply the contamination file when creating ACIS response files:
- mkarf
- mkgarf
- mkwarf
- mkinstmap
As well as the scripts which use them:
- specextract (calls mkwarf and mkarf)
- fullgarf (calls mkgarf)
- fluximage (calls mkinstmap)
- merge_obs (calls mkinstmap)
Time-dependent ACIS Gain (T_GAIN) Files for -120 C Data

There are new, time-dependent ACIS Gain (T_GAIN) files for August 2013 - October 2013 (Epoch 55):
```
acisD2013-05-01t_gainN0005.fits
acisD2013-05-01t_gainN0006.fits
acisD2013-08-01t_gainN0002.fits
acisD2013-08-01t_gain_biN0002.fits
```
The files are applicable to -120 C focal plane temperature only.

Users working with ACIS data taken since 2013 May 01 may wish run chandra_repro and reprocess the data to improve the T_GAIN calibration. The DATE-OBS header keyword records the observation start date.

Note that only spectra with several hundreds of counts and/or prominent features (in emission or absorption) will show changes from the T-gain refinement that exceed the uncertainties from the gain calibration.

HRC-I Imaging Data

HRC-I Gain Maps

The latest time-dependent HRC-I Gain Map (GMAP) and the corresponding PI background spectrum—to be applied to observations after 2013 September 16, have been released.
```
hrciD2013-09-16sampgainN0001.fits 
hrciD2013-09-16pibgspecN0001.fits 
```
Note that HRC-I gain maps are only useful to observers doing hardness ratio or PI spectral analysis studies with HRC-I.

As the HRC gain continues to slowly decline, the gain maps are updated annually. The change in gain correction is less than 1% on-axis and roughly 5-10% off-axis. Towards the edge of the microchannel plate, the change is up to 15%. Technical details on the gain maps are described in the gain maps section of the HRC calibration website and the in the HRC-I Gain Map Why Document.

The corresponding background spectrum applicable to the same HRC-I GMAP time period have been generated.

Users working with HRC-I data may wish run chandra_repro to reprocess the data and improve the gain calibration.

How CALDB 4.5.8 Affects Your Analysis

CALDB 4.5.8 Release Notes (release 22 Aug 2013)

ACIS Imaging and Grating Data

Time-dependent ACIS Gain (T_GAIN) Files for -120 C Data

There are new, time-dependent ACIS Gain (T_GAIN) files for May 2013 - Jul 2013 (Epoch 54):
```
acisD2013-02-01t_gainN0005.fits
acisD2013-02-01t_gainN0006.fits
acisD2013-05-01t_gainN0002.fits
acisD2013-05-01t_gain_biN0002.fits
```
The files are applicable to -120 C focal plane temperature only.

Users working with ACIS data taken since 2013 Feb 01 may wish to reprocess to improve the T_GAIN calibration in their data. The DATE-OBS header keyword records the observation start date.

Note that only spectra with several hundreds of counts and/or prominent features (in emission or absorption) will show changes from the T-gain refinement that exceed the uncertainties from the gain calibration.

How CALDB 4.5.7 Affects Your Analysis

CALDB 4.5.7 Release Notes (release 28 June 2013)

ACIS Imaging and Grating Data

Time-dependent ACIS Gain (T_GAIN) Files for -120 C Data

There are new, time-dependent ACIS Gain (T_GAIN) files for Feb 2013 - Apr 2013 (Epoch 53):
```
acisD2012-11-01t_gainN0005.fits
acisD2012-11-01t_gainN0006.fits
acisD2013-02-01t_gainN0002.fits
acisD2013-02-01t_gain_biN0002.fits
```
The files are applicable to -120 C focal plane temperature only.

Users working with ACIS data taken since 2012 Nov 01 may wish to reprocess to improve the T_GAIN calibration in their data. The DATE-OBS header keyword records the observation start date.

Note that only spectra with several hundreds of counts and/or prominent features (in emission or absorption) will show changes from the T-gain refinement that exceed the uncertainties from the gain calibration.
ACIS Blank-sky Background File (Group F ACIS BKGRND)

The Group F ACIS blank-sky background files is now completed for observations starting Autumn 2009, when the particle background levels began decreasing with the receding solar cycle.
```
acis5sD2009-09-21bkgrndN0001.fits
acis7sD2009-09-21bkgrndN0001.fits
```
The ACIS-5 and ACIS-7 background files included in this release are non-CTI corrected, applicable to datasets taken in GRADED mode, where CTI correction is available for back-illuminated chips.

The CTI-corrected Group F background files previously released have been upgraded to version 2.
```
acis0iD2009-09-21bkgrnd_ctiN0002.fits
acis1iD2009-09-21bkgrnd_ctiN0002.fits
acis2iD2009-09-21bkgrnd_ctiN0002.fits
acis3iD2009-09-21bkgrnd_ctiN0002.fits
acis6iD2009-09-21bkgrnd_ctiN0002.fits

acis2sD2009-09-21bkgrnd_ctiN0002.fits
acis3sD2009-09-21bkgrnd_ctiN0002.fits
acis5sD2009-09-21bkgrnd_ctiN0002.fits
acis6sD2009-09-21bkgrnd_ctiN0002.fits
acis7sD2009-09-21bkgrnd_ctiN0002.fits
acis8sD2009-09-21bkgrnd_ctiN0002.fits
```
These files now include populated DETX,DETY and SKYX,SKYY coordinates, making file visualization easier. The coordinates are artificial, applied only to show the non-dithered distribution of events in SKY coordinates for the convenience of the user, and meaningful coordinates can be applied with reproject_events.

The Using the ACIS "Blank-Sky" Background Files thread contains instructions on how to select a file and match it to a specific observation.

Note that the ACIS background files are not included in the main CALDB tarfile. There is a separate background files tarfile available via ciao-install or from the CALDB Download page.
HRC-S T_GMAP with GRATING=NONE

The time-dependent gain maps (T_GMAP) in the CalDB, since version 4.2.0, have been identified by the calibration team as not being applied to HRC-S observation when no gratings are present. Rather, the old time-independent gain maps (GMAP) have been applied by default in both chandra_repro and in the standard data processing with hrc_process_events.
```
hrcsD1999-07-22t_gmapN0002.fits
hrcsD2012-03-29t_gmapN0002.fits
```
The cause have been identified in the T_GMAP CalDB files having been indexed with GRATING=LETG in the file header, causing errors in the HRC branch index.

The time-dependent and time-indpendent gain map applied to a HRC-S events list is stored in the GAINCORF header keyword, so users may identify whether or not their data is affected by the problem.
```
unix% dmkeypar *evt2.fits GAINCORF echo+
```
If the results are either of the two following file names:
```
hrcsD1999-07-22t_gmapN0002.fits
hrcsD2012-03-29t_gmapN0002.fits
```
the user need not do anything, as the time-dependent gain map has been applied to the dataset. If the returned filename is:
```
hrcsD1999-07-22gmapN0001.fits
```
then the old gain map is applied, and should be updated.

Affected users may simply reprocess their data with chandra_repro to correct the problem once CalDB is upgraded to version 4.5.7 or later.

How CALDB 4.5.6 Affects Your Analysis

CALDB 4.5.6 Release Notes (release 28 February 2013)

ACIS Imaging and Grating Data

Time-dependent ACIS Gain (T_GAIN) Files for -120 C Data

There are new, time-dependent ACIS Gain (T_GAIN) files for Nov 2012 - Jan 2013 (Epoch 52):
```
acisD2012-08-01t_gainN0005.fits
acisD2012-08-01t_gainN0006.fits
acisD2012-11-01t_gainN0002.fits
acisD2012-11-01t_gain_biN0002.fits
```
The files are applicable to -120 C focal plane temperature only.

Users working with ACIS data taken since 2012 Aug 01 may wish to reprocess to improve the T_GAIN calibration in their data. The DATE-OBS header keyword records the observation start date.

Note that only spectra with several hundreds of counts and/or prominent features (in emission or absorption) will show changes from the T-gain refinement that exceed the uncertainties from the gain calibration.
ACIS Blank-sky Background File (Group F ACIS BKGRND)

A new set of ACIS blank-sky background files have been released for observations starting Autumn 2009, when the particle background levels began decreasing with the receding solar cycle.
```
acis0iD2009-09-21bkgrnd_ctiN0001.fits
acis1iD2009-09-21bkgrnd_ctiN0001.fits
acis2iD2009-09-21bkgrnd_ctiN0001.fits
acis2sD2009-09-21bkgrnd_ctiN0001.fits
acis3iD2009-09-21bkgrnd_ctiN0001.fits
acis3sD2009-09-21bkgrnd_ctiN0001.fits
acis5sD2009-09-21bkgrnd_ctiN0001.fits
acis6iD2009-09-21bkgrnd_ctiN0001.fits
acis6sD2009-09-21bkgrnd_ctiN0001.fits
acis7sD2009-09-21bkgrnd_ctiN0001.fits
acis8sD2009-09-21bkgrnd_ctiN0001.fits

acis0D2009-09-21bgstow_ctiN0001.fits
acis1D2009-09-21bgstow_ctiN0001.fits
acis2D2009-09-21bgstow_ctiN0001.fits
acis3D2009-09-21bgstow_ctiN0001.fits
acis5D2009-09-21bgstow_ctiN0001.fits
acis6D2009-09-21bgstow_ctiN0001.fits
acis7D2009-09-21bgstow_ctiN0001.fits
```
The Using the ACIS "Blank-Sky" Background Files thread contains instructions on how to select a file and match it to a specific observation.

Only CTI-corrected events cases have been included in this release. ACIS back-illuminated chip events, without CTI-corrections for GRADED mode data will be added in a future release.

Note that the ACIS background files are not included in the main CALDB tarfile. There is a separate background files tarfile available via ciao-install or from the CALDB Download page.
ACIS Substitute CTI Files (September-October 2005)

During the period of 2005 September 16 to 2005 October 15, there was an onboard ACIS anomaly causing an uncontrolled focal plane temperature, leading to FP_TEMP values to be to be +1.3 K than expected, affecting 71 ACIS ObsIDs, during this time period.

While this problem is fixed in archive Reprocessing IV, two substitute temperature-dependent CTI files have been added to CALDB 4.5.6 to correct for the +1.3 K FP_TEMP discrepancy to adjust the temperature-dependent CTI corrections to their proper values.
```
acisD2005-09-16ctiN0005B.fits
acisD2005-09-16ctiN0007.fits
```
With the introduction of temperature-dependent CTI correction in CALDB 4.4.0, any data from the 71 ObsIDs that has been reprocessed with the CTI-correction since then, will have propagated the FP_TEMP error into their temperature-dependent CTI-corrected event lists. Users that had reprocessed any of these datasets with the correction in CIAO would have selected on of the following files from CALDB:
```
acisD2002-08-01ctiN0007.fits (DATAMODE=FAINT or VFAINT)
acisD2002-08-01ctiN0005B.fits (DATAMODE=GRADED or CC33_GRADED)
```
The file used will be stored in the CTIFILE header keyword of a reprocessed L2 events list. Those users should consider reviewing their results and either reprocessing their data in CIAO with CALDB 4.5.6 or obtain the Repro IV data from the archives when available. Note that CIAO 4.5 is needed for this correction to be applied properly if reprocessing.
Calibration Validity End Date Correction (AFFECTS OBSID 5954 ONLY)

The calibration validity end dates were reset in the T_GAIN files associated with five previously unidentified ACIS ObsIDs (OBS_ID 5954 and 4 calibration observations), to address a problem with these files described here. Users analyzing any of the affected observations, who have not recently reprocessed the data, should do so using CalDB 4.5.6 in order for the ACIS time-dependent gain correction to be applied during manual reprocessing with acis_process_events or through chandra_repro.
ACIS EVTSPLT Validity Overlap Correction (AFFECTS OBSID 6318 ONLY)

The calibration validity dates overlapped between two existing ACIS EVTSPLT thresholds files caused T_GAIN file selection failures for ObsID 6318.

How CALDB 4.5.5 Affects Your Analysis

CALDB 4.5.5 Release Notes (release 13 December 2012)

ACIS Imaging and Grating Data

Time-dependent ACIS Gain (TGAIN) Files for -120 C Data

There are new time-dependent ACIS Gain (TGAIN) files for Aug - Oct 2012 (Epoch 51):
```
acisD2012-05-01t_gainN0005.fits
acisD2012-05-01t_gainN0006.fits
acisD2012-08-01t_gainN0002.fits
acisD2012-08-01t_gain_biN0002.fits (new default for SDP current processing)
```
The files are applicable to -120 C focal plane temperature only.

Users working with ACIS data taken since 01 May 2012 may wish to reprocess to improve the TGAIN calibration in their data. The DATE-OBS header keyword records the observation start date.

Note that unless you are fitting a spectra with oxygen emission lines, the gain refinement is unlikely to have an effect on the spectrum larger than the uncertainties in determining the gain.
ACIS Time-varying QEU files

The ACIS Quantum Efficiency Uniformity (QEU) has been varying over time due to event grade migration, a Charge Transfer Inefficiency (CTI) effect. This time dependence is corrected with the addition of new ACIS CALDB QEU files in 2-year intervals. This is a 3% correction for CTI-corrected S3 events (4% CTI-uncorrected) and a 1% correction for FI chips over the life of the mission.
```
acisD2000-01-29qeuN0007.fits
acisD2002-02-01qeuN0007.fits
acisD2004-02-01qeuN0007.fits
acisD2006-02-01qeuN0007.fits
acisD2008-02-01qeuN0007.fits
acisD2010-02-01qeuN0007.fits
acisD2000-01-29qeuN0005B.fits
acisD2002-02-01qeuN0005B.fits
acisD2004-02-01qeuN0005B.fits
acisD2006-02-01qeuN0005B.fits
acisD2008-02-01qeuN0005B.fits
acisD2010-02-01qeuN0005B.fits
```

HRC Imaging and Grating Data

HRC-I GMAP Upgrades

To continue to address the decline of the gain of the HRC-I, a new set of time-dependent gain maps are available in CALDB 4.5.5 which differ from the previous set in two ways: the maps are based on (and intended for use with) the scaled sum of amplifier signals (SUMAMPS) instead of PHA, as SUMAMPS is a better proxy for spectral response than PHA; and the time dependence of the gain decline is corrected using an inverse exponential plus linear function fit to temporal correction factors derived from AR Lac, G21.5-0.9 and HZ 43, instead of a pure linear function fit to the median AR Lac PHAs.

By default, PI is now calculated from scaled SUMAMPS ("SAMP") instead of PHA. The new gainmaps, listed below, convert SAMP to PI, correcting for the gain declines which have occurred since launch.
```
HRC-I 
hrciD1998-10-30sampgainN0001.fits
hrciD1999-10-04sampgainN0001.fits
hrciD2000-12-12sampgainN0001.fits
hrciD2002-01-26sampgainN0001.fits
hrciD2003-02-22sampgainN0001.fits
hrciD2004-11-25sampgainN0001.fits
hrciD2005-10-17sampgainN0001.fits
hrciD2006-09-20sampgainN0001.fits
hrciD2007-09-17sampgainN0001.fits
hrciD2008-09-07sampgainN0001.fits
hrciD2009-09-24sampgainN0001.fits
hrciD2010-09-25sampgainN0002.fits
hrciD2011-09-19sampgainN0001.fits
hrciD2012-09-27sampgainN0001.fits

HRC-S 
hrcsD1999-07-22t_gmapN0002.fits
hrcsD2012-03-29t_gmapN0002.fits
```
HRC-I PIBGSPEC Upgrades

The particle background in the HRC, which had been increasing due to the drop in solar activity, has begun to decrease again. Yearly background spectra obtained from AR Lac observations with HRC-I are now available in CALDB 4.5.5; The CIAO "HRC-I Background Spectra Files" thread describes how they can be used to filter on PI and reduce background.
```
hrciD1999-10-04pibgspecN0001.fits
hrciD2000-12-12pibgspecN0001.fits
hrciD2002-01-26pibgspecN0001.fits
hrciD2003-02-22pibgspecN0001.fits
hrciD2004-11-25pibgspecN0001.fits
hrciD2005-10-17pibgspecN0001.fits
hrciD2006-09-20pibgspecN0001.fits
hrciD2007-09-17pibgspecN0001.fits
hrciD2008-09-07pibgspecN0001.fits
hrciD2009-09-24pibgspecN0001.fits
hrciD2010-09-25pibgspecN0001.fits
hrciD2011-09-19pibgspecN0001.fits
hrciD2012-09-27pibgspecN0001.fits
```
HRC-S QE version N0012 Upgrade

On 29 March 2012, the High Voltage settings of the HRC-S microchannel plates were increased to address a loss in quantum efficiency as a function of wavelength, caused by a sag in gain over time. Analysis of data prior to and after the adjustment resulted in a wavelength-dependent correction to the HRC-S QE, now available in CALDB 4.5. Users can expect an increase in counts of 4.5-6% dependent upon wavelength, which is corrected by the revised QE map. More refinements to the QE for HRC-S will likely follow in future years.
```
hrcsD1999-07-22qeN0012.fits
hrcsD2012-03-29qeN0012.fits
```

Installation

Users should be aware of these installation items before installing CIAO 4.5. Additional problems which are seen less frequently are listed on the Installation & Smoke Tests bug page.

Supported Platforms

CIAO 4.5 is supported on:
- Linux 32 bit and 64 bit (tested on CentOS 5.8 and Red Hat Enterprise 6 )
- Mac OS X 10.6 Intel 64 bit
- Mac OS X 10.7 & 10.8 Intel 64 bit
The following platforms are no longer supported:
- Mac OS X 10.5
- Mac PPC
- Mac 32 bit
- Solaris 10

Updates to the ciao-install Installation Script

The script will not run if called with sudo, i.e. "sudo bash ciao-install". If ciao-install encounters a permissions problem, it prints instructions on what to fix in order to install CIAO and gives the user the chance to specify a different installation directory.

Remove old parameter files

With every new CIAO release, some parameter files have changed: new parameters may be added and occasionally old ones removed or renamed. Deleting or renaming the local parameter directory ensures that the correct parameter files will be accessed the first time a tool is run:
```
unix% rm ~/cxcds_param4/*
```

Remove the fontconfig cache

The GUI applications (e.g. prism, ChIPS, ds9) used in CIAO may not run correctly if there are incompatible .fontconfig files cached on the user's system. To remove the .fontconfig cache:
```
unix% rm ~/.fontconfig/*
```

IPython Settings

CIAO 4.5 now includes a newer version of IPython, 0.13. If upgrading from CIAO 4.4 or earlier, the ipython configuration files stored in the users home directory are incompatible with the new version; a new copy will be installed in a new directory: $HOME/.ipython-ciao4.5. If you have made customizations with earlier version of CIAO, you will need to manually migrate those changes to work with CIAO 4.5.

Tools

acis_find_afterglow

Updated the logic to correctly determine if an event should be removed from the search for hotpixels based on whether the pixel is part of an afterglow or not.

addresp

Added additional error checking to catch situation where the inputs are invalid (empty, zero row PHA file) and when various inputs are missing (lookupTab).
The ranges of columns in input file are now copied to the output.

calquiz

add two command-line options, "echo_qual" and "echo_fidel", to return the values of CAL_QUAL and FIDELITY respectively.

dmcoords

The internal help failed to mention the MSC command to set mirror spherical coordinates.

dmimgfilt

Two new sigma-clipping filters are now available: 3sigmedian and 3sigmean.

dmextract

PHA files made by "dmextract opt=pha1" write a BACKSCAL keyword containing a normalized extraction area. The usual normalization uses the TLMAX/TLMIN keyword values, but for ACIS a fixed value of (8192)**2 is now used in order to correctly handle the new behaviour of the reprojection tools. This value is set in the $ASCDS_INSTALL/data/cxo.mdb file.

dmhistory

Correct problem if comment contain "%" sign.

dmmerge

Set (TLMIN,TLMAX) for output SKY and POS to the union of the ranges (TLMIN,TLMAX) from each input file. Allows event files to be merged to create mosaics larger than default image sizes.

dmsort

Fix problem when first sort column is string and multiple rows have same value.

hrc_process_events

Update to support gain tables with arbitrary number of rows.

mkacisrmf

Improved error reporting. Will now warn users when the input grid is truncated to match the calibration data.

mkexpmap

Fixed units from "sec" to "s"
GRATING keyword is now copied from input instrument maps

mkgrmf

When interpolating or extrapolating the grating efficiencies, any resulting values less than zero will be set to zero. (Change is in ardlib)
DETCHANS, HDUCLAS2, etc keywords were added to the EBOUNDS extension of the output file for better OGIP compliance.
The value of the HDUCLAS2 keyword in the output file has been corrected.

mkpsfmap

Users can now supply a spectrum and the psfmap will be spectrally weighted.

By setting
```
energy=INDEF
```
user can then set the spectrum parameter to a file name. The input is the same format as used by mkinstmap and user can use the same make_instmap_weights tool to create it.

mkpsf

Withdrawn. Users should use either ChaRT or MARX to simulate the point spread function.

modelflux

A new option to calculate the unabsorbed flux has been added. If the user does choose this new option, then both absorbed and unabsorbed fluxes are calculated and reported to stdout, and written to the modelflux parameter file.

prop_colden

Output coordinates are now identified as having been converted to B1950
Fixed issue where later entries in a file of input positions were handled incorrectly

reproject_events

Update TLMIN/TLMAX for SKY and POS only, such that it represents the union of the actual data range with the "legal data range". Allows for mosaics of large datasets that exceed the standard image sizes.

tg_resolve_events

Units on columns are copied to the output.

tg_findzo

This new tool can be used to locate the 0-th order location in those observations that suffer from pileup or were performed with the 0th order events masked out.

wavdetect

Internal updates various parts of the code to increase performance and address memory leaks.
Wavelet scales > 100 are now fully supported.

wcs_match

A new option has been added that will compute a simple translation-only solution based on the matched sources. This can be used when the full translate, rotate, and scale solution is biased by spatial distribution of matched sources.

Parameter Files

A summary of parameter files changes is provided in this section. Refer to the Tools and Analysis Scripts sections of these release notes for complete details.

We recommend deleting all the old parameter files or renaming the parameter-file directory before running any new version of CIAO to avoid conflicts. More information is available in the FAQ.

aprates

The min values for alpha and beta are now set to 0

calquiz

add two parameters, "echo_qual" and "echo_fidel", to return the values of CAL_QUAL and FIDELITY respectively.

dmimgfilt

The function parameter has two new options: 3sigmean and 3sigmedian.

mkpsfmap

New spectrum parameter allows output to be spectrally weighted.

wcs_match

A new translation only solution is available via the new method parameter. The default "rst" uses the full rotate, scale, and translate solution where as the new translation only option is selected with method=trans.

Data Model

Enhancements & Bug Fixes

It is now possible to discriminate between 0 and small numerical values (e.g. values smaller than ~ 10^^-16) in a Data Model filter.

With the file:

unix% cat tst.dat
#data
0.0
1.0e-13
1.0e-15
1.0e-16
1.0e-18
1.0e-10
1.0e-20

CIAO 4.5:

unix% dmlist "tst.dat[data = 0]" data,clean
#  data
                    0
unix% dmlist "tst.dat[data > 0]" data,clean
#  data
                1E-13
                1E-15
                1E-16
                1E-18
                1E-10
                1E-20

CIAO 4.4:

unix% dmlist "tst.dat[data = 0]" data,clean
#  data
                    0
                1E-16
                1E-18
                1E-20
unix% dmlist "tst.dat[data > 0]" data,clean
#  data
                1E-13
                1E-15
                1E-10

The Data Model filter expression
```
[sky=circle(a,b,c),y=m:n,p:q]
```
where the same columns (in this case y are involved in both a spatial filter and a range filter) is ambiguous - is the combination meant to intersect or union the two filters?

The correct form of the filter is to use the && or || terms to indicate how the filters should be combined, but this did not work in CIAO 4.4.

It is now possible (in CIAO 4.5) to say
```
unix% dmcopy "evt.fits[sky=circle(4096,4096,100)&&y=4020:4100,4250,4350]" filt1.fits
unix% dmcopy "evt.fits[sky=circle(4096,4096,100)||y=4020:4100,4250,4350]" filt2.fits
```
There was also a related problem when using && to combine filters on the same field; CIAO 4.5 allows you to use expressions like
```
unix% dmlist "acisf00635_000N001_evt3.fits[(grade=2)&&(grade=0)]" counts 
```
The dmcopy tool no longer displays a warning message saying Filter data type mismatch when filtering on BIT or BOOLEAN (logical) columns.
Improve the error message generated if an exclude filter is used after a range filter in the same virtual file expression, since these can not be combined.
The FITS keywords BSCALE and BZERO have been added to the list of reserved keywords used by the Data Model.
Improvements have been made in reporting errors in the syntax of column filters.
The Ascii Kernel for the Data Model has seen improvements in the handling of quoted strings.

PyCrates

The get_filename method of a crate now returns any Data Model filter used to open the file; in CIAO 4.4 it just returned the file name.

CIAO 4.5:

chips-1> cr = read_file("4948.asol[#row=:100][cols ra,dec,roll]")
chips-2> cr.get_filename()
         '4948.asol[#row=:100][cols ra,dec,roll]'

CIAO 4.4:

chips-1> cr = read_file("4948.asol[#row=:100][cols ra,dec,roll]")
chips-2> cr.get_filename()
         '4948.asol'

The add_col routine will now create a column name - of the form COLUMNm, where m is the column number - if the Crate data object has no name (in CIAO 4.4 the column would have been named "()" when the data was written out to a file).
The set_key routine has been added; it either creates a new keyword, or updates an existing one, in the file header.
The read_pha routine, and the PHACrateDataset object, now use the block called SPECTRUM as the "most-interesting block", that is, consider this the current block, regardless of its position in the file (in CIAO 4.4 the first block would be used, which could be a WMAP block).
There were problems writing out files if a compressed version (i.e. *.gz) of the file already existed.
It is no longer necessary to use the snip method of the parent object of a Crate - the object returned by the get_dataset method - to free up memory from a crate. The following code will still work, but the call to snip now does nothing and this method is deprecated, and will be removed in a forthcoming major release of CIAO.
```
try:
    cr = read_file(filename)
    ... process contents ...

finally:
    cr.get_dataset().snip()
```

ChIPS

Major enhancements to the ChIPS GUI

It is now possible to add annotations - labels, lines, points, regions, or color bars - directly from the ChIPS GUI.

Users can add an annotation by selecting the Annotation sub-menu from the Right-Mouse-Button menu of the ChIPS window:

or from the Annotate menu item in the main GUI window:
The "position" values shown in the GUI - e.g. for axis limits and for the coordinates of objects such as labels - have been updated to support scientific and sexagesimal notation and to provide better precision. Values can be adjusted with the keyboard, using the up and down buttons, or by scrolling the mouse scroll button up and down.
It is now possibile to change the position - and coordinate system - for most annotations, such as labels, from the main GUI window. You can still select these objects and drag them with the mouse, or move them with the arrow keys.
The GUI interface for using, and setting, manual (i.e. arbitrary) tick positions on axes (e.g. as can be done with the set_arbitrary_tick_positions command), has been greatly improved:
- when a row is highlighted return or space will allow you to edit the cell, up/down arrow keys will allow you to change rows, left/right arrow keys will allow you to change column, and tab will move the focus from the columns to the Add button;
- when editing a cell tab, return, or an arrow key will allow you commit the change;
- and after clicking on the Add button a new row is added, the table scrolls down to that row and sets the focus to that row.

Other changes

In CIAO 4.4 the ~/.chips.rc file that ChIPS creates (if one does not exist) did not contain any of the GUI preference settings. To check, you can
```
unix% mv ~/.chips.rc ~/.chips.rc.old
unix% chips -b
unix% diff ~/.chips.rc.old ~/.chips.rc
```
and adjust the ~/.chips.rc file for settings that have changed and you wish to keep the old values.

In the above, the -b option makes ChIPS run in "batch mode", where it exits after processing any command-line script; in this case as we supply no file it just runs the ChIPS set up code, which includes creating the ~/.chips.rc file, and then exits.
Curves containing a single point were not being displayed.
Support for curves containing values which vary by very small amounts, or are very small, has been improved.
When given a file name or a Crate, make_figure>, add_curve, and add_histogram now support multiple values in a single row, as used by PHA2 files.
Creating images from images, Crates, or numpy arrays with single dimension will now produce a 1 x n image, where n is the number of elements in the array. For instance:
```
chips> add_image(np.arange(256))
chips> set_data_aspect_ratio('')
chips> limits(Y_AXIS, 0.5, 1.5)
```
The add_histogram call has been fixed so that
```
chips> add_histogram(x, y, ['*.color', 'red'])
```
now sets the fill.color attribute.
Adding closed regions - i.e. those regions where the first and last coordinates are the same - caused the ChIPS server to crash.
The first coordinate pair returned by pick or get_pick was sometimes incorrect.
The boundary used to highlight a selected label was too large when the label contained greek characters, or was invalid for regions created in a world coordinate data system.
It was not possible to drag plots which were embedded within other plots (e.g. the smaller plot in the Placing one plot within another plot ChIPS gallery example).
The binary state file created by save_state now contains settings for the axis ticklabel.style attribute. This means that CIAO 4.5 state files can not be read into earlier versions of CIAO, but earlier versions can still be read into CIAO 4.5.
The make_script output now uses the correct coordinate system for annotations and improvements have been made to support visualizations containing multiple data systems (e.g. multiple axis pairs in a plot).
The set_arbitrary_tick_positions command now works correctly when the label for the first tick mark is an empty string (e.g. see the Adjusting axis labels Gallery example).

Axes that were set to arbitrary mode before a call to bind_axes would not update to display the correct tick positions.

Ascii scripts created by make_script did not handle axes with arbitrary tick marks.

The coordsys attribute of attribute objects - such as ChipsLabel - could not be used to specify the coordinate system when creating an annotation.

chips> lbl = ChipsLabel()
chips> lbl.coordsys = FRAME_NORM
chips> lbl.size = 18
chips> lbl.color = "olive"
chips> lbl.font = "times"
chips> lbl.halign = 1
chips> add_label(0.95, 0.92, "A label", lbl)

Improvements in the formatting of the numeric labels of axes:
- changing the tick axis format to %Z or %# could cause array bounds errors and memory leaks;
- binding of WCS axes and then changing the tickformat did not re-display the majorgrid lines correctly;
- the GUI did not always display the correct setting for the tickformat in the axis properties area when sexagesimal formats such as %ra, %RA, or %DEC were used.
The handling of user-defined fonts - i.e. those loaded with the load_font command - has been improved in the GUI.
Several corner cases have been fixed that could cause invalid PDF files to be created.
Several minor improvements to the GUI, including:
- there were problems in over-writing files when using the File/Save As menu item to save either Python scripts or a binary state file;
- the color selection are has been enhanced so that - when displaying the list of color names, rather than the color wheel - the currently-selected color is highlighted (if there is a match), and information on the value of the default color is provided (when selected);
- regions can now be deleted from the ChIPS plot by selecting the region (so that the selection handles are displayed) and then use the delete key on the keyboard;
- the default option of "Pan mode" was not working correctly when the visualization was created (users would have to switch modes and then back to Pan mode to get it to work);
- the show_gui command did not correctly set the path of the embedded Python console when called with an argument;
- the interactive Python shell now traps attempts to end the shell, such as calls to quit or exit;
- the Save Dialogs now add saved files to the Recently-Used list, which can then be viewed in later uses of Save/Open file;
- terminology has been cleaned up to use 'Frame' consistently in the help and menus of the GUI (in CIAO 4.4 'Frame' and 'Pane' were used inter-changeably);
- and the FAQ menu item has been removed.
ChIPS will now exit faster when the plot contains a large number of objects.

Sherpa

Grouping

Fixes to retain filters after dynamic grouping
A new rule, that when a dynamic grouping scheme is applied to a data set, it is also automatically applied to all associated background data sets. Commands such as:
```
group_snr(3)
```
now apply this grouping condition to both source and background. It is no longer necessary to manually group the background(s). The filter is also correctly updated for both source and background.

Note that this means that source and background will have *different* arrays of grouping flags. If the background has fewer counts, for example, then it will naturally have fewer groups if all groups must meet the condition that the SNR >= 3. The user can still set a different background grouping scheme with:
```
group_snr(3)
group_snr(5, bkg_id=1)
```
Here, both source and background will be grouped with SNR >= 3 by the first command; but the background alone will then be grouped by SNR >= 5 by the second command.

Models

The Sersic2d model has been added. This model is a generalization of the de Vaucouleurs profile (i.e., the exponent is allowed to be values other than 4).

Fake

fake_pha will now include simulated background in the faked dataset.

User Interface

A new function, sample_flux, has been added to calculate the unabsorbed energy flux due to a Sherpa model. It returns a sample of parameters with a corresponding flux and a flux uncertainty for a model component or a combination of model components. The model components have to be previously defined and used in the fit. The samples are generated from the multi-variate normal distributions with the scales defined by covariance (if at the best fit) or supplied (as "scales"). The flux is calculated for each set of new parameters. The returned flux value is given by a sample's median with the lower and upper quantiles defined by the confidence level supplied to the function.

GUIs (Graphical User Interfaces)

Prism

Fixed issues with some icons not displaying in prism (e.g the tab icon display the status of the file)

Obsvis

New tri-state toggle switch to identify optional ACIS CCDs.
New functionality was added to show and hide the target, nominal aimpoint, and optical axis markers. Toggles in the Display Parameters frame (left side of the obsvis gui above the button row) allow per FOV selection of which markers are visible. In addition, the initial settings of the markers can be specified in the Preferences dialog.
Changed startup script from ksh to bash

Analysis Scripts

The CIAO scripts package is considered a required part of the software installation.

Version names

The labeling of the scripts package has changed from a date to a numeric system; this version number is included in the $ASCDS_CONTRIB/VERSION.CIAO_scripts file - e.g.
```
unix% cat $ASCDS_CONTRIB/VERSION.CIAO_scripts 
scripts 4.5.1 Thursday, December 13, 2012
```

specextract

The script has been updated so that it will exit with an error if the input event file is missing a CTI_APP keyword (see the note on the removal of the check_ctiapp.sh script below).

Removed: check_ctiapp.sh

The check_ctiapp.sh script - which is no longer needed because the data issues in the archive it was needed for have long-since been fixed - has been removed from the scripts package with the CIAO 4.5 release. Please see the CTI why topic for more information.

Deprecated: merge_all

The merge_all script is deprecated and has been removed from CIAO. Please use one or more of merge_obs, reproject_obs, and flux_obs instead. See the Using merge_obs to combine observations and create exposure-corrected images thread for more information.

Deprecated: mkBgReg.pl and mkSubBgReg.pl

The mkBgReg.pl and mkSubBgReg.pl scripts are deprecated and will be removed in a later release of CIAO. Please use the roi tool and the splitroi script since they are more powerful than the scripts they replace, as discussed in the An Image of Diffuse Emission thread.

Libraries

cxcparam

The cxcparam python module has gained the plist function has been added to retrieve the list of all parameter names associated with a parameter file:
```
>>> import paramio as pio
>>> pio.plist("dmlist")
['infile', 'opt', 'outfile', 'rows', 'cells', 'verbose', 'mode']
```

stack

The stack Python module has been completely re-written in CIAO 4.5. The new module is called stk and returns a native Python list of strings rather than an opaque wrapper around the underlying C structure.

CIAO 4.5:

>>> import stk
>>> myfiles = stk.build("a1,b2[cols x,y],c3")
>>> len(myfiles)
3
>>> myfiles[1]
'b2[cols x,y]'
>>> myfiles = stk.build("")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
Exception: Empty stack string.

CIAO 4.4:

>>> import stack as stk
>>> mystk = stk.stk_build("a1,b2[cols x,y],c3")
>>> stk.stk_count(mystk)
3
>>> stk.stk_read_num(mystk, 2)
'b2[cols x,y]'
>>> mystk = stk.stk_build("")
>>> stk.stk_count(mystk)
1
>>> stk.stk_read_num(mystk, 1)
''

The new module returns a list of strings which is easier to iterate over using native Python controls. Note that building a stack from an empty string now raises an exception that may need to be dealt with differently than before.

caldb4-lib

The following changes are not going to affect most users:
- Reorganize the querying thread in the order of telescope/instrument/product and other general parameters, boundary condition parameters, start and stop date/time specification, data fidelity and data quality. The order conforms to new CALDB4 Requirements.
- Rewrite the algorithm for extracting the largest start and smallest stop from the valid data set.
- Modify handling of END_TIME NULL value (nullval=0.0) in key-config (internally re-assigned to nullval=999999.0).
- Add checks of date/time formats and MJD to UTC conversions before starting any querying. Print out warnings of any inconsistency between the Indexed values and the run-time calculation.
- Retain double precision of REF_TIME/END_TIME all the way to applications.
- Add debug-level macros for consistent appearance of output messages.
- Print out intermediate query results for diagnosis and understanding of the final output at high verbosity levels.
- Make "bad" data returnable for "cal_qual=5" queries.
- Fix the output message for "match=firstwarn" or "match=lastwarn".
- Allow double oquted text in Index boundary condition parameters.

Environment

Off-the-Shelf (OTS) Package Versions

The following OTS packages are included with CIAO 4.5. For more information on how the OTS packages are built for use with CIAO, refer to the INSTALL_SOURCE file distributed with the software.

Executables
- ds9 v7.1
Python
- IPython v0.13 (this does not include the extra modules needed to take advantage of the interactive notebook capabilities of IPython, for instance IPython.parallel or IPython.zmq).
- NumPy v1.5.1
- pyds9 v1.3
- Python v2.7.2
Libraries
- CCfits v2.4
- cfitsio v3.290
- fftw v3.3.1
- fontconfig v2.8.0
- freetype v2.4.4
- gsl v1.15
- readline v6.2
- vte v0.25.1
- VTK v5.6.1
- wcssubs v3.8.3
- xpa v2.1.13
- XSpec v12_7_1 (models only)

Documentation

Analysis Threads

The CIAO and Sherpa Threads are in the process of being tested and updated for the CIAO 4.5 release. Please check the "Last Update" information at the top of each thread to see if it has been reviewed. The ChIPS Threads have been updated.

/ ciao / releasenotes / ciao_4.5_release.html

CIAO 4.5 Release Notes

ACIS Imaging and Grating Data

ACIS QE Contamination Model vN0008

Time-dependent ACIS Gain (T_GAIN) Files for -120 C Data

HRC-I Imaging Data

HRC-I Gain Maps

ACIS Imaging and Grating Data

Time-dependent ACIS Gain (T_GAIN) Files for -120 C Data

ACIS Imaging and Grating Data

Time-dependent ACIS Gain (T_GAIN) Files for -120 C Data

ACIS Blank-sky Background File (Group F ACIS BKGRND)

HRC-S T_GMAP with GRATING=NONE

ACIS Imaging and Grating Data

Time-dependent ACIS Gain (T_GAIN) Files for -120 C Data

ACIS Blank-sky Background File (Group F ACIS BKGRND)

ACIS Substitute CTI Files (September-October 2005)

Calibration Validity End Date Correction (AFFECTS OBSID 5954 ONLY)

ACIS EVTSPLT Validity Overlap Correction (AFFECTS OBSID 6318 ONLY)

ACIS Imaging and Grating Data

Time-dependent ACIS Gain (TGAIN) Files for -120 C Data

ACIS Time-varying QEU files

HRC Imaging and Grating Data

HRC-I GMAP Upgrades

HRC-I PIBGSPEC Upgrades

HRC-S QE version N0012 Upgrade

Supported Platforms

Updates to the ciao-install Installation Script

Remove old parameter files

Remove the fontconfig cache

IPython Settings

acis_find_afterglow

addresp

calquiz

dmcoords

dmimgfilt

dmextract

dmhistory

dmmerge

dmsort

hrc_process_events

mkacisrmf

mkexpmap

mkgrmf

mkpsfmap

mkpsf

modelflux

prop_colden

reproject_events

tg_resolve_events

tg_findzo

wavdetect

wcs_match

aprates

calquiz

dmimgfilt

mkpsfmap

wcs_match

Enhancements & Bug Fixes

CIAO 4.5:

CIAO 4.4:

PyCrates

CIAO 4.5:

CIAO 4.4:

Major enhancements to the ChIPS GUI

Other changes

Grouping

Models

Fake

User Interface

Prism

Obsvis

Version names

specextract

Removed: check_ctiapp.sh

Deprecated: merge_all

Deprecated: mkBgReg.pl and mkSubBgReg.pl

cxcparam

stack

CIAO 4.5:

CIAO 4.4: