/ ciao / releasenotes / ciao_4.12_release.html

CIAO 4.12 Release Notes

CIAO 4.12 is distributed for the following platforms:

Linux 64 bit
Apple macOS 10.12 (Sierra) through macOS 10.15 (Catalina)

[New] CIAO is now available via the conda package manager: conda builds are available for Linux and Mac for Python 3.7, 3.6, and 3.5. More information can be found in the conda section of the Installation notes below.

There is no support for 32 bit operating systems, older Linux (CentOS 5 era) or older macOS platforms (OS-X El Capitan and earlier). More details can be found on the Platform Support page.

Notable changes and improvements in CIAO 4.12
How CALDB 4.9.3 Affects Your Analysis
How CALDB 4.9.2.1 Affects Your Analysis
How CALDB 4.9.1 Affects Your Analysis
How CALDB 4.9.0 Affects Your Analysis
Installation
Tools
Parameter Files
ChIPS
Sherpa
Graphical User Interfaces
Analysis Scripts
Python Modules
Libraries
Environment
Documentation

Notable changes and improvements in CIAO 4.12

CIAO 4.12 has several important event processing updates including ACIS badpixel updates to exclude frame-store shadow region, HRC updates to gain calibration, HRC degap calibration file changes, and improvements to the ACIS temperature dependent CTI correction.

CALDB 4.9.0 version requirements

CIAO 4.11 and earlier users analyzing ACIS data should NOT upgrade to CALDB 4.9.0. While the new format of the temperature dependent CTI correction calibration files can be read by earlier versions CIAO, the higher order coefficients are not included. The new columns are ignored resulting in an inaccurate temperature adjustment being applied.

CIAO 4.12 HRC users must update to CALDB 4.9.0 due to the changes in the degap file. Similarly, CALDB 4.9.0 should not be used with earlier versions of CIAO. In both cases doing so will trigger an earlier degap algorithm that performs badly at the edges of the plates.
The ChIPS plotting package has been withdrawn from CIAO 4.12. Users should migrate their sherpa and plotting scripts to use matplotlib. The CXC has provided a conversion guide to assist users to modify their scripts to use matplotlib. prism has been updated to create plots using matplotlib.
CIAO can now be installed using the conda package manager. This is still experimental. We have performed our standard set of tests using generic, vanilla conda installations however given the large number of packages and installation options it is possible users may encounter unforseen problems. Please report these to the CXC Helpdesk. Users can install the conda edition with Python 3.7, 3.6, or 3.5 on Linux and Mac.

Feedback Requested

The CXC is actively soliciting user feedback on this installation method; users can submit their comments to the CXC Helpdesk.

Please review the known problems on the conda installation page before contacting the helpdesk.
CIAO includes version 8.1 of SAOImage ds9. Users are reminded that they now need to change the Edit mode setting in order to select or create regions in recent versions of ds9. Please see the watchout page for more information on this.

SAOImageDS9 v8.1 highlights include:
- New support for overlaying masks based on world, physical or image coordinates. Each mask is rotated, scaled, and centered upon the specified coordinate system. There is no limit to the number of masks which can be displayed.
- An all new plot tool is now available. Each plot tool dialog can contain one or more graphs, arranged in row, column, or grid order. In addition, a new strip plot layout is also available. Support for line, bar, or scatter graphs is provided. Each graph has its own set of display parameters and can contain one or more data sets. Each plot dialog maybe printed as a postscript image, or exported as tiff, png, jpeg, or gif.
- The movie feature now supports animated GIFS. 3D data cubes, or a sequence of multiple frames can be animated, and saved as an animated GIF to be displayed on the desktop, in twitter or on a web site.
- Contours have been reimplemented to provide high quality resolution and have been threaded to take advantage of multiple CPUs, providing fast rendering, especially in the case of 3D data cubes.
Sherpa includes both the changes made in the Sherpa 4.11.1 release and Sherpa 4.12.0. Notable changes include:
- Plotting is now handled by Matplotlib as ChIPS has been removed in CIAO 4.12.
- Update to version 12.10.1n of the XSPEC Model library, which adds support for the following additive models: xsagnsed, xscph, xskyrline, xsqsosed, xsvcph, xszbknpower and xszlogpar.
  
  Default XSPEC abundance and cross-section settings
  
  The 12.10.1 XSPEC model library now sets default values based on the contents of your $HOME/.xspec/Xspec.init file if it exists.
  
  If you do not have this file then the default cross-section table has changed: it now defaults to vern when it was bcmc in CIAO 4.11 and earlier.
- The plot_xxx family of routines - such as plot_data and plot_fit - can now accept plot settings as keyword arguments. The allowable keywords are the same as those returned by the get_xxx_plot_prefs routines, such as xlog, ylog, and color.
  
  As an example,
```
plot_fit(xlog=True, ylog=True)
```
  will use logarithmic axes for both axes of the plot.
- Sherpa now supports the use of a PSF image with a finer resolution than the resolution of 2D data images. 1D models can also be evaluated on a finer grid than the grid of the input data arrays. The models are evaluated on the resolution given by the PSF pixel scale or the 1D arbitrary grid. The evaluated models are rebin to match the original data scales for calculating the statistic during the fit.
- Asymmetric errors are now supported for calculating parameter uncertainties via bootstrap. The data with asymmetric errors can be entered using a new load_ascii_with_errors function or Data1DAsymmetricErrs class.
- The paging performed by the show_xxx set of commands - such as show_data and show_all - is now handled by Python rather than using an external command. This means that the output will now appear in Jupyter notebooks, and that the PAGER environment variable is no longer checked.
As a reminder, Sherpa is also available as a stand alone system, accessible from the Sherpa GitHub repository or from the standalone Sherpa page.
The CXC Datamodel has several updates related to reading various ASCII files.
Several tools have been updated including
- Updates to the temperature dependent ACIS CTI correction algorithm in acis_process_events. These changes will be activated by future CALDB updates; until then the temperature dependent CTI corrections will be unchanged.
- Rows at the bottom of the ACIS CCDs are now being marked as bad due to the "shadow" cast by the frame store covering.
- HRC updates to support pending gain calibration changes and to fix a technical issue with the previously released updated degap files.
- wavdetect now creates additional diagnostic columns to help identify duplicate sources.
- Many tools have been updated to properly copy the image units from the input to the output image.
prism, peg, and taskmonitor now use GTK3. prism has been updated to use matplotlib for plotting.
There have been several script updates since the CIAO 4.11 release including:
- chandra_repro updates to support new badpixel CALDB updates (frame store shadow).
- Updates to several script to use matplotlib for plotting.
- Updates to dax include using ds9 for plotting, updates to simplify existing analysis tasks, and new radial profile fitting option.
- Support for Version 2 of the Chandra Source Catalog.

How CALDB 4.9.3 Affects Your Analysis

CALDB 4.9.3 Release Notes (release 15 October 2020)

ACIS Imaging and Grating Data

Time-dependent ACIS Gain (T_GAIN) Files for -120 C Data

The new, combined time-dependent ACIS Gain (T_GAIN) file for May-July 2020 (Epoch 82) and February-April 2020 (Epoch 81) are introduced. The combined Epochs 79+80 file has also been updated since CALDB 4.9.2.1. Therefore, the new T_GAIN affects observations taken since 2019 May 02. Other observations are unaffected by these new calibration files.

Users working with ACIS data taken since 2019 Mayy 02 may wish to run chandra_repro and reprocess the data to improve the T_GAIN calibration. The DATE-OBS header keyword records the observation start date.

See the time-dependent ACIS T_GAIN why page for more information.

The gain corrections are at the nominal and expected levels of less than 1% of the energy value and users interested in CCD spectroscopy may benefit in applying the new gain adjustment. Grating spectroscopy benefits as well, in the form of improved order sorting. It is unnecessary to apply this adjustment for only doing timing or imaging analysis, although doing so will not have a negative effect.

Note that only spectra with several hundreds of counts and/or prominent features (in emission or absorption) will show changes from the T-gain refinement that exceed the uncertainties from the gain calibration.

How CALDB 4.9.2.1 Affects Your Analysis

CALDB 4.9.2.1 Release Notes (release 09 July 2020), includes the CalDB 4.9.2 updates.

ACIS Imaging and Grating Data

Time-dependent ACIS Gain (T_GAIN) Files for -120 C Data

This update affects all ACIS observations made at the -120 C focal plane temperature, covering Epochs 1-80 (29 Jan 2000-31 Jan 2020). The chandra_repro script and reprocess the data thread to improve the T_GAIN calibration may be used to apply the new calibration results. The DATE-OBS header keyword records the observation start date.

The time-dependent ACIS gain files applies a new algorithm to improve statistical averaging for data with decreasing counting statistics, first introduced in the ACIS T_GAIN Epoch 75-76 file (CalDB 4.8.3, May 2019), to all ACIS time-dependent gain files. The algorithm has resulted in considerable improvement in the resulting noise level in the corrected PHA/ENERGY data in the events lists.

See the time-dependent ACIS T_GAIN why page for more information.

While the gain corrections improves results for imaging spectroscopy, grating spectroscopy benefits as well, in the form of improved order sorting. It is unnecessary to apply this adjustment for only doing timing or imaging analysis, although doing so will not have a negative effect.
ACIS Blank-sky Background Files

This update to the blanksky background files changes header keywords, expanding the focal plane temperature boundaries to let them be selected using acis_bkgrnd_lookup when working on ObsIDs with warmer ACIS focal plane settings.

HRC Data

HRC-S Time-Dependent Gain Maps

Rapid loss of HRC-S gain since the year 2015 has led to problems with filtering background events from HRC-S/LETG LETG/HRC-S observation spectra, requiring the development of a new time-dependent method to generate the gain maps. Making use of these new T_GMAPs require changes to hrc_process_events which were introduced in CIAO 4.12.

This results in being able to filter on pulse height in LETG spectra as a function of dispersed wavelength in order to reduce the background by over 50% for wavelengths >20 Å, compared with standard Level 2 processing.
HRC-S QE Files

Regular calibration observations have shown that the HRC-S QE is declining at longer wavelengths (energies <200 eV, >60 Å), decaying linearly since the 2012-03-29 high-voltage adjustment onboard. This spectral variation in the QE loss is accounted for using a series of time-displaced QE files introduced in this CALDB.

These CIAO tools and scripts automatically apply the HRC-S QE files when creating response files:
- mkgarf
- fullgarf
- mktgresp
HRC-I Time-Dependent QE Files

The HRC-I quantum efficiency QE has been decaying for some time, particularly since the year 2011, when a 5% to 10% per year reduction had been measured. To address this issue, the HRC-I Calibration Team had introduced a time-dependence into the QE files in 2018 which are now extended in time to the latest epoch of observations. While the time-dependent modeling of the QE has improved, this update introduces a retroactive change of ~0.5%, even for the oldest epochs. The QE update affects the results from mkinstmap and fluximage and is applicable to all previous epochs of HRC-I observations.

How CALDB 4.9.1 Affects Your Analysis

CALDB 4.9.1 Release Notes (release 03 April 2020)

ACIS Imaging and Grating Data

Time-dependent ACIS Gain (T_GAIN) Files for -120 C Data

The new, combined time-dependent ACIS Gain (T_GAIN) file for November 2019-January 2020 (Epoch 80) and August-October 2019 (Epoch 79) are introduced. The combined Epochs 77+78 file has also been updated since CALDB 4.8.4.1. Therefore, the new T_GAIN affects observations taken since 2019 February 01. Other observations are unaffected by these new calibration files.

Users working with ACIS data taken since 2019 February 01 may wish to run chandra_repro and reprocess the data to improve the T_GAIN calibration. The DATE-OBS header keyword records the observation start date.

See the time-dependent ACIS T_GAIN why page for more information.

The gain corrections are at the nominal and expected levels of less than 1% of the energy value and users interested in CCD spectroscopy may benefit in applying the new gain adjustment. Grating spectroscopy benefits as well, in the form of improved order sorting. It is unnecessary to apply this adjustment for only doing timing or imaging analysis, although doing so will not have a negative effect.

Note that only spectra with several hundreds of counts and/or prominent features (in emission or absorption) will show changes from the T-gain refinement that exceed the uncertainties from the gain calibration.
DET_GAIN and P2_RESP FI CCDs Upgrade

For some time, the ACIS calibration team has been aware of a small gain drop for each front-illuminated CCD, persisting since the beginning of the mission. Currently, these "gain droops" have some energy dependence in their depth as well as their spatial extent, seen in the mid-chip region of the FI CCDs corresponding to: CHIPX=~480-540, and CHIPY=~100-900. The current ACIS DET_GAIN file version N0007 (along with earlier versions) does not correct for this effect. The new version N0008 DET_GAIN file accounts for and corrects for the gain droop on all eight of the FI chips. Along with a corresponding P2_RESP N0008 file (which is the input for CIAO 4.12 tool mkacisrmf, and must be matched to the DET_GAIN file), the new DET_GAIN file is applicable to non-GRADED DATAMODE cases, wherein the CTI_APP keyword is set to "PPPPPBPBPP", indicating the configuration where all ACIS chips are CTI-corrected. At present, there is no corrected DET_GAIN or P2_RESP applicable to GRADED mode data sets.

Note that the BI chips ACIS-S1 and ACIS-S3 are not affected.

These CIAO tools and scripts automatically apply the gain map files:
- acis_process_events
- chandra_repro
and the P2_RESP files are used by:
- specextract
- mkacisrmf

HRC Data

HRC-S Time-Dependent QEU Files

The HRC-S time-varying quantum efficiency uniformity files have been replaced with a new set, calculated based on unfiltered calibration observations. This latest set (N0013) shows very minor change over the prior set of QEU files, but incorporates results from the latest HZ43 calibration observations; however the newest QEU for observations taken since 2019 August 22 includes the ~5% drop of QE over the 0.060-0.652 keV range, above that range is essentially unchanged. The set of QEU files are unevenly distributed in effective dates and affects every HRC-S observation since the start of the mission.

These CIAO tools and scripts automatically apply the HRC-S QE and QEU files when creating response files:
- mkgarf
- fullgarf
- mktgresp
- mkinstmap
- fluximage
- merge_obs

How CALDB 4.9.0 Affects Your Analysis

CALDB 4.9.0 Release Notes (release 17 December 2019)

ACIS Imaging and Grating Data

ACIS Bad Pixel Library

The latest CalDB ACIS bad pixel files have been updated to make use of status bit 17, so that the ACIS framestore shadow can be accounted for in TE mode observations. The ACIS framestore shadow is a rectangular region due to shadowing by the framestore cover, which differs from chip-to-chip, that includes the first several rows of each chip, and is henceforth designated as bad pixel regions.

Events falling in these regions are being handled differently in the software, so they may be re-included manually, but by default are filtered out in level 2 event file.

Users working with ACIS data may wish run chandra_repro to reprocess the data, since this change directly affects the acis_build_badpix tool. This change will not be incorporated in to the Standard Processing pipeline until early 2020. Futhermore, the framestore shadow is currently not accounted for in CC-model data sets.

Installation

Users should be aware of these installation items before installing CIAO 4.12. Additional problems which are seen less frequently are listed on the Installation & Smoke Tests bug page.

Supported Platforms

CIAO 4.12 is supported on:
- Linux 64 bit
- Apple macOS 10.12 (Sierra) through macOS 10.15 (Catalina)

Updates to the ciao-install Installation Script

Various updates to support a single Linux and a single Mac download.
Updated ciao-install to check for strings utility before trying to use it.
The ciao-install script now retrieves the CIAO tar files from via using encrypted https://. The CALDB tar files continue to be served using ftp://.
Improved support for internationalization during installation.
Changed behavior of the --caldb option to correctly deal with linked CALDBs when applying updates.
The smoke tests have been updated to avoid conflicts due to the different versions of IPython used by CIAO (e.g. CIAO 4.11 provided IPython 6 whereas CIAO 4.12 contains IPython 7).
The smoke tests have also been updated for the removal of chips, and SKIP some tests if running in an environment with X11 ( eg headless or remotely logged into another system).

Installing with the conda package manager

CIAO can now be installed using the conda package and environment management system.

The conda release is available from the custom CXC channel, and can be installed with the following command if you have conda installed (see the conda installation instructions for more information, including known problems and incompatabilities):
```
$ conda create -n ciao-4.12 -c https://cxc.cfa.harvard.edu/conda/ciao/ ciao sherpa ds9 caldb_main marx
```
With the conda edition of users can choose to install CIAO with Python 3.7 (the default), 3.6, or 3.5. Other OTS such as numpy and matplotlib will also be be different based on the python version.

Source builds

Improved consistency of build flags such as compiler optimization.
Remove invalid C++ warning flags

IPython Settings

Users will be prompted to update their IPython profiles the first time they run sherpa

Remove old parameter files

With every new CIAO release, some parameter files are changed: new parameters may be added and occasionally old ones removed or renamed. Deleting or renaming the local parameter directory ensures that the correct parameter files will be accessed the first time a tool is run:
```
unix% rm ~/cxcds_param4/*
```

Tools

axbary

Updated to recognize MJD-OBS keyword and convert from the unsupported MJD_OBS keyword.

acis_build_badpix

Added support for new badpixel status bit 17 : frame store shadow.

There is a "shadow" cast on the bottom rows of each ACIS CCD by the frame-store covering. Since there are fewer X-ray events the calibration in these rows is more uncertain and are now marked as bad.

The default bitflag parameter has been updated to include bit 17.
Fixes possible memory corruption when setting bad status values.

acis_process_events

Added support for new badpixel status bit 17 : frame store shadow.

There is a "shadow" cast on the bottom rows of each ACIS CCD by the frame-store covering. Since there are fewer X-ray events the calibration in these rows is more uncertain and are now marked as bad.
A number of obsolete parameter have been removed. This includes: alignmentfile, time_offset, tstart, tstop, and instrume.
Implemented new polynomial coefficients for the temperature dependent CTI corrections.

With CALDB 4.9.0, the higher order coefficients are all 0; therefore there is currently no change in the temperature dependent calibration. Future CALDB updates will include updated calibrations which are expected to affect data obtained at warm temperatures.
Update rules for calculating and propagating PHA_RO values from input file based on calibration files and parameter settings.
Update CTI_APP keyword based on input file, available calibrations, and parameter settings.
Added support for upcoming changes to aspect solution files.

acis_streak_map

Fix syntax error with higher verbose messages.

addresp

Updated to handle XMM files where the first channel is 0 instead of 1.

celldetect

Correct bug misidentifying double sources when using fixed cell sizes.

dither_region

Remove constraint that stack of asol files share the same NOM values since those values now are obtained from the wcsfile.

dmcoords

Fixes interactive bug where a string value was copied over itself potentially causing memory corruption.

dmcontour

Fix bug closing polygons when contour hits the edge of the image.

dmfilth

Now correctly propagates units from the input image to the output image.

dmimgadapt

Now correctly propagates units from the input image to the output image.

dmimgcalc

Now correctly propagates units from the input image to the output image.

dmimgdist

Now correctly propagates units from the input image to the output image.

dmimgfilt

Now correctly propagates units from the input image to the output image.

dmimgpm

Now correctly propagates units from the input image to the output image.

dmimgreproject

Now correctly propagates units from the input image to the output image.

dmkeypar

Update handling for unsigned long keyword values.

dmmaskbin

Now correctly propagates units from the input image to the output image.
Corrects initialization error when pixel at 0,0 is NaN.

dmmaskfill

Now correctly propagates units from the input image to the output image.

dmnautilus

Now correctly propagates units from the input image to the output image.

dmregrid2

Now correctly propagates units from the input image to the output image.

dmsort

Now will correctly sort columns with IEEE special values: NaN < -Inf < Real Numbers < Inf.

hrc_build_badpix

Update to new degap calibration file to use modified HRC_OFFX and HRC_OFFY keywords. Previous calibration files used non-standard 9-character keyword names.

hrc_process_events

Updated to use MJD-OBS keyword.
New PI gain calibration algorithm.
Update to new degap calibration file to use modified HRC_OFFX and HRC_OFFY keywords. Previous calibration files used non-standard 9-character keyword names.

reproject_image

Now correctly propagates units from the input image to the output image.

reproject_image_grid

Now correctly propagates units from the input image to the output image.

skyfov

A new algorithm to create a smaller bounding polygon has been implemented using the new method=convexhull parameter.

However, for observations with large changes in the aspect solution DY,DZ columns, this new FOV boundary does clip the observation. A redefinition of the aspect solution files is being developed which eliminates this concern.

The default method=minmax implements the original algorithm.
The output FOV will now only contain polygons for the chips included in the input file (ie after any filters have been applied).
Improved consistency checks between input file, aspect solution, and mask file.
Improved error checking for cases with 0 rows in aspect solution (eg all bad time).
Use _AVG keyword keywords from the input file is asol file is omitted.

tgdetect

Restore the xoffset and yoffset parameters.

wavdetect

Three new vector columns have been added to assist users when dealing with duplicate source detections:
1. CORRELATION_MAX(X_CORRELATION_MAX,Y_CORRELATION_MAX) : The location of the correlation maximum with the wavelet. For well behaved sources, the correlation maximum will be close to the source centroid (X,Y), and will be close to the location of the flux maximum. Some duplicate sources will have large distances between the correlation_max and the flux_max.
2. FLUX_MAX(X_FLUX_MAX,Y_FLUX_MAX) : The location of the local maximum closest to the correlation maximum. (See correlation_max discussion above.)
3. SCALES(X_WAVELET_SCALE,Y_WAVELET_SCALE): the actual wavelet scale the source was reconstructed using. Users should find that the scales are different for duplicate sources.

Parameter Files

A summary of parameter files changes is provided in this section. Refer to the Tools and Analysis Scripts sections of these release notes for complete details.

We recommend deleting all the old parameter files or renaming the parameter-file directory before running any new version of CIAO to avoid conflicts. More information is available in the FAQ.

acis_build_badpix

The default bitflag=00000000000000122221100020022222 now includes bit 17 for the frame store shadows.

acis_process_events

A number of obsolete parameter have been removed. This includes: alignmentfile, time_offset, tstart, tstop, and instrume.

skyfov

A new method parameter has been added. Users should leave the parameter at its default method=minmax to use the standard FOV algorithm. The new method=convexhull algorithm should not be used at this time.

tgdetect

Reintroduced the xoffset and yoffset parameters.

ChIPS

The CXC has decided to dropped support for ChIPS and move to using Matplotlib, alongside DS9, as the primary plotting and visualization systems in CIAO.

A ChIPS to Matplotlib conversion guide is provided for users who would like to transition from ChIPS to Matplotlib. Please contact the CXC Helpdesk if you have further questions or encounter problems.

The version of Matplotlib and choice of backend (the library that provides the visualization and any interactive capabilities) depends on how CIAO was installed. The ciao-install installation provides version 3.0.2 of Matplotlib and the TkAgg back end. The conda installations, in particular for Python 3.6 and 3.7, provide newer versons.

Sherpa

Sherpa is developed as a community project, and is available on GitHub. Comments, fixes, and additions from the community are welcomed. There are additional releases for this version throughout the year, and can be installed either directly from GitHub or using the conda package manager, as described on the standalone Sherpa documentation page.

Switching to Matplotlib

The default behavior is for Sherpa to use Matplotlib for its plotting in CIAO 4.12.

If you have used the Sherpa application with previous versions of CIAO then your sherpa configuration file is likely to still be set up to use ChIPS rather than Matplotlib. In this case you will see the following warning message when you start Sherpa:
```
WARNING: chips is not supported in CIAO 4.12+, falling back to matplotlib.
WARNING: Please consider updating your $HOME/.sherpa.rc file to suppress this warning.
```
and Sherpa will switch itself to using Matplotlib. Please see the Sherpa FAQ entry on how to change the resource file to stop seeing this warning.

Tip

Unlike CIAO 4.11, users of the Sherpa application no-longer have to use either the %matplotlib IPython magic command, or say import matplotlib.pyplot as plt to have access to Matplotlib in interactive mode.

Sherpa contains a number of functions that change the plot preferences (i.e. how new plots will be displayed), such as set_xlog. However, users may be used to using ChIPS commands - such as limits or log_scale - to change existing plots. Please see the ChIPS to Matplotlib conversion guide to find out the appropriate replacements (which in this example are plt.xlim, plt.ylim, plt.xscale, and plt.yscale).

Plotting improvements

The plot functions - that is, all functions beginning plot_ (but not plot) can now accept keyword arguments to over-ride the plot preferences. This lets you ensure the plot has logarithmic axes, such as
```
plot_source(xlog=True, ylog=True)
```
or overplot fits while using a consistent color scheme for the data and model:
```
plot_fit(1, color='blue')
plot_fit(2, color='orange', overplot=True)
```
The supported keywords are the same as the entries in the dictionaries returned by get_data_plot_prefs() and get_model_plot_prefs().
Two new plotting routines have been added - plot_fit_ratio and plot_bkg_fit_ratio - which create a figure containing two components, where the top plot is the fit, and the bottom one is the ratio of the model to the data. These functions complement the existing plot_fit_resid and plot_fit_delchi routines.

There are no ahelp pages for these new commands. Please use the native Python help system - i.e. at the sherpa prompt say:
```
help(plot_fit_ratio)
```
The plot_fit_resid, plot_fit_delchi, and (new) plot_fit_ratio functions now create figures where the top plot is larger than the bottom plot. In CIAO 4.11 this was not the case with Matplotlib (but did work correctly when using ChIPS).

The order of the model and data in plot_fit has been adjusted with the Matplotlib backend (changes in Matplotlib 2.0 meant that plots created with CIAO 4.11 and Maplotlib had the model line overdrawn by the error bars).
The plot_pvalue function has seen improvements when fitting PHA data.

XSpec

The XSPEC models have been updated to version 12.10.1n.

The following additive models have been added: xsagnsed, xscph, xskyrline, xsqsosed, xsvcph, xszbknpower, and xszlogpar.

The update to version 12.10.1 of XSPEC has meant that the default cross-section table has changed: it now defaults to vern when it was bcmc in CIAO 4.11 and earlier. Note that this default setting is now read from your $HOME/.xspec/Xspec.init configuration file (if it exists), using the XSECT setting.

The default abundance table will now be set from your $HOME/.xspec/Xspec.init file, if the ABUND line is set.

Notable changes

The calc_ftest function has been updated to better match the XSPEC version and the documentation.

Bug fixes

The full list of changes made in this version of Sherpa are:
Remove ChiPS dependency from Sherpa startup script

Sherpa's IPython profile startup script does not assume ChiPS is installed and it skips the version check. The code that imports the ChiPS commands into the root namespace has been left for compatibility. Users upgrading to CIAO 4.12 and then going back to CIAO 4.11 will be asked if they want to "update" their profile scripts. If they don't, and they have ChiPS as their preferred backend, ChiPS will be imported in the shell. Additionally, Sherpa now warns the user if ChiPS is selected as a backend but it is not available in the installation. In this case Sherpa will also try and fall back to the pylab backend.

Enable Matplotlib in sherpa shell

Since ChiPS has been removed from CIAO, the shell wrapper will now run %matplotlib when using the pylab backend so Matplotlib plots will appear on the screen when plot commands are issued.

Update ipython_config.py to work with ipython 7.2

Modified the ipython_config script used for the 'sherpa' command shell to work with ipython 6 or 7. The prompt api changed between the two ipython versions and broke compatibility.

Remove ChiPS reference in Sherpa wrapper and update the rc file check in ciao_app_setup

The references to ChiPS have been removed from the Sherpa wrapper and ciao_app_setup.sh.

Update white spacing in ipython profile compatibility warnings

Updated white spacing in ciao_app_setup.sh so that the messages issued by Sherpa contain less spaces.

User Model: unable to change parameter values (Fix #609)

A regression introduced in Sherpa 4.10.0 prevented users from changing user-model parameter values through direct access. This issue has been fixed. Several tests were added.

update ciao default plotter to Matplotlib

The default Sherpa plotting package has been changed to Matplotlib for CIAO (it had been the default for standalone Sherpa for several years). The update will be applied when the `sherpa` command is run. Users of the previous versions would need to edit/regenerate the sherpa.rc file.

stop using numpy.typeNA which is now deprecated

A numpy deprecation warning was fixed by removing the usage of typeNA, which was not documented and will be removed in a future release of numpy. The print_fields function has been changed to include a default mapping using the current typeNA implementation.

update pager code (fix #445 #561)

Replace the use of an external pager (such as more or less) with a Python one. This means that Sherpa no-longer uses the PAGER environment variable, and that the screen output from the show_*series of commands should now appear in the correct location when using a Jupyter notebook or the spyder application.

Fix regression with responses `get_x` method (Fix #572)

Version 4.10.1 introduced a regression which made the get_x() method fail when called on RMF and ARF objects. This has been resolved.

fix LevMar when dof < 0 and Fix LevMar when dof < 0 and harmonise the handling of dof is zero or negative in fit and calc_stat_info (fix #565)

The fit and calc_stat_info methods now use the same code to calculate the reduced statistic (rstat) and "quality of fit" (qval) values. This avoids a TypeError when the number of degrees of freedom is negative in calc_stat_info and ensures that NaN is returned for both values when the goodness-of-fit calculation has failed (e.g. the reduced statistic is zero or less or the statistic is negative), whereas in previous versions the qval value could be None or 1 depending on the code path.

fix gammapy/gammapy#1905, catch errors with psf pixel size

Sherpa did not properly catch errors when checking the PSF pixel size matches the image pixel size. This has now been fixed.

Fix XSPEC models: mtable table models and ismabs (fix #535 #538 #540)
This PR fixes:
- #535 XSPEC multiplicative table models (mtable models loaded with load_xstable_model)
- #538 incorrect evaluation of the ismabs model
- #540 the kerrd model evaluated to 0 for XSPEC 12.10.0 and later
For the first two, the models were being multiplied by the value of the last parameter of the model. At best this would lead to fits where the overall model amplitude was incorrect but could be corrected. However, it is likely that such fits were not robust (due to the accidental correlation between the overall amplitude and the last parameter of the model), and so it is not guaranteed that the best-fit would be found. The kerrd model was due to a change in the naming of the XSPEC functions which had not been reflected in the XSPEC model.dat file in XSPEC 12.10.0 or 12.10.1b, which meant that the kerrd model in XSPEC 12.10.0 or later was wrong in Sherpa.
XSPEC 12.10.1 support and mtable fix

Sherpa now supports XSPEC 12.10.1. Several issues were fixed to ensure compatibility with this release. Note that there are known issues with version 12.10.1 (no patches), so at least v12.10.1a must be used.

ensure XSPEC 12.10.0 uses ATOMDB 3.0.9 by default

Ensure that the default AtomDB version is 3.0.9 when using XSPEC 12.10.0. This is to fix an issue with the models-only XSPEC installation in XSPEC 12.10.0 which uses a default version of 3.0.7 but only provides data files for 3.0.9. Without this change models such as XSapec would return all 0's (unless the AtomDB version was set manually by the user).

Be explicit about "invalid escape characters" in strings

Convert strings that contain "invalid" escape characters, which newer versions of Python complain about, to raw strings by preceding them with a 'r' character. This avoids the test suite from reporting DeprecationWarnings.

Ensure the ui modules have a unique list of exported symbols (fix #502)

Remove the sherpa.utils.erf symbol from the two ui modules as it is "over-written" by the ModelWrapper-created symbol when applied to sherpa.models.basic.Erf. Tests have been added to point out if this ever happens again: for example, if any new models are added to Sherpa, they could conflict with an existing symbol, and these tests will pick that up.

Fix "get_int_proj does not work when recalc=True" (#543)

get_int_proj did not work when recalc=True on Python 3. This has now been fixed.

Asymmetric Errors

Sherpa now supports asymmetric error bars. Errors can be read through a new `load_ascii_with_errors` high level function, or through the new `Data1DAsymmetricErrs` class. Sherpa uses bootstrap for estimating the uncertainties.

plot_pvalue

Updates to utilize the appropriate response files (ARF and RMF) for X-ray spectra and changes to the p_value output to 1/(number of simulations) when p_value is 0 and the number of simulations in not large enough.

Run optimization algorithms over multiple cores

This PR enables the user to run the optimization algorithms (DifEvo, LevMar, and NelderMead) on multi-cores.

PSF rebinning (fix #43)

Sherpa now supports using a PSF with a finer resolution than 2D data images. If Sherpa detects that the PSF has a smaller pixel size than the data, it will evaluate the model on a "PSF scale" that has the same resolution (pixel size) as the PSF and the same footprint as the data image, then rebin the evaluated model back to the data space for calculating the statistic.

Refactor data classes (fix #563, #627, #628)

Sherpa's basic data classes have been refactored and cleaned up to help facilitating fixing bugs and implementing new features. New tests were added to reduce the chances of introducing regressions in the future.

Reduce integration tolerance (tol)

Changed integration routine tolerance from double epsilon to float epsilon since optimization routines have a tolerance of ~1.0e-7.

enable run-time-option to test cache (default=True)

Capability to allow the user to turn off caching at runtime for testing purpose.

Do not create warnings about error bars in plots if no error bars are to be shown (fix #621)

Remove the "The displayed errorbars have been supplied with the data or calculated using chi2xspecvar..." warnings that appear for data, residual/ratio, and fit plots when the user has explicitly turned off the display of errorbars (by setting the yerrorbars plot-preference setting to False).

add plot_fit_ratio and plot_bkg_fit_ratio functions

Add the plot_fit_ratio function to the sherpa.ui layer (to match plot_fit_resid and plot_fit_delchi) and plot_bkg_fit_ratio to the sherpa.astro.ui layer. Added tests for a number of plot types.

fix 'tuple index out of range' error in the fit.est_errors method

Sherpa's Confidence class is confused if a parameter is frozen then thawed after a fit resulting with an exception of the form: IndexError: tuple index out of range. The error does not show up if conf() is called from the ui level.

fix numpy hist warnings

When plotting a probability density function (PDF) Sherpa now calls numpy.histogram with density=True|False rather than normed=True|False. There is no change visible to the user other than a warning that will not be issued anymore. normed=True was broken for nonuniform bins, but our code always produces uniform bins, so we should never hit the problematic case.

Handle files with MJD_OBS or MJD-OBS keywords (datastack module)

The sherpa.astro.datastack module has been updated to deal with files that use the MJD-OBS keyword instead of MJD_OBS. This is only used by the show_stack routine, which prints out several keywords from each file in the stack.

Remove python 2.7 support (fix #498)

Remove support for Python 2.7 from Sherpa. The metadata used by pip now requires python 3.5 or higher (but not version 4),

XSPEC: build against 12.10.1m by removing support for models

Remove support for XSrelline, XSrelline_lp, and XSrelline_lp_ext models. These were added in 12.10.1 but removed in 12.10.1m.

Remove the unused parameter in sherpa.astro.optical.LogEmission model (fix #219)

Remove the unused parameter in the sherpa.astro.optical.LogEmission model. The LogEmission model only has four parameters (fwhm, pos, flux and skew) consistent within the calc method. The hidden parameter limit and it docs were removed.

Remove ChiPS from the documentation

Remove mention of ChiPS from the documentation (both on read the docs and in the sherpa.astro.ui and sherpa.ui modules), replacing with Matplotlib where appropriate.

Improve documentation and testing for get_source_plot when sent lo/hi arguments

Improve the documentation for the sherpa.astro.ui.get_source_plot routine, describing how touse the result when the lo or hi arguments are sent.

Add default config options

Whenever an option is read from the configuration file (e..g. ~/.sherpa.rc) provide a default option in case the configuration file is missing (or is missing this option).

jointplot now respects the ratio argument (top plot is larger) with Matplotlib backend

The matplotlib backend now makes the main (top) plot taller than the secondary (bottom) plot when using the sherpa.ui.plot_fit_resid and sherpa.ui.plot_fit_delchi routines (this also holds for the sherpa.astro.ui variants).

Add link/unlink parameter tests

Added three new tests for sherpa's un/link commands. The tests make sure that all the members of the Parameter class are correct for the low and ui levels of sherpa's link function.

Address indexing DeprecationWarning/IndexError from NumPy for PHA data with filter + ignore_bad (fix #361)

Add warning about masked array to sherpa.astro.ui.load_arrays

Updated docstring to contain warning consistent with PR#698.

modelCacher1d needs to have a couple of deep instead of shallow copy()

Update to make deep copy of cached function values (fix to IS#673).

Improve Matplotlib plot_fit plots (order of data and model)

Changes to 'zorder' of plot objects drawn by Matplotlib to make plots more readable.

Allow users to override plot preferences when creating a plot

Change to allow kwargs to be specified to change plot preferences at creation time.

Add warnings in documentation about masked arrays

Updated docstring to indicate that Sherpa doesn't support numpy masked arrays.

change calc_ftest from delta dof to dof (similar to XSPEC) - fix for IS#641

Updated input to calc_ftest() to reflect setting specified in the documentation and also aligned with the XSPEC interface.

a fix for internal regression test issues

add warning when ChiPS is selected but cannot be imported (rebase of #648)

Sherpa now warns the user if ChiPS is selected as a backend but it is not available in the installation.

raise ModelErr when model using Data1DInt have overlapping bins

Addition of a check to 'regrid' function to make sure that the integrated grid has non-overlapping bins. The PR is a fix for issue #569.

Graphical User Interfaces

prism, peg, and taskmonitor have been updated to use GTK 3. The applications will have a different look than before.

prism

prism now use Matplotlib for plotting.
prism now displays an error message when trying to modify a compressed file.

peg

Updates to support GTK 3.

taskmonitor

Updates to support GTK 3.

obsvis

IPAC-2mass survey has been added

Internal work to use ds9 command line options
Added NED@CDS, SIMBAD@CDS, VIZIER@SAO and VIZIER@CDS to the obsvis list of name servers.
Add an option to specify edge and center blanking for HRC-I and HRC-S through obsvis.

Analysis Scripts

There have been numerous updates to the contributed scripts package since CIAO 4.11 was released. These changes are specific to the CIAO 4.12.1 contributed scripts release.

chandra_repro

Updated to include new badpixel bit, 17, when recomputing badpixels.

check_ciao_version

Updated to work with CXC secure web server
Updated to work with conda edition of CIAO.

check_ciao_caldb

Updated to work with CXC secure web server

combine_grating_spectra

Remove the TG_M and SPEC_NUM keywords from the output file. Previously the values were taken from 1st file in the merge.

combine_spectra

Remove any TG_M filter from the output file subspace. This cleans up the final output file to remove extraneous subspace components.

dax

Updated to create residual plots for spectral fits and radial fits.

deflare

Updates to support matplotlib 3.1

make_instmap_weights

The default setting for the xsect parameter has changed from bcmc to vern to match the behavior of the XSPEC 12.10.1 model library.

tgsplit

Fixes a bug when trying to work with TYPE:II pha files created by the tgextract2 tool.

ciao_contrib.runtool

Updated for parameter changes in this release.

lightcurves

The module has been updated to work with matplotlib 3.1.

ciao_contrib.all

This module no-longer imports the chips_contrib module.

sherpa_contrib.utils

The plot_instmap_weights routine has been updated to use matplotlib preference settings (color) rather than ChIPS ones (linecolor and linethickness) from get_data_plot_prefs. The preferences can also be over-ridden when the plot is created.

sherpa_contrib.xspec.xsconvolve

Added the load_xskyconv routine to support the kyconv convolution model. Please note that support for XSPEC convolution models is considered experimental and there are known problems, so use carefully.

chips_contrib

The chip_contrib.decorators, chips_contrib.helix, chips_contrib.images, chips_contrib.regions, chips_contrib.scatter, and chips_contrib.utils modules have been removed, as ChIPS is no longer distributed as part of CIAO. The Matplotlib plotting package should be used instead.

ChIPS to Matplotlib conversion guide

Please see the ChIPS to Matplotlib conversion guide if you need help. The CXC Helpdesk can be used if you still have problems.

Python Modules

caldb

Remove hard coded build dependencies.

crates

CrateData objects can now take unit=None and desc=None. These are converted to empty strings.

keywords and columns can now be added at a specific index (order) using add_key() and add_column().
Update to correctly copy image units
Fixed bug where a 1x1 image was returned as a 1D array.
Speed improvements

group

Improved configuration to support various python versions.

history

Extended interface to allow for creating a HistoryRecord object without any parameters.

paramio

Increased maximum string length to 2048 characters to match C library.

Improved error checking when string length max is exceeded.
Improved configuration to support consistent optimization levels.

pixlib

Allow greater flexibility when setting parameters (duck typing)

Improved error handling: throwing exceptions rather than setting hidden error status codes.
Improved pkgconfig dependencies for easier builds

psf

Improved configuration to support consistent optimization levels.

region

Improved configuration to support consistent optimization levels.
Region shapes can now be sliced to obtain a subset of the region
```
>>> from region import *
>>> srcs = circle(10,10,10)+circle(20,20,20)
>>> excld = field()-srcs
>>> slce = excld[1:]
```
The shapes that are returned are always included and are logically OR'ed together.

stk

Improved configuration to support consistent optimization levels.

transform

Improved configuration to support consistent optimization levels.

Libraries

ardlib

CIAO now provides a pkgconfig file to make building tools with this library easier.

cxcparam

Updates to allow for mode=H. This mode forces exit when illegal parameters are encountered rather than prompting the user for a legal value.

Improved error checking for invalid mode values.

Additional internal code cleanup.

These updates were provided by Diab Jerius.
Updated float to string conversion to omit ends (end string) from C++ classs.

dm

CIAO now provides a pkgconfig file to make building tools with this library easier.
Improved composite filtering using NULL values. Filters such as [(tg_m!=0)&&(tg_part!=99)] could allow the tg_m NULL values through the filter. These are now properly omitted.
Miscellaneous internal code clean up.
ASCII TSV files with duplicate columns names are now displayed correctly.
ASCII files with long column or keyword descriptions can now be processed.
Improved parsing of strings with leading numeric values: for example when column contain dates of the form: 2019-12-15.

dslib and hdrlib

Chandra has been improperly using the keyword MJD_OBS in file headers. FITS WCS paper II specifies that the correct keyword name is MJD-OBS (a dash instead of an underscore). This has lead to some minor incompatibilities interpreting the World Coordinate Systems when using Chandra data products with 3rd party applications. This change ensures that the proper MJD-OBS keyword is created and will be corrected when propagating header information.

This change affects the output from most CIAO tools.

pixlib

Internal code cleanup.

region

CIAO now provides a pkgconfig file to make building tools with this library easier.

Environment

Environment variables

The ASCDS_WORK_PATH environment variable is used to specify the temporary directory used by several CIAO tools. The ciao setup scripts will now use the same values as the TMPDIR environment variable if it is set. Otherwise, it will default to the /tmp.
The ASCDS_OTS environment variable is now set for both bash and tcsh shells.

Off-the-Shelf (OTS) Package Versions

The following OTS packages are included with CIAO 4.12 installed with ciao-install. For more information on how the OTS packages are built for use with CIAO, refer to the INSTALL_SOURCE file distributed with the software.

These do not hold for the conda-installed version of CIAO

The conda-installed version of CIAO 4.12 has different OTS versions, and it also depends on the Python version you have chosen to use.

Executables
- ds9 8.1
Python
- Python 3.5.4
  Frameworks on OSX have been disabled to avoid a problem with users picking up the wrong version of NumPy.
- NumPy 1.15.4
- ipython 7.2.0
- six 1.12.0
- pip 19.1.1
- setuptools 28.8.0
- cython 0.285
- Matplotlib 3.0.2
- jupyter notebook 6.0.0
- pytest 4.0.2
Libraries
- cfitsio 3.470
- CCfits 2003.12.08
- wcssubs 5.19
- readline 6.3
- ncurses 5.7
- XSPEC v12.10.1n (models only)
- fftw 3.3.8
- gsl 2.5
- xerces 3.2.2
- GTK 3.22.30
- adwaita-icon-theme 3.20
- glib 3.56.2
- hicolor theme 0.17
- fontconfig 2.13.1
- freetype 6.17.0
- jpeg 9.3.0
- tcl 8.6.9
- tk 8.6.9
- xpa 2.1.18
- openSSL 1.0.1e

Compiler versions

The following versions of the gcc compiler were used to compile the binary CIAO releases installed with ciao-install. The conda-installed compiler versions are different.

System Build System Version

Linux CentOS 6 gcc 4.8.5

macos Sierra llvm 8.0.0

System	Build System	Version
Linux	CentOS 6	gcc 4.8.5
macos	Sierra	llvm 8.0.0

Documentation

Many pages have been updated to reflect the changes in this release. Several analysis threads are still under review.

/ ciao / releasenotes / ciao_4.12_release.html

CIAO 4.12 Release Notes

ACIS Imaging and Grating Data

Time-dependent ACIS Gain (T_GAIN) Files for -120 C Data

ACIS Imaging and Grating Data

Time-dependent ACIS Gain (T_GAIN) Files for -120 C Data

ACIS Blank-sky Background Files

HRC Data

HRC-S Time-Dependent Gain Maps

HRC-S QE Files

HRC-I Time-Dependent QE Files

ACIS Imaging and Grating Data

Time-dependent ACIS Gain (T_GAIN) Files for -120 C Data

DET_GAIN and P2_RESP FI CCDs Upgrade

HRC Data

HRC-S Time-Dependent QEU Files

ACIS Imaging and Grating Data

ACIS Bad Pixel Library

Supported Platforms

Updates to the ciao-install Installation Script

Installing with the conda package manager

Source builds

IPython Settings

Remove old parameter files

axbary

acis_build_badpix

acis_process_events

acis_streak_map

addresp

celldetect

dither_region

dmcoords

dmcontour

dmfilth

dmimgadapt

dmimgcalc

dmimgdist

dmimgfilt

dmimgpm

dmimgreproject

dmkeypar

dmmaskbin

dmmaskfill

dmnautilus

dmregrid2

dmsort

hrc_build_badpix

hrc_process_events

reproject_image

reproject_image_grid

skyfov

tgdetect

wavdetect

acis_build_badpix

acis_process_events

skyfov

tgdetect

Switching to Matplotlib

Plotting improvements

XSpec

Notable changes

Bug fixes

prism

peg

taskmonitor

obsvis

chandra_repro

check_ciao_version

check_ciao_caldb

combine_grating_spectra

combine_spectra

dax

deflare

make_instmap_weights

tgsplit

ciao_contrib.runtool

lightcurves

ciao_contrib.all

sherpa_contrib.utils

sherpa_contrib.xspec.xsconvolve

chips_contrib