This page is for the obsolete CIAO 4.16 release. Please see the CIAO home page for the latest release.

CIAO 4.13 Release Notes

---

CIAO 4.13 is distributed for the following platforms:

• Linux 64 bit
• Apple macOS 10.13 (High Sierra) through macOS 10.15 (Catalina)

Visit our platform support page for our statement about support for macOSX 11: Big Sur .

CIAO is available using the conda package manager: conda builds are available for Linux and Mac for Python 3.8, 3.7, and 3.6. More information can be found in the conda section of the Installation notes below.

There is no support for 32 bit operating systems, older Linux (CentOS 6 era) or older macOS platforms (macOSX Sierra and earlier). More details can be found on the Platform Support page.

---

• Notable changes and improvements in CIAO 4.13
• How CALDB 4.9.6 Affects Your Analysis
• How CALDB 4.9.5 Affects Your Analysis
• How CALDB 4.9.4 Affects Your Analysis
• Installation
• Tools
• Parameter Files
• Sherpa
• Graphical User Interfaces
• Analysis Scripts
• Python Modules
• Libraries
• Environment
• Documentation
• ChIPS

---

Notable changes and improvements in CIAO 4.13

---

How CALDB 4.9.6 Affects Your Analysis

ACIS Imaging and Grating Data

• Time-dependent ACIS Gain (T_GAIN) Files for -120 C Data

  The new, combined time-dependent ACIS Gain (T_GAIN) file for May-July 2021 (Epoch 86) and February-April 2021 (Epoch 85) are introduced. The combined Epochs 83+84 file has also been updated since CALDB 4.9.4. Therefore, the new T_GAIN affects observations taken since 2020 November 01. Other observations are unaffected by these new calibration files.

  Users working with ACIS data taken since 2020 November 01 may wish to run chandra_repro and reprocess the data to improve the T_GAIN calibration. The DATE-OBS header keyword records the observation start date.

  See the time-dependent ACIS T_GAIN why page for more information.

  The gain corrections are at the nominal and expected levels of less than 1% of the energy value and users interested in CCD spectroscopy may benefit in applying the new gain adjustment. Grating spectroscopy benefits as well, in the form of improved order sorting. It is unnecessary to apply this adjustment for only doing timing or imaging analysis, although doing so will not have a negative effect.

  Note that only spectra with several hundreds of counts and/or prominent features (in emission or absorption) will show changes from the T-gain refinement that exceed the uncertainties from the gain calibration.

HRC Data

• HRC-S Time-Dependent Gain Maps and PI Region Filter Cut-off

  The HRC-S gain correction has been unsatisfactory since early 2018 where certain assumptions in prior calibration procedures have proven faulty. Improved time-dependent gain calibration for HRC-S has since been developed.

  Shortly after the release of the HRC-S gain maps introduced in CalDB 4.9.2, it was found that the HRC-S/LETG background filter was sometimes removing an excessive number of valid X-ray events. The filter is intended to be limited to remove 1% of eventts, but the excess losses were seen primarily at negative-orders for observations beginning in 2017 an increasing over time, with loses up to 6% at some wavelengths by late 2020. It was found that the event pulse height distributions were subtly distorted at very low gains and having large effects at higher PI channels— where background filtering primarily occurs—causing more real X-ray events be mis-identified as background events. Corrections have now been applied to the HRC-S TGAIN coefficients so that the 99th percentile PI values for X-ray events, rather than the medians, are constant.

  Optimal background event filtering comes from an accurate time-dependent gain map applied using a corresponding PI region filter (TGPIMASK2).

  The PI region background filter can be safely applied for all observations prior to the HRC-S high voltage change on 2021 May 14; however, it must not be applied to ObsIDs taken after that date. A modified chandra_repro from the updated contributed scripts package 4.13.3 will work automatically with CalDB 4.9.6 to handle the PI region filtering appropriately for all cases.

  CIAO 4.12 users will need to reprocess manually to apply the new background filtering using the "LETG/HRC-S Grating Spectra" thread, replacing the letgD1999-07-22pireg_tgmap_N0001.fits region file to letgD1999-07-22pireg_tgmap_N0002.fits in the example. Again, the PI region background filtering step must be skipped for LETG/HRC-S ObsIDs taken after 2021-05-14T00:00:00.

• HRC-S Time-Dependent QE and QEU Files

  Regular calibration observations have shown that the HRC-S QE is declining at longer wavelengths (energies <200 eV, >60 Å), and decaying with the 2012-03-29 and 2021-05-14 high-voltage adjustments onboard. This spectral variation in the QE loss is accounted for using a series of time-displaced QE files, with the latest high-voltage adjustment introduced in the N0016 set in this CALDB.

  The HRC-S time-varying quantum efficiency uniformity files have been replaced with a new set, calculated based on unfiltered calibration observations. This latest set (N0015) shows very minor change over the prior set of QEU files, but incorporates results from the latest HZ43 calibration observations. The newest QEU for observations taken since 2021 May 14 includes updates to the outer plates' wavelength-dependent correction. The set of QEU files are unevenly distributed in effective dates and affects every HRC-S observation since the start of the mission.

  These CIAO tools and scripts automatically apply the HRC-S QE and QEU files when creating response files:

  • mkgarf
  • fullgarf
  • mktgresp
  • mkinstmap
  • fluximage
  • merge_obs

• HRC-I Time-Dependent QE Files

  The HRC-I quantum efficiency QE has been decaying for some time, particularly since the year 2011, when a 5% to 10% per year reduction had been measured. To address this issue, the HRC-I Calibration Team had introduced a time-dependence into the QE files in 2018 which are now extended in time to the latest epoch of observations. Included in this update is a new file coinciding to the recent HRC-I high-voltage adjustment affecting observations after 2021 February 16. The QE update affects the results from mkinstmap and fluximage and is applicable to all previous epochs of HRC-I observations.

---

How CALDB 4.9.5 Affects Your Analysis

ACIS Imaging and Grating Data

• Time-dependent ACIS Gain (T_GAIN) Files for -120 C Data

  The new, combined time-dependent ACIS Gain (T_GAIN) file for November 2020-January 2021 (Epoch 84) and August-October 2020 (Epoch 83) are introduced. The combined Epochs 81+82 file has also been updated since CALDB 4.9.3. Therefore, the new T_GAIN affects observations taken since 2020 May 02. Other observations are unaffected by these new calibration files.

  Users working with ACIS data taken since 2020 May 02 may wish to run chandra_repro and reprocess the data to improve the T_GAIN calibration. The DATE-OBS header keyword records the observation start date.

  The previous T_GAIN update release (CalDB 4.9.3, October 2020) included an error in the focal plane temperature boundary condition specification. In particular, the CBD20001 keyword values were set to "FP_TEMP(148.16-164.16)K", an older specification, rather than the new value of "FP_TEMP(148.15-168.16)K", as had been specified and approved for all ACIS temperature-dependent calibration files in CalDB 4.9.2 (July 2020). This error affects no ACIS science observations to date, but could affect T_GAIN file selection in chandra_repro for very warm ObsIDs that will be taken in the future. Hence all users of ACIS data going forward should upgrade their CalDB to version 4.9.5.

  See the time-dependent ACIS T_GAIN why page for more information.

  The gain corrections are at the nominal and expected levels of less than 1% of the energy value and users interested in CCD spectroscopy may benefit in applying the new gain adjustment. Grating spectroscopy benefits as well, in the form of improved order sorting. It is unnecessary to apply this adjustment for only doing timing or imaging analysis, although doing so will not have a negative effect.

  Note that only spectra with several hundreds of counts and/or prominent features (in emission or absorption) will show changes from the T-gain refinement that exceed the uncertainties from the gain calibration.

• Time- and Temperature-dependent ACIS-S3 (ACIS-7) CTI Correction Files

  New time- and temperature-dependent correction to the ACIS charge transfer inefficiency (CTI) correction file for ACIS-S3 is introduced. The new CTI corrections only affect ACIS-S3 FAINT, VFAINT, and CC33_FAINT mode observations (not GRADED or CC33_GRADED modes) taken since 2020 January 01. No other ACIS chips are affected by this change. There is no CTI correction for back-illuminated chips with GRADED or CC33_GRADED modes.

  Users working with ACIS-S3 observations made since 2020-01-01 may wish to run chandra_repro and reprocess the data to improve the CTI gain shift calibration.

---

How CALDB 4.9.4 Affects Your Analysis

ACIS Imaging and Grating Data

• ACIS QE Contamination Model vN0014

  The ACIS QE contamination model has been upgraded to N0014:

  acisD1999-08-13contamN0014.fits 
  

  Recent calibration work has shown that observations of previously-observed cosmic sources, such as clusters of galaxies and supernova remnants, has shown that the accumulation of contaminant has defied systematic predictions.

  This model builds on the prior contamination model, increasing the predictive central contamination build up on both ACIS-S and ACIS-I. This update most readily affects extended sources observed since 2018 and should use the new CalDB to generate contamination-dependent data products.

  More information about the N0014 contamination models, and prior versions, can be found in the ACIS QE contamination model Why document. A detailed presentation of the derivation of the new model—and the resulting changes in the estimated time-dependent effective area of ACIS-I and ACIS-S—is in the technical details section of the CALDB 4.9.4 Release Notes.

  These CIAO response tools automatically apply the contamination file when creating ACIS response files:

  • mkarf
  • mkgarf
  • mkwarf
  • mkinstmap

  As well as the scripts which use them:

  • specextract (calls mkwarf and mkarf)
  • fullgarf (calls mkgarf)
  • fluximage (calls mkinstmap)
  • merge_obs/flux_obs (calls mkinstmap)
  • mktgresp (calls mkgarf)

HRC Data

• HRC-S Time-Dependent QEU Files

  The HRC-S time-varying quantum efficiency uniformity files have been replaced with a new set, calculated based on unfiltered calibration observations. This latest set (N0014) shows very minor change over the prior set of QEU files, but incorporates results from the latest HZ43 calibration observations. The newest QEU for observations taken since 2020 November also accounts for the switch to the detector's electronics 'B-side'. The set of QEU files are unevenly distributed in effective dates and affects every HRC-S observation since the start of the mission.

  These CIAO tools and scripts automatically apply the HRC-S QE and QEU files when creating response files:

  • mkgarf
  • fullgarf
  • mktgresp
  • mkinstmap
  • fluximage
  • merge_obs

• HRC-S/LETG PI Region Filters

  The update of the time-dependent gain maps (T_GMAPs) introuduced in CalDB 4.9.2 in July 2020 necessitated an update to the PI region filter file (TGPIMASK2), but was not included due to an oversight.

  Because the existing (old) PI Region Filter file is not matched with the new T_GMAPs, some significant loss of real X-ray events is now occurring when that old filter is applied to the newly-calibrated data. This affected HRC-S/LETG data being analyzed for the whole mission using CalDB 4.9.2-4.9.3. The TGPIMASK2 file is used in chandra_repro via dmcopy.

---

Installation

Supported Platforms

• CIAO 4.13 is supported on:

  • Linux 64 bit
  • Apple macOS 10.13 (High Sierra) through macOS 10.15 (Catalina)

Installing with the conda package manager

• CIAO can be installed using the conda package and environment management system.

  The conda release is available from the custom CXC channel, and can be installed with the following command if you have conda installed (see the conda installation instructions for more information, including known problems and incompatibilities):

  $ conda create -n ciao -c https://cxc.cfa.harvard.edu/conda/ciao/ ciao sherpa ds9 ciao-contrib caldb_main marx
  

  With the conda edition users can choose to install CIAO with Python 3.8 (the default), 3.7, or 3.6. Other OTS such as numpy and matplotlib will also be be different based on the python version.

Updates to the ciao-install Installation Script

• Changes to prevent CIAO from being installed on unsupported platforms (CentOS6 and earlier as well as macOSX Sierra and earlier).

Source builds

• Fixes to configure various OTS packages to allow for easier source builds.

• Fix to the prism wrapper script when building on Ubuntu system.

• CIAO now requires the C++ compiler implements the GNU++11 standard. Users with GCC 6 and earlier may need to explicitly set CXX="g++ --std=g++11" when running configure to build.

  unix% ./configure --with-fits --with-ascii CXX="g++ --std=g+11"
  

IPython Settings

• Users will be prompted to update their IPython profiles the first time they run sherpa

Remove old parameter files

• With every new CIAO release, some parameter files are changed: new parameters may be added and occasionally old ones removed or renamed. Deleting or renaming the local parameter directory ensures that the correct parameter files will be accessed the first time a tool is run:

  unix% rm ~/cxcds_param4/*
  

---

Tools

acis_find_afterglow

• Internal updates to support newer C++ standards.

acis_process_events

• Add support for Level 3 aspect solution files (different CONTENT keyword).

addresp

• Fix for XMM/Newton data which does not have LIVETIME and ONTIME in the ARF.

asp_offaxis_corr

• 

  This tool updates an input aspect solution file by correcting the pointing (ra, dec, roll, quaternions) for the offsets measured by the FID lights (dy, dz, dtheta).

dmarfadd

• Fix the value of TG_M keyword in the output ARF files.

dmcoords

• A bug has been fix which prevented the tool from properly computing Chandra grating coordinates. Specifically users can now supply the zeroth order location (ra_zo, dec_zo), the grating arm and order, and a location (eg in sky or chip coordinates), and the tool will now report the correct grating angular coordinated (GAC: tg_r, tg_d), as well as the expected Energy.

• The tool will now error out nicely if the CALDB is not installed.

dmfilth

• Improved error checking with method=POLY when there are no valid source or background regions.

dmhedit

• Now properly interprets quoted numbers, eg "0334", as string rather than as numbers.

dmhistory

• The HISTNUM keyword is no longer required for the tool to access HISTORY records.

dmimgpick

• Improved error reporting, including exit status.

dmlist

• Now properly display infinities as Inf and -Inf instead of as NaN.

dmpaste

• Updates to prevent duplicate column, including vector column, names when combining files.

dmstat

• Fix to recognize subspace filters regardless of axis grouping (two 1-D axes vs, one 2-D axis).

hrc_process_events

• Removed the obsolete parameters: aligmnentfile, instrume, tstart, tstop, and time_offset.

mkgrmf

• Added missing keywords OBS_ID, DS_IDENT, and CHECKSUM to output files.

mkrmf

• Internal cleanup to support newer C++ standards.

mkwarf

• Fix to remove unnecessary check for CALDBALIAS environment variable.

pget

• Internal cleanup to support newer compilers.

  This applies to all parameter tools: paccess, pdump, pget, pline, plist, pquery, and punlearn.

• Tool will now error out if it cannot write to any of the user-writable directories specified by the PFILES, PDIRS, or UPARM directories. The user-writable part is the colon separated list of directories listed first before the semi-colon.

skyfov

• Internal cleanup of memory management.

tgdetect2

• Fix problem with Python 3 compatibility when running tg_findzo.

• Improved error checking (ensure warnings are printed to the terminal).

tg_resolve_events

• Removed obsolete parameters: alignmentfile and time_offset.

• Update HRC standard event definition, stdlev1_HRC, to include additional columns: AU/V 1-3 , and SUMAMPS.

wcs_match

• Expanded the header meta-data output by the tool and various improvements to existing keywords (units,comments).

• Log handling has been cleaned up.

• Initialize some variables used with higher verbose|debug settings.

• Added new select=auto or select=manual parameter. With select=manual, users can interactively choose which matching source pairs to include or exclude from the final solution. With select=auto the tool will run without interaction.

  At verbose=1 and higher, additional diagnostic information is now printed to the screen to facilitate manual selection.

  If select=manual and method=trans and all the source pairs are excluded, the tool will produce a No-Op (unit) transform.

  A new keyword, MAN_UPDT is written to the output file indicating whether the source selection was manual (True) or automatic (False).

• Fix a bug with method=trans reporting an error when there is only 1 matching source pair. The output was correct in this case but the tool reported it as an error.

---

Parameter Files

hrc_process_events

• The following parameter have been removed from the tool: alignmentfile, instrume, time_offset, tstart, and tstop.

tg_resolve_events

• The following parameters have been removed from the tool: alignmentfile and time_offset.

• The stdlev1_HRC event defintion parameter has been updated to include additional columns including: sumamps and au[1-3]/av[1-3]

wcs_match

• A new parameter has been added: select. When select=auto the tool will automatically determine the best set of matching source pairs to use to compute the astrometric correction. When select=manual, users can interactively include and excludes source pairs to use to determine the astrometric correction coefficients.

---

Sherpa

Switching to Matplotlib

• Sherpa in 4.13 only supports plotting using matplotlib. Users upgrading from CIAO 4.11 and earlier may see the following warning when you start Sherpa:

  WARNING: chips is not supported in CIAO 4.12+, falling back to matplotlib.
  WARNING: Please consider updating your $HOME/.sherpa.rc file to suppress this warning.
  

  and Sherpa will switch itself to using Matplotlib. Please see the Sherpa FAQ entry on how to change the resource file to stop seeing this warning.

  Sherpa contains a number of functions that change the plot preferences (i.e. how new plots will be displayed), such as set_xlog. However, users may be used to using ChIPS commands - such as limits or log_scale - to change existing plots. Please see the ChIPS to Matplotlib conversion guide to find out the appropriate replacements (which in this example are plt.xlim, plt.ylim, plt.xscale, and plt.yscale).

Changes to the Sherpa configuration file

• The default Sherpa configuration file has been updated for CIAO 4.13 to add the multiprocessing section, which contains the multiprocessing_start_method, and to remove the chips section, as it is no-longer used.

  If you have trouble running Sherpa it is worth removing your existing resource file and then re-starting Sherpa (which will create a new version) to see if this fixes problems: for example

  unix% mv ~/.sherpa.rc ~/.sherpa.old.rc
  unix% sherpa
  ...
  

Multiprocessing, Python 3.8, and macOS

• In Python 3.8 on macOS the multiprocessing module has changed how it starts processes. As the new default does not work for Sherpa, the multiprocessing.multiprocessing_start_method setting has been added to the Sherpa configuration file to allow this setting to be changed, but please contact the CXC Helpdesk for help if you find you need to change this setting.

Plotting improvements

• CIAO 4.13 continues the improvements made in CIAO 4.12 to the plotting functionality, and includes:

  • changes to the display of PHA and histogram data and models;
  • the transparency of the data or model can be changed by setting the alpha argument to a value between 0 and 1;
  • the plot function can now accept keyword arguments, with the settings applied to each plot;
  • and the overplot option now works for the plot_fit_xxx and plot_bkg_fit_xxx family of commands (e.g. plot_fit_ratio and plot_bkg_fit_resid).

  As an example, the following shows the output of plot_fit_delchi(xlog=True, ylog=True) for a PHA dataset in CIAO 4.12:

  

  and CIAO 4.13:

  

Flux calculations

• Extending the CIAO 4.12.1 changes to the calc_energy_flux and calc_photon_flux commands, there have been a number of improvements to the sample_energy_flux and sample_photon_flux commands. The main change is the introduction of the model parameter, so you can now calculate the flux for a component (e.g. the unabsorbed flux). The documentation has been improved and the support for the scales array no correctly handles both NumPy and list inputs.

  The How do we calculate a flux in Sherpa Jupyter notebook has been updated to discuss this new functionality.

XSpec

• The XSPEC models have been updated to version 12.10.1s from version 12.10.1n in CIAO 4.12 (so there is essentially no change).

  Users who want to use XSPEC 12.11.1 will need to build Sherpa themselves.

  The XSPEC convolution models - such as xscflux, xszashift, and xsgsmooth - can now be used in Sherpa models. For example, to attach the XSPEC cflux convolution model to just the power-law component in a model you can say:

  sherpa> set_source(xsphabs.gal * xscflux.cmdl(powlaw1d.pl))
  sherpa> pl.ampl.freeze()
  sherpa> print(get_source())
  (xsphabs.gal * xscflux.cmdl(powlaw1d.pl))
     Param        Type          Value          Min          Max      Units
     -----        ----          -----          ---          ---      -----
     gal.nH       thawed            1            0       100000 10^22 atoms / cm^2
     cmdl.Emin    frozen          0.5            0        1e+06        keV
     cmdl.Emax    frozen           10            0        1e+06        keV
     cmdl.lg10Flux thawed          -12         -100          100        cgs
     pl.gamma     thawed            1          -10           10
     pl.ref       frozen            1 -3.40282e+38  3.40282e+38
     pl.ampl      frozen            1            0  3.40282e+38
  
  sherpa> print(cmdl)
  xscflux.cmdl
     Param        Type          Value          Min          Max      Units
     -----        ----          -----          ---          ---      -----
     cmdl.Emin    frozen          0.5            0        1e+06        keV
     cmdl.Emax    frozen           10            0        1e+06        keV
     cmdl.lg10Flux thawed          -12         -100          100        cgs
  
  

  

  Default XSPEC abundance and cross-section settings

  The 12.10.1 XSPEC model library now sets default values based on the contents of your $HOME/.xspec/Xspec.init file if it exists.

  If you do not have this file then the default cross-section table has changed: it now defaults to vern when it was bcmc in CIAO 4.11 and earlier.

PHA data handling

• There have been a number of improvements and fixes to PHA data handling. In particular fitting a model to the background dataset has been re-worked and can result in different results if the source and background PHA datasets have different exposure times.

  The load_pha command will now use the id argument when loading in a PHA2 file: it defines the first dataset which will be loaded whereas previously it always started at 1.

  The ungroup and unsubtract commands will no-longer error out if used on ungrouped or unsubtracted data.

Notable changes

• Documentation

  The Sherpa ahelp files have been updated to match the Python docstrings. Each command now has its own ahelp file, rather than combining multiple commands into a single file.

  Jupyter notebooks

  Many of the objects created by Sherpa will now take advantage of Jupyter notebooks to display a HTML table or an actual plot. As an example, load a PHA file with load_pha and then call get_data, or call get_source to display a model.

  An example notebook shows some of the features supported by Sherpa.

  Handling asymmetric errors

  The resample_data routine, used to estimate the effect of asymmetric errors, has seen a number of improvements in behavior and the data that is returned.

  Numpy masked arrays

  Numpy masked arrays can now be used to create a dataset with load_arrays or any of the Data classes.

  Model combination

  The dimensionality of models is now checked, so it is no longer to add a 1D model to a 2D model.

  The voigt1d and pseudovoigt1d models

  The absorbtionvoigt and EmissionVoigt models were problematic and have been removed. They have been replaced by the more-generic voigt1d and pseudovoigt1d models.

Bug fixes

• The full list of changes made in this version of Sherpa are:

  Sherpa Wrapper Default Version Check

  Updates CIAO's Sherpa wrapper in Conda to print the version based on version of Sherpa found by Python.

  Cache code: make the parameter optional and fix some missing cases

  Pytest opt

  An initial release of simultaneous fit on multicores (slower for most

  Distributes the evaluations of the multiple independent data sets using the multi-cores built-in the user's workstation. The current default setting for this PR is to evaluate the multiple independent data sets sequentially since the overhead for distributing the workload across multi-cores is high if the evaluation of the data sets is not time consuming.

  Add HTML representations of common classes for IPython display (fix #345)

  An initial version of HTML display support for Sherpa objects for users of IPython/Jupyter notebooks.

  Added invitation for native software citation

  Adds a citation method to the sherpa module and updates CITATION to refer to this new functionality:

  sherpa> sherpa.citation('latest')
  

  Address matplotlib linestyle removal in Matplotlib version 3.3

  Matplotlib version 3.3 now requires you to split out the drawstyle and linestyle arguments. This set of PRs changes the code so that the two arguments are now set: linestyle defaults to solid and drawstyle to default for plots and step-mid for histograms. This appears to replicate the old version, and should be backwards compatible (drawstyle is supported in Matplotlib 2.2.5 and 1.5.3)

  Residual-style plots ignore the ylog setting (fi#586)

  Residual, ratio, and delchi plots always use a linear scale for the y axis, no matter what the ylog setting is.

  resample_data: when the error range on a bin includes -1 and no-longer restrict sampling to +/-1 sigma

  Fixes resample_data/ReSampleData so that it correctly handles bins where the range of data values (i.e. low to high limit) includes the value -1.

  Fix issue 638, guess needs to update fwhm/sigma bounds

  Adds support for guessing the fwhm or sigma parameters of the gauss2d, normgauss2d, and sigmagauss2d models

  Add xspec convolution api

  Adds support for XSPEC convolution-style models.

  Add docs for setting up all dependencies of the source build with conda

  Adds documentation on how to use conda to install source build dependencies

  No error for no-ops in `ungroup` and friends

  Removes DataErrs when ungrouping a dataset that is not grouped or unsubtracting a dataset that was not subtracted

  Replace EmissionVoigt/AbsorptionVoigt models by Voigt model (fix #597)

  Replaces the emissionvoigt and AbsorptionVoigt models with a single model, voigt1d. The emissionvoigt and AbsorptionVoigt models will error out when an instance is created, pointing users to voigt1d (as the parameter definitions have changed).

  Add XSPEC 12.11.0 support (HEAsoft 6.27)

  Adds support for XSPEC 12.11.0 (released March 31 2020)

  Add a pyproject.toml file

  Adds a pyproject.toml file to the top level (PEP 518) to document build requirements

  fix issue #788, fit using moncar with verbose=1 and/or numcores!=1

  Fixes NameError in moncar when verbose setting >0

  Fix ignore/notice error-ing out when all bins have been set bad #790

  Allow notice and ignore to be called on a dataset which has no "good" bins after ignore_bad has been called

  Accept masked arrays for Data XXX creation

  Allows use the mask of numpy arrays when initializing DataXXX

  Updates to sampling of energy and photon fluxes: bug fixes and calculate unabsorbed components

  Fixes and improvements to the energy_flux and photon_flux set of commands: sample_energy_flux, sample_photon_flux, plot_energy_flux, plot_photon_flux, get_energy_flux_hist, and get_photon_flux_hist.

  Remove all __future__ imports

  Removes __future__ imports that were in place when sherpa supported python 2

  Add unsubtract and ungroup to datastack

  Adds corresponding unsubtract and ungroup methods to match subtract and group

  fix typo in rst docs

  Fixes a typo in the rst docs

  Add hyperlink to similarly named SHERPA package

  Documentation update to include a link to thethe similarly named package "SHERPA" for hyperparameter tuning of machine learning models.

  Fix scaling of staterror when reading PHA file with rate instead of counts

  Fixes scaling of STAT_ERR when reading a PHA spectra with a RATES column instead of COUNTS

  xspec table models: add out-of-bound check to avoid segfault

  Adds explicit out of bounds checks to avoid segv when calling an XSPEC table model

  Updates to the Read-The-Docs build environment

  Update the Read-The-Docs configuration to the latest version (2), and switch to a cleaner build (using conda) for the documentation. The minimum Sphinx requirement is now 1.8 (updated from 1.3).

  Support setting the id value in load_pha with pha2 files (fix #666)

  Updates load_pha to set the data ids of PHA2 datasets to: id to id + nfiles - 1 (when id is an integer), or f"{id}1" to f"{id}{nfiles}" when id is a string.

  Minor documentation improvements to ReadTheDocs

  Adds minor updates to the convolution, regrid, and model evaluation sections of the ReadTheDocs documentation

  Clean up of the XSPEC interface code

  Internal changes to the XSPEC interface code, which reduces the amount of similar (sometimes identical) code. There is no change to the behavior of the XSPEC models.

  Minor documentation fixes

  Several documentation fixes: XSPEC parameter names, avoiding confusion over links on references (Sphinx pages), and adding some basic documentation to the sherpa.astro.background module

  Add parameter-clipping strategy to routines that generate samples (fix #846)

  The addition of the clip parameter lets users control how parameter values are clipped before use in sample_energy_flux, sample_photon_flux, plot_energy_flux, plot_photon_flux, get_energy_flux_hist, and get_photon_flux_hist.

  Add delete_pileup_model (fix #441), list_pileup_model_ids, list_psf_ids functions, fix list_models (fix #749)

  Add the delete_pileup_model function to allow a pileup model to be removed from a fit (issue #441), and list_psf_ids and list_pileup_model_ids routines to list those datasets with an associated PSF or pileup model. The list_models routine no-longer returns an iterator but a list when given an option (issue #749).

  Add MacOS LDFLAGS warning to devdocs

  Copies the warning about setting PYTHON_LDFLAGS from the install page to the developer docs

  Move logic from ui layer to DataPHA class: background responses (fix #879, #880)

  Moves the logic for adding a background response, if one doesn't exist, from the UI layer to the DataPHA class to clear up several edge cases

  Support vector backscales and bugfix for background modeling

  Supports fitting backgrounds to PHA datasets which have a variable BACKSCAL array (rather than a scalar), which can come from combining spectra (e.g. the CIAO contrib script combine_spectra) or from the data extraction process. In doing so a number of routines related to the scaling of background-to-source aperture data have seen adjustments to behavior and some enhanced functionality (such as sherpa.astro.ui.get_bkg_scale and the sherpa.astro.data.DataPHA.get_background_scale method).

  Add support for XSPEC 12.11.1

  Allows Sherpa to be built against XSPEC 12.11.1. There are no new or changed models in this release compared to XSPEC 12.11.0.

  Update likelihood descriptions

  Updates likelihood description in several doc-strings and clarifies descriptions of statistics.

  Ensure that 1D and 2D models are not combined in an expression.

  Checks that models have the correct dimensionality when combining them, so expressions like gauss2d.src + const1d.bgnd will now raise a ModelErr.

  Improve and add support for histogram plots

  Updates to the display of histogram-style plots, in particular for 1D integrated datasets and some model display for PHA data. The histograms now cover the full length of each bin (previously they only showed half the bin for the first and last bins), and gaps in the histogram (where the high edge of a bin is less than the lower edge of the next bin) are now correctly displayed.

  FEATURE: support alpha transparency for matplotlib plots

  Supports the 'alpha' preference setting for most plots and contours generated by Matplotlib

  Refactor: remove _get_model/source methods

  Removes the _get_source and _get_model methods as they are the same as get_source and get_model

  Docs: fix several minor issues

  Documentation clean up - the load_template_interpolator function was named incorrectly in the example, and the examples for get_source_component_plot and get_model_component_plot were missing the trailing _plot for the function names

  Docstring: changes for ahelp

  Formatting changes to the docstrings for several routines that are useful for SDS in generating ahelp files

  fix an issue with PHA filtering that affects plot_model

  Fixes an issue when applying filters to generate the plot_model and plot_model_component plots for PHA datasets

  enable regrid for the BinaryOpModel class (rebased #798)

  Enables composite models (created by a binary operation between two models) to be regridded. The composite model is evaluated at the new grid, and it is only the combined model expression that is rebinned to the data grid.

  channel settings with grouped PHA data and model plotting

  Fixes a bug when filtering a grouped PHA dataset using analysis=channel. The selected bin ranges did not always match the versions you would have received when doing the same operation with energy or wavelength analysis (the first or last bin may have been different).

  Updated to ignore the .vscode directory

  updates system .gitignore to skip over vscode directories

  Support overplot option in plot_fit_xxx (issue #700)

  The overplot argument can now be used with the plot_fit_xxx and plot_bkg_fit_xxx routines (e.g. plot_fit_ratio).

  Update Data1DInt / DataPHA data plots to use the histogram plot style

  Switch the plot_data/plot_bkg plots to draw the data as histograms for Data1DInt and DataPHA plots. This will change the behavior of code that accesses the plot data - e.g. get_data_plot or the dataplot element of get_fit_plot - since for DataPHA and Data1DInt datasets the data will no-longer have an x attribute but xlo and xhi. To reduce the need for code changes for existing scripts - as many people use get_model_plot and get_data_plot to get the data - the histogram plots will return (xlo+xhi)/2 when asked for the x attribute.

  Docs: note XSPEC convolution models are new in 4.12.2

  Documentation only change noting that the support for XSPEC convolution models is new to 4.12.2.

  Minor documentation improvements

  Fixed several minor issues in the existing documentation

  Improve handling of the default id with PHA background datasets (fix issue #943)

  The set_default_id call no-longer sets the default identifier for background ids, which are now kept as the value 1. This avoids several issues when using set_default_id with the background components of PHA datasets

  Documentation fixes

  Minor documentation updates and adding ReSampleData to the RTD documentation

  docstring fixes

  Cleans up typos in documentation

  Docs: include the example notebook in the RTD pages

  Adds the example notebook (SherpaQuickStart.ipynb) to the Sphinx documentation under the "Notebooks" heading

  Update notebook support for Data1DInt/PHA data

  Improves the display of Data1DInt and DataPHA objects when displayed directly by Jupyter notebook and add a new notebook showing off the notebook support

  ocs: note plot_fit_xxx overplot change and add info on notebook support

  Adds documentation updates including notes on overplot support in plot_fit_xxx and plot_bkg_fit_xxx functions and RTD notes on adding notebooks

  Docs: avoid invalid escape sequence warning

  Very-minor tweak to the Chi2 docstring

  Fix 2d image filtering (fix #965)

  Fix problems with ignore2d and notice2d when multiple regions are used.

  Support Python 3.8 for MacOSX

  Add a new multiprocessing_start_method option to sherpa.rc and sherpa-standalone.rc and init code to set start multiprocessing method to fork by default.

  Docs: fix sphinx warnings

  Fix documentation in sherpa.sim.sample on the ReadTheDocs site.

  Update Sherpa version example and Python versions in CONTRIBUTING.md

  Doc updates to remove obsolete python 2.7 reference

  Improve plots of PHA data sets and models when the response grid is not ideal

  Improve the display (plots) of PHA data and models when the response grid contains minor numerical differences. (fixes #977)

  Address set_xlog/ylog problems with DataPHA/Data1DInt classes

  Fixes an issue with set_xlog/ylog routines for PHA and 1D integrated datasets

---

Graphical User Interfaces

prism

• Quick plots are now "new" plots (rather than "over" plots) by default.

• Internal cleanup: gtk-sheets.

• Removed the Analysis Menu.

• Fix for issue with cursor hovering over long column name.

obsvis

• Facility added to display error bars on aim point. The calibration file updated to handle error bar magnitudes based on instrument dither mode. New Adjust-FoV menu item indicating whether errors shown are in active dither mode or in stare mode. The preferences are updated to allow default setting of active dither mode or no dither mode.

---

Analysis Scripts

acis_check_pha_range

• Script to estimate the actual energy range as discussed in the Understanding ACIS on-orbit energy filters page.

chandra_repro

• If the script can only find the original CONTENT=ASPSOL aspect solution, it will not run the new asp_offaxis_corr script to apply the boresight corrections to the aspect solution, creating a pcadfNNNN_repro_asol1.fits file (where NNNNN is the OBS_ID). This can be overridden by setting asol_update=no.

• Updates to support the changes to tg_resolve_events and hrc_process_events parameter files.

• Replace the hard-coded HRC-S+LETG background PI calibration file name with an automated CALDB lookup.

• Now uses a clean (level-2 like) event file with tgdetect2.

convert_xspec_script

• This script (which is experimental) will convert a XSPEC save file to a file that can be used in Sherpa or a Python program. It is intended to simplify checking out fits from XSPEC in Sherpa, but is not guaranteed to create exactly the same results due to differences in how XSPEC and Sherpa work.

convert_xspec_user_model

• The script to allow XSPEC user models to be used in Sherpa has been added back in (it was removed in CIAO 4.11). The interface is similar to before but the model compilation is handled differently. The script remains experimental and users are advised to contact the CXC Helpdesk for help.

dax

• Updates to plotting to match changes to sherpa PHA plot style.

download_chandra_obsid

• These scripts should be more robust to future changes to the Chandra Data Archive.

find_chandra_obsid

• These scripts should be more robust to future changes to the Chandra Data Archive.

merge_obs

• The warning message about not using the merged event file for calculating instrument responses now restricts the message so that the value does not include excess precision, but is limited to the limit value. The tangent point in the screen output has also seen its accuracy reduced (this only affects the screen output).

mktgresp

• Fix for HRC-I + LETG combination. The channel grid was mismatched between the RMF and the PHA files. Note: there are no calibrations for gratings used with HRC-I, so a diagonal RMF is created.

reproject_obs

• The warning message about not using the merged event file for calculating instrument responses now restricts the message so that the value does not include excess precision, but is limited to the limit value. The tangent point in the screen output has also seen its accuracy reduced (this only affects the screen output).

regphystocel

• A new script to convert regions stored in physical coordinates, including FITS region files, to celestial coordinates stored in ds9 format. This is especially important when providing user supplied regions to srcflux when combining data.

rank_roi

• The output from the roi tool will generally exclude the overlapping area between multiple sources. This new script will assign that overlapping area to the "best" source; users can select the figure-of-merit (brightest, biggest, etc) used to determine source ranking.

specextract

• Internal changes, cleaning up code and logic.

srcflux

• When multiple event files are specified, users will now also get a flux estimate from all the observations combined. Currently model independent fluxes are not combined; but rates, photon fluxes, and model fluxes (absorbed and unabsorbed) are computed. Uncertainties are computed using the aprates tool. Variable sources will likely yield incorrect flux estimates.

• In addition the energy band parameter has been changed to band=default. This allows both ACIS (default is broad band) and HRC (default is wide band) to work without explicitly changing this parameter.

• A new random_seed parameter has been added which is passed to the simulate_psf script when psfmethod=marx. The default random_seed=-1 will use the current time to seed the random stream.

• Updates to minimum number of events to simulate with MARX. Also updates to deal with floating point precision of MARX PSF images.

---

Python Modules

crates

• Fix for extraneous field() being added to region subspace when copying files.

• Improved argument vetting in get_subspace_data().

• Fix issue preventing modifying individual vector column values.

• Added read_dataset and write_dataset to the high-level user interface (HLUI).

• Improved error checking in get_transform() method.

---

Libraries

caldb4

• Updates for newer C++ compiler standards

• Fix for rpath substitution after rebuilding from source.

cxcparam

• If the user's parameter directory does exist but does not contain the specified parameter file, pset would copy the system file into the user's param directory and still fail. Updated the cxcparam library to return the copied filename instead of the original filename and write parameter to the copied file. This change affects pset but does not change the behaviour of any other partool.

• clean up compiler warnings when using gcc 4.8.5 and 7.4.0

datamodel

• Internal cleanup needed to eventually support additional datatypes

• Fixed a problem when accessing virtual columns (e.g. world coordinates) and applying a filter at the same time. For example:

  % dmcopy "evt.fits[sky=circle(4096,4096,100)][cols ra,dec]" filter.evt
  

  Under some circumstances the output could contain garbage values.

• Fix for tsv format files with long column definition strings (including comments). This was affecting tools such as dmlist and prism when trying to display output from the Chandra Source Catalog.

• Fix to correctly interpret ASCII files with a header but with 0 rows.

• Fix for files containing keywords similar to DM reserved keywords (eg DSTYPE vs. the reserved DSTYP#).

• Fix error when opening same file multiple times with different read+write permissions. Affected crates users using read_file and the TABLECrate or IMAGECrate objects directly.

dslib

• Updates to support copying WCS from image axes to table columns.

grplib

• Updates to configure scripts.

hrdlib

• The DS_IDENT keyword is no longer populated based on the OBS_ID value. It is now simply copied from the input file to the output.

  The use of dataset identifiers has been deprecated. Chandra data is now being archived with DOI values.

pixlib

• Internal cleanup to support newer compilers

• Corrects a problem reading the wrong grating angle value for the HRC-I + MEG grating arm. This is a highly atypical configuration and this bug affects only a small number of datasets.

regionlib

• Internal cleanup to support newer compilers.

---

Environment

Off-the-Shelf (OTS) Package Versions

• The following OTS packages are included with CIAO 4.13 installed with ciao-install. For more information on how the OTS packages are built for use with CIAO, refer to the INSTALL_SOURCE file distributed with the software.

  

  These do not hold for the conda-installed version of CIAO

  The conda-installed version of CIAO 4.13 has different OTS versions, and it also depends on the Python version you have chosen to use.

  Executables

  • ds9 8.2

  Python

  • Python 3.7.6
    Frameworks on OSX have been disabled to avoid a problem with users picking up the wrong version of NumPy.
  • NumPy 1.18.1
  • ipython 7.13
  • six 1.14.0
  • pip 20.0.2
  • setuptools 28.8.0
  • cython 0.29.15
  • Matplotlib 3.2.0
  • jupyter notebook 6.0.0
  • pytest 5.3.5

  Libraries

  • cfitsio 3.490
  • CCfits 2003.12.08
  • wcssubs 5.19
  • readline 6.3
  • ncurses 5.7
  • XSPEC v12.10.1s (models only)
  • fftw 3.3.8
  • gsl 2.5
  • xerces 3.2.2
  • GTK 3.22.30
  • adwaita-icon-theme 3.34.3
  • glib 3.56.2
  • hicolor theme 0.17
  • fontconfig 2.13.1
  • freetype 6.17.0
  • jpeg 9.3.0
  • tcl 8.6.10
  • tk 8.6.10
  • xpa 2.1.19
  • openSSL 1.0.1e

Compiler versions

• The following versions of the gcc compiler were used to compile the binary CIAO releases installed with ciao-install. The conda-installed compiler versions are different.

  System	Build System	Version
  Linux	CentOS 7	gcc 4.8.5
  macOSX	High Sierra	llvm 9.1.0

• Several CIAO applications have been updated to support the GNU++11 standard.

---

Documentation

---

ChIPS

---