CIAO 4.17 Release Notes

CIAO 4.17 is distributed for the following platforms:

• Linux 64 bit
• Apple macOS 12 (Monterey) through macOS 15 (Sequoia), Intel/x86 and ARM (Apple Silicon)

CIAO is available using the conda package manager; conda builds are available using Python 3.11 for both Linux and Mac. More information can be found in the conda section of the Installation notes below. The ciao-install script can be used to automate the conda installation process.

There is no support for 32 bit operating systems, older Linux (CentOS 7 era), older macOS platforms (macOSX Big Sur and earlier), or any version of Windows (including using WSL). More details can be found on the Platform Support page.

• Notable changes and improvements in CIAO 4.17
• How CALDB 4.11.6 Affects Your Analysis
• Installation and Setup
• Tools
• Parameter Files
• Sherpa
• Graphical User Interfaces
• Analysis Scripts
• Python Modules
• Libraries
• Environment
• Documentation

Notable changes and improvements in CIAO 4.17

How CALDB 4.11.6 Affects Your Analysis

HRC Data

• HRC-I Time-Dependent QE Files

  The HRC-I quantum efficiency QE has been decaying for some time, particularly since the year 2011, when a 5% to 10% per year reduction had been measured. To address this issue, the HRC-I Calibration Team had introduced a time-dependence into the QE files in 2018 which are now extended in time to the latest epoch of observations. Included in this update (fileset N0016) are two new files coinciding to the recent HRC-I high-voltage adjustments affecting observations after 2024 March 30 and 2024 September 20, respectively. The QE update affects the results from mkinstmap and fluximage and is applicable to all previous epochs of HRC-I observations.

Installation and Setup

Supported Platforms

• CIAO 4.17 is supported on:

  • Linux 64 bit
  • Apple macOS 12 (Monterey) through macOS 15 (Sequoia), Intel/x86 and ARM (Apple Silicon)

  Unlike previous versions of CIAO, CIAO 4.17 only supports Python 3.11.

  CIAO can be installed using either the ciao-install script or conda package manager.

Install using the ciao-install Installation Script

$ bash ciao-install

--remove marx

Installing with the conda package manager

$ conda create -n ciao-4.17 -c https://cxc.cfa.harvard.edu/conda/ciao -c conda-forge \
ciao pyciao sherpa ds9 marx ciao-contrib caldb_main

Smoke Tests

$ ciao
$ cd $ASCDS_INSTALL
$ bash test/smoke/bin/run_smoke_tests.sh

• Users of previous versions of CIAO will note that the order of the tests is different; in the event of a test failure be sure to include the name of the test, not just the test number when contacting helpdesk.

Source builds

$ bash ciao-install --add ciao-src --add sherpa-src

$ ciao
$ cd $ASCDS_INSTALL
$ bash ./ciao_build.sh --prefix `pwd`

Remove old parameter files

unix% rm ~/cxcds_param4/*

Tools

acis_process_events

• New CTI_TMAP keyword has been added to identify which CTI correction algorithm was applied.

• Fix for memory problem causing fatal crash on macOS Sequoia.

aconvolve

• FFT convolutions are now performed at double precision. The increased dynamic range provides better numeric accuracy especially for small pixel values in the presence of a large range of pixel values.

• Bug fix when the input image was an unsigned byte (8-bit) image. Previously the data was read in as a signed byte.

acrosscorr

• FFT convolutions are now performed at double precision. The increased dynamic range provides better numeric accuracy especially for small pixel values in the presence of a large range of pixel values.

apowerspectrum

• FFT convolutions are now performed at double precision. The increased dynamic range provides better numeric accuracy especially for small pixel values in the presence of a large range of pixel values.

arestore

• FFT convolutions are now performed at double precision. The increased dynamic range provides better numeric accuracy especially for small pixel values in the presence of a large range of pixel values.

collect_chandra_dois

• This tool will recursively scan the files in the input directory looking for the Chandra DS_IDENT keyword. It will generate a list of all the unique the Digital Object Identifier (DOI) values it discovers.

csmooth

• FFT convolutions are now performed at double precision. The increased dynamic range provides better numeric accuracy especially for small pixel values in the presence of a large range of pixel values.

dmcontour

• Fix for repeated and blank error messages.

dmcopy

• Fix bug causing opt=all to corrupt good time intervals. This was frequently the case when users have applied a time filter to a Chandra gratings dataset and then used dmcopy with opt=all to copy the EVENTS and REGION blocks; the GTIs would become corrupted along with the ONTIME, LIVETIME, and EXPOUSURE keywords. The dmappend work around is no longer required.

  Note: for some historical datasets (eg ROSAT) that do not explicitly specify which block the GTI is associated with, the datamodel will now only associate the GTI with a single, first interesting, block rather than all blocks.

dmextract

• Fix for HDUCLAS3 and HDUCLAS4 keyword in spectra (opt=pha or pha2); the keywords were swapped. Also removed the deprecated HDUVERS1 keyword. The HDUVERS keyword has been updated to "1.2.1"

dmgti

• Fix for "~=" (substring) operator bug causing memory corruption with newer compilers.

dmhistory

• Fix for excess white space after tool name.

dmmakereg

• Fix for unused array elements; they are now set to NaN instead of '0'. This includes columns such a 'R', 'ROTANG', as well as 'X' and 'Y' (especially when Polygon shapes are used).

  This makes dmmakereg consistent with the region library including the python module, as well as the other tools that create region files.

dmtcalc

• Fix for "~=" (substring) operator bug causing memory corruption with newer compilers.

get_src_region

• Fix bug preventing clobbering certain files.

hrc_process_events

• Updates to support newer GCC compilers.

skyfov

• Fix problem with repeated error messages and warnings.

tgdetect

• Improve warning when the initial SNR check fails to find a source to use for the zero order location.

vtpdetect

• Updates required for latest version of C-Lang compilers on macOS.

wcs_match

• Fix for a crash that would occur if there were 0 matches between the input and reference source lists.

• Fix to use standard error reporting library to conform to system standards.

• Fix to suppress duplicate/redundant error message.

wcs_update

• Fix for bug that would quietly fail to update the WCS of the input event files and images if the infile and the wcsfile were the same file.

Parameter Files

Sherpa

changes

• The full list Sherpa updates is given in the 4.17.0 and 4.16.1 release notes for the two standalone releases that are incorporated into CIAO 4.17. The major updates were made to plotting data, fitting, data access, and bug fixes. The release highlights include:

  XSPEC

  • Updated to version 12.14.0k of the XSPEC model library.

  • The addition of the 50 new additive models in this release.

  • New functions: show_xsabund, get_xsabundances, and set_xsabundances.

  • Updates to the default parameter values to reflect changes made in the XSPEC 12.14 series.

  Filtering and grouping

  There is better support for PHA files which start the channels at 0 rather than 1. Filter ranges in channel space may be changed because of this (the same data values are used but using a channel filter from CIAO 4.16 may change the noticed range slightly).

  It is possible for notice and ignore calls to fail after calling ignore_bad. When this happens there will now be a screen message reporting this failure rather than appearing to succeed.

  Models

  Added the calc_source and calc_model commands to calculate the predicted model values. These routines should be used in preference to the get_model_plot command for accessing the model values as an array, as the output does not depend on the plot options (such as set_analysis for PHA data).

  Model expressions no-longer contain excessive brackets, so xsphabs.gal * (xsapec.therm + xspowerlaw.pl + xsgaussian.line) rather than (xsphabs.gal * ((xsapec.therm + xspowerlaw.pl) + xsgaussian.line)).

  When fitting data, Sherpa now automatically includes any linked parameters (that are thawed) even if the model is not included explicitly in the model expression. The previous work around, of adding the model component but multiplying it by 0, can still be used but is no-longer necessary.

  The guess command can now be used with PHA data, although the normalization limits should be checked to make sure they remain sensible.

  Data access

  Data with asymmetric errors - e.g. loaded with load_ascii_with_errors - can now be plotted after filtering. There have also been improvements to allow the statistic and optimized to be changed when resampling the data.

  The way that data is loaded has been changed to reduce differences when loading data with the Crates or pyfits backends. This may mean that slightly different messages are displayed, compared to CIAO 4.16, when loading in data or if there is an issue with the data file.

  Plotting improvements

  The new plot_model_components and plot_source_components commands will now separate and plot additive components in a source expression. So a source model like "gal * (pl + line)" will display the model (or source) plots of "gal * pl" and "gal * line".

  Residual plots of histogram and PHA data have been changed to match the data and model versions. The residual plot objects now have xlow and xhi attributes rather than xerr, and the vaues are now correct when switching to wavelength units.

  Axis labels used in plots can now be customized by calling get_data and then using the set_xlabel and set_ylabel methods on the returned object.

  There have been a number of improvements to the plot and contour commands. They now work correctly when used with overplot/overcontour arguments; arguments can now be given as a sequence, which means per-plot values can be given; and the layout can now be changed by setting the cols and rows arguments.

  The 1D and 2D confidence plots - e.g. created by int_proj, int_unc, reg_proj, and reg_unc - now correctly handle the log parameter when set.

  The plot_rmf command will now display the X axis in Angstroms if the analysis setting for the backend has been changed to "wavelength", and in Channels if the setting was changed to "channel".

  There have been improvements to the bokeh backend.

  Documentation

  This release also includes a number of typing annotations added to some of the routines. The Ahelp page files have been updated to move information about the expected types of function arguments from the SYNTAX to the PARAMETERS section.

Graphical User Interfaces

Obsvis

• Updated default acis chip selections for CfP27

Analysis Scripts

fine_astro

• Automates the typical steps needed to apply fine astrometric registration corrections to a set of Chandra observations.

mkrprm

• Compute approximate PSF aperture correction for radial profiles. See also the Python sherpa_contrib.matrix_model

  There is a new Why? Radial Profile Aperture Corrections page discussing the need to sometimes include aperture corrections when extraction radial profiles.

patch_hrc_ssc

• Identify and patch (replace) corrupt dead time factor values due to Secondary Science Corruption in some HRC data.

chandra_repro

• Added a new parameter: patch_hrc_ssc, which when set to "yes", will run the new patch_hrc_ssc script to look for and patch Secondary Science Corruption events in some HRC data.

download_obsid_caldb

• Updated to support multiple P2_RESP CALDB files used by mkacisrmf which will be available for multiple focal plane temperatures (FP_TEMP).

sherpa_contrib.matrix_model

• Create a new MatrixModel. This is a type of convolution model that will multiply a square matrix with the model. This can be used with the output from the mkrprm script.

ciao_contrib.runtool

• The module has been updated to match CIAO 4.17 parameter files including: chandra_repro, fine_astro, patch_hrc_ssc, and mkrprm.

sherpa_contrib.notebook_plotter

• The tutorial notebook located at $ASCDS_INSTALL/share/notebooks/notebook_plotter_tutorial.ipynb had been updated.

Python Modules

pytransform

• Fix to replace deprecated imp module required for eventual Python 3.12 support.

• Fix for memory leaks identified with latest compilers.

Libraries

datamodel

• Will now recognize the FITS standard dead-time keyword: DEADC, in addition to the DTCOR keyword which is a Chandra convention. This allows users to use CIAO tools to filter data from NuSTAR and other mission and have the correct ONTIME, LIVETIME, and EXPOSURE keywords.

  However, some individual tools including dmextract, pfold, and glvary do not yet use DEADC so they will not include the dead time when computing count rates for data from missions like NuSTAR.

• Fix a portability issue when casting IEEE NaN values to integers.

• Fix for missing warning message when requesting data be cast to an invalid data-type using [opt type={}].

• Fixes to support CFITSIO 4.3.0's change to allow for long/continued keyword comments.

cxcparam

• Fix memory leaks identified using new compilers.

stklib

• Fix for problem when input stack happens to be exactly 1022 characters. Also fix problem when recursively expanding stacks and when appending DM filters to stacks.

pixlib

• Updates to support latest version of GCC/C-lang compilers

• Fix for using pixlib to cast negative double precision floats to long integers; values were being truncated instead of being rounded. This mostly has been seen in hrc_process_events with data outside the aspect solution time interval which is eventually filtered as being outside the good time intervals.

Environment

Off-the-Shelf (OTS) Package Versions

• The following OTS packages are included with CIAO 4.17 installed with ciao-install.

  

  Differences in OTS for conda installation

  The conda-installed version of CIAO 4.17 has different OTS versions, and it also depends on the Python version you have chosen to use.

  Executables

  • ds9 8.6

  Libraries

  • cfitsio 4.4.0
  • xpa 2.1.20
  • XSPEC 12.14.0k (models only)

  Python (major packages)

  • Python 3.11.6 (unlike previous releases there is only one version of Python supported by CIAO 4.17)
  • NumPy 1.26.4
  • Matplotlib 3.9.2
  • jupyterlab (notebook) 1.1.1

Compiler versions

• The following versions of the gcc compiler were used to compile the binary CIAO releases installed with ciao-install. The conda-installed compiler versions are different.

  System	Build System	Version
  Linux	RedHat Enterprise Linux 8	gcc 12.2.0
  macOSX Intel	Big Sur	clang 14.0.6
  macOSX ARM	Big Sur	clang 14.0.6

• Several CIAO applications have been updated to support the GNU++14 standard.

Documentation