CIAO 4.15.1 Release Notes

---

CIAO 4.15.1 is distributed for the following platforms:

• Linux 64 bit
• Apple macOS 10.15 (Catalina) through macOS 13 (Ventura)

Visit our platform support page for our statement about support for Apple's M1/M2 (ARM) chip support used with macOS 11 (Big Sur) and 12 (Monterey), and 13 (Ventura).

CIAO is available using the conda package manager; conda builds are available for Linux and Mac for Python 3.10, 3.9, and 3.8. More information can be found in the conda section of the Installation notes below.

There is no support for 32 bit operating systems, older Linux (CentOS 6 era) or older macOS platforms (macOSX High Sierra and earlier). More details can be found on the Platform Support page.

---

• Notable changes and improvements in CIAO 4.15.1
• Notable changes and improvements in CIAO 4.15
• How CALDB 4.10.7 Affects Your Analysis
• How CALDB 4.10.4 Affects Your Analysis
• How CALDB 4.10.2 Affects Your Analysis
• Installation and Setup
• Tools
• Parameter Files
• Sherpa
• Graphical User Interfaces
• Analysis Scripts
• Python Modules
• Libraries
• Environment
• Documentation

---

Notable changes and improvements in CIAO 4.15.1

---

Notable changes and improvements in CIAO 4.15

---

How CALDB 4.10.7 Affects Your Analysis

HRC Data

• HRC-S Time-Dependent QEU Files

  The HRC-S time-varying quantum efficiency uniformity files have been replaced with a new set, calculated based on unfiltered calibration observations. This latest set (N0017) shows decline in the net count rate of 9.5%, 17%, and 9.7% for zeroth-, positive-, and negative-orders respectively between the latest set of HZ43 calibration observations taken over 2023 June and when they were last observed over 2021 November-December. There is also a notable decline in the positive-order gain and QE at the longest wavelengths. The set of QEU files are unevenly distributed in effective dates and affects every HRC-S observation since the start of the mission.

  These CIAO tools and scripts automatically apply the HRC-S QE and QEU files when creating response files:

  • mkgarf
  • fullgarf
  • mktgresp
  • mkinstmap
  • fluximage
  • merge_obs

---

How CALDB 4.10.4 Affects Your Analysis

ACIS Imaging and Grating Data

• Time-dependent ACIS Gain (T_GAIN) Files for -120 C Data

  The new, combined time-dependent ACIS Gain (T_GAIN) file for February 2022-January 2023 (Epochs 89+90+91+92) are introduced. The combined Epochs 87+88 file has also been updated, which was introduced in CALDB 4.9.7. Therefore, the new T_GAIN affects observations taken since 2021 November 01. Other observations are unaffected by these new calibration files.

  

  Note

  ACIS T_GAIN updates will occur annually going forward rather than semi-annually.

  Users working with ACIS data taken since 2021 November 01 may wish to run chandra_repro and reprocess the data to improve the T_GAIN calibration. The DATE-OBS header keyword records the observation start date.

  See the time-dependent ACIS T_GAIN why page for more information.

  The gain corrections are at the nominal and expected levels of less than 1% of the energy value and users interested in CCD spectroscopy may benefit in applying the new gain adjustment. Grating spectroscopy benefits as well, in the form of improved order sorting. It is unnecessary to apply this adjustment for only doing timing or imaging analysis, although doing so will not have a negative effect.

  Note that only spectra with several hundreds of counts and/or prominent features (in emission or absorption) will show changes from the T-gain refinement that exceed the uncertainties from the gain calibration.

---

How CALDB 4.10.2 Affects Your Analysis

ACIS Imaging and Grating Data

• ACIS QE Contamination Model vN0015

  The ACIS QE contamination model has been upgraded to N0015:

  acisD1999-08-13contamN0015.fits 
  

  Recent calibration work with observations of previously-observed cosmic sources, such as clusters of galaxies, supernova, and blazars, has shown that the accumulation of contaminant on the optical blocking filter has defied systematic predictions.

  This model updates the prior N0014 contamination model for both ACIS-S and ACIS-I, which showed an undercorrection fo the contamination effect in the most recent calibration observations. This update most readily affects observations in 2020 and beyond. For this time frame should use the new CalDB to generate contamination-dependent data products.

  More information about the N0015 contamination models, and prior versions, can be found in the ACIS QE contamination model Why document. A detailed presentation of the derivation of the new model—and the resulting changes in the estimated time-dependent effective area of ACIS-I and ACIS-S—is in the technical details section of the CALDB 4.10.2 Release Notes.

  These CIAO response tools automatically apply the contamination file when creating ACIS response files:

  • mkarf
  • mkgarf
  • mkwarf
  • mkinstmap

  As well as the scripts which use them:

  • specextract (calls mkwarf and mkarf)
  • fullgarf (calls mkgarf)
  • fluximage (calls mkinstmap)
  • merge_obs/flux_obs (calls mkinstmap)
  • mktgresp (calls mkgarf)

---

Installation and Setup

Supported Platforms

• CIAO 4.15.1 is supported on:

  • Linux 64 bit
  • Apple macOS 10.15 (Catalina) through macOS 13 (Ventura)

Installing with the conda package manager

• CIAO can be installed using the conda package and environment management system.

  

  CIAO 4.15.1 conda install requires conda-forge channel

  Updating to the latest version of XSpec requires CIAO to use a newer version of libgfortran, which is not available in the default channel.

  Therefore users will need to include the conda-forge channel when they install CIAO 4.15.1. Because of this, users are encouraged to create a new conda environment for CIAO 4.15.1 rather than trying to upgrade an existing environment.

  The conda release is available from the custom CXC channel.

  $ conda create -n ciao-4.15.1 -c https://cxc.cfa.harvard.edu/conda -c conda-forge \
  ciao sherpa ds9 marx ciao-contrib caldb_main
  

  See the conda installation instructions for more information, including known problems and incompatibilities.

  With the conda edition users can choose to install CIAO with Python 3.10, 3.9, or 3.8. Other OTS such as numpy and matplotlib will also be be different based on the python version.

• The conda installation now requires that matplotlib version be > 3.3.0.

Updates to the ciao-install Installation Script

• The outdated ciao.bash setup script has been removed. bash, ksh, and zsh shell users need to source the ciao.sh setup script.

• The problem with text wrapping not working on some non-Fedora based systems such as Ubuntu (and other Debian based distributions) has been fixed.

Source builds

• Users with GCC 11 and newer will need to use CXX="g++ --std=gnu++14" since CIAO is not compatible with the C++17 standard

• macOS users with clang 14.0 or newer need to use CXX="g++ -Wno-implicit-function-declaration". to compile CIAO.

Smoke Tests

• There are not changes to the smoke tests.

IPython Settings

• Users will be prompted to update their IPython profiles the first time they run sherpa

Remove old parameter files

• With every new CIAO release, some parameter files are changed: new parameters may be added and occasionally old ones removed or renamed. Deleting or renaming the local parameter directory ensures that the correct parameter files will be accessed the first time a tool is run:

  unix% rm ~/cxcds_param4/*
  

---

Tools

acis_find_afterglow

• Internal cleanup

acis_streak_map

• The script now writes proper history records

asphist

• Now supports the object-centered aspect solution created by sso_freeze.

axbary

• The script now writes proper history records

arfcorr

• The script now writes proper history records

celldetect

• The WCS on the input image is now copied to the X,Y columns in the output table. This includes the name of the WCS vector column, which means that EQSRC will be replaced with EQPOS for most data sets.

• The combination of using fixedcell with convolve=yes has been fixed. However, there are still serious limitation when using convolve=yes and its usage is not recommended.

create_bkg_map

• The script now writes proper history records

dither_region

• Corrects typo in verbose output

dmcontour

• A bug has been fixed that prevented creating some contours when the first few coordinates overlap (ie has a width equal to 0).

• The levels parameter can now be either a stack of values (eg comma separted list), or as a grid (eg min:max:step).

• The WCS on the input image is now copied to the X,Y columns in the output table.

dmcopy

• See also the Datamodel notes below.

• The tool name is now lowercase in any error or warning messages

dmdiff

• Now correctly reports diffs between +Inf and -Inf.

• Fix to continue to check headers if either file has zero header keywords.

dmellipse

• Updated the tool to recognize real-valued NULLs and to recognize valid data ranges.

• The WCS on the input image is now copied to the X,Y columns in the output table.

dmextract

• Fixed a bug that when an error occurs would leave quasi-random files in the users current directory.

dmfilth

• The method=POISSON option has been rewritten to use the GNU Scientific Library's Poisson distribution, replacing the approximated distribution previously used.

  The method=DIST and method=GLOBAL have been updated to correctly handle non-integer data, and data with arbitrary dynamic range.

• Internal cleanup.

dmgti

• The script now writes proper history records

dmimg2jpg

• Fixed the cool.lut color lookup table.

dmimgblob

• Updated the tool to recognize real-valued NULLs and to recognize valid data ranges.

dmimgcalc

• The op parameter can now take a stack list for input. Since stacks support line continuation via a trailing "\" character, they can support arbitrary long expressions. For example

  unix% cat my_op.lis
  imgout=\
  (img1*img1_expousre+\
  img2*img2_exposure+\
  ...
  img30*img30_exposure)/\
  (img1_exposure+\
  img2_exposure+\
  ...
  img30_exposure)
  unix% dmimgcalc @myimgs.lis op=@my_op.lis out=my_out.fits mode=h clob+
  

• Improve error checking for unsupported output data types.

dmimgdist

• Updated the tool to recognize real-valued NULLs and to recognize valid data ranges.

dmimgfilt

• Four new filters have been added:

  • ridge : If the pixel value is greater than or equal to all the pixels in the mask region, use it, otherwise it is replaced with a NaN. See also 'peak'.
  • plain : If the pixel value is less than or equal to all the pixels in the mask region, use it, otherwise it is replaced with a NaN. See also 'valley'.
  • q10: Use the 10% quantile value of the pixels in the mask.
  • q90: Use the 90% quantile value of the pixels in the mask.

• A new randseed parameter has been added to control the filter=jitter random selection.

• Updated the tool to recognize real-valued NULLs and to recognize valid data ranges.

dmimghull

• The WCS on the input image is now copied to the X,Y columns in the output table.

• Updated the tool to recognize real-valued NULLs and to recognize valid data ranges.

dmimglasso

• The WCS on the input image is now copied to the X,Y columns in the output table.

• Updated the tool to recognize real-valued NULLs and to recognize valid data ranges.

dmimgpick

• Updated the tool to recognize real-valued NULLs and to recognize valid data ranges.

dmimgpm

• Updated the tool to recognize real-valued NULLs and to recognize valid data ranges.

dmimgproject

• Updated the tool to recognize real-valued NULLs and to recognize valid data ranges.

dmimgreproject

• Updated the tool to recognize real-valued NULLs and to recognize valid data ranges.

dmimgthresh

• Fixes bug when using scaled integer images (output values has scaling applied twice).

dmkeypar

• The tool name is now lowercase in any error or warning messages

dmlist

• The excess white space padding preceding bit values has been removed. Values with more than 1024 bits will now be truncated to 1024 characters.

• The tool name is now lowercase in any error or warning messages

dmmakepar

• The output of dmmakepar has changed. The parameter description is now starts with the keyword unit whereas previously the units appeared at the end of the description. This is preserve the units when written back out to a FITS file where the convention is to have the units first.

dmmaskfill

• Updated the tool to recognize real-valued NULLs and to recognize valid data ranges.

dmnautilus

• Updated the tool to recognize real-valued NULLs and to recognize valid data ranges.

dmradar

• Updated the tool to recognize real-valued NULLs and to recognize valid data ranges.

dmreadpar

• The tool name is now lowercase in any error or warning messages

dmregrid2

• Updated the tool to recognize real-valued NULLs and to recognize valid data ranges.

dmsort

• The tool name is now lowercase in any error or warning messages

dmstat

• The tool name is now lowercase in any error or warning messages

hrc_dtfstats

• Adds tool name to error messages.

imgmoment

• Updated the tool to recognize real-valued NULLs and to recognize valid data ranges.

mean_energy_map

• The script now writes proper history records

mkgrmf

• Updated to support HRC-I+HETG, including diagonal RMFs and for orders greater than 1.

• The tool has been updated for the eventual release of calibration data that will allow for enclosed counts fractions for off-axis sources.

  For this to work, users must use the L2 event file, with the attached REGION extension, for the obsfile rather than using the pha2 file.

mkpsfmap

• Updated the tool to recognize real-valued NULLs and to recognize valid data ranges.

mkwarf

• Added error checking when outfile and weightfile are set the same file name.

pfold

• Fixed crash when the period=0, and improved support for periods much, much less than 1.0 seconds.

• The periodgrid can now be specified using the standard CIAO grid syntax. This includes min:max:step, min:max:#steps, and grid(filename).

reproject_aspect

• The script now writes proper history records

reproject_events

• Now uses the standard CIAO library to parse coordinates. This fixes an issue with using negative zero declination value, eg -00:xx:yy.

• Fixed typo in error message if missing certain keywords

reproject_image

• Updated the tool to recognize real-valued NULLs and to recognize valid data ranges.

reproject_image_grid

• Now using the standard CIAO library to parse coordinates.

• Updated the tool to recognize real-valued NULLs and to recognize valid data ranges.

rmfimg

• Adds tool name to error message and warnings

srcextent

• The input srcfile value can now have DM filters applied, eg srcfile=acis_evt.fits[energy=500:7000]

• The script now writes proper history records

• (4.15.1) Updated to support numpy 1.24

tgdetect

• The script now writes proper history records

tgdetect2

• The script now writes proper history records

tg_findzo

• The script now writes proper history records

tg_resolve_events

• Updated to properly perform order sorting for HRC-I+HETG in the region where the HEG and MEG arms overlap.

vtpdetect

• Fixed a bug that would cause unpredictable behavior if the input image or table has negative physical coordinates.

• The WCS on the input image is now copied to the X,Y columns in the output table. This includes the name of the WCS vector column, which means that EQSRC will be replaced with EQPOS for most data sets.

wavdetect

• Convolutions are now performed in double floating point precision. This greater dynamic range improves the stability of detecting low significance sources.

• A bug has been fixed that would allow wavdetect to report two sources at essentially the same coordinates. This has no effect on other existing sources, but has been shown in rare situations to allow for new detections of previously omitted sources.

• The script now writes proper history records

• The WCS on the input image is now copied to the X,Y columns in the output table. This includes the name of the WCS vector column, which means that EQSRC will be replaced with EQPOS for most data sets.

wcs_update

• Previously the WCS parameters and the NOM keywords were computed subtly differently (difference approaching the machine percision level). This has been correct so that the NOM keywords and the WCS parameters are now computed identically.

---

Parameter Files

dmimgfilt

• The new randseed parameter is used to control the random number stream used by the jitter function.

• The function parameter has 4 new enumerated options: q10, q90, ridge, and plain.

---

Sherpa

changes

• The full list Sherpa updates is given in the 4.15.0 and 4.14.1 release notes for the two standalone releases that are incorporated into CIAO 4.15.1. The major updates were made to data validation, plotting backend and infrastructure. The release highlights:

  XSPEC

  Updated to version 12.12.1c of the XSPEC model library.

  The chi2xspecvar statistics in fitting PHA data now matches XSPEC when the background is subtracted and the source or background group contains 0 counts.

  Models

  Models can now be frozen or thawed when accessing the model objects directly.

  Filtering and grouping

  Change to the filter is reported to the screen in the interactive session when using notice or ignore - including the variants notice_id and ignore2d

  Plotting improvements

  CIAO 4.15.1 continued the improvements to the plotting functionality, and includes internal changes to the plotting code and documentation.

  Documentation

  The Sherpa ahelp files have been updated to match the Python docstrings. Each command has its own ahelp file, rather than combining multiple commands into a single file.

---

Graphical User Interfaces

Obsvis

• Updated obsvis.cal to version 1.16 to include new aimpoints and default dither parameters for ACIS.

---

Analysis Scripts

aplimits

• Compute upper limits for the detection of a source, using data obtained from background apertures in event lists, images, and exposure maps. It is based on the article: On computing upper limits to source intensities (Kashyap et al. 2010, ApJ, 719, 900); https://ui.adsabs.harvard.edu/abs/2010ApJ...719..900K

blanksky

• The stowedbg parameter has been added to allow the stowed background files to be used. This is only useful for CTI-corrected ACIS observations.

bkg_fixed_counts

• Create a background region with a fixed number of counts. The region center is specified and then an annulus is created which encloses the requested number of counts (as a minimum). The inner radius of the annulus is based on the PSF size at the location, set by the enclosed counts fraction, and overlapping source regions if set.

chandra_repro

• A problem when multiple SSO eph1 files are found has been fixed.

• Work around a DM bug when tg_zo_position=detect which prevented tgdetect2 from choosing the tg_find_zo algorithm.

convert_xspec_user_model

• The script has been updated to account for changes in the interface to the XSPEC model library. The script will also now attempt to exclude code that XSPEC's initpackage call may have added to the directory. Please contact the CXC helpdesk if you are having problems with this script.

correct_periscope_drift

• The script has been updated to account for Sherpa plotting updates.

dax

• dax now properly updates the WCS when ds9 has blocked the image being analyzed (this is different to zoom or bin, which did not need updating).

• Three new tasks have been added:

  • Coordinates -> Interactive Grating Coordinates Vector
  • Regions -> PSF Contour
  • Regions -> Fixed Counts Background

• The list of Image Processing -> Non-Linear Filters includes the new dmimgfilt filters: ridge, plain, q10, q90, and jitter.

• Changes to Sherpa fitting routines to clean up the textual output.

mktgresp

• As mkgrmf can now create a diagonal matrix for HRC data, the work-around to handle the HRC-I + HETG case has been removed.

psf_contour

• Create a source region file from a simulated mono-chromatic PSF that is created with MARX. There are a number of ways that the region can be constructed based on the desired fraction of counts: a contour; a lasso, which is similar to the contour option but uses dmimglasso rather than dmcontour; a convex hull, which should reduce the complexity compared to the contour or lassoo options; an ellipse fit to the data; and an ellipse that encloses the fraction.

specextract

• The resp_pos parameter has been added. Previously, specextract would use the location of the maximum pixel in the extraction region, but the default is to use the center from the extraction region, but the resp_pos parameter can be used to change this.

• The way that the tools are run in parallel has been reworked to improve the behavior.

• The screen output - when used in an interactive terminal - has been updated to better handle the case where multiple files are being processed.

• A fix for an odd bug with the name of the combined files not matching the outroot parameter has been fixed.

srcflux

• The plugin concept has been added to srcflux, which allows users to add extra code and merge the results to the tables and outputs that srcflux creates. There are two types of plugin:

  • run after each energy band for each observation
  • run after each energy band for the merged data

  Examples are provided that

  • Perform a spectral file (per obsid, and on merged data)
  • Combine images and exposure maps on merged data
  • Compute hardness ratios (per obsid)
  • Extract radial profiles (per obsid)
  • Perform blind deconvolution using arestore (per obsid)
  • Estimate source extent (per obsid)
  • Get upper limits on missed detections and count rates for false detections using the new aplimits tools (per obsid)

  See also the Using srcflux plugins thread.

• With psfmethod=marx, the PSFs will now be simulated in parallel (based on the 'parallel' and 'nproc' parameters).

• The default background region inner radius has been increased. Previously it was the same as the source radius; it is now 1.7 times the source radius. This improves the background estimate around very bright sources; events spread by the PSF could bias the background estimate.

• When merging observations, a new "INSIDE_FOV" column is added to the merged .flux output file. It is an array of boolean values (True|False) which indicates which observations the source was imaged in.

statmap

• Compute simple statistics for events in each map region. This allows simple statistics to be computed from adaptive binning tools, such as dmnautilus and dmradar.

ciao_contrib.runtool

• The module has been updated to reflect parameter changes and the new scripts added in this release. The module now points out those scripts which can not be run - as they do not have a parameter file - and points users to alternative ways to run them.

sherpa_contrib.chart

• The module has been updated to account for Sherpa plotting updates.

sherpa_contrib.marx

• The module has been updated to account for Sherpa plotting updates.

coords.gratings

• Introduces a new derived class, TGPixlib, to help convert from sky coordinates to grating coordinates: dispersion angles and energy.

---

Python Modules

pycrates

• Fix problem reading images with blank WCS names, such as with ROSAT WMAP files.

• (4.15.1) Updated to support numpy v1.24.

pyregion

• Now preserves the logic when slicing regions. Previously all shapes were OR'd together.
• Fixed a bug that caused incorrect results when using non-contiguous numpy arrays ; non-contiguous numpy arrays can be generated when arrays are sliced.

---

Libraries

caldb4

• Fix to preserve the users timezone; mostly an issue when using the Python module.

• Internal cleanup for compiler issue.

cxcparam

• remove a hardwired number in a string which can be automatically inserted by the preprocessor.

  remove explicit prototype of a system routine

  revert RSSize() argument to void* to be consistent with internal usage.

• corrects the compiler error for multiple definitions of ipar_lineno variable in macOS/Ubuntu source build with GCC 10

datamodel

• behavior of [opt null=#] with BLANK. Correction to the datamodel library handling of null=# in combination with an associated BLANK image key. Previously, there was some conflation of the values, where both BLANK and # were considered NULL, and the BLANK value was modified. Now, the null=# value is assigned to image data cells outside any region/bin/filter in that command. This operation is independent of any BLANK value associated with the image.

• Fix issues with reading wrong GTI.

• Fixes a bug when using "@" GTI filters when binning into an image.

• Cleanup various compiler warnings/issues

hdrlib

• Cleanup issue with Makefiles when running "uninstall"

pixlib

• Updates for newer compilers

region

• Updates for newer compilers

---

Environment

Off-the-Shelf (OTS) Package Versions

• The following OTS packages are included with CIAO 4.15.1 installed with ciao-install. For more information on how the OTS packages are built for use with CIAO, refer to the INSTALL_SOURCE file distributed with the software.

  

  Differences in OTS for conda installation

  The conda-installed version of CIAO 4.15.1 has different OTS versions, and it also depends on the Python version you have chosen to use.

  Executables

  • ds9 8.4.1

  Python

  • Python 3.9.9
    Frameworks on OSX have been disabled to avoid a problem with users picking up the wrong version of NumPy.
  • NumPy 1.22
  • ipython 8.0.0
  • six 1.16.0
  • pip 22.0.4
  • cython 0.29.26
  • Matplotlib 3.5.1
  • jupyter notebook 6.4.10
  • pytest 6.2.5

  Libraries

  • cfitsio 4.1.0
  • CCfits 2.6
  • wcslibs 7.3.1
  • wcssubs 3.9.5
  • readline 8.1
  • XSPEC v12.12.1c (models only)
  • fftw 3.3.9
  • gsl 2.7
  • xerces 3.2.3
  • jpeg 9d
  • tcl 8.6.10
  • tk 8.6.11
  • xpa 2.1.20

Compiler versions

• The following versions of the gcc compiler were used to compile the binary CIAO releases installed with ciao-install. The conda-installed compiler versions are different.

  System	Build System	Version
  Linux	CentOS 7	gcc 7.4
  macOSX	Catalina	llvm 11.0.3

• Several CIAO applications have been updated to support the GNU++14 standard.

---

Documentation

ahelp

• The generic "tools" ahelp context has been replaced with a more specific qualifier. This helps provide more relevant grouping on the ahelp context list page.

---