CIAO 4.16 Release Notes

CIAO 4.16 is distributed for the following platforms:

• Linux 64 bit
• Apple macOS 11 (Big Sur) through macOS 14 (Sonoma), Intel/x86 and ARM/M1/M2/M3

CIAO is available using the conda package manager; conda builds are available for Linux and Mac for Python 3.11, 3.10, and 3.9. More information can be found in the conda section of the Installation notes below. The ciao-install script can be used to automate the conda installation process.

There is no support for 32 bit operating systems, older Linux (CentOS 7 era), older macOS platforms (macOSX Catalina and earlier), or Windows (including using WSL). More details can be found on the Platform Support page.

• Notable changes and improvements in CIAO 4.16
• How CALDB 4.11.5 Affects Your Analysis
• How CALDB 4.11.3 Affects Your Analysis
• How CALDB 4.11.2 Affects Your Analysis
• How CALDB 4.11.1 Affects Your Analysis
• How CALDB 4.11.0 Affects Your Analysis
• Installation and Setup
• Tools
• Parameter Files
• Sherpa
• Graphical User Interfaces
• Analysis Scripts
• Python Modules
• Libraries
• Environment
• Documentation

Notable changes and improvements in CIAO 4.16

How CALDB 4.11.5 Affects Your Analysis

HRC Data

• HRC-S Time-Dependent QE and QEU Files

  Regular calibration observations have shown that the HRC-S QE is declining at longer wavelengths (energies <200 eV, >60 Å), and decaying with the 2012-03-29, 2021-05-14, and 2024-09-20 high-voltage (HV) adjustments onboard. This spectral variation in the QE loss is accounted for using a series of time-displaced QE files, with the latest high-voltage adjustment introduced in the N0017 set in this CALDB. The last of the four files corresponds to the most recent HV change, and raises the QE by 8%, across all chips and at all wavelengths.

  The HRC-S time-varying quantum efficiency uniformity files have been replaced with a new set, calculated based on unfiltered calibration observations. This latest set (N0020) extends the prior N0019 set, adding two extra QEU files. The first, effective 2024 July 24, is derived from the most recent HZ 43 (ObsID 29452), from August 2024, with the previous HV settings. The last file, effective 2024 September 20, is actually derived from the new HV portion of ObsID 28427, taken in June 2024.

  These CIAO tools and scripts automatically apply the HRC-S QE and QEU files when creating response files:

  • mkgarf
  • fullgarf
  • mktgresp
  • mkinstmap
  • fluximage
  • merge_obs

How CALDB 4.11.3 Affects Your Analysis

HRC Data

• HRC-S Time-Dependent QEU Files

  The HRC-S time-varying quantum efficiency uniformity files have been replaced with a new set, calculated based on unfiltered calibration observations. This latest set (N0019) extends the prior N0018 set, adding a new QEU file based on the most recent HZ 43 observation, effective since 2024 March 4. The wavelength-independent drop in QE since the previous November 2023 observation is 2-3.5%, depending on dispersed orders. The wavelength-dependent corrections required on S1 and S3 continue to follow long-observed trends. The data used to derive the previous two wavelength-dependent models (2022-09-09 and 2023-09-08) have considerably better statistics than was previously the case.

  These CIAO tools and scripts automatically apply the HRC-S QE and QEU files when creating response files:

  • mkgarf
  • fullgarf
  • mktgresp
  • mkinstmap
  • fluximage
  • merge_obs

How CALDB 4.11.2 Affects Your Analysis

HRC Data

• HRC-I Time-Dependent QE Files

  The HRC-I quantum efficiency QE has been decaying for some time, particularly since the year 2011, when a 5% to 10% per year reduction had been measured. To address this issue, the HRC-I Calibration Team had introduced a time-dependence into the QE files in 2018 which are now extended in time to the latest epoch of observations. Included in this update is a new file coinciding to the recent HRC-I high-voltage adjustment affecting observations after 2023 September 27. The QE update affects the results from mkinstmap and fluximage and is applicable to all previous epochs of HRC-I observations.

How CALDB 4.11.1 Affects Your Analysis

ACIS Data

• Warm Temperature ACIS-I CTI Correction Files

  Improved time- and temperature-dependent correction to the ACIS charge transfer inefficiency (CTI) correction files (N0012) for the ACIS-S2 and ACIS-S3 chips, joining the four ACIS-I chips with corrections at warm focal-plane temperatures up to -107 C introduced in N0011. No changes are made for data taken with focal-plane temperatures below -117 C, or prior to 2005 on ACIS-I. The CTI corrections for ACIS-S0, -S1, -S4, and -S5 remain the same as the prior N0009 version of the CTI corrections.

  Users working with observations using any of the ACIS-S2, ACIS-S3, or any of the chips in the ACIS-I array made after 2005 or with focal-plane temperatures greater than -117 C may wish to run chandra_repro and reprocess the data to improve the CTI gain shift calibration.

HRC Data

• HRC-S Time-Dependent QEU Files

  The HRC-S time-varying quantum efficiency uniformity files have been replaced with a new set, calculated based on unfiltered calibration observations. This latest set (N0018) shows decline in the net count rate of 1-2% between the latest set of HZ43 calibration observations, effective since 2023 September 8. The set of QEU files are unevenly distributed in effective dates and affects every HRC-S observation since the start of the mission.

  These CIAO tools and scripts automatically apply the HRC-S QE and QEU files when creating response files:

  • mkgarf
  • fullgarf
  • mktgresp
  • mkinstmap
  • fluximage
  • merge_obs

How CALDB 4.11.0 Affects Your Analysis

ACIS Data

• Warm Temperature ACIS-I CTI Correction Files

  Improved time- and temperature-dependent correction to the ACIS charge transfer inefficiency (CTI) correction file (N0011) for the four ACIS-I chips, in particular, introduction of corrections for for warm focal-plane temperatures up to -107 C. No changes are made for data taken with focal-plane temperatures below -117 C, or prior to 2005 on ACIS-I. The CTI corrections for all six chips on ACIS-S remain the same as the prior N0009 version of the CTI corrections.

  Users working with observations using any of the chips in the ACIS-I array made after 2005 or with focal-plane temperatures greater than -117 C may wish to run chandra_repro and reprocess the data to improve the CTI gain shift calibration.

HRC Data

• HRC-I Time-Dependent QE Files

  The HRC-I quantum efficiency (QE) has been decaying for some time, particularly since the year 2011, when a 5% to 10% per year reduction had been measured. To address this issue, the HRC-I Calibration Team had introduced a time-dependence into the QE files in 2018 which are now extended in time to the latest epoch of observations, effective 2022 September 4.

  This N0014 version of the QE maps now account for QE loss at higher energies, prior versions were limited to wavelengths >60 Å, but is now extended over the range of 0 Å < λ < 135 Å. The QE update affects the results from mkinstmap and fluximage and is applicable to all previous epochs of HRC-I observations.

Installation and Setup

Users should be aware of these installation items before installing CIAO 4.16. Additional problems which are seen less frequently are listed on the Installation & Smoke Tests bug page.

Note

The steps needed to install CIAO have changed in CIAO 4.16 for both installation methods: conda and ciao-install, as discussed below.

Supported Platforms

• CIAO 4.16 is supported on:

  • Linux 64 bit
  • Apple macOS 11 (Big Sur) through macOS 14 (Sonoma), Intel/x86 and ARM/M1/M2/M3

Installing with the conda package manager

$ conda create -n ciao-4.16 -c https://cxc.cfa.harvard.edu/conda/ciao -c conda-forge \
ciao pyciao sherpa ds9 marx ciao-contrib caldb_main python=3.11

Install using the ciao-install Installation Script

$ bash ciao-install

Source builds

$ bash ciao-install --add ciao-src --add sherpa-src

$ ciao
$ cd $ASCDS_INSTALL
$ bash ./ciao_build.sh --prefix `pwd`

Smoke Tests

$ ciao
$ cd $ASCDS_INSTALL
$ bash test/smoke/bin/run_smoke_tests.sh

• Users of previous versions of CIAO will note that the order of the tests is different; in the event of a test failure be sure to include the name of the test, not just the test number when contacting helpdesk.

• The obsvis smoke tests have been simplified to reduce the amount of windows that are created (and usually taking focus).

Remove old parameter files

unix% rm ~/cxcds_param4/*

Tools

acis_process_events

• Added support for new temperature dependent CTI file format.

• Added a new keyword, CHKVFPHA, to indicate whether the check_vf_pha algorithm has been applied.

acis_build_badpix

• Fix warning about finding multiple CALDB files.

acis_streak_map

• Fix for compatibility issue with Numpy 1.25 and newer.

asphist

• Corrected the units from "degrees" to "deg" as per the FITS standard.

asp_offaxis_corr

• Fix for problem when dithering across the RA=360|0 boundary.

celldetect

• Some error handling cleanup.

• Corrected the units from "degrees" to "deg" as per the FITS standard.

dmarfadd

• Internal: add missing function prototypes

• Updates to support newer versions of C-Lang compiler

dmcontour

• Some error handling cleanup.

dmcoords

• Code cleanup to remove some compiler warnings.

dmcopy

• Fixed an issue when copying a FITS image to ASCII (TEXT/DTF) format that messed up the image subspace.

dmextract

• When using opt=generic or opt=generic2 with images, dmextract now will copy the world coordinate system from the input image to the output table's X and Y columns. The real "RA" and "Dec" columns are now removed as the values can now be accessed as WCS virtual columns.

dmfilth

• Some error handling cleanup.

• Corrected the units from "degrees" to "deg" as per the FITS standard.

• Images that do not provide a full world coordinate system can now be used. The only requirement is that the regions used must be in physical coordinates rather than celestial coordinates.

dmimgfilt

• Internal: add missing function prototypes

dmimghull

• Internal: add missing function prototypes

dmimglasso

• Some error handling cleanup.

dmimgpick

• The WCS on the input image is now copied to the output table. The X,Y output columns are now a vector column rather than individual scalar columns.

dmlist

• Fixed issue when listing contents of a file which contains vector columns of either strings or booleans data type.

dmmaskbin

• Internal: add missing function prototypes

dmnautilus

• Internal: add missing function prototypes

dmregrid

• Some error handling cleanup

evalpos

• Updated to support modified output from dmimgpick tool.

get_src_region

• The WCS on the input image is now copied to the output table.

• Some error handling cleanup.

mkgrmf

• A bug has been fixed which caused the enclosed energy fraction, due to the spectral extraction cross-dispersion width, to be incorrect for orders than +/-1. The magnitude depends on wavlength, but was at maximum 3% in an order's total effective area. (This factor occurs in a grating RMF normalization.)

reproject_image_grid

• Corrected the units from "degrees" to "deg" as per the FITS standard.

rmfimg

• Added support for RMF files where the MATRIX and EBOUNDS extensions are in arbitrary order.

sso_freeze

• Corrects the CRVAL and CRPIX values shown in the verbose output.

tgextract

• A bug has been fixed in the spectral extraction regions written to the output PHA files for polygon regions. These had first order grids instead of wavelengths scaled by diffraction order. These regions are used by mkgrmf to compute the enclosed energy fraction in the cross-dispersion direction as a function of wavelength. The factor becomes a normalization term in the grating RMF. This error only affected non-standard products. Default grids for HETG are boxes, which have correct coordinates. For LETG/HRC-S, by default, only a first order region is produced.

vtpdetect

• Some error handling cleanup

wcs_match

• If the last matching pair was a poor match, the code mistakenly deleted that pair and the prior, valid matched pair. The logic has been updated to prevent the valid pair from being deleted.

• Fixed a bug when using manual matching and duplicate values were input.

• In manual matching mode, users can now specify values in an external file using the @file_name syntax as well as supporting ranges in the form a:b.

wrecon

• Some error handling cleanup.

• Corrected the units from "degrees" to "deg" as per the FITS standard.

wtransform

• Some error handling cleanup.

• Corrected the units from "degrees" to "deg" as per the FITS standard.

Parameter Files

Sherpa

changes

• The full list Sherpa updates is given in the 4.16.0 and 4.15.1 release notes for the two standalone releases that are incorporated into CIAO 4.16. The major updates were made to data filtering and grouping, addition of a new plotting backend and bug fixes. The release highlights include:

  XSPEC

  • Updated to version 12.13.1e of the XSPEC model library.

  • Fix an issue with the redshift parameter for XSPEC table models.

  • Allow XSPEC table models which include the ESCALE parameter.

  • Provide experimental support for writing out XSPEC table models.

  Filtering and grouping

  The grouping commands like group_counts and group_snr now default to only grouping within the noticed range of channels, which is a change in behaviour. The filter status is now reported for calls to the group commands and set_analysis, to follow the behavior of notice and ignore which were changed to report the filter status in CIAO 4.15.

  Models

  Added support for use of arbitrary python functions when linking model parameters.

  Data access

  • The save_arf and save_rmf calls have been added.

  • A number of changes have been made to improve the output of the save_all command.

  • Routines have been added to convert an RMF to an image to help visualize the instrument response (they have a similar capability to the rmfimg tool).

  Random number generation

  It is now easier to ensure that Sherpa is repeatable, taking advantage of the NumPy Random Number Generator along with the new set_rng and get_rng calls.

  Plotting improvements

  New plotting backend: Users can now choose between matplotlib and bokeh (experimental, users need to install the package)

  A number of plot issues related to PHA data have been addressed.

  Documentation

  Improved documentation across the system.

Graphical User Interfaces

Obsvis

Note

Obsvis is no longer available as a separate, stand-alone download. Users need to install CIAO to use obsvis.

• Update default ACIS-I/S y/z offset values to 0.0 to match the obsvis.cal values used since 2015.

• Updated obsvis to fix an issue with FoV roll being reset when the image frame is binned in ds9

• Updated code to preserve target string (ie. keep spaces) so that the parser to properly handle target names containing multiple words (ie. 'ar lac' or 'eta carina').

• Updated to add "Configuration File Version" option to Help pulldown menu. Selecting the option invokes a message dialog indicating the loaded .obsvis,.cal file and latest detected version available for download.

Analysis Scripts

dax

• The model editor now checks that any values that have been modified have been set (by pressing the Return or Enter key) before allowing the user to proceed with certain actions such as Fit or Plot.
• The image tasks will now handle images that have been blocked by DS9.
• Re-ordered and renamed a number of tasks:

  Original name	New name
  PI	Spectrum (PI)
  PHA	Spectrum (PHA)
  TIME	Ligthcurve (Time)
  EXPNO	Lightcurve (Exposure Number)

energy_hue_map

• A new technique to create true color images from a continuous energy spectrum rather than traditional tri-color images created using 3 energy bands. The inputs include a map estimating the median (or mean) energy in each pixel (for example created using the statmap tool), and an image of the counts or flux.

hexgrid

• Creates a regular grid of hexagonal shaped map regions. Hexagons can best approximate circles as they are the highest order regular polygon that can uniformly tile a plane.

vtbin

• Creates a map file based on the Voronoi Tessellation of points in the input image. By default it uses the location of the local maxima or users can supply their own locations (via the sitefile).

centroid_map

• This script runs vtbin and then computes the centroid of the pixel values in each map region. It then repeats the Voronoi Tessellation using the centroid values N-many times, where N is the numiter parameter.

pathfinder

• This script generates map regions based on the local gradient. It follows the steepest ascent to the local maxima. All pixels that reach the same local maxim are grouped together. Better for point sources and other "clumpy" emission.

mkregmap

• Turn a stack of regions into a map file.

map2reg

• A utility tool to take a map file and converts it into a region file.

merge_too_small

• Some of these algorithms can create small regions needed to "fill in" the space between other more statistically significant regions. The merge_too_small script will re-assign small groups to the largest neighboring group; where "small" is either area below threshold or counts below threshold.

apply_fov_limits

• The script now defaults to creating images with the i4 datatype (4-byte integer) rather than i2 (2-byte integer) it used to use, as the old behavior could lead to data loss. This does increase the memory needed to store the images, so the datatype parameter can be used to change back to the old behaviour.

blanksky

• blanksky has been updated to make use of the CHKVFPHA keyword, introduced to acis_process_events in CIAO 4.16, to determine if "status=0" event filtering should be applied to ACIS VFaint observations, but will fall back on the old technique of parsing the input event file header to make the determination if the new keyword is not available. The blanksky_sample script will make use of the reference 'infile' keywords for the optional combined infile/sampled background products instead of the 'Merged'-valued keywords as a result of using dmmerge.

check_ciao_version

• The script has been updated to handle the updated ciao-install script and provide initial support for using mamba/micromamba to install CIAO.

convert_xspec_script

• Support for the ENERGIES command from XSPEC has been added.

convert_xspec_user_model

• The script has been updated to match changes to Sherpa in CIAO 4.16.

correct_periscope_drift

• The script has been updated to support changes to Sherpa plotting in this release.

reproject_obs

• The script has been updated so that file validation checks at its initialization will not have a dependency on mask files, which are otherwise not used.

specextract

• The script has been updated so that event files generated by MARX are supported and better handling of the ancillary file parameters when set to "none" values. In particular when the bad pixel file parameter was set to "none", occasionally it would revert to using the default bad pixel file from the CalDB, which is now fixed.

srcflux

• Added new option to create "optimized" source and background regions based on the PSF.
• The marx_root parameter is also added to specify the directory where marx is installed.
• In addition, there is a numerical precision bug fix when computing the combined PSF fraction when merging datasets.

crates_contrib.masked_image_crate

• This module introduces the MaskedIMAGECrate class. It is a subclass of the standard crates IMAGECrate class but extends it to include a valid method to check if the pixel location is a valid pixel (inside subspace and not NULL/NaN/Inf). It only supports 2D images.

ciao_contrib.runtool

• The module has been updated to reflect parameter changes and the new scripts added in this release.

sherpa_contrib.profiles

• The routines have been updated to support changes to Sherpa plotting in this release.

Python Modules

pycrates

• Internal documentation tweaks.

Libraries

datamodel

• Fix memory leaks with variable length arrays.

• Fix for some non-Chandra/CIAO files with bogus DSTYP keywords. Specifically some XMM files filtered with non-Chandra tools had DSTYP keywords exceeding 80 characters which could cause tools to crash.

• Updated to behave more robustly when a file contains references to subspace blocks: GTIs, MASKs, and REGIONs; but those blocks do not exist. This may happen when using non-CIAO tools to filter or save files.

• Updated to be more robust when subspace keywords, DSVALn, are blank/invalid.

• Fixed an issue affecting REGION blocks with non-standard headers (missing HDU keywords).

cxcparam

• When prompting for a parameter value, if the min or max value was redirected to another parameter, the final redirected value will now be shown in the parameter prompt rather than the redirection.

• Users can now get the parameters data type using the .p_type modifier. For example:

  $ pget dmlist infile.p_type
  f
  

  which shows the type a "f", ie. a file name.

  Users may not set or change the data type of a parameter.

• Fixed a problem accessing parameter files if the parameter file directory contained the sub-string ".par". This was most often seen during the CIAO smoke tests when the username of the user started with "par"

Environment

Off-the-Shelf (OTS) Package Versions

• The following OTS packages are included with CIAO 4.16 installed with ciao-install.

  

  Differences in OTS for conda installation

  The conda-installed version of CIAO 4.16 has different OTS versions, and it also depends on the Python version you have chosen to use.

  Executables

  • ds9 8.5

  Libraries

  • cfitsio 4.3.1
  • xpa 2.1.20
  • XSPEC 12.13.1e (models only)

  Python (major packages)

  • Python 3.11.6
  • NumPy 1.26
  • Matplotlib 3.8.1
  • jupyterlab (notebook) 4.0.8

Compiler versions

• The following versions of the gcc compiler were used to compile the binary CIAO releases installed with ciao-install. The conda-installed compiler versions are different.

  System	Build System	Version
  Linux	RedHat Enterprise Linux 8	gcc 7.4
  macOSX	Big Sur	llvm 11.0.3

• Several CIAO applications have been updated to support the GNU++14 standard.

Documentation