Last modified: 1 November 2022

URL: https://cxc.cfa.harvard.edu/ciao/releasenotes/ciao_2.2_release.html

CIAO 2.2 Release Notes


Contents:


Tools

acis_detect_afterglow 
  . Bug fix for not checking ccdid, could have tagged a few extra
    events as 'glowing' that were not.
    
acis_fef_lookup ** NEW ** 
  . Back in the baseline CIAO distribution.
  . Determines the FEF file and filter needed for the source position.
    
acis_process_events 
  . New algorithm to compute sky positions for CC mode events. This
    works to remove the dither in the CHIP Y direction.
  . should stop warning about finding two calibration files for the
    grade file.
  . New algorithm to check for pixel values in outer 5x5 island (very
    faint mode only) greater than split threshold. Will set status bit 23.
    AKA "very faint" checking and/or "part55".
  . Includes 2 new parameters for very faint checking: check_vf_pha to
    determine whether the check is done or not and "trail" which is an
    ad-hoc adjustment factor for CTI effects.
  . Fix to deal with bad input obs.par file
  . Added new parameter to control PHA randomization
  . Fixed bug in TE Graded mode where badpixel could have been missed.
    (not an issue for SDP, only user supplied badpixels isolted to a
    single pixel rather than 3x3 island)
  . worked history so that duplicate history records aren't written
    out.
  . Bug fix to not override TIMPIXR if already exists when using an
    obs.par file
    
acisreadcorr ** NEW ** 
  . Tool to identify the 'streak' of events in the chip read-out
    direction that coincides with a bright source. These are events
    that are detected during the frame shift/read-out.
  . Sets times for those events so some rudimentary time analysis can
    be done.
    
aconvolve 
  . fixed incomplete error message
    
acrosscorr 
  . no significant changes
    
add_grating_orders ** NEW ** 
  . Previously part of the optional download, this script is now
    packaged as part of the CIAO distribution.
  . Adds together spectra from different grating orders.
    
add_grating_spectra ** NEW ** 
  . Previously part of the optional download, this script is now
    packaged as part of the CIAO distribution.
  . Adds grating spectra (eg MEG and HEG)
    
apowerspectrum 
  . no significant changes
    
asp_apply_sim ** deprecated ** 
  . No longer necessary with the new 'asphist'
  . May not be shipped in later CIAO releases.
    
asp_calc_offsets ** deprecated ** 
  . No longer necessary with the new 'asphist'
  . May not be shipped in later CIAO releases.
    
asphist 
  . New asphist that works directly from the aspect solution files (asol).
  . Will continue to work with the output of asp_apply_sim (backwards
    compatible)
  . fixed clobbering problems for corrupted outfile
    
axbary 
  . This was a fundamental but minor bug that led to errors of the
    order of less than 1 us in the barycenter correction.
  . There was a error condition related with keeping track of the
    position in the orbit ephemeris file that was not properly taken
    care of.
  . Apply TIMEZERO: a bug that affected XTE corrections
  . Output writable
    
celldetect 
  . Added source cell locations and double detection flags in the
    celldetect output. New output columns CELL_X, CELL_Y, DOUBLE, and
    DOUBLE_ID.
  . Sources which are on blocking boundaries and are detected in two
    successive blocking passes are flagged in DOUBLE and DOUBLE_ID
  . checks for successful creation of the logfile
  . Fixed src centers for even size cells
  . Fixed problem with clobbering files
  . Enhanced duplicate suppression based on the .5 pixel centers for
    even sized cells. Now it can also reject duplicate sources that
    are slightly further apart. User's may see their source lists
    shrink slightly.
  . Upgraded default parameter file for the latest psfsize table
  . Fixed problem with src ellipse axis lengths when data are blocked
  . The region output files have the revised ascii region format
    acceptable to both ds9 and the DM.
    
csmooth 
  . Fixes clobber problem. Also increases length of filename arrays.
  . fixed bug for input scale map when only 1 pixel used a certain
    scale.
  . fixed incomplete history
  . Fixed waiting until end to test whether to clobber outputs
    
destreak 
  . Updated so clobber messages doesn't appear when verbose=0
    
dmappend 
  . Fixed dmappend not returning appropriate exit value
    
dmarfadd 
  . no significant changes
    
dmcontour 
  . Fix some long standing issues when the contours included regions
    that should really be excluded. Regions will now properly be
    included/excluded. There where problems in the past when regions
    where inside each other, the logic wasn't correct.
  . Fixed problem when no contours are found
    
dmcoords 
  . fixed bad behavior when bad inputs given for celfmt
    
dmcopy 
  . Updated so clobber messages doesn't appear when verbose=0
    
dmextract 
  . The POISSERR keyword is properly set in the counts case to reflect
    when the errors are Poisson or not.
  . proper error case when the RA/Dec cannot be determined.
  . Updates to deal with doubles and floats in images
  . Updates to catch DBL_EPSILON problems on linux
  . Now allow the use of Poisson or Gaussian for the pha/histogram
    case.
  . Added a warning if the user tries to use a variance file
  . Reset the defaults for the error values
  . The exposure maps were not correctly accounted for. Dmextract was
    weighting the images by the binning factor.
  . There was a problem in the way values were being indexed in image
    source extraction.
  . Fixes problem w/ normalization of exposure maps
  . Added reading units from exposure maps to normalize counts
    correctly
  . Fixed PHA_MIN/PHA_MAX bug
  . Support for BKG subtraction in PHA/histogram space
  . We are going to be multiplying the (cm^2.s) maps by the livetime
    in the file. If the user inputs a exp file, but not a bkg_exp file
    or vice-versa, We will ignore the exp values, and calculate
    without them and print out an error message.
    
dmfilth ** NEW ** 
  . "FIL in The Hole"; used to create background maps by excluding
    source regions from the image.
  . Various method s are presented; some are good for 'counts' images
    (eg Poisson), some are good for smoothed images, eg bilinear
    interpolation.
    
dmgroup 
  . Fixes case where writes out grouping keyword & column
    
dmgti 
  . fixed bug when using string keywords (truncating the string by 1
    character).
    
dmhedit 
  . Added checks against bad input list files
  . Made operation, key, etc automatic so user will be queried (if not
    filelist is present, eg filelist="")
  . Fixed problem when given a file list with nothing in it
    
dmimg2jpg 
  . Cleaned up the color lookup tables. There were extra entries at
    the end of the files that were not part of the original color
    schemes
  . changed region support to use region lib parsing for more general
    use.
  . fixed minor annoyance in clobber output
    
dmimgcalc 
  . Added support for infile2="none" to allow simple scalar
    multiplication of images
  . Get rid of aim point tests for WCS matching.
    
dmimghist 
  . Added column for normalized counts
  . pixels above and below boundary are not counted in end bins. This is
    controlled thru new "strict" parameter.
    
dmimgthresh ** NEW ** 
  . Sets all values in an image above/below a certain limit to a new
    value.
    
dmjoin ** NEW ** 
  . Interpolates columns from one table onto another by joining the
    two files on a common column.
    
dmkeypar 
  . Fixed problem on Alpha where string-length was truncated.
    
dmlist 
  . Will report "Null" values
  . Increase precision of output for doubles/floats.
  . Fixed problem dmlisting 1x1 array
    
dmmakepar 
  . no significant changes
    
dmmakereg 
  . fixed clobbering problem
  . better error handling when bad input is supplied
    
dmmerge 
  . fixed clobbering problems for corrupted outfile
    
dmpaste 
  . Fix problems where error messages were causing segv's.
  . Added warnings when columns being pasted from 'pastefile' exist in
    'infile'; 'pastefile' columns are skipped.
  . Fixed broken error message when no. rows in files not the same.
    
dmreadpar 
  . no significant changes
    
dmrega2fits ** REMOVED **
  . Removed from CIAO. Use dmmakereg to create ASC-FITS-Region style
    region files. Use tg_create_mask to create grating region files.
    
dmregrid 
  . no significant changes
    
dmsort 
  . changes, but some are untested. Should continue to only use in
    defined CIAO threads.
    
dmstat 
  . fixed the incorrect sum problem when centroid=yes.
  . Added checking for NaN's and TNULL's in columns
  . Added support for NaNs in floating point images. and NULL values in
    integer images.
  . Changed output format to use "%g" rather than "%f" to get a fixed
    precision.
  . Removed code that did coord transforms automatically. Users can
    now use the datamodel to specify columns with WCS, eg dmstat
    infile.fits[cols ra,dec]
  . Will now report centroid positions in physical coords in addition to
    logical coords.
  . Will now report the location (in physical coords) of the min
  . only the first instance of the min/max are reported.
  . Reports location of min/max regardless of centroid being computed.
  . WARNING: the output format has changed slightly, which may break 
    scripts that use dmstat

dmtcalc 
  . Fixed divide by zero SIGFPE.
  . Fixed scientific notation so that you can enter "3E12" rather than
    needing to enter "3.E12"
  . Fixed problem with column smoothing and keeping original value
    unless explicitly over-written, so foo=goo:3 will have the
    original values of "goo" in the outfile, but goo=goo:3 will have
    the smoothed values.
  . Now propagates the WCS attached to various columns to the output.
  . Should also now be setting TLMIN/TLMAX's and also propagating the
    DSS (GTIs) properly.
  . fixed bug when using string keywords (truncating the string by 1
    character).
    
dmtype2split 
  . no significant changes
    
dmwritefef ** REMOVED **
  . No longer packaged with CIAO.
    
get_src_region 
  . no significant changes
    
hrc_process_events 
  . fixed bug that was allowing PI values above 255
  . Fix to deal with bad input obs.par file
  . worked history so that duplicate history records aren't written
    out.
    
lightcurve 
  . no significant changes
    
mkarf 
  . Ardlib options permitted on the mirror parameter (e.g., AREA=1).
  . The name of the current working directory is written to the
    history keyword to give meaning to relative path names. 

mkexpmap 
  . The name of the current working directory is written to the
    history keyword to give meaning to relative path names.

mkgarf
   . CALDB support for the OSIP file.
   . IRMXFIL header renamed to OSIPFILE
   . The engrid parameter now permits "compute" to be used to
     indicate that a default grid is to be used that is consistent
     with the resolution of the instrument.
   . The name of the current working directory is written to the
     history keyword to give meaning to relative path names.

mkgrmf  ** NEW **
  . This program computes a grating RMF.

mkinstmap 
  . BUNIT keyword added to the output file.
  . The name of the current working directory is written to the
    history keyword to give meaning to relative path names.
mkpsf 
  . If user requests a position outside the current psf library file, the
    tool will return the closest match (and issue a warning). Previously,
    this was a fatal error.
  . Correct bug which sometimes identified two interpolation data
    columns when no interpolation was needed.
  . Allow alternate filepaths with autonaming for psflib data output
    files, check for pre-existing files and add clobber param for
    psflib files.
  . ability to extract individual psf images in addition to interpolated
    one.
  . Added additional output header keywords so that detect tools can
    run on psf's that are output
  . Added naming options for output psflib data files,
  . fixed bin/rotate bug for interpolated image
  . add ROLL_NOM keyword to output files
  . remove user option to specify roll angle.
  . fix for not write out RA/DEC in header when in DET coords
  . Added CALDB automatic lookup to pick the "best" PSF file ONLY when an
    input event file is provided.
  . Fixed bug which caused an error in the output interpolated PSF
    image when the PSF SIZE parameter was chosen to exactly match the
    PSF image size after rebinning.
  . Added bug fix for incorrect energy hdr keyword in output psflib
    data files when selected energy was out of range.
    
mkrmf 
  . Includes new "weights" parameter. When set, a weighted RMF based on
    the weights of the FEF regions will be created.
  . When "weightfile" is set, any filter appearing in 'infile'
    expression will be ignored
  . Allows user to input abbreviated "axis1=energy" when weights file
    is used since it uses energy grid from weights file
  . Still backwards compatible with the existing single FEF inputs.
  . Fix for SEGV when logfile name was missing or ""
  . Clobber checked at the beginning of run
  . Added CALDB lookup when using a weight file (ONLY when using a weight
    file).
  . Speed enhancements
  . add message to stdout to inform multi-region precessing status
  . add CHECKSUM/DATASUM/CREATOR keywords
  . remove spatial filter from FEFFILE keyword
  . quite unnecessary warning message
  . fix for incomplete error msg when bad FEF region is requested
    
mkwarf ** NEW ** 
  . new tool to create a weighted ARF file based on an input WMAP.
  . second (necessary) output is a set of weights needed by mkrmf to
    create the corresponding weighted RMF.
  . Users must supply "[wmap]" on input file name if using a wmap from
    dmextract.
    
mtl_build_gti 
  . fixed bug when using string keywords (truncating the string by 1
    character).
    
paccess 
  . no significant changes
    
pdump 
  . no significant changes
    
pget 
  . no significant changes
    
pline 
  . no significant changes
    
plist 
  . no significant changes
    
pquery 
  . no significant changes
    
psextract ** NEW ** 
  . Was originally on the optional download list, this is now part of
    the basic CIAO installation.
  . Extracts the spectra and computes the arf and rmf for the given
    source region.
    
pset 
  . Now internally sets tool name. The upshot being that you can get
    help on the tool you are psetting parameter for.
    
psf_project_ray ** NEW ** 
  . Post CIAO2.2, the CXC will allow users to generate custom ray
    files thru the CXC's mirror model, "SAO-SAC". This will be a web
    interface.
  . psf_project_ray will take these rays and project them onto the
    detector configuration (including non-standard SIM locations) to
    produce a PSF at the exact desired location.
    
punlearn 
  . no significant changes
    
reproject_events 
  . New functionality to set a random time w/in the GTI of the match file.
    This allows the "background" datasets to be used w/in CIAO
  . New parameter to control time/position randomization.
  . various error checking on bad inputs
    
rmfimg ** NEW ** 
  . Creates a 2D image from an input RMF file. Can be displayed in
    imager and includes WCS.
  . Will optionally multiply by the ARF.
    
sso_freeze 
  . added check to verify at least 2 sso ephem records exist prior to
    first event.
    
stk_build 
  . no significant changes
    
stk_count 
  . no significant changes
    
stk_read_num 
  . no significant changes
    
tg_create_mask 
  . Fixes problem with HETG mask in CC mode.
  . added hetg and letg width factor parameters. width_factor defaults to
    redirect of hetg.
  . Removed HESF parameters
  . Added WPSF keyword to properly catch what CALDB resolved to at
    run-time
  . made output more consistent with CXC region format so it can be
    used as a dm filter.
  . Added history information about infile, outfile, and calibration
    files used.
    
tg_resolve_events 
  . Fix bug where 0th order photons were not getting assigned dispersion
    coordinates.
  . Fixed problem with CC mode data not being de-dithered if the aspect
    solution files are used rather than the aspect offsets.
  . Fixed ACIS CC+LETG case
  . Changed ACISDE keyword to OSIPFILE to properly catch which CALDB
    file was used.
  . Now gives out warnings only if GRATING keyword is HETG but the
    user enters LETG to the grating_obs parameter or vice versa
  . Removed HSEF code to clean things up.
  . made changes so it can now handle the cases where the center of
    the zero-order source falls in a chip gap
  . changed the chip_minrange & chip_maxrange variable from short to
    double. Also changed the logic to check if an event's coordinates
    lie within the valid range. The changes should make the tool
    resolve events that are very close to the edge of the detector.
  . fixed minor un-noticed bug that could have effected users iff they
    had tried to force the LETG grating type.
  . The tool can now sort order (for acis) by using 2 new parameter values
    (osort_lo, osort_hi) from the par file. This order-sorting method will
    be used only if the osipfile (used to be the rmfile parameter)
    parameter is set to "NONE".
  . Changes in parameter interface: 1)rename the rmfile parameter to
    osipfile 2)add osort_lo and osort_hi parameter
    
tgdetect 
  . Fixed problem when no. counts per pixel greater than 32K
    
tgextract 
  . made tg_lam,tg_d into a vector column to support region filtering.
  . Fixed problem with entering "+" before number
  . added MTYPE/MFORM keywords to make output regions be valid FITS
    region files.
  . Added INREGION keyword to properly catch what CALDB resolved to at
    run-time
  . deleted the tg_d_units parameter from tgextract.par it can only
    handle tg_d in degrees, so it's meaningless to have tg_d_units in
    the par file
    
tgidselectsrc 
  . no significant changes
    
tgmatchsrc 
  . no significant changes
    
vtpdetect 
  . The region output files have the revised ascii region format
    acceptable to both ds9 and the DM.
    
wavdetect 
  . The region output files have the revised ascii region format
    acceptable to both ds9 and the DM.
  . fixed output file autonaming extensions.
  . Fixed problem with src ellipse axis lengths when data are blocked
    
wrecon 
  . Fixed autonaming problem
  . Fixed problem with src ellipse parameters when data are blocked
  . Fixed problem with src ellipse axis lengths when data are blocked
  . The region output files have the revised ascii region format
    acceptable to both ds9 and the DM.
    
wtransform 
  . Fixed segv when exposure map size does not match input image size


Sherpa

New (as opposed to enhanced) functionality:

  . EQWIDTH: [command] computes the line equivalent width.
  . FTEST:   [command] computes the F-test model comparison test significance.
  . MLR:     [command] computes the Maximum Likelihood Ratio model
             comparison test significance.
  . UNLINK:  [command] undoes previous linkage of a specified parameter.

  . GRIDMODEL: [model] allows one to use a numerical model in Sherpa fits;
    exposure maps may be specified via GRIDMODEL (see also EXPMAP).

  . sherpa_eval() has been added to allow execution of Sherpa commands
    in S-lang scripts.
  . One may read in systematic errors in addition to statistic errors.

Changes to I/O:

  . For PHA files, filtering and fitting can be done in either energy *or* 
    wavelength space, if an instrument is specified.
  . DATASPACE now includes HISTOGRAM (binning) option.
  . One may specify additional filters after issuing the command READ FILTER.
  . One may specify additional filters after issuing the command IGNORE BAD.
  . New functions (e.g., sherpa_update_mdl()) have been written to create MDLs
    and store them in S-Lang variables.

Changes to General Fitting:

  . Generalized background: source and background datasets no longer need to
    be of the same size.
  . Background data may be fit alone, with no source data specified.
  . Background datasets may have their own instrument models.
  . In image analysis, one may specify that the unfiltered background dataset
    is the same as the source dataset using the command BACK => DATA.
  . PSFFROMFILE: [model] is now faster for 2D convolution and has an improved
    interface for 1D convolution.
  . PILEUP model stacks may now be specified.
  . XSPEC models may be used even when analysis is performed in wavelength 
    space.

Changes to Sherpa Models:

  . New model: JDPILEUP, a pileup model.
  . New model: NBETA1D, a normalized beta model for line fitting.
  . Sherpa user models can now be written as S-Lang scripts.
  . More scripts for doing high resolution spectroscopy (GUIDE).
  . One can apply XSPEC v11.1 models.

Changes to Visualization:

  . Visualization of RESIDUALS now shows (data - model); DELCHI shows 
    (data - model)/error.
  . More background visualization options: BFIT, BRESIDUALS, etc.
  . IMAGE FIT now tiles images in ds9 by default.
  . Contour and surface plots updated to handle arbitrarily filtered data;
    also, multiple surface, contour plots can be displayed at once.
  . New OPLOT command to overplot several curves in one drawing area.
  . WRITE has been updated to provided energies or wavelengths of bins;
    and filtered or unfiltered data.
  . Filtered statistics images can now be visualized.
  . Use of new store/restore with chips; the plotted data is stored with the 
    Chips FITS file.


User Interface (UI)

ChiPS
-----

   - introduced chips state object, which signficantly improves
     ease and flexibility of chips customization

   - will read .chipsrc file at startup, if found

   - more streamlined/stable use of S-Lang

   - can now be used outside of CIAO, via import("chips") in
     any application that embeds S-Lang (like slsh, or ISIS, etc)

   - chips_eval() function introduced for executing chips commands
     within S-Lang scripts

   - store/restore: has been extended to correctly save/restore plots
     generated from S-Lang variables

   - undo/redo: has been extended to correctly save/restore plots
     generated from S-Lang variables

   - several enhancements to limits functionality

   - new --batch and --slscript startup options

   - deprecated 'rename' command

GUIs
----

   - cell-based table and image editing has been added to PRISM,
     including capability to add rows and columns

   - PRISM now maintains a recently-visited files list

   - replaced non-editable quicklook plots in PRISM with full chips
     plotting functionality

   - improved Analysis menu organization in all GUIs

   - FILTWIN layout enhanced

   - support for new RA/DEC entry formats in OBSVIS

   - OBSVIS roll/visibility plot calculations now use auxiliary datafile
     rather than hardcoded numbers

Varmm/S-Lang
------------

   - writefits(): extended ISIS fitsio functionality and encapsulated it
     an within an easy-to-use function, thus providing powerful FITS
     table and image writing capability from S-Lang in CIAO

   - will read .varmmrc file at startup, if found

   - introduced varmm state object, to improve customization of
     varmm/s-lang behavior in CIAO

   - introduced new VarmmFunc class to encapsulate S-Lang function
     as a C++ object

   - S-Lang structures returned by varmm I/O functions now use underscore
     prefix to denote fields which contain metadata

   - structure fields which reflect table column names are no longer 
     lowercased, but rather reflect the case used in the file on-disk

   - min/max/step arrays are now present in the S-Lang structures 
     generated by the image I/O routines

Miscellaneous
-------------

   - ISIS spectral analysis package now bundled with CIAO, and accessible
     within ChIPS and SHERPA commandlines via import("isis")

   - several enhancements to ahelp (command line help), including
     better performance and access to either local or web-based content

   - proposal/mission planning tools (OBSVIS and proposal planning tooklit)
     merged into CIAO distribution

   - significantly improved user documentation


Libraries

datamodel 
  . Fix for area calculations not being truncated by field size
  . Support for row number filtering: dmcopy "evt.fits[#row=100:200,...]"
    However [cols #row] will fail.
  . Support to access WCS/Coordinate columns as virtual columns dmlist
    "foo_evt.fits[cols ra,dec]"
  . Fixed problems with string length when complex regions were used
  . Fix for bug in rebinning non-square images: dmcopy "image.fits[bin
    x=4,y=4]" rebin_image.fits
  . DM enhancements to support the definition of a null value when making
    images, use "[opt null=value]"
  . Problems with many-point-polygon region filtering fixed.
  . Can suppress BTYPE/BSCAL FITS rescaling using "[opt scale=no]"
  . Can rename columns on-the-fly with "[cols new=old,...]" syntax.
    
parameter lib 
  . Fixed some error messages to indicate which parameter files was
    being accessed.
    
regions 
  . Fixed bug that didn't parse "'" correct for arcmin.
  . new "bounds()" functionality to get a bounding rectangle around
    whatever region is supplied
    
caldb 
  . Added ability to override any/all parameters from the command
    line.
    
stacklib 
  . Now allows one to escape "," in file names using "\,"
  . Blank lines (strlen()==0) in input list files (@foo.lis) are now
    ignored.


Documentation

threads (documents_threads.html)
   . a number of new threads developed
   . existing threads reviewed and updated to include CIAO 2.2 capability
   . threads now available as pdf documents

scripts (download_scripts.html)
   . new and major revisions for ciao scripts
   . most widely used scripts taken from the 'user contributed' area,
   and incorporated as a part of the standard CIAO build.  these
   scripts are "psextract", "acis_fef_lookup", "add_grating_orders",
   and "add_grating_spectra".

manuals (documents_manuals.html)
   . significant revisions to the data model manual(s) and detect guide

help system (ahelp) (documents_ahelp.html)
   . added the "-i" option to access the "i"nternet version of the
   ahelp file, at
   "http://cxc.harvard.edu/ciao/ahelp/[toolname].html".  this
   complements the existing "-w" flag, which accesses the *local* html
   version of the ahelp document.
   . cleaned up the appearance of online ahelp files and created a
   subject-ordered list
   . ahelp files now available as pdf documents

www
   . redesigned the ATODMB (../atomdb) web pages