| AHELP for CIAO 4.5 | ciao_runtool |
Context: contrib |
Synopsis
Run CIAO tools from Python as if they were functions (CIAO contributed package).
Syntax
from ciao_contrib.runtool import *
Run the dmcopy tool:
dmcopy("evt2.fits[bin sky=::8]", "img.fits")
Run dmstat:
dmstat.punlearn()
dmstat.median = True
dmstat("src.dat[cols z,fx]", verbose=0)
print("The medians are {0}".format(dmstat.out_median))
Description
The ciao_contrib.runtool module allows CIAO tools to be run as if they were Python functions, and supports a pset-like mode where tool parameters can be set before the tool is run, named parameters when calling the tool, and easy access to the parameter values after the tool has completed. Parameter values are given using native Python types - e.g. True and False for boolean parameters.
The ciao_contrib.runtool module is provided as part of the CIAO contributed scripts package.
Loading the routines
The module can be loaded into ChIPS, Sherpa or a Python script by saying:
from ciao_contrib.runtool import *
Note that it is not provided by the ciao_contrib.all module, unlike the other ciao_contrib modules.
Tools and parameter files
The runtool module provides a function for each major tool in the CIAO bin and contrib/bin directories that has a parameter file. It also provides functions to set and get settings for those parameter files without an associated tool, such as ardlib, lut and colors.
Extra routines
The following routines are also provided:
- get_pfiles() - return the user or system portion of the PFILES environment variable;
- set_pfiles() - easily change the user portion of the PFILES environment variable;
- make_tool() - create a routine that can be used to call a CIAO tool;
- list_tools() - returns a list of the tools and parameter files that are covered by this module;
- new_tmpdir() - a context manager that creates a temporary directory and ensures that the directory is removed once the block has finished;
- new_pfiles_environment() - a context manager that creates a temporary directory and uses it as the user path of the PFILES environment variable for the duration of the block; on exit the PFILES variable is restored to its original value and the temporary directory is removed;
- add_tool_history() - adds a CIAO history block to one or more files to indicate that a tool has been run with a given set of parameters (this block can be read by the dmhistory tool).
Running a tool
A tool can be run by calling the routine with all the arguments it needs; required arguments are given either by positional order or as a named argument whereas optional arguments always have to be given as a named argument. As an example
chips> dmcopy("in.fits[sky=region(src.reg)][bin sky=::1]", "img.fits")
chips> dmstat(infile="img.fits", median=True, centroid=False)Parameters can also be set using a parameter-style interface of <toolname>.<parname> before calling the tool as shown below:
chips> dmcopy.outfile = "img.fits" chips> dmcopy.infile = "in.fits[sky=region(src.reg)][bin sky=::1]" chips> dmcopy() chips> dmstat.median = True chips> dmstat.centroid = False chips> dmstat(infile="img.fits")
You can mix both approaches, as shown above in the last dmstat call.
Clearing, setting, and viewing parameter settings
Each tool has a punlearn() method which will reset the values to the CIAO defaults. As shown above, parameter values can be set either when calling the tool or via the <toolname>.<parname> approach, and the current settings can be displayed using print. As an example
chips> dmjoin.punlearn()
chips> dmjoin.interpolate = "closest"
chips> print(dmjoin)
Parameters for dmjoin:
Required parameters:
infile = Input file
joinfile = Input join file
outfile = Output file
join = Join col
Optional parameters:
interpolate = closest Interpolate mode
verbose = 0 Debug Level(0-5)
clobber = False Clobber
There are also write_params() and read_params() methods that can be used to create a .par file from the current settings or read values in from a given .par file. See "help <toolname>.read_params" or "help <toolname>.write_params" for more information on these methods.
Support for unique parameter prefixes
As with the command line, parameter names can be shortened to any unique prefix. This means that dmjoin.inf can be used to read or write the infile parameter of dmjoin, but dmstat.joi will fail since it matches both the join and joinfile parameters.
This support also holds when calling a tool, so
chips> dmcopy(infile, outfile, cl=True)
calls dmcopy with the clobber parameter set to yes.
An error will be raised if the prefix is not unique; for instance
chips> dmjoin.joi AttributeError: Multiple matches for dmjoin parameter 'joi', choose from: joinfile join
A SyntaxError will occur if a Python reserved word - such as "in" or "for" - is used as the parameter name.
Stack support
There is limited support for stacks in this module: if a string or file parameter is sent an array then the elements are turned into a comma-separated string and this is used as the parameter value.
The following example creates the average of all the files that match imgn.fits, with n being from 1 to 9, to create avg.fits:
chips> import glob
chips> from ciao_contrib.runtool import dmimgcalc
chips> dmimgcalc.punlearn()
chips> infiles = glob.glob("img[1-9].fits")
chips> nf = len(infiles)
chips> inames = ["img{0}".format(i) for i in range(1,nf+1)]
chips> opstr = "imgout=({0})/{1}.0".format("+".join(inames), nf)
chips> dmimgcalc(infiles, out="avg.fits", op=opstr)The build() routine from the stk module can be used to convert a parameter value to an array of values, and the make_stackfile() routine from the ciao_contrib.stacklib module can be used to create a temporary stack file from an array.
Parameter constraints
If a parameter can only be one of a set of values then the parameter value can be set to a unique substring, just as on the command line. As an example, the interpolate parameter of dmjoin can be one of "linear", "first", "last", "closest", "furthest", "minimum", or "maximum", so the following are all valid:
chips> dmjoin.interp = "min"
chips> dmjoin("a.dat","b.dat","c.dat","x",interp="lin")An error occurs if a parameter is set to an invalid value; e.g. either using a number outside the valid range or if there are a limited set of choices, as shown below:
chips> dmjoin.verbose = 10 ValueError: dmjoin.verbose must be <= 5 but set to 10 chips> dmjoin.interp = "foo" ValueError: The parameter interpolate was set to foo when it must be one of: linear first last closest furthest minimum maximum
The return value
If there is no screen output from the tool then the tool returns None, otherwise it returns a single string containing both the STDOUT and STDERR channels. This output can either be stored in a variable or - if used from an interactive environment such as ChIPS and Sherpa - it will be automatically displayed. In the following, the dmcopy call creates no screen output, so it returns None which is not displayed by ChIPS, whereas the dmstat output is displayed
chips> dmcopy(infile, outfile)
chips> dmstat(outfile)
y
min: 10 @: 1
max: 87.3 @: 5
mean: 35.3
sigma: 28.55548984
sum: 176.5
good: 5
null: 0
chips> out = dmstat(outfile)
chips> lines = out.split("\n")
chips> print(lines[1])
min: 10 @: 1
What happens if the tool fails?
An IOError is raised if the tool returns a non-zero exit status. The error contains the screen output of the tool, as shown below where I used an invalid file:
chips> dmstat("simple.dat[cols z]")
IOError: An error occurred while running 'dmstat':
# DMSTAT (CIAO 4.5): could not open infile simple.dat[cols z].
The return value can be retrieved using the get_runtime_details() method of the tool, which is described below.
Unfortunately not all CIAO tools set the exit status correctly on error, so these will appear to have run successfully even on failure.
How long did the tool run for?
The get_runtime_details() method returns a dictionary which lists information on the last time the tool was run using this interface. This information is cleared when the punlearn() method is used. The keyword and values returned are listed below. Note that no keyword is guaranteed to be present and that the timings are not intended for precise bench-marking so should not be considered to be accurate to more than about one second.
| Keyword | Value |
|---|---|
| code | The return value for the tool (0 indicates success, otherwise there was a failure). |
| start | The start time, as returned by the time.localtime() routine. |
| end | The end time, as returned by the time.localtime() routine. |
| delta | A string representing the run time. It is useful for a simple display, but use the start and end fields for a more accurate representation. |
| args | An array of the command-line arguments used to run the tool, stored as (name,value) pairs. |
| output | The screen output for the tool (includes both STDOUT and STDERR in the same string). This is not cleaned up to remove trailing white space as is done for the return value when calling the tool. |
See the documentation for the Python time module for more information on how to use the start and end values.
Example 1
Providing all the arguments in the call
In this example we run the dmcopy tool and provide all the necessary arguments when the routine is called. The dmcopy.punlearn routine is used to ensure the settings are at their default values.
chips> dmcopy.punlearn()
chips> dmcopy("evt2.fits[ccd_id=7]", "ccd7.fits", clobber=True)We adapt this to loop through the ccd_id values 0, 1, 2, 3, and 6 and create the files "ccd<n>.fits". We also take advantage of the support for using parameter prefixes to shorten "clobber" to "cl".
chips> for ccd in [0,1,2,3,6]:
infile = "evt2.fits[ccd_id={0}]".format(ccd)
outfile = "ccd{0}.fits".format(ccd)
dmcopy(infile, outfile, cl=True)
print("Created: {0}".format(outfile))
Created: ccd0.fits
Created: ccd1.fits
Created: ccd2.fits
Created: ccd3.fits
Created: ccd6.fits
Example 2
Setting the arguments before the call
The "pset" approach can also be used to set the arguments before the call. The original example above can be re-written as:
chips> dmcopy.punlearn() chips> dmcopy.infile = "evt2.fits[ccd_id=7]" chips> dmcopy.outfile = "ccd7.fits" chips> dmcopy.cl = True chips> dmcopy()
Example 3
Mixing both approaches
Both approaches can be combined. For instance, if you wish to call a tool multiple times with some parameters fixed you may wish to use the pset approach for those parameters and call the tool with those arguments that change.
Here we set the clobber argument and then call dmcopy with the infile and outfile arguments.
chips> dmcopy.punlearn() chips> dmcopy.cl = True chips> dmcopy(infile, outfile)
Example 4
Accessing parameter values after a tool runs
Some tool set parameter values after being run. These values can be inspected and used as shown below, which includes setting the parameter value for the mkexpmap tool based on the output of get_sky_limits.
chips> get_sky_limits("img.fits", verbose=0)
chips> dmf = get_sky_limits.dmfilter
chips> print("Filter: {0}".format(get_sky_limits.dmf))
Filter: X=4047.5:4098.5:#51,Y=3920.5:3971.5:#51
chips> mkexpmap.xygrid = get_sky_limits.xygridThe whole set of parameters can also be displayed:
chips> print(get_sky_limits)
Parameters for get_sky_limits:
Required parameters:
image = img.fits Image for which you want to know the binning
Optional parameters:
precision = 1 Precision [# decimal points] for output numbers
dmfilter = X=4047.5:4098.5:#51,Y=3920.5:3971.5:#51 DM filter syntax to match image
xygrid = 4047.5:4098.5:#51,3920.5:3971.5:#52 xygrid parameter for mkexpmap to match image
verbose = 0 Debug Level (0-5)
Example 5
Multiple copies of a tool
Multiple versions of a tool can be created, each with its own set of parameters. As a simple case, if you want to run dmstat with the median parameter set to True or False you could try the following:
chips> dms1 = make_tool("dmstat")
chips> dms1.median = False
chips> dms2 = make_tool("dmstat")
chips> dms2.median = TrueThen using dms1 or dms2 will run dmstat with the median parameter set to False and True repsectively:
chips> dms1("lc.fits[cols count_rate]")
chips> dms2("lc.fits[cols count_rate]")Example 6
Changing the temporary directory used by a tool
Several tools allow you to change the location of the temporary directory used to store intermediate products. This parameter should be changed if multiple copies of the tool are being run simultaneously, to avoid clashes and potential tool failure. One way to do this is to use the new_tmpdir context manager, which creates a temporary directory and guarantees it's clean up. As an example:
with new_tmpdir() as tmpdir:
# tmpdir has been created and can be used in this block
bmap = make_tool("create_bkg_map")
bmap.tmpdir = tmpdir
...
bmap(...)
# at this point the temporary directory is guaranteeds to have been
# deleted, even if there is was an error in the block of code above
#
...
Routines with no corresponding tool
There are a number of CIAO parameter files which have no corresponding tool - for instance ardlib, geom, colors and lut. These are provided in this module for reading and setting parameters, but you can not call them. Also, as discussed below in the "PARAMETER FILES" section, setting these parameters does not change the on-disk parameter file unless you call the write_params method.
Here is an example of accessing the lut parameter file:
chips> print(lut)
Parameters for lut:
a = ${ASCDS_CALIB}/a.lut Color lookup table
aips = ${ASCDS_CALIB}/aips.lut Color lookup table
b = ${ASCDS_CALIB}/b.lut Color lookup table
bb = ${ASCDS_CALIB}/bb.lut Color lookup table
blue = ${ASCDS_CALIB}/blue.lut Color lookup table
color = ${ASCDS_CALIB}/color.lut Color lookup table
cool = ${ASCDS_CALIB}/cool.lut Color lookup table
green = ${ASCDS_CALIB}/green.lut Color lookup table
grey = ${ASCDS_CALIB}/grey.lut Color lookup table
halley = ${ASCDS_CALIB}/halley.lut Color lookup table
heaob = ${ASCDS_CALIB}/heaob.lut Color lookup table
heat = ${ASCDS_CALIB}/heat.lut Color lookup table
hsv = ${ASCDS_CALIB}/hsv.lut Color lookup table
i8 = ${ASCDS_CALIB}/i8.lut Color lookup table
rainbow1 = ${ASCDS_CALIB}/rainbow1.lut Color lookup table
rainbow2 = ${ASCDS_CALIB}/rainbow2.lut Color lookup table
ramp = ${ASCDS_CALIB}/ramp.lut Color lookup table
red = ${ASCDS_CALIB}/red.lut Color lookup table
sls = ${ASCDS_CALIB}/sls.lut Color lookup table
staircase = ${ASCDS_CALIB}/staircase.lut Color lookup table
standard = ${ASCDS_CALIB}/standard.lut Color lookup table
chips> hl = lut.halley
chips> print(os.path.expandvars(hl))
/soft/ciao-4.5/data/halley.lut
(assuming CIAO is installed in /soft/ciao-4.5/) but it can not be called
chips> lut(b="tst.lut") TypeError: 'CIAOParameter' object is not callable
Parameter files
Using these routines does not change any parameter file on disk, unless you use one of the routines listed below. This means it is safe to use this module to run tools from a script whilst using CIAO tools from the command line; the parameter files used by the command-line tools will not be over-written or changed by the Python script.
Any routine that is an instance of the CIAOToolParFile class - which, as of CIAO 4.4, is all tools except for modelflux, fullgarf, and dmgti - will not change the on-disk parameter file.
Routines that do change the on-disk parameter file
Running the modelflux, fullgarf, or dmgti routines - i.e. instances of the CIAOToolDirect class - will change the on-disk parameter file.
How to check whether a parameter file will be changed?
The class of the routine - which can be returned by the type() call or checked using isinstance() - determines whether a parameter file will be changed ("CIAOToolDirect") or not ("CIAOToolParFile"). For instance,
chips> type(dmgti)
<class 'ciao_contrib.runtool.CIAOToolDirect'>
chips> type(dmcopy)
<class 'ciao_contrib.runtool.CIAOToolParFile'>
chips> type(wavdetect)
<class 'ciao_contrib.runtool.CIAOToolParFile'>
These class names are rather confusing and may be changed in a later release.
WARNING: use of ardlib and other such parameter files
Many instrument-specific tools use parameter files like ardlib and geom to find certain files. When the tool is run, they still use the on-disk version of the file and not the values stored in the Python object of the same name. So the setting
chips> ardlib.AXAF_ACIS0_BADPIX = "bpix.fits[ccd_id=0]"
will not be picked up when a tool such as mkinstmap is run. One solution is to use the write_params method, which is described further below, to write out the ardlib parameter file to disk after making changes.
chips> ardlib.AXAF_ACIS0_BADPIX = "bpix.fits[ccd_id=0]" chips> ardlib.write_params()
Updating a parameter file
If you want to write the current set of parameters to disk, you can use the "write_params" method. As an example, where e1, e2 and e3 are numbers that have been calculated elsewhere
chips> expr = "imgout=(img1*{0}+img2*{2})+{3}".format(e1,e2,e3)
chips> dmimgcalc.operation = expr
chips> dmimgcalc.write_params()will write the expression out to the dmimgcalc parameter file (eg try "!plist dmimgcalc" to see the result).
To write to a specific file, give a file name as the argument to write_params. A ".par" suffix will be appended if the filename does not contain one.
Updating to match a parameter file
The default values - and those used when the punlearn() method is used - are taken from the CIAO defaults. To update the values to match those from an on-disk file use the "read_params" method:
chips> dmimgcalc.read_params()
To read from a specific file, give a file name as the argument to write_params. A ".par" suffix will be appended if the filename does not contain one.
Changing the PFILES environment variable
The set_pfiles() routine changes the user portion of the PFILES environment variable. This controls where the tools look for ancillary parameter files (e.g. ardlib) and where the read_params and write_params methods default to when no explicit filename is given. See 'help set_pfiles' for more information.
The get_pfiles() routine can be used to get the current user or system portions of the PFILES environment variable. See 'help get_pfiles' for more information.
The new_pfiles_environment() context manager can be used to automatically handle the creation and clean up of a directory, as well as changing the PFILES environment variable. For instance:
def getemap(ccd, gridstr):
"""Create an ACIS exposure map for the given CCD and using gridstr
as the xygrid parameter of mkexpmap.
The input files are hard coded to simple names for simplicitly.
"""
with new_pfiles_environment(ardlib=True):
afile = "asphist{0}.fits".format(ccd)
ah = make_tool("asphist")
ah(infile="asol1.fits", outfile=afile,
evtfile="evt2.fits[ccd_id={0}]".format(ccd),
clobber=True)
ifile = "imap{0}.fits".format(ccd)
mki = make_tool("mkinstmap")
mki(outfile=ifile, monoenergy=1.7, pixelgrid="1:1024:1,1:1024:1",
obsfile=afile+"[asphist]", detsubsys="ACIS-{0}".format(ccd),
maskfile="msk1.fits", pbkfile="pbk0.fits", clobber=True)
efile = "emap{0}.fits".format(ccd)
mke = make_tool("mkexpmap")
mke(asphistfile=afile, outfile=efile, instmapfile=ifile,
xygrid=gridstr, clobber=True)
defines a routine, getemp(), which will run the asphist, mkinstmap and mkexpmap tools using a temporary directory and PFILES setting. This directory will be automatically removed, and the PFILES environment variable restored at the end of the routine, even if one of tools fails.
Differences to CIAO tools
No mode parameter
All tools are run with the mode parameter set to "hl". This is to avoid the tool trying to prompt you for missing parameter values, which would lead to it appear to hang as the prompt would not be seen by the user.
Missing parameters
As well as the mode parameters, there are a few tools which have a parameter that can only be set to a single value. These parameters are excluded from the interface.
Parameter values
Parameters should be set using the appropriate Python type, namely booleans, integers, floats or strings. For floating-point values, None is used to indicate the "INDEF" setting. None can be used for filenames and strings to indicate the value "". File names are given as strings, but you do not need to add extra quotes if it contains a space - so the following is valid
chips> dmstat("evt2.fits[sky=region(src.reg)][cols energy]")Redirected values
There is limited support for the use of parameter redirects - the use of ")<parname>" to use the value from another parameter - but it only supports redirects to the same parameter file.
Parameter names
When required, parameter names are adjusted to make sure they are valid Python. At present the only change used is to convert any "-" characters to "_"; this is only required for several parameters in ardlib. Purely numeric parameter names are ignored; this only occurs for the "helper" parameters of dmtabfilt.
Changes in the December 2012 release
The module was updated to support the new tools in the CIAO 4.5 release (tg_findzo), the removed tools (mkpsf), and for changes in parameters of existing tools (e.g. modelflux). Support for INDEF with numeric parameters has been improved.
Bugs
Use of parameter redirects to other parameter files
Parameter values of the form ")toolname.parname", which use the parameter value from another parameter file, are not supported. So, for example tgdetect, which has its thresh parameter set to ")celldetect.thresh", will not run unless you manually set these parameters to a valid value. For example:
chips> from ciao_contrib.runtool import * chips> print(tgdetect.thresh) )celldetect.thresh chips> tgdetect.thresh = celldetect.thresh chips> print(tgdetect.thresh) 3.0
See the bugs pages for an up-to-date listing of known bugs.
Refer to the CIAO bug pages for an up-to-date listing of known issues.
![[CIAO Logo]](../imgs/ciao_logo_navbar.gif){kind=logo}
