Last modified: December 2022

URL: https://cxc.cfa.harvard.edu/ciao/ahelp/calindex.html
AHELP for CIAO 4.16

calindex

Context: Tools::CALDB

Synopsis

Create an index file or append data on an existing one of calibration database (CALDB) products.

Syntax

calindex index keyconf calfile [--clobber y] [--verbose #]

Description

calindex generates an "index" file from calibration data source directed to "calfile" with user's specification in "key.conf". The index then will be served in CALDB product retrievals by other applications. The index has a single FITS extension "CIF" with multi-columns, including a set of mendatory columns of predefines and the optional ones of run-time specification. A header-key configuration table, "keyconf", allows user to extend the index table with defined column data types, data formats, and other entries. The name field, "calfile", sets access of the calibration data products for the indexing.

calindex appends new index data on an existing "CIF" extension or appends new index extension on an existing file if "clobber" is default or set to no|n|0.


Examples

Example 1

calindex xfile.1.fits keys_1.cfg datadir --clobber yes

Create index file named as "xfile.1.fits" to index all FITS files in directory, "datadir" and its recursively down directories. The header-key config file, "keys_1.cfg", is a text file and has following columns and rows. As clobber is set to "yes" (yes|y|1), "xfile.1" will be removed first if existed.

     #keys_01.cfg 
     #colName   dataType        format	hdrKey		query	null
     TELESCOP   string		a10	TELESCOP        yes     ""
     INSTRUME   string		a10     INSTRUME        yes	""
     CAL_VSD    string		a10	DATE-OBS	no	""
     CAL_VST    string		a8	DATE-OBS	no	""
     CAL_DESC   string		a70	CDES		no	""
     CAL_CLAS   string		a4	FOO		no	""

     

The lines starting with '#' are comments and will be ignored at run time. Only the first three columns (colName, dataType, and format) are interesting for index creation and the last 3 columns are served for CALDB product retrieval in other applications. What it says here is that, for example, "TELESCOP", a column name entry, has "string" data type and data format of 10 ASCII characters ("a10"). The rows of the "TELESCOP" values are extracted from all FITS files under "datadir" and then indexed in "xfile.1.fits". The other five rows of "keys_1.cfg" follow the same suit. calindex gets right CALDB FITS files by checking the file postfix, "fits" or "fit", which is case-insensitive.

In this example, the output of "xfile.1.fits" with "CIF" block is created with 14 columns, as shown below. Column #1,2,11-14 are user's addition as specified in "keys_1.cfg" above and the rest are defaults, common to any index file.

--------------------------------------------------------------------------------
Columns for Table Block CIF
--------------------------------------------------------------------------------
 
ColNo  Name                 Unit        Type             Range        Comment
   1   TELESCOP                          String[10]                   Telescope
   2   INSTRUME                          String[10]                   Instrument
   3   CAL_DEV                           String[20]                   CALDB Device
   4   CAL_DIR                           String[70]                   CALDB Directory
   5   CAL_FILE                          String[40]                   CALDB File
   6   CAL_CNAM                          String[20]                   CALDB Product
   7   CAL_CBD[9]                        String(9)                    Condition
   8   CAL_XNO                           Int2           -             Extension #
   9   CAL_QUAL                          Int2           -             Quality
  10   CAL_DATE                          String[10]                   date
  11   CAL_VSD                           String[10]                   Start Date
  12   CAL_VST                           String[8]                    Start Time
  13   CAL_DESC                          String[70]                   description
  14   CAL_CLAS                          String[4]                    class

Example 2

calindex xfile.1a.fits keys_1.cfg mycaldb1+ --clobber y
or
calindex xfile.1a.fits keys_1.cfg mycaldb1/+ --clobber y

Create index, "xfile.1a.fits", with the same optional columns as above example but having different data products indexed. The index rows of the output are filled from all FITS files in the directory "mycaldb1" and in its recursive down directories. The symbol, '+', following directory name and or '/' indicates a recursive readout of calibration files from the current location.

Example 3

setenv CALDB mycaldb7
calindex xfile.7.fits keys_7.cfg CALDB --clobber y

Create index, "xfile.7.fits" with "CALDB" to "calfile". The "CALDB" is an environmental variable and it, in the current example, is pointed to "mycaldb7+" by pre-commanding "setenv CALDB mycaldb7". In the current cae, mycaldb7 contains 3 FITS files for feeding calindex.

"keys_7.cfg" has two more rows, "CAL_VED" and "CAL_VET", added over the "keys_1.cfg", as shown below

#keys_7.cfg
#colName        dataType        format  hdrKey          query           null
TELESCOP        string          a10     TELESCOP        yes             ""
INSTRUME        string          a10     INSTRUME        yes             ""
CAL_VSD         string          a10     ""              yes             ""
CAL_VST         string          a8      ""              yes             ""
CAL_VED         string          a10     ""              yes             ""
CAL_VET         string          a8      ""              yes             ""
CAL_DESC        string          a70     CDES            no              ""
CAL_CLAS        string          a4      FOO             no              ""

Example 4

setenv CALDB mycaldb6
calindex xfile_6b.fits keys_1.cfg "CALDB,mycaldb2" --clobber y

Index file, "xfile_6b.fits", is created on directory list, CALDB and "mycaldb2". CALDB is aliased to "mycaldb6" in which there are 3 FITS files and one sub-dir containging one file. "mycaldb2" has only one FITS file within. So, the output has, at least, 4 FITS files indexed.

Example 5

calindex xfile.5.fits keys_1.cfg @cfile_5.lst --clobber y

This example shows an alternative to assign calfile value with named file list, marked by '@' before the named file, "cfile_5.lst". Below is what the list looks like

     #cfile_5.lst
     cfile_5A.fits
     cfile_5B.fits

When calindex detects the first char, '@', of the calfile string and it opens up the file list, "cfile_5.lst", and feeds "cfile_5A.fits" and "cfile_5B.fits" in to the array of calibration files.

Example 6

calindex xfile.1g.fits keys_1.cfg ./*.foo --clobber y

Create index, "xfile.1g.fits", to all files ending with ".foo", in the current directory.

Example 7

calindex xfile.1h.fits keys_1.cfg '' --clobber y
or
calindex xfile.1h.fits keys_01.cfg "NONE" --clobber y

Create index, "xfile.1h.fits", to all FITS files in and under the CALDB directory tree. Note that calindex, with any empty or 'NONE' input to 'calfile', will look into CALDB directory tree.

Example 8

setenv CALDB /data/regression_test/development/CALDB/data/chandra
calindex xfile.8a.fits keys_1.cfg $CALDB/acis/cpf+ --clobber y

Create index, "xfile.8a.fits" to all FITS files in "$CALDB/acis/cpf" directory and its resursive subdirectories.

Example 9

calindex xfile.9.fits keys_9.cfg "cfile.9A.fits,cfile.9B.fits"
--clobber y

As usual, calindex creates "xfile.9.fits" to index two calibration files, "cfile.9A.fits" and "cfile.9B.fits". However "keys_9.cfg" adds a "MYDETNAM" column name, an incoventional entry, as shown below. With this in check, calindex geneartes the output containing "MYDETNAM" column.

#keys_9.cfg
#colName        dataType        format  hdrKey          query         null
TELESCOP        string          a10     TELESCOP        yes           ""
INSTRUME        string          a10     INSTRUME        yes           ""
CAL_VSD         string          a10     ""              yes           ""
CAL_VST         string          a8      ""              yes           ""
MYDETNAM        string          a20     "dnam"          yes           "NONE"
CAL_DESC        string          a70     CDES            no            ""
CAL_CLAS        string          a4      FOO             no            "DATA"

Example 10

calindex xfile.9.fits ./acis/key.conf ./acis/det_gain

As "clobber" is default(to 0 ), the tool appends new index rows, extracted from ./acis/det_gain directory, on the extension, "CIF", of the disk file, "xfile.9.fits" if existed or it creates "xfile.9.fits" if not existed.


Parameters

name type ftype def min max reqd
index file output       yes
keyconf file input       yes
calfile file input       yes
--clobber boolean   no      
--verbose integer   0 0 5  

Detailed Parameter Descriptions

Parameter=index (file required filetype=output)

Output file name

The named file, created either from scratch or from an existing one, has an extension named after "CIF", Calibration Index File. The "CIF" table contains columns internally defined (mandatory) as defaults and optionally defined by users. The mandatory columns are

         CAL_DEV, CAL_DIR, CAL_FILE, CAL_XNO, 
         CAL_CNAM, 
         CAL_CBD, 
         CAL_QUAL,
         CAL_DATE

conforming with the current CALDB version 4. The set of optional columns are specified in an external text file, an input to 'keyconf'. The default name of 'index' will be given to ./caldb.indx' if the value of 'index' is empty or 'NONE' or 'none'.

Parameter=keyconf (file required filetype=input)

Input file name

The key config file is used for user to define optional columns of the Index. The frequent conventional columns in the CALDB category include

         TELESCOP, 
         INSTRUME, 
         CAL_VST, CAL_VSD, 
         CAL_VET, CAL_VED, 
         DETNAM 

In addition, user can adds on any names other than the above, like FIDELITY, MYDETNAM, etc.

There are two special notes about the creation of the keyconf file. The column names of "TELESCOP" and "INSTRUME" must appear in keyconf file, so do in index file, as they will be served as the starting point of CALDB query later. Though the first three columns, or colName, dataType, and format, will be only interesting to index generation, the format of the entire set of columns have to be carried on as required by the software.

Parameter=calfile (file required filetype=input)

Input file name

"calfile" file, as the name explains, serves as a source directories of calibration data products. Following entries are allowed by the tool.

	 1> CALDB (the root of calibration data tree), 
	 2> single file, 
	 3> file stack, and 
	 4> named file list. 

The stacking files, as usual, are separated in comma, ','. The file list is indicated by '@' symbol in front of the named file string. Below is to summarize the possible syntaxes for the input string to 'calfile':

	 [directory]
	 [directory]+  or  [directory]/+
	 CALDB   
	 [file1, file2, ..., filen]
	 [directory]/*.fits
	 [directory]/*.[foo]
	 @[file-list]
	 NONE, none or empty 

Parameter=--clobber (boolean default=no)

The prefix, "--", to "clobber" is an optional tag, followed by a given value. If the value is set to 1 or "yes", calindex removes the existing file and create a new file with an index, "CIF", extension. Otherwise for the default (no|n|0), calindex either 1) appends new data to an existing "CIF" extension of the said output file, or 2) creates a "CIF" extension appending on to the said output file which does not have CIF extension yet, or 3) creates a "CIF" index output while the file does not exit in the disk.

Parameter=--verbose (integer default=0 min=0 max=5)

Verbosity level of terminal display information to user.

There is no output message for default but more details printed out in ascending levels.


Bugs

There are no known bugs for this tool.

See Also

calibration
ardlib, caldb
tools::caldb
calmerge, calquiz, calvalid
tools::utilities
check_ciao_caldb