Chandra X-Ray Observatory
	(CXC)
Skip to the navigation links
Last modified: 08 Dec 2014

URL: http://cxc.harvard.edu/ciao/threads/ciao_build_linux/

Rebuilding CIAO 4.7 source with pre-built OTS binaries on Linux

CIAO 4.7 Science Threads


Overview

Synopsis:

CIAO depends on a number of external software components we refer to as OTS (off-the-shelf) software. The following instructions let you build CIAO 4.7 from source on Linux systems, but assume you are using pre-built binaries for the OTS software. With these directions, you can rebuild CIAO entirely, or rebuild selected tools in the source code.

Related Links:

Last Update: 08 Dec 2014 - First issue. Merges INSTALL_SOURCE text file into CIAO website.


Contents


A few notes before you start

To prevent any old variables from affecting the CIAO build environment, it is advised to open up a new terminal before continuing.

Where should CIAO be installed?

CIAO can be installed in any location and does not need to be installed with any root or super-user privileges.


Off-the-shelf software

The CIAO off-the-shelf packages include adapted OTS and some dependencies on specific versions of OTS. For example, we use a modified version of VTK for CIAO, as well as a specific snapshot of gtkglext. As such, it is highly beneficial (and will save you a lot of headaches) to build CIAO with our OTS binaries. If you do need to build CIAO completely from source, contact the CXC Helpdesk.

You can view the list of the OTS and their versions here.


CIAO Source Build Quickstart

The following are the basic directions for building CIAO 4.7 from source with pre-built OTS binaries on Linux operating systems (32- and 64-bit supported). If you are only rebuilding selected directories, skip step 4. For detailed information and explanation of the steps, please continue to the following sections.

  1. Download GCC and gfortran, if not already installed
  2. Download and unpack CIAO binaries and source code via ciao-install; cd into ciao-4.7
    % cd <path-to>/ciao4.7/
          
  3. Create $ASCDS_INSTALL variable pointing to location of ciao-4.7 directory
    % export ASCDS_INSTALL=`pwd`       (for bash)
    % setenv ASCDS_INSTALL `pwd`       (for tcsh)
          
  4. Delete contents of lib/ and binexe/.

    If only rebuilding selected directories, skip this step.

    % rm -rf binexe/* lib/*
          
  5. Check to see if you have Python 2.7.x on your system:
    % python --version
          
    If you do, go to the next step.

    If not, set the following variables:
      (for bash)
    % export CIAO_PYTHON=CIAO
    % export PATH=${ASCDS_INSTALL}/bin:${PATH}
    
      (for tcsh)
    % setenv CIAO_PYTHON CIAO
    % setenv PATH ${ASCDS_INSTALL}/bin:${PATH}
          
  6. Set $PKG_CONFIG_PATH to include paths to ${ASCDS_INSTALL}/ots/lib/pkgconfig and the path to your system pkgconfig:
    % export PKG_CONFIG_PATH=${ASCDS_INSTALL}/ots/lib/pkgconfig:/usr/lib64/pkgconfig       (for bash)
    % setenv PKG_CONFIG_PATH ${ASCDS_INSTALL}/ots/lib/pkgconfig:/usr/lib64/pkgconfig       (for tcsh)
          
    For some newer Redhat based builds (e.g., CentOS 6), you need to add /usr/share/pkgconfig to $PKG_CONFIG_PATH.
    % export PKG_CONFIG_PATH=${ASCDS_INSTALL}/ots/lib/pkgconfig:/usr/lib64/pkgconfig:/usr/share/pkgconfig       (for bash)
    % setenv PKG_CONFIG_PATH ${ASCDS_INSTALL}/ots/lib/pkgconfig:/usr/lib64/pkgconfig:/usr/share/pkgconfig       (for tcsh)
          
  7. Set $LD_LIBRARY_PATH to ${ASCDS_INSTALL}/ots/lib:
    % export LD_LIBRARY_PATH=${ASCDS_INSTALL}/ots/lib   (for bash)
    % setenv LD_LIBRARY_PATH ${ASCDS_INSTALL}/ots/lib   (for tcsh)
          
  8. Setup the CIAO environment and run configure. If the source code and binaries were unpacked by hand (i.e., using bash ciao-install --download-only), then configure must first be run without any arguments.Also, if using the CIAO Python distribution (i.e., you do not have Python 2.7.x), ciaorun must be run in line with configure.
    % ./configure    (...run first ONLY IF you unpacked the source code and binaries by hand)
    
      (start ciao session)
    % . bin/ciao.sh            (for bash)
    % source bin/ciao.csh      (for tcsh)
    
    % ciaorun ./configure --with-ascii --with-fits --<other-options>    (using CIAO Python (if you DO NOT have Python 2.7.x))
    % ./configure --with-ascii --with-fits --<other-options>            (using system Python)
          
  9. Run make install
    % ciaorun make install    (using CIAO Python (if you DO NOT have Python 2.7.x))
    % make install            (using system Python)
          
    If only rebuilding a certain tool, cd into that tool's directory and run make install there.

  10. Run ciao-python-fix
    % bash src/config/ciao-python-fix
          
  11. Generate AHELP pages
    (start CIAO session)
    % . bin/ciao.sh -o        (for bash)
    % source bin/ciao.csh -o  (for tcsh)
    % ahelp -r
          
  12. Run smoke tests
    % cd test
    % make -k test 2>&1 | tee ~/ciao_test.log  (for bash)
    % make -k test |& tee ~/ciao_test.log         (for tcsh)
          

Requirements

Operating Systems

Building CIAO with the pre-built OTS is supported on 32- and 64-bit Linux systems.

Note: we also provide instructions for building CIAO with the pre-built OTS on Mac OS X 10.6-10.9.


Compilers

Building CIAO requires C, C++ and FORTRAN compilers. We strongly recommend using GCC and gfortran. Building CIAO on Linux system has been tested with GCC and gfortran versions 4.2.x, 4.4.7, and 4.9.2.

Check to see if you have gfortran and GCC on your system:

% gcc --version
% gfortran --version
      

If either gives you an error, you will have to download and install that compiler.

For Linux 32-bit systems

Our binary copy of the XSPEC libraries depends on libgfortran.so.1, which may cause a conflict with versions of GCC / gfortran higher than 4.1.x. If you are using gfortran-v4.2 or higher, please follow the directions in For Linux 32-bit systems to properly install CIAO with the XSPEC models.


Source files + OTS binaries

Building CIAO from source using the pre-built binary OTS requires the ciao-4.7-src-*.tar.gz packages as well as the platform specific ciao-4.7-bin-<Linux|Linux64>.tar.gz binaries. Linux 32-bit systems should grab the Linux binaries; 64-bit systems should use the Linux64 binaries.

First, install CIAO using the ciao-install script. Go to the Downloading CIAO page. Under Custom Installation, choose your platform and check the "Standard CIAO installation" and "Full source build" boxes. This will select all of the necessary binaries (which contain the OTS) and will download the source code. CALDB, the background event files and the science scripts package are optional; include or exclude these packages by checking the boxes next to them. Finally, click the "Download the ciao-install Script" button.

From a terminal, run the ciao-install script. ciao-install will unpack all the files for you [recommended].

% bash <path-to>/ciao-install
      

If you want to unpack the files yourself, use the --download-only switch:

% bash <path-to>/ciao-install --download-only
      

and then unpack the tar.gz packages. For example, the commands for unpacking the "Standard CIAO installation" Linux 64-bit packages are shown below.

% tar -xzf ciao-4.7-bin-chips-Linux64.tar.gz
tar -xzf ciao-4.7-bin-core-Linux64.tar.gz
tar -xzf ciao-4.7-bin-graphics-Linux64.tar.gz
tar -xzf ciao-4.7-bin-obsvis-osx64.tar.gz
tar -xzf ciao-4.7-bin-prism-Linux64.tar.gz
tar -xzf ciao-4.7-bin-sherpa-Linux64.tar.gz
tar -xzf ciao-4.7-bin-tools-Linux64.tar.gz
tar -xzf ciao-4.7-contrib-DEV.tar.gz
tar -xzf ciao-4.7-src-chips.tar.gz
tar -xzf ciao-4.7-src-core.tar.gz
tar -xzf ciao-4.7-src-obsvis.tar.gz
tar -xzf ciao-4.7-src-prism.tar.gz
tar -xzf ciao-4.7-src-sherpa.tar.gz
tar -xzf ciao-4.7-src-tools.tar.gz
tar -xzf caldb_4.6.5_main.tar.gz
      

Please note that the list of files to untar depends on the selections made when downloading the ciao-install script.

If installing CALDB with --download-only switch

Unpacking caldb_4.6.5_main.tar.gz will create directories data/, docs/, and software/. You will need to create a CALDB directory and move data/, docs/ and software/ into CALDB/, then softlink CALDB to CIAO after the build.

> # in directory where caldb_4.6.5_main.tar.gz is
> mkdir CALDB
> mv docs/ data/ software/ CALDB/
	

Note: ciao-4.7-bin-obsvis-<plat>.tar.gz is NOT needed as no OTS are packaged in that binary distribution.

Note: If you are having trouble with the ciao-install script, refer to "Downloading the Software" in Manually Installing CIAO.


Build Preparation

  1. Untarring the ciao-4.7-*.tar.gz files creates a directory ciao-4.7; cd into it:
    % cd ciao-4.7
    % ls
    bin             config.h.in    COPYING.LIB     LICENSE           README.obsvis     VERSION_obsvis
    binexe          config.log     data            LICENSE.SAO       share             VERSION_prism
    CALDB           config.status  doc             Linux_build.html  src               VERSION.prop_tk
    ciao_installed  config.sub     include         Mac_build.html    test              VERSION_sherpa
    ciao-type       configure      INSTALL_BINARY  Makefile          VERSION           VERSION_tools
    confdefs.h      configure.in   install-sh      ots               VERSION_chips
    config          contrib        INSTALL_SOURCE  param             VERSION_core
    config.guess    COPYING        lib             README            VERSION_graphics
    
          
    You will make many references to the ciao-4.7 directory during the build configuration. Creating an environment variable for the path to ciao-4.7 will make the build commands easier:
    % export ASCDS_INSTALL=`pwd`        (for bash)
    % setenv ASCDS_INSTALL `pwd`        (for tcsh)
    % echo $ASCDS_INSTALL
    /<path-to>/ciao-4.7
          
    If you only want to rebuild certain directories of the CIAO source code, skip to step (3).

  2. To make sure that libraries and executables are newly created by the source build on your system, delete the contents of the lib/, and binexe/ directories:
    % rm -rf lib/* binexe/*
          
  3. To build CIAO, you need Python 2.7.x. In a terminal, check for your version of Python:
    % python --version
    Python 2.7.5 :: Anaconda 1.8.0 (64-bit)
          
    If you have Python 2.7.x, continue onto Step (4). The Python version in the example output above is compatible for building CIAO.

    If you have another version of Python or no Python at all, you will need to set the following variables to point to the Python distribution shipped in the CIAO binaries:
      (for bash)
    % export CIAO_PYTHON=CIAO
    % export PATH=${ASCDS_INSTALL}/bin:${PATH}
    
      (for tcsh)
    % setenv CIAO_PYTHON CIAO
    % setenv PATH ${ASCDS_INSTALL}/bin:${PATH}
          
  4. CIAO uses GTK for chips and prism. The necessary GTK files are packaged with the CIAO graphics binary tarfile in the pkgconfig directory. We discovered that combining the system and CIAO GTK produces a more stable system than the CIAO GTK alone. Therefore, you must build with both the system and the CIAO GTK pkgconfig. To do this, set the following variable $PKG_CONFIG_PATH:
    % export PKG_CONFIG_PATH=${ASCDS_INSTALL}/ots/lib/pkgconfig:/usr/lib64/pkgconfig      (for bash)
    % setenv PKG_CONFIG_PATH ${ASCDS_INSTALL}/ots/lib/pkgconfig:/usr/lib64/pkgconfig      (for tcsh)
          
    • Note: On some newer Redhat Based builds (CentOS 6), you need to add /usr/share/pkgconfig to $PKG_CONFIG_PATH:
      % export PKG_CONFIG_PATH=${ASCDS_INSTALL}/ots/lib/pkgconfig:/usr/lib64/pkgconfig:/usr/share/pkgconfig      (for bash)
      % setenv PKG_CONFIG_PATH ${ASCDS_INSTALL}/ots/lib/pkgconfig:/usr/lib64/pkgconfig:/usr/share/pkgconfig      (for tcsh)
      	  
    • Note: On 32-bit systems, the pkgconfig libraries are in /usr/X11R6/lib/pkgconfig:
      % export PKG_CONFIG_PATH=${ASCDS_INSTALL}/ots/lib/pkgconfig:/usr/X11R6/lib/pkgconfig      (for bash)
      % setenv PKG_CONFIG_PATH ${ASCDS_INSTALL}/ots/lib/pkgconfig:/usr/X11R6/lib/pkgconfig      (for tcsh)
            

For Linux 32-bit systems

Our binary copy of the XSPEC libraries depends on libgfortran.so.1, which may cause a conflict with versions of GCC / gfortran higher than 4.1.x. If you cannot use a copy of gfortran that uses the libgfortran.so.1 library, you will either need to re-build XSPEC (more involved; instructions at HEASARC), or download our XSPEC package built with libgfortran.so.3 (easier). If you use our XSPEC library, download the XSPEC tarfile, unzip it, and then remove the old XSPEC libraries from ${ASCDS_INSTALL}/ots/lib

% gunzip -c xspec-gfortran-4.4.7.tar.gz | tar xf -
% rm ${ASCDS_INSTALL}/ots/lib/libXS*
% cd xspec-gfortran-4.4.7
      

Copy libSX.a, libXSFunctions.so, libXSModel.so and libXSUtil.so from the lib/ directory into ${ASCDS_INSTALL}/ots/lib:

% cp lib/libXS.a lib/libXSFunctions.so lib/libXSModel.so lib/libXSUtil.so ${ASCDS_INSTALL}/ots/lib
      

This will make sure the XSPEC models are included in Sherpa after building CIAO.


Build Configuration

Before building CIAO, we must configure the build script for our system. This involves choosing which compilers to use, setting library locations, and adding other options to the build. All of this is handled by a configuration script.

Basic configuration

  1. If you unpacked the source code and binaries by hand, run the configuration script without any arguments before continuing. This links the CIAO Python and other CIAO tools to the CIAO build; skipping this step would result in a CIAO environment without access to all the CIAO tools.

    Otherwise, skip to Step 2.
    % ./configure
    	  
  2. Start a CIAO session.
    % . bin/ciao.sh          (for bash)
    % source bin/ciao.csh    (for tcsh)
    	  
  3. Run the configuration script. The most basic configuration call to have a successful CIAO build using the default build options (i.e., using the Python distribution packaged with CIAO):
    % ciaorun ./configure --with-fits --with-ascii    (using the default build options)
    	  
    or, if using your system Python distribution:
    % ./configure --with-fits --with-ascii   (using system Python)
    	  
    The --with-fits and --with-ascii switches include the FITS and ASCII kernels when building the CXC DataModel (necessary to build CIAO).

    Additional arguments, described in the following subsections, may be used in the configure call to further specify the configuration.

If configure fails, see the Troubleshooting section. A likely cause of failure is the environment variables $PATH and $PKG_CONFIG_PATH are not properly set, or that there are multiple compilers on your $PATH and configure is picking-up the wrong compiler(s).

The following subsections contain details on setting the compilers, the configuration options, and compiler optimization levels.


Setting the compilers

Specifying a specific path to a compiler can be achieved by using the appropriate keyword:

  • CC - Name and location of the c compiler
  • CXX - Name and location of c++ compiler
  • F77 - Name and location of FORTRAN compiler
  • FC - Name and location of FORTRAN compiler
  • F95 - Name and location of FORTRAN compiler

Usage example:

% ciaorun ./configure --with-ascii --with-fits \
  FC=<path-to>/bin/gfortran-4.2 \
  F77=<path-to>/bin/gfortran-4.2 \
  F95=<path-to>/bin/gfortran-4.2 \
  CC=<path-to>/bin/cc \
  CXX=<path-to>/bin/c++
      

(Exclude ciaorun from the command if building with your system Python.)

To specify the FORTRAN compiler, the FC, F77 and F95 variables must be the same.

The compiler must also be in your PATH. If <path-to-compiler> is not in $PATH, add it using

% export PATH=${PATH}:<path-to-compiler>       (for bash)
% setenv PATH ${PATH}:<path-to-compiler>       (for tcsh)
      

Configuration flags/options

Here is the list of configure options available for CIAO:

Option Description
--prefix=<location> Directory to install CIAO. Default location will be '.'
--x-includes=<DIR> directory DIR in which the X include files are located
--x-libraries=<DIR> directory DIR in which the X library files are located
--with-debug Compile with debug mode enabled
--with-opt=<level> Set optimization level (default=-O3)
--with-fits Include FITS kernel when building CXC DataModel.
Note: This switch is required to build the DataModel. It indicates that the setup is for a compile and not just a binary install.
--with-ascii Include the ASCII kernel when building the CXC DataModel.
Note: This switch is required if you need to read ASCII files into ChIPS.
--with-cfitsio Link against libcfitsio instead of libcxccfitsio
--with-f95loc=<DIR> Location of libf95 (if using g95)
--with-gfortranloc=<DIR> Location of libgfortran (if using gfortran) Unnecessary if gfortran and libgfortran are in the same tree (e.g. if gfortran is in /usr/bin, and libgfortran is in /usr/lib, they’re in the same tree)
--with-imagerdir=<saord> Path to directory/tree containing DS9

The flags --with-ascii and --with-fits are REQUIRED for CIAO to build correctly. All other flags are optional.


Optimization and compilers

Optimization is turned on by default, -O3 for GCC. If you wish to use a different optimization level or turn off optimization, use the --with-opt= switch. To turn off optimization, use --with-opt=-O0. See GNU’s Optimization Options for more details.


Build Instructions

Now, the source tree should be properly set up to build. The directions here are for rebuilding the entire CIAO source code (Rebuilding CIAO), or for rebuilding a selected directory (Rebuilding selected source code).

Rebuilding CIAO

To build and install the code, type "make install". CIAO currently requires that "make install" be issued as a single command rather than as separate commands "make; make test; make install". This is due to build dependencies on installed libraries and packages.

% ciaorun make install   (using default build options) 
% make install           (using system Python)
      

Note that this will take a few minutes.


Rebuilding selected source code

There are a few tools that depend on local libraries that if not installed may require rebuilding first. Most src directories can be rebuilt after running configure by simply stepping into the directory and running "make install".

For example, say a user rebuilt libfftw to take advantage of their specific hardware. After placing the libfftw files in $ASCDS_INSTALL/ots/lib, they’d need to re-link the tools in $ASCDS_INSTALL/src/da to their system by doing

% cd ${ASCDS_INSTALL}/src/da
% ciaorun make install   (using default build options) 
% make install           (using system Python)

      

to rebuild the da tools with their custom libfftw build.


Post-processing

After the build completes, run the script ciao-python-fix:

% bash src/config/ciao-python-fix
      

This post-processes your configuration, to byte-compile included Python files for your installation, and to ensure all CIAO packages with Python interfaces can be built. Once this script is run, the CIAO installation cannot be moved or relocated to a different directory without repeating the configure and ciao-python-fix steps.

Linking to CALDB: if you downloaded CALDB and unpacked the tarfile by hand (e.g. ran ciao-install with the --download-only switch), you can link it to CIAO at this point by creating a softlink to the CALDB directory in $ASCDS_INSTALL:

% pwd
/<path-to>/ciao-4.7
% ln -s <path-to>/CALDB CALDB
      

AHELP Indexing

After installation you will need to generate an index file used by the CIAO help documentation application (ahelp). The ciao.*sh script needs to run to set up dynamic library paths used by ahelp.

% . bin/ciao.sh -o        (for bash)
% source bin/ciao.csh -o     (for tcsh)
  

Then run the ahelp command with the -r (re-index) option.

% ahelp -r
  

This will generate the help indexes. If this completes successfully, you can execute the smoke tests.


Smoke Tests

To run the tests, you should do something like:

% cd test
% . ../bin/ciao.sh       (Note: Only needed if you have not
                                setup for CIAO in a prior step)
% make -k test 2>&1 | tee ~/ciao_test.log    (for bash)
  

or

% cd test
% source ../bin/ciao.csh   (Note: Only needed if you have not
                                setup for CIAO in a prior step)
% make -k test |& tee ~/ciao_test.log    (for tcsh)
  

A successful test will look like:

        [1/41] Running test FOO-smoke001 ... PASS
  

where 'FOO' is the name of an individual test. If you see a message that reports:

        [1/41] Running test FOO-smoke001 ... FAIL
  

then there is an unexpected problem with the tests. Note that if you see additional warnings/errors and the test reports PASS, then the test is good (tests sometimes do test error conditions).

All the test outputs are written to $ASCDS_WORK_PATH/smoke.$LOGNAME where $LOGNAME is your system login name. The tests need about 60 Mb to run. The test scripts clean-up the disk space upon successful completion. Tests which fail leave their test output so you may check the results. When all of the tests PASS, the smoke test scripts should remove $ASCDS_WORK_PATH/smoke.$LOGNAME and all its sub-directories.

Note

$ASCDS_WORK_PATH is set in the $ASCDS_INSTALL/bin/ciao.*sh setup scripts as /tmp by default. If the machine does not have a /tmp or you don't have write permission, you will need to edit the appropriate ciao.*sh script to change the $ASCDS_WORK_PATH location or set the $ASCDS_WORK_PATH environment variable. If this variable is set, ciao.*sh will not modify it.

If a test FAILS, you may want to double-check that you’re using the right versions of GCC and gfortran, and that your environment variables $PATH and $PKG_CONFIG_PATH are set correctly. Also check Bugs: Installation & Smoke Tests for other documented problems and solutions; if you run into any other problems, please contact the CXC Helpdesk. If you decide to re-run the build process, repeat the configuration and building process in a new terminal to prevent the old environment from affecting the new build.


Troubleshooting

Below are some common problems you may run into while building and installing CIAO. You can also check Bugs: Installation & Smoke Tests for other documented problems and solutions. If you run into any other problems, please contact the CXC Helpdesk.

Configure script

  1. Error "cfitsio is not found"
    checking for pkg-config... no
    configure: error: cfitsio is not found. Please specify the path using
        '--with-cfitsio=<<path>' or
        'setenv PKG_CONFIG_PATH <path>/lib/pkgconfig'
    	  

    This error shows that the $PKG_CONFIG_PATH variable is not set correctly. $PKG_CONFIG_PATH should contain
     (for Linux 64-bit)
    ${ASCDS_INSTALL}/ots/lib/pkgconfig:/usr/lib64/pkgconfig
    
     (for 64-bit Cent OS 6)
    ${ASCDS_INSTALL}/ots/lib/pkgconfig:/usr/lib64/pkgconfig:/usr/share/pkgconfig   
    
     (for Linux 32-bit)
    ${ASCDS_INSTALL}/ots/lib/pkgconfig:/usr/X11R6/lib/pkgconfig   
    	  
  2. For other errors, check that:
    • your compilers are in your $PATH
    • $PKG_CONFIG_PATH is set correctly
    • ${ASCDS_INSTALL}/ots/bin is in your $PATH

Build

If the build fails at any point during "make install", check that

  1. you have compatible GCC and gfortran compilers (see Compilers for details)
  2. your compilers are in your $PATH.
  3. ${ASCDS_INSTALL}/ots/bin is in your $PATH.
  4. $PKG_CONFIG_PATH is set correctly

Then, run "make -k clean", then remove the contents of lib/ and binexe/ again,

% rm -rf binexe/* lib/*
      

re-configure the build with

% . bin/ciao.sh -o          (for bash)
% source bin/ciao.csh -o    (for tcsh)
% ciaorun ./configure --with-fits --with-ascii    (Exclude "ciaorun" from the command if building with your system Python.)
      

and run make install again. If it still fails, try opening-up a new terminal so that you have a clean environment to work with. Check that you have compatible compilers and that you’ve set $PKG_CONFIG_PATH and your $PATH variables correctly defined.


History

08 Dec 2014 First issue. Merges INSTALL_SOURCE text file into CIAO website.


Last modified: 08 Dec 2014
Smithsonian Institute Smithsonian Institute

The Chandra X-Ray Center (CXC) is operated for NASA by the Smithsonian Astrophysical Observatory. 60 Garden Street, Cambridge, MA 02138 USA.   Email:   cxchelp@head.cfa.harvard.edu Smithsonian Institution, Copyright © 1998-2017. All rights reserved.