Chandra X-Ray Observatory
Skip to the navigation links
Last modified: 13 March 2017

Why does using an exclude filter take so long?

The [exlude sky=region(ciao.reg)] filter, especially with dmextract, can be extremely slow; sometimes requiring hours or days to complete. This is often noticed when trying to exclude hundreds or even thousands of sources from, for example, some type extended emission.

The [exclude ] directive does not simply change the "is this point inside" logic, it actually inverts the logic of the region itself so it can be stored in the output file's subspace. It uses De Morgan's Law to invert the logic in the region. Since CIAO regions support complex logical AND (*), OR (+) , and NOT (!) operations, the general solution to inverting a region is an order O(N2) kind of operation; that is the run-time of the tool grows exponentially with number of sources being excluded. To reduce this, CIAO does have some optimizations to deal with groups of shapes whose bounding boxes overlap; however, this does not help for dense source regions that cover the same area.

While CIAO does support complex AND, OR, and NOT operations, most users, expecially in this scenerio, are not making use of complicated logic patterns. Most often the region being excluded simply contains ellipses (or maybe circles) generated by one of the source detect tools, eg wavdetect.

unix% cat ciao.reg

In this scenario, it is easier and faster for the user to manually invert the region and avoid using the [exclude ] syntax altogether. Since each shape is being excluded from the entire field, this can be explicitly written out as field()-circle()-circle(), where users are reminded the minus sign, "-", is CIAO short hand for "*!" (ie "AND NOT"). It is very easy to do this with regions stored in ASCII format. (Users can use dmmakereg with kernel=ascii to convert FITS region files to ASCII).

unix% cat ciao.reg | awk ' BEGIN { print "field()" } {print "-"$0}' > exclude_ciao.reg
unix% cat exclude_ciao.reg

Users should check the files carefully. Some ASCII regions contain comment lines (begin with "#") and ds9 format region files contain a global line. Neither should not have those lines prefixed with a '-'.

Users unfamiliar with the awk utility can use their favorite scripting language to get the same effect. This file can the be used as a normal CIAO filter:

unix% dmextract "evt.fits[sky=region(exclude_ciao.reg)][bin sky=annulus(4096,4096,0:1000:10)]" ...etc...

This will run much faster than with the [exclude ] syntax.

There are of course examples of more complicated regions: sources that have other nearby sources excluded, sources that include a polygon for the field of view, or maybe exclude a rectangular region for the readout streak. These regions cannot be as easily inverted as the example shown. In those cases the region would have be inverted by-hand or left to the [exclude ] filter to work out.

Last modified: 13 March 2017
Smithsonian Institute Smithsonian Institute

The Chandra X-Ray Center (CXC) is operated for NASA by the Smithsonian Astrophysical Observatory. 60 Garden Street, Cambridge, MA 02138 USA.   Email: Smithsonian Institution, Copyright © 1998-2017. All rights reserved.