CIAO mask filtering syntax
Masks are two dimensional filters that can be used to include or exclude data from a given file. They can be used as filters similar to the ways regions are used.
Masks are 2D images. Pixel coordinates that map to values equal to 0 are excluded from the input, pixel values not equal to 0 are included. The mask can either be inclusive or exclusive.
The basic mask filtering syntax is
Masks complement the existing region filtering syntax. Masks provide a way to create complex 2D filters without needing to express it with a potentially long and complicated region expression. Common uses will be to exclude a large number of point sources embedded in extended emission and to simplify the footprints for overlapping fields of view for merged datasets.
For a general introduction to table and image filtering in CIAO, refer to "ahelp dmfiltering".
unix% dmcopy "evt.fits[sky=mask(mask_file)]" filtered_evt.fits unix% dmcopy "img.fits[sky=mask(mask_file)]" filtered_img.fits
The same syntax (and mask) is used to filter both tables (eg event files) and images.
unix% dmcopy "img.fits[exclude sky=mask(mask_file)]" filtered_img.fits unix% dmcopy "img.fits[filter sky=mask(mask_file)]" filtered_img.fits
The "exclude" directive inverts the mask logic. The rarely used "filter" directive can also be used; this is the default behavior.
Note that excluded masks are stored in the file subspace with the mask values inverted.
unix% dmcopy "evt.fits[ccd_id=3,sky=mask(mask_file)]" out.fits
Mask filters can be combined with other column filters.
unix% dmcopy "evt.fits[(x,y)=mask(mask_file)]" out.fits
Mask filters can be applied to vector columns ('sky') or any arbitrary pair of columns.
unix% dmcopy "evt.fits[sky=bounds(mask(mask_file))]" out.fits
The "bounds()" directive can be used with masks to generate a rectangular region around the edge of the mask file.
unix% dmcopy "evt.fits[sky=mask(mask_file),sky=region(ciao.reg)]" out.fits unix% dmcopy "evt.fits[sky=mask(mask_file),sky=circle(1000,1000,20)]" out.fits
Mask filters and region filters can be used concurrently.
unix% dmcopy "evt.fits[sky=mask(mask_file),x=1000:4000]" out.fits
Mask filters can also be used with range filters.
unix% dmcopy "evt.fits[sky=mask(first_file)||sky=mask(second_file)]" out.fits unix% dmcopy "evt.fits[sky=mask(first_file),sky=mask(second_file)]" out.fits
Mask filters on the same column can be combined using the logical-and (",") and logical-or ("||") operators.
unix% dmcopy "evt.fits[sky=mask(sky_mask),(time,pi)=mask(another_mask)]" out.fits
Separate mask filters can be applied to multiple columns. Masks are not restricted to spatial axes.
unix% dmcopy "evt.fits[sky=mask(mask_file)+region(ciao.reg)]" out.fits ERROR unix% dmcopy "evt.fits[sky=mask(mask_file)+circle(4095,4096,100)]" out.fits ERROR unix% dmcopy "evt.fits[sky=field()-mask(mask_file)]" out.fits ERROR
Masks cannot be logically combined with regions using the region logic operators.
unix% dmcopy "evt.fits[sky=mask(mask_file)][exclude sky=circle(4095,4096,100)]" out.fits ERROR unix% dmcopy "evt.fits[exclude sky=mask(mask_file)][sky=circle(4095,4096,100)]" out.fits ERROR
Using both include and exclude filters is not supported. This is true for any/all DM filters.
unix% dmcopy "evt.fits[sky=mask(sky_mask),(x,time)=mask(another_mask)]" out.fits ERROR
A single column, eg 'x', can only be used with one mask at a time.
How to Create a Mask
Masks are simply 2D images, so any CIAO tools can be used. Some common examples are shown below.
dmimgthresh may be used on an exposure map to create a mask that provides the footprint (field of view) of the observation.
unix% dmimgthresh exposure_map.fits mask.fits cut=10% value=0 clob+
The cutoff value can be expressed in percentage of the max or as an absolute value. Note: dmimgthresh replaces values that are strictly less than cutoff value. Since all the mask filtering logical deals with is pixel values = 0 or those not equal to 0, there is no need to replace the values > the 10% cutoff, however, users wishing to can continue using dmimgthresh
unix% dmimgthresh mask.fits mask_0s_and_1s.fits cut=:10% value=1 clob+
The output image will now contain values equal to 0, 1 (and possibly NaN's if the original input image contained NaN values).
Another common technique will be to create an image that is all 1's or all 0's and then use DM region filters to include/exclude pixels.
unix% dmimgcalc counts.fits none all_ones.fits op="imgout=((img1-img1)+1)" unix% dmcopy "all_onts.fits[sky=region(ciao.reg)][opt full]" mask.fits
using [opt full] will preserve the size image. A mask file can be more efficient and faster when dealing with a large number of individual sources (shapes). It is also trivial to invert the mask by simply subtracting it from 1.
unix% dmimgcalc mask.fits none inverted_mask.fits op="imgout=(1-img1)"
One advantage of using the tools in this way is that the WCS will automatically be preserved.
Crates allows for easy manipulation of individual mask pixels. When using crates it is often easiest to start with a donor image file and then use crates to read and python to modify the pixel values. This way the image size is already established along with the necessary WCS.
>>> from pycrates import read_file >>> import numpy as np >>> img = read_file("img.fits") >>> orig_pix_vals = img.get_image().values >>> zeros = np.zeros_like( orig_pix_vals ) >>> zeros[999,999] = 1 >>> img.get_image().values = zeros >>> img.write("delta.fits", clobber=True)
The output image has all the pixel values replaced with a delta function at pixel 1000,1000 (in image/logical coordinates). Numpy arrays are 0-based indexed, so to get pixel 1000,1000 in logical coordinates you subtract 1. Also note that Numpy arrays are accessed as [y,x] so setting: zeros[199,49]=1, would set logical pixel(50,200)=1.
Since only the pixel values were replaced, the WCS (if any) is preserved. This method can only be used when replacing the pixel value array with an array of the same size and shape. Using a different size/shape array will invalidate the WCS.
How to Visualize a Mask
As masks are simply 2D images, any visualization software should be able to view them.
SAOImage ds9 can directly display the binary mask by specifying the block name
unix% ds9 "evt.fits[mask]"
and just like other images, the mask and counts data can both be loaded into separate frames and users can alternate/blink between them.
unix% ds9 evt.fits "evt.fits[mask]" -blink
When working with images (not event files), ds9 also has support for overlaying a mask with adjustable transparency
unix% ds9 img.fits -mask color green -mask "img.fits[mask]"
The transparency of the mask can be adjusted by going to Analysis -> Mask Parameter ... Technically, the ds9 mask functionality does also work for event files, but the mask must be full resolution (so for ACIS, would be an 8k x 8k image).
How to Manipulate a Mask
Depending on how the mask is created it may be desirable to adjust the mask pixel values. As these are just 2D images, any tools that operate on images can be used. Below are some common operations that may be performed.
As shown above, one simple way to modify a mask file is to filter the pixel values. Using [opt full] retains the original size of the mask which is often desireable.
unix% dmcopy "mask.fits[sky=region(ds9.reg)][opt full]" filt_mask.fits
This is only useful to change non-zero pixels to 0.
Since masks can be any datatype, it is possible apply thresholds to an image to create different masks. One example of an exposure map was shown above.
unix% dmimgthresh expmap.fits mask.fits cut=10% value=0
Another example may be to select a mask from a map such as is produced by wavdetect (the "cell image"), dmimgblob, or dmnautilus. Each of these tools produce a map where the integer pixel value are used to group pixels together. The values usually go from 1 to N where N is the total number of groups. To turn the map into a mask then requires applying a threshold at the desired group number. A simple example might look like
unix% aconvolve img.fits sm_img.fits "lib:gaus(2,5,5,3,3)" method=fft unix% dmimgblob sm_img.fits map.fits threshold=0.5 srconly=yes unix% dmimgthresh map.fits mask.fits cut=5:5 value=0
dmimgblob identifies groups of connected pixels above the threshold in the smoothed input image and outputs a map that provides the groupings. A single group can be selected with dmimgthresh by setting the upper and lower limits to be the same and setting value=0; so all pixels not associated with, in this example, group #5 would be excluded.
Masks can be fine-tuned using common image processing techniques.
An image can be "dilated" using the "max" function in dmimgfilt
unix% dmimgfilt img.fits mask.fits function=max mask="box(0,0,3,3)"
which increases the size of the include area by 1 pixel in all directions. Similarly, the image can be "eroded" using the "min" function
unix% dmimgfilt img.fits mask.fits function=min mask="box(0,0,3,3)"
which decreases the included area by 1 pixel in all directions.
These basic gray-scale morphological operations can be especially useful when used together. For example if the mask contains many isolated points and the desire is to create a mask that includes the entire area, then the "closing" operation can be achieved by dilating then eroding the image
unix% dmimgfilt img.fits - max "box(0,0,3,3)" numiter=10 | \ dmimgfilt - mask.fits min "box(0,0,3,3)" numiter=10
The "closing" operation acts to fill in small holes in the mask while retaining the original size and outline. The reverse can also be done by "opening" the image
unix% dmimgfilt img.fits - min "box(0,0,3,3)" numiter=10 | \ dmimgfilt - mask.fits max "box(0,0,3,3)" numiter=10
which is useful when the goal is to increase the excluded regions while retaining the original outline.
Mask images can be logically combined using ordinary math operations available in dmimgcalc.
To create the union of two masks use the summation, sum, operation. (Note: this assumes that the pixel values are positive!).
unix% dmimgcalc mask1.fits mask2.fits mask_union.fits sum
If the pixel values could be negative, then use the full calculator expression:
unix% dmimgcalc mask1.fits,mask2.fits none mask_union.fits op="imgout=fabs(img1)+fabs(img2)"
To create the intersection of two masks use the multiply, mul, operation
unix% dmimgcalc mask1.fits mask2.fits mask_intersect.fits mul
To combine more than two, the full dmimgcalc operation expression can be used
unix% dmimgcalc m1,m2,m3,m4 none mask_intersect.fits op="imgout=(img1*img2*img3*img4)" unix% dmimgcalc m1,m2,m3,m4 none mask_union.fits op="imgout=(img1+img2+img3+img4)"
or, if the mask pixel values are >=0 then stacks can be used with dmimgfilt using the "min" function for intersection and "max" function for union.
unix% dmimgfilt @img.stk mask_intersect.fits fun=min mask="point(0,0)" unix% dmimgfilt @img.stk mask_union.fits fun=max mask="point(0,0)"
where "point(0,0)" tells dmimgfilt to use the same point from each image in the stack when applying the filter.
Summary of Mask Filters Properties
The mask file can be any 2D image (integer or floating-point). Pixels value equal to 0 are excluded, pixel values not equal to 0 are included. Negative pixel values, <0, are not equal to 0 and are also included. Pixel locations outside the mask are always excluded. Regardless of the input dataype, masks are always stored as single Byte type images in the output subspace.
If the file being filtered has a World Coordinate System (WCS), and the mask file has a WCS, then the two must be equivalent. That is when filtering images the pixel size must be same and have the same projection parameters (eg tangent point). If the mask or the file being filtered do not contain a WCS then then no consistency checking is performed.
Any pair of image axes (table columns) can be filtered with masks; masks are not restricted to just spatial filtering.
Filtering Tables vs. Images
Images and tables (eg event files) are filtered using the same syntax. For real-valued table columns rows that have values within +/- 0.5 of the pixel center are filtered with the pixel value.
NaN's and NULL's
Mask pixels with NaN or integer NULL values in the mask are treated as 0. All IEEE special values, eg Inf, are treated as 0.
The mask filter is stored as part of the data subspace. The mask is always stored as an image in a MASK extension. The Byte type mask will only contain pixel values = 0 or 1.
unix% dmlist evt.fits blocks -------------------------------------------------------------------------------- Dataset: evt.fits -------------------------------------------------------------------------------- Block Name Type Dimensions -------------------------------------------------------------------------------- Block 1: PRIMARY Null Block 2: EVENTS Table 15 cols x 54 rows Block 3: GTI3 Table 2 cols x 3 rows Block 4: GTI2 Table 2 cols x 2 rows Block 5: GTI1 Table 2 cols x 2 rows Block 6: GTI0 Table 2 cols x 2 rows Block 7: MASK Image Byte(2634x2630) % dmlist evt.fits subspace -------------------------------------------------------------------------------- Data subspace for block EVENTS: Components: 4 Descriptors: 16 -------------------------------------------------------------------------------- --- Component 1 --- ... 8 sky Real4 TABLE MASK MASK(MASK) Field area = 6.71089e+07 Region area = 398 ...
The WCS, if any, is saved with the mask. Other header keywords in the original mask file are not saved.
The bounds() operation returns a rectangle that encloses the sub-image of the mask that is non-zero.
The inside-outside logic for pixels in the mask is inverted when the [exclude column=mask(file)] syntax is used. Pixels outside the mask are always excluded irrespective of the [exclude ] directive.
If a filter is applied to an image with a mask, both the image and mask will be cropped to the size of the bounding-box around the filter.
unix% dmcopy "img.fits[sky=mask(mask.fits)]" a.fits unix% dmcopy "a.fits[sky=region(ciao.reg)]" b.fits
The output image in b.fits will be cropped to the bounding box around the region file. Simultaneously, the MASK extension will be cropped to the same bounding box.
When event files (tables) containing masks are merged with dmmerge, the masks will be combined. The output will be the union of the masks from the files in the input stack.
unix% dmcopy "evtA[sky=mask(a.mask)]" a.evt unix% dmcopy "evtB[sky=mask(b.mask)]" b.evt unix% dmmerge a.evt,b.evt merged.evt
Masks are automatically included in the calculation of the area by tools like dmlist and dmextract.
Comparison to Regions
Masks cannot be used with region operators "+", "-", "!", "*", "&", and "|". While this command
unix% dmcopy "img.fits[sky=mask(mask.fits)®ion(ciao.reg)]" out.fits ERROR
does not work, this command
unix% dmcopy "img.fits[sky=mask(mask.fits),sky=region(ciao.reg)]" out.fits
The ciao tools that regrid images and event files do not update the coordinates for regions in the subspace including the mask subspaces. This includes dmregrid, dmregrid2, reproject_image, and reproject_image_grid. Users should regrid their data before filtering.
CIAO tools that modify the WCS do not automatically update the WCS of the MASK extensions. This includes wcs_update which is run by reproject_aspect. Users should apply any astrometric corrects before filtering.
dmmerge will attempt to merge the region subspaces, including masks, when combining event files. However, other tools that combine data from multiple input files generally do not attempt to combine (union or intersect) the subspace information. For example using dmimgcalc to sum image does not automatically combine the regions|masks.
Multiple FITS Blocks
Users trying to append extra blocks onto a file (eg with dmmappend) should carefully check the output. The results when appending two blocks, each with different masks is error prone.
Masks should not be used to filter two arbitrary axes within a 3D or higher dimension images.
The mask created when merging multiple event files may be incorrect if the input files have multiple subspaces (ie multiple GTI blocks). Users should apply mask filters after the files are dmmerge'ed together.
- coords, level, pileup, times
- autoname, ciao, ciao-install, ciaorc, history, parameter, stack, subspace
- dm, dmascii, dmbinning, dmfiltering, dmopt, dmregions
- convert_ds9_region_to_ciao_stack, dither_region, dmcontour, dmgroupreg, dmimgdist, dmimgfilt, dmimghull, dmimglasso, dmmakereg, get_src_region, mkbgreg, mksubbgreg, rank_roi, regphystocel, roi, splitroi, tg_create_mask