next up previous
Next: 4. Sherpa Optimization Methods Up: Sherpa Reference Manual Previous: 2. Sherpa Display

Subsections


3. Sherpa Filtering


3.1 Introduction

Filtering within Sherpa may be performed using one or more of the following means:


3.2 Filtering with IGNORE | NOTICE Commands

The IGNORE and NOTICE commands specify data-portions to be included or excluded, for 1- and 2-D datasets. The syntax for the commands is:

sherpa> {IGNORE | NOTICE} $ [$#$ ]$ {$ <$arg$ >$ | FILTER $ <$filterExpr$ >$}

where # specifies the number of the dataset on which the command will act (default dataset number is 1). To specify an entire dataset or image, $ <$arg$ >$ is ALL. To specify pha channels with quality flags different than 0, $ <$arg$ >$ is BAD (this argument is valid only with the IGNORE command).

See Sections 1.44 and 1.55 for examples.


3.2.1 Interactive Filtering from a 1-D Plot

To interactively specify a filter region from a plot data display, the IGNORE or NOTICE command argument $ <$arg$ >$ is PLOT:

  1. Plot the data (see the LPLOT command, Section 1.51).
  2. Issue the command IGNORE $ [$#$ ]$ PLOT or NOTICE $ [$#$ ]$ PLOT.
  3. Place the cursor on the plot, at the desired minimum x-axis value for the filter, and click once with the left mouse button.
  4. Place the cursor at the desired maximum x-axis value, and click again with the left mouse button. This will set the data filter to exclude or include the marked region of the dataset. Note that the character e may be used as an alternative to clicking the left mouse button.

Note: If you are interactively filtering on any dataset other than dataset number one, you must specify the dataset number when issuing the command IGNORE # PLOT or NOTICE # PLOT. Otherwise, the filter you specify from the data display will be applied to dataset number 1.

Please see the LPLOT command and Chapter 2 for further information regarding plotting capabilities within Sherpa .

See Sections 1.44 and 1.55 for additional examples.

Examples:

  1. Exclude a 1-D data range, specified from a plot display: 
    sherpa> READ DATA example.dat
sherpa> LP DATA
sherpa> NOTICE ALL
sherpa> IGNORE PLOT
<left-mouse-click once at the desired minimum x-axis value>
<and then left-mouse-click again at the desired maximum x-axis value>
sherpa> LP FILTER
sherpa> LPLOT DATA
    In this example, 1-D data is first displayed. Then the data filter is set to include the entire dataset, with the command NOTICE ALL. The command IGNORE PLOT readies the cursor for selecting the desired filter region, and the user should then left-mouse-click first at the desired minimum x-axis location and then again at the maximum x-axis location. After the second left-mouse-click, the Sherpa command prompt is returned. The command LPLOT FILTER will then plot the data region that you have marked for exclusion. Finally, the dataset is plotted again, with the command LPLOT DATA, showing that the selected regions have been properly excluded.


3.2.2 Interactive Filtering from a 2-D Image

To interactively specify a filter region from an image data display, the IGNORE or NOTICE command argument $ <$arg$ >$ is IMAGE:

  1. Display the image and mark the filter region(s) (see the IMAGE command, Section 1.45).
  2. Issue the command IGNORE $ [$#$ ]$ IMAGE or NOTICE $ [$#$ ]$ IMAGE. This will set the data filter to exclude or include the marked region of the dataset.

Note: If you are interactively filtering on any dataset other than dataset number one, you must specify the dataset number when issuing the command IGNORE # IMAGE or NOTICE # IMAGE. Otherwise, the filter you specify from the data display will be applied to dataset number 1.

Also note: If you re-display an image, following filtering, the entire image will be displayed, even though the filter has been applied to the data.

Please see the IMAGE command and Chapter 2 for further information regarding imaging capabilities within Sherpa .

See Sections 1.44 and 1.55 for additional examples.

Examples:

  1. Exclude a 2-D region, specified from an image display: 
    sherpa> READ DATA example_img.fits
sherpa> IMAGE DATA
sherpa> NOTICE ALL
<mark regions for exclusion>
sherpa> IGNORE IMAGE
sherpa> IMAGE FILTER
    In this example, 2-D image data is first displayed. Then the data filter is set to include the entire image, with the command NOTICE ALL. Next, the user chooses regions, by placing markers on the displayed image. The command IGNORE IMAGE sets the data filter to exclude the marked regions. The command IMAGE FILTER will display the image data regions that have been marked for exclusion.


3.2.3 1-D Filter Expressions - Syntax Option A

To specify a data filter using a filter expression, there are two different syntax options available for the IGNORE or NOTICE command argument $ <$filterExpr$ >$:

For Syntax Option A: This syntax is a list of data value ranges, and/or specific data values, combined using logical operators. Data value ranges are formed using either a data column name and comparison symbols, or the colon range syntax:

$ <$filterExpr$ >$ $ =$ $ <$colname$ >${$ <$symbol$ >$# | $ =$$ <$range$ >$} $ \backslash$
$ [$$ <$operator$ >$ $ <$colname$ >${$ <$symbol$ >$# | $ =$$ <$range$ >$} ...$ ]$

where $ <$colname$ >$ is the name of an appropriate data column. Note that columns must have assigned names when the data are read into Sherpa , in order to use this syntax option. These column names for a dataset may be viewed using the SHOW DATA command.

The following tables show the available comparison symbols, $ <$symbol$ >$, and logical operators, $ <$operator$ >$:

Table 3.1: 1-D Filter Expression Symbols - Syntax Option A
$ <$symbol$ >$ Description
$ >$ greater-than
$ >$$ =$ greater-than or equal-to
$ <$ less-than
$ <$$ =$ less-than or equal-to
!$ =$ not equal-to
$ =$ equal-to
   

Table 3.2: 1-D Filter Expression Operators - Syntax Option A
$ <$operator$ >$ Description
& and
$ \wedge$ exclusive or
| inclusive or
   

With this syntax, # is a data value being specified. Or, a range may be specified with one of the following colon range syntax options:

Table 3.3: 1-D Filter Expression Range Syntax - Syntax Option A
$ <$range$ >$ Description
value1:value2 from numerical data value1 to value2, inclusive
#: all data values greater-than or equal-to #
:# all data values less-than or equal-to #
# all data values equal-to #

Examples: 

  1. sherpa> READ DATA "example_bin.fits[columns x1, x2]"
sherpa> NOTICE ALL
sherpa> IGNORE FILTER x1=4 | x1=8: | x1=1:3
    The first command inputs a data file containing named columns, and then all data points are included with NOTICE ALL. The filter expression then specifies column x1 data values equal-to 4, greater-than or equal-to 8, or 1 thru 3, inclusive. This filter expression is equivalent to Example 1 of Subsection 3.2.4, when Axis 0 is named x1. The following commands are also equivalent: 
    sherpa> IGNORE FILTER x1=4 | x1>=8 | x1=1:3
sherpa> IGNORE FILTER x1=4 | x1>=8 | (x1>=1 & x1<=3)
    Note that expressions with logical operators can be grouped using parentheses, for control over logical order of operations. 

  2. sherpa> IGNORE ALL
sherpa> NOTICE FILTER x1=:2 | x1=8
    This filter expression specifies column x1 data values less-than or equal-to 2, or equal-to 8. This filter expression is equivalent to Example 2 of Subsection 3.2.4, when Axis 0 is x1. The command 
    sherpa> NOTICE FILTER x1<=2 | x1=8
    is also equivalent. 

  3. sherpa> NOTICE ALL
sherpa> IGNORE FILTER x1=:2 | x1=7:
    This filter expression specifies column x1 data values less-than or equal-to 2, or greater-than or equal-to 7. This filter expression is equivalent to Example 3 of Subsection 3.2.4, when Axis 0 is x1. The command 
    sherpa> IGNORE FILTER x1<=2 | x1>=7
    is also equivalent. 

  4. sherpa> IGNORE ALL
sherpa> NOTICE FILTER x1>5 & x1<8
    This filter expression specifies column x1 data values greater-than 5 and less-than 8 (i.e. 6 thru 7 inclusive). This filter expression is equivalent to Example 4 of Subsection 3.2.4, when Axis 0 is x1. The following commands are also equivalent: 
    sherpa> NOTICE FILTER x1=6:7


3.2.4 1-D Filter Expressions - Syntax Option B

To specify a data filter using a filter expression, there are two different syntax options available for the IGNORE or NOTICE command argument $ <$filterExpr$ >$:

For Syntax Option B: This syntax is a comma-delimited list of data value ranges, and/or specific data values, for Axis 0. Data value ranges are formed using a colon, :.

$ <$filterExpr$ >$ $ =$ $ <$range$ >$ $ [$, $ <$range$ >$, ...$ ]$

where $ <$range$ >$ is one of the following:

Table 3.4: 1-D Filter Expression Range Syntax - Syntax Option B
$ <$range$ >$ Description
value1:value2 from numerical data value1 to value2, inclusive
#: all data values greater-than or equal-to #
:# all data values less-than or equal-to #
# all data values equal-to #
   

Examples: 

  1. sherpa> READ DATA "example_bin.fits[columns x1, x2]"
sherpa> NOTICE ALL
sherpa> IGNORE FILTER 4, 8:, 1:3
    The first command inputs a data file containing two columns, and then all data points are included with NOTICE ALL. This filter expression specifies Axis 0 data values equal-to 4, greater-than or equal-to 8, or 1 thru 3, inclusive. This filter expression is equivalent to Example 1 of Subsection 3.2.3, when Axis 0 is x1. 

  2. sherpa> IGNORE ALL
sherpa> NOTICE FILTER :2, 8
    This filter expression specifies Axis 0 data values less-than or equal-to 2, or equal-to 8. This filter expression is equivalent to Example 2 of Subsection 3.2.3, when Axis 0 is x1. 

  3. sherpa> NOTICE ALL
sherpa> IGNORE FILTER :2, 7:
    This filter expression specifies Axis 0 data values less-than or equal-to 2, or greater-than or equal-to 7. This filter expression is equivalent to Example 3 of Subsection 3.2.3, when Axis 0 is x1. 

  4. sherpa> IGNORE ALL
sherpa> NOTICE FILTER 6:7
    This filter expression specifies Axis 0 data values greater-than 5 and less-than 8 (i.e. 6 thru 7 inclusive). This filter expression is equivalent to Example 4 of Subsection 3.2.3, when Axis 0 is x1.


3.2.5 2-D Filter Expressions

Filter expressions for 2-D data utilize filter descriptors that are combined using logical operators. Data value ranges are formed as a comma-delimited list, contained within parantheses as part of the filter descriptor.

$ <$filterExpr$ >$ $ =$ $ [$$ <$colname1$ >$,$ <$colname2$ >$ $ =$$ ]$ $ <$descriptor$ >$ $ \backslash$
$ [$$ <$operator$ >$ $ <$descriptor$ >$ ...$ ]$

where $ <$colname1$ >$ and $ <$colname2$ >$ are the names of appropriate data columns. Note that specifying column names is not required; if not specified, Axis 0 ($ <$colname1$ >$$ =$x1) and Axis 1 ($ <$colname2$ >$$ =$x2) are the defaults. The following tables show the available filter descriptors, $ <$descriptor$ >$, and logical operators, $ <$operator$ >$:

Table 3.5: 2-D Filter Expression Descriptors
$ <$descriptor$ >$ Description
BOX(x,y,width,height) x and y are the center coordinates of the box, width is twice the distance along the first axis from the center to one edge, and height is twice the distance along the second axis from the center to the edge.
BOX(x,y,width,height,angle) x and y are the center coordinates of the box, width is twice the distance along the first axis from the center to one edge, height is twice the distance along the second axis from the center to the edge, and angle is the counter-clockwise rotation angle in degrees, relative to the first axis.
CIRCLE(x,y,radius) x and y are the center coordinates of the circle, and radius is the distance from the center to the edge of the circle.
ELLIPSE(x,y,width,height) x and y are the center coordinates of the ellipse, width is the distance along the first axis from the center to the edge, and height is the distance along the second axis from the center to the edge.
ELLIPSE(x,y,width,height,angle) x and y are the center coordinates of the ellipse, width is the distance along the first axis from the center to the edge, height is the distance along the second axis from the center to the edge, and angle is the counter-clockwise rotation angle in degrees, relative to the first axis.
POINT(x,y) x and y are the coordinates of a specific point.
POLYGON(x1,y1,x2,y2,x3,y3$ [$,...,xn,yn$ ]$) Each x,y pair specifies one vertex of a polygon, up to n=32 vertices.
PIE(x,y,start,stop) x and y are the apex coordinates for a filter in the shape of a pie wedge. The wedge is swept out starting at the angle start and stopping at the angle stop. The angles are given in degrees, increasing in the counter-clockwise direction, relative to the first axis.
   

Table 3.6: 2-D Filter Expression Operators
$ <$operator$ >$ Description
& and
$ *$ and
$ -$ and not
$ \wedge$ exclusive or
| inclusive or
$ +$ inclusive or
   

Examples: 

  1. sherpa> DATA example_img.fits
sherpa> NOTICE ALL
sherpa> IGNORE FILTER x1,x2 = BOX(250,250,50,50)
    This filter expression specifies a box of width 50, height 50, and center at x1,x2 = 250,250. The command 
    sherpa> IGNORE FILTER BOX(250,250,50,50)
    is also equivalent.

    \includegraphics{plots/box.ps} 

  2. sherpa> NOTICE ALL
sherpa> IGNORE FILTER BOX(250,250,50,50) & CIRCLE(275,225,36)
    This filter expression specifies an area enclosed both by the specified box and a circle of radius 36 centered at 275,225. The result is a pie wedge.

    \includegraphics{plots/box_and_circle.ps} 

  3. sherpa> NOTICE ALL
sherpa> IGNORE FILTER BOX(250,250,50,50) | CIRCLE(275,225,36)
    This filter expression specifies an area enclosed either by the specified box or the specified circle.

    \includegraphics{plots/box_or_circle.ps} 

  4. sherpa> NOTICE ALL
sherpa> IGNORE FILTER BOX(250,250,50,50) - CIRCLE(275,225,36)
    This filter expression specifies an area enclosed by the specified box, and not of a circle of radius 36 centered at 275,225.

    \includegraphics{plots/box_andnot_circle.ps}


3.3 Filtering with an Input Filter File

A file that contains filter assignments for each data point may be input using the READ command. The filter assignment value should be either 1 for inclusion, or 0 for exclusion of the data point. The syntax of the READ command for an input filter file is:

sherpa> READ FILTER $ [$#$ ]$ $ <$filespec$ >$

See the READ command for the $ <$filespec$ >$ definition and list of accepted file formats.

Examples:

  1. Input ascii data and filter files: 
    sherpa> READ DATA example1.dat 1 2
sherpa> READ FILTER example1.dat 1 3
sherpa> SHOW FILTER
    The first command reads columns 1 and 2 of the ascii data file example1.dat, as dataset number 1. Column 3 of example1.dat contains a filter assignment for each of the data points (1 for the data point to be included; 0 for the data point to be excluded). Current filter assignments for each data point can be reported with the command SHOW FILTER.


3.4 Filtering Using Input Data Manipulation Syntax

The CIAO Data Manipulation (DM) software allows data to be filtered during the input/output process. As part of the CIAO software package, Sherpa may utilize this capability. A short description of the Data Manipulation filtering concept and syntax is given herein. For a more detailed description of the Data Manipulation software please see the Data Manipulation Guide.

In any CIAO software package program using the DM interface (including Sherpa ), the user can pass a ``virtual file'' where an input file is expected. The virtual file is a file that is filtered (and/or binned) from an original data file, during run-time. For example, given a large 32K square HRC image, the user may wish to input only a subset of that file, with say x values within only a certain range. Using the DM filtering syntax, the user may input this filtered file, without first creating an intermediate disk file. That is, Sherpa will only ``see'' the ``virtual'' filtered file that has been specified. With the DM software, the virtual file is implicitly created from a true disk file by filtering or binning, but does not get created on disk or even formatted into a UNIX pipe. Instead, I/O calls in the program return records of the virtual file by internally accessing the actual disk file and figuring out which data passes the filter.

The full syntax for specifying a DM filter is a filename followed by a filtering and/or binning command argument, $ <$virtual_file_syntax$ >$. This command argument consists of a series of optional qualifiers within brackets, $ [$ $ ]$:

filename[block][filter][columns][binning]

Any qualifier my be omitted, but those which are used must appear in the order given above.

DM syntax may be used with the following file types:

$ <$filetype$ >$: Extension: Description:
FITS fits 2-D fits image
FITSIMAGE fits 2-D fits image
FITSBIN fits 1-D fits binary table
IMH imh 2-D IRAF image
QP qp 2-D PROS qpoe

See the READ command for information regarding the Sherpa data input syntax for these file types.

Examples:

  1. Input various data columns from fits binary data files: 
    sherpa> READ DATA 2 "example_bin.fits[2][cols TIME, EXPNO]" FITSBIN
sherpa> READ DATA 3 "example_bin.fits[EVENTS][cols time, expno]" FITSBIN
    The first command reads columns time and expno, from the second extension of the fits binary table file example_bin.fits, as dataset number 2. The second command reads columns time and expno, from the EVENTS extension of the fits binary table file example_bin.fits, as dataset number 3. Note that columns may be specified by case insensitive name. Also, the fits extension can be specified by either the number or the name of the extension (in this example, the name of the second extension is EVENTS).

  2. Input a portion of a 2-D fits image data file: 
    sherpa> READ DATA "example_img.fits[#1=100:200, #2=100:400]" FITS
    This command reads the specified portion (from Axis 0 coordinates 100 to 200 and from Axis 1 coordinates 100 to 400) of the 2-D fits image example_img.fits. Note that, by default, the data is taken from the first fits block for which NAXIS is nonzero. The following command is equivalent: 
    sherpa> READ DATA "example_img.fits[100:200, 100:400]" FITS

  3. Input various data columns from a fits binary data file: 
    sherpa> READ DATA 1 "example_bin.fits[2][columns #1, #2]" FITSBIN
    This command reads the first two columns from the second extension of the fits binary table file example_bin.fits, as dataset number 1. Note that column numbers or names must always be specified when reading fits binary table files. The following commands are equivalent: 
    sherpa> READ DATA 1 "example_bin.fits[2][columns #1, #2]" FITSBIN
sherpa> READ DATA 1 "example_bin.fits[2][cols #1, #2]" FITSBIN
sherpa> DATA "example_bin.fits[2][cols #1, #2]" FITSBIN

  4. Input a 2-D image by binning columns from a qpoe data file: 
    sherpa> DATA "example.qp[bin time=32, pha=32]" FITSIMAGE
    This command creates and inputs into Sherpa an image, using the pha and time columns of of the qpoe file example.qp. In this example bin sizes of 32 are used, which will in this case reduce an 8192x8192 image to 256x256. Note that the FITSIMAGE argument is required by the Sherpa READ command syntax (since it is ultimately a fits image that is being input to Sherpa ).

  5. Input a 2-D image by binning columns from a fits binary data file: 
    sherpa> DATA "example_bin.fits[bin chipx, chipy]" FITSIMAGE
    This command inputs into Sherpa the fits binary table example_bin.fits, but bins the table to create an image using the chipx and chipy columns. Note that the FITSIMAGE argument is required by the Sherpa READ command syntax.

  6. Input a 2-D image by filtering and binning a column, from a fits binary data file: 
    sherpa> DATA "example_bin.fits[bin chipx=200:400:4, chipy=300:400:4]" FITSIMAGE
    This command creates and inputs into Sherpa , an image using the chipx and chipy columns of the binary fits table file example_bin.fits. In this example, ranges for the axes, and bin sizes, are given. The resultant image size is restricted. Note that the FITSIMAGE argument is required by the Sherpa READ command syntax. Note however, that the following command is not equivalent: 
    sherpa> DATA "example_bin.fits[chipx=200:400:4, chipy=300:400:4][bin chipx, chipy]" $\backslash$ \\ 
FITSIMAGE
    since this command does not restrict the overall size of the resultant image.

  7. Input a 2-D image by filtering and binning columns, from a fits binary data file: 
    sherpa> READ DATA "example_bin.fits[EVENTS][time=37277062.1:37277062.2] $\backslash$ \\
[bin X=16350:16420:1,Y=16350:16420:1]" FITSIMAGE
    This command creates and inputs into Sherpa , an image using the X and Y columns of the binary fits table file example_bin.fits. In this example, filtering is performed with two columns, for filtering both spatially and in time. The spatial subspace is the subset of the event file for X between 16350 and 16420 and Y in the same range, binned in steps of 1. The time range is from 37277062.1 to 37277062.2 seconds. Note that in this case, the order in the syntax is important. This command first goes to the events block, next filters the data in time, and then makes an image by binning over the requested X and Y region. You can't filter after binning in the same operation. In addition, once the binning is done, the data no longer contains the time information. Note that the FITSIMAGE argument is required by the Sherpa READ command syntax.

next up previous
Next: 4. Sherpa Optimization Methods Up: Sherpa Reference Manual Previous: 2. Sherpa Display
cxchelp@head.cfa.harvard.edu