Last modified: 11 October 2018

Why is there data being plotted or fit outside the energy range I noticed?

In the following example, we notice energies in the 0.3 to 7.0 keV range, but when the we plot the data, there is a data point at almost 9 keV. Why?

sherpa> load_data("out.pi")
read ARF file out.arf
read RMF file out.rmf
sherpa> group_counts(10)
sherpa> notice(0.3,7.0)
sherpa> plot_data()

The answer is because the data have been group'ed.

It is important to recognize that there are two distinct spectral axes: energy (usually keV) and detector channel (PI). When you group() your data, this is applied to the data on the detector channel grid. When you notice() an energy range, this is applied to the energy grid the model is evaluated over. The RMF provides the mapping between two grids.

The model is only ever evaluated on the energy grid within the notice'ed energy range. The fit() is only done over the notice()'d energy range.

What is being plotted are the group'ed data. More specifically, the plot shows the middle of each group. As is often the case, in this example the final (right most) group has some channels that correspond energies inside the noticed energy range as well as some channels that correspond to energies outside the noticed energy range. Since the last group includes some channels that are included in the fit, the group -- the entire group -- is shown on the plot.

You can inspect the group boundaries by looking at the data's grouping flags together with the nominal energy-to-channel relation contained in the RMF files "EBOUNDS" extension.

More information

A full discussion of spectral file formats and conventions is beyond the scope of this FAQ. Interested readers can review the documents "The OGIP Spectral File Format" and "The Calibration Requirements for Spectral Analysis"

Sherpa stores the grouping information for a dataset in the arrays: grouping and quality. Groups are encoded as a sequence of values: 1 (start of a new group), -1 (continuation of the group), and 0 (ungrouped). We can use this to get the channel that corresponds to the beginning of each group.

sherpa> mydata = get_data()
sherpa> start_of_groups = np.where( mydata.grouping == 1 )

To map channel to energy we can then use the EBOUNDS information from the RMF (the RMF was automatically loaded with the data).

sherpa> myrmf = get_rmf()
sherpa> groups_in_keV = myrmf.e_min[start_of_groups]
sherpa> print(groups_in_keV)
[  7.30000017e-03   2.33600006e-01   2.62800008e-01   3.06600004e-01
   3.94199997e-01   4.67200011e-01   5.25600016e-01   5.83999991e-01
   6.13200009e-01   6.42400026e-01   6.71599984e-01   7.00800002e-01
   7.44599998e-01   8.02999973e-01   8.32199991e-01   8.75999987e-01
   9.34400022e-01   9.92799997e-01   1.02199996e+00   1.06579995e+00
   1.10959995e+00   1.16799998e+00   1.24100006e+00   1.29939997e+00
   1.38699996e+00   1.44540000e+00   1.54760003e+00   1.66439998e+00
   1.75199997e+00   1.91260004e+00   2.04399991e+00   2.45280004e+00
   2.73020005e+00   3.29959989e+00   3.66459990e+00   4.49679995e+00
   6.80359983e+00   1.09354000e+01   1.47606001e+01]

From this we see that the final plotted group is the one that starts at a channel that corresponds to 6.8keV, and extends upto a channel that corresponds to 10.9 keV. There is also one group that is not shown on the plot because it does not contain any channels within the noticed energy range. Since users may notice different energy ranges during their analysis session, the notice command does not change the grouping flags of the dataset.

Therefore, even though there is a data point at ~9keV, it only contains data from the group with channels less than 7keV and the model itself is only evaluated upto 7keV.

Users should be aware that this is also happening with the first group that is being plotted. That group extends from 0.2628keV to 0.3066keV, which has a mid-group energy=0.28keV which is also outside the noticed energy range. It is just that often the groups at lower energies are smaller so it is more difficult to visually identify this behavior. The same discussion above applies to this end of the spectrum.