Chandra Data Reprocessing

The job of turning telemetry from Chandra into images that a scientist can analyze and the average person can admire falls to the Data Systems group under the direction of Giuseppina (Pepi) Fabbiano, with major help from other groups, especially the Science Data Systems and Science Operations Team.

The day begins early at the Chandra Data Systems Operations Center. Around 7:00 a.m., the data that arrived during the night are examined for problems, and each observation is sent through minimal processing to produce a quicklook image (see Figures 15 and 16). Each quicklook image is delivered immediately to the Science Operations Team, who uses the image to verify instrument performance and target acquisition. There have been very few problems of this nature during the Chandra mission, but it is often the Data Systems Operations staff that spots a problem and raises an alert.

**Figure 15:** Quicklook image of HRC-S LETG observation of the white dwarf HZ 43 (Obsid 2584). The zeroth order of the source appears star-shaped due to a coarse-support structure in the grating. The bright line in the cross-dispersion direction that intersects the source is due to the configuration of the fine-support structure in the LETG. There is also increased background near the edges of the detector segments, caused by higher ``background" from the free edges of the MCPs, and by event reconstruction errors.
$\begin{figure}\centering \resizebox{\textwidth}{!}{\includegraphics{obs2584fig1.ps}}\end{figure}$

**Figure 16:** Closeup view of the source in the quicklook image.
$\begin{figure}\centering \resizebox{\textwidth}{!}{\includegraphics{obs2584_centerfig2.ps}}\end{figure}$

Throughout the day these quicklook images are produced as soon as data arrive from Chandra, and are generally available in between 1 and 12 hours (depending on the length of the observation and number of events to be processed).

The telemetered data do not always arrive from Chandra unscathed. Every 8 to 12 hours, Chandra data are downlinked to one of several antennae at one of the Deep Space Network (DSN) support stations in Australia, Spain, and California. The captured data are then transferred to the central DSN data center at JPL. DSN handles data from many satellites at once, and the Chandra data must be sorted out and transferred to the Chandra Operations Control Center (OCC) in Cambridge. At the OCC, the data are read into a database system and then delivered to Data Systems. It takes about 6 to 8 hours for a single data dump of the onboard recorders to reach Data Systems. The data can arrive out of order, be declared ``lost" and then ``found" or be very late in arriving if Chandra is unable to dump the onboard recorders because of ground receipt problems at the DSN support station. However, these problems are actually quite rare.

It's not uncommon to lose a few seconds of data here and there during the downlink and ground transfer. Generally a little data loss is not a problem, unless a critical piece of status information is lost. In that case, considerable manual effort is required to reconstruct the necessary information for processing. So Data Systems Operations has to carefully check each batch of incoming data to be sure all necessary data are present, and to develop work-arounds for problems.

Data Systems Operations keeps it's finger to the pulse of the mission, closely monitoring what goes on onboard to detect any issues that will affect processing. For example, occasionally one or more guide stars are not acquired and accommodation must be made in the processing. Interrupted observations due to spacecraft issues often need special handling also.

In addition, there is the weather - space weather - to worry about. Solar flares, which have been frequent over the last year, are dangerous to Chandra's sensitive instruments, especially the solar storms that send a blizzard of high energy particles Chandra's way. The instruments must be shut down until the solar storm is over, which means that the whole process of sending instructions and receiving data from Chandra has to be interrupted.

Data Systems Operations also stands ready to support the Science Operations Team in case of spacecraft or instrument anomaly, and to support immediate processing of Targets of Opportunity (TOOs) when planning for a follow up observation requires a review of the first one. Request for rapid processing of an observation is a decision of the Chandra Director's Office.

From the processing perspective: To process the Chandra data for distribution and archiving, the operations team oversees the standard reduction and analysis of Chandra data via AP, the data systems Automatic Processing manager application. AP is configured via a registry system that identifies a pipeline along with its input and output products. AP sends a trigger when data are available that kicks off the appropriate pipeline and provides it with all of its input data. A pipeline consists of a set of computer programs that are run in series to perform a particular job. We have processing levels defined for the pipelines - Level 0 is telemetry processing where the input stream is decommutated, Level 1 is photon event processing where instrumental corrections are applied to the data and sky positions are computed, and Level 2 filters the data and performs data analysis. After a pipeline runs, data are then available as input to other pipelines and the data products created along the way are staged for storage in the Chandra archive. In all there are an average of 73 pipelines and 244 computer programs that run to contribute to 1 scientific observation.

From the scientific perspective: Standard processing involves figuring out exactly where the telescope was pointing at each moment of the observation by tracking fiducial alignment lights on the spacecraft and the positions of well-known stars using the Aspect camera. The spacecraft motion is removed from the data and the data are corrected for features peculiar to each instrument. Other temporal features like temperature, radiation rates, and aspect quality are also monitored and contribute to the ``good time" filters determined in level 1 and applied to the photon event list in level 2. Finally, after applying temporal and spatial filters to an observation, a few simple data analysis tasks including source detection complete data processing.

Figures 17 and 18 show the images produced by the standard data processing for the same observation as seen in Figures 15 and 16. While most of the improvements made to the data in this complete processing are not obvious by looking at an image, one can still see the effects of the removal of the instrumental signature and background determination.

**Figure 17:** Obsid 2584 processed through Level 2.
$\begin{figure}\centering \resizebox{\textwidth}{!}{\includegraphics{o2584fig3.ps}}\end{figure}$

**Figure 18:** Closeup view of the source in the fully processed Level 2 image.
$\begin{figure}\centering \resizebox{\textwidth}{!}{\includegraphics{o2584_ctrfig4.ps}}\end{figure}$

The processed data are reviewed by a V&V (validation and verification) Scientist and then distributed to the scientist who proposed the observation, and placed in the Chandra data archive. The archive currently contains 4.5 million files. These are files from the approximately 3600 targets that Chandra has observed so far. At present the archive has a storage capacity of just over 1 terabyte of compressed data.

Custom processing can be performed for cases where rapid processing is required (e.g., TOO, instrument anomaly). Custom processing has been completed 4 to 6 hours after receipt of the data in previous instances of rapid processing requests. Custom processing is also used to support calibration testing.

The day ends late at Chandra Data Systems Operations. The last shift leaves the building around 8:00pm, but they leave with their pagers, ready to respond if needed.

The long and complex river of data running from the Chandra X-ray Observatory to the scientist requires many teams of skilled and dedicated people who carefully lay plans for a smooth trip, but who are also ready, willing, and able to ride out the rapids.

Processing Chandra Data