SE08 Orbital Activation and Checkout Thread Rehearsal Debrief

Overview

The first rehearsal of the SE08 Orbital Activation and Checkout (OAC) Thread was an exercise covering several weeks and culminating in an all-day session of simulated commanding of the AXAF science instruments (ACIS, HRC, and ACA) on 1998 April 13. The post-rehearsal debrief was held 1998 April 21. The goals of the test were to develop, execute, and test procedures to be used during OAC (and to support ground testing) and to test the interfaces that will be used during OAC:

FOT and SOT/IPI
OCC and FPSIs
OCC FDDI ring and EGSE
OCC and ASCDS telemetry processing

The original plan for the test was to develop and run procedures for the following portions of OAC:

ACIS

Open Little Vent Valve
Open Large Vent Valve
Acquire data with internal cal source
Acquire focus plate data

Activate fiducial lights and ACA
Post SSD fine attitude initialization

Initial turn-on (HRC-I config)
Door Open
Anitco Shield on
HRC-I MCP HV on
Switch from HRC-I to HRC-S

This list of procedures would test the FOT to SOT/IPI team interface in the development of procedures and the implementation of command sequences prior to the test day and in their execution on the test day. The test day execution would also form the basis of testing the OCC to FPSI interfaces.

On the day of the test OPSIM would be used to generate telemetry which would be distributed to the science instrument EGSE as well as captured by the OCC ONLS for subsequent transmission to the ASC.

Some of the OAC rehearsal work for the HRC was ultimately pre-empted by preparation for a test of OCC commanding of the real science instruments. The rehearsal was quite beneficial to our preparedness for the commanding test.

Rehearsal Results

General

The rehearsal was for the most part a success in that we:

learned how long it takes to develop and validate all the parts required for an activity (procedures, scripts, command loads,...)
learned how to communicate between the FOT personnel commanding the spacecraft and the science team
developed software to convert UDP packets on the OCC FDDI ring into TCP for distribution to science instrument EGSE
identified "missing" procedures
learned about implementing instrument modules in OPSIM

Version control of procedures, scripts, command sequence definitions, command "hex-code" patterns and any other products is needed; as is visibility of the version number and number of the parent on the product. Procedures need to be established for a timely review by the SOT and IPI teams of the products generated by the FOT. Making all procedures available electronically as ASCII text will help facilitate this review.

Written procedure development was a bottle-neck during rehearsal preparation due to the fact that only one person, Eric Martin, was writing the procedures. The responsibility for generating the written procedures must be more widely distributed.

The cross referencing between the written procedure and the executing script must be improved to make it easier to follow where the procedure is in execution.

The SOT needs to become more familiar with the standard operating procedures within the OCC ( voice protocol, appropriate contacts,...). The CAMBRIDGE OPS position is the correct contact for problems with the facility, including workstations, voice headsets, and the UDP to TCP translator machine. It is planned for there to be console handbooks at all the EHS workstations with general reference information.

The SOT staffed the ACIS and HRC portions of the rehearsal differently with ASC personnel in the both the Control and TST rooms during the ACIS portion and only in the Control room during the HRC portion. We also switched who was the SOT representative in the control room based on the active instrument. It is an open question whether this will continue to be the case during OAC; we should re-think what we want to do based on total staff available and the typical planned schedule of activity.

The software to translate UDP to TCP for the EGSEs worked well. We anticipate future versions to simplify/automate establishing the socket connection to the EGSE. Data drop-outs due to UDP packet loss have been seen in the past and may have occurred during this test. A future test should be run to monitor the percentage of dropped packets. The science teams should be made aware of the possibility of dropped packets.

On the test day, the OCC to ASCDS interface was not tested. There is no documented procedure for the SOT to request for the captured telemetry to be transfered to the ASCDS for processing and as a result this was not done at specified points during the test. A NRT data capture request was performed on the following day and the data file transfered to outside the OCC. The strippers/extractors for ACA-image, ACA-cal, HRC, EPHIN, Engineering. The output is available in:

/data/astage/test/R4B2.0/indata/telem/tmp.all/run/ ACIS extractor not yet ready so it has not been run on this data. Review of these products is underway.

ACIS

While procedures and products for all of the planned activities were available only the first three of the four ACIS activities listed above were tried on the day of the test. The plate focus measurements were skipped due to time constraints. It is hoped to be able to use OPSIM to play back all the selected test data in the near future.

The capability of OPSIM to be able to playback ACIS telemetry was largely due to the efforts of Peter Ford. On the day of the test there were some initial problems getting the ACIS module in OPSIM up and running. While running there were some commands that did not operate properly; some of these problems were due to differences between the OCC CDB mnemonics used by OPSIM and what was used by the ACIS module.

In developing the ACIS command loads and scripts several problems were identified in the OFLS and ONLS

The AXAF to AXAF-OCC ICD describes a real-time format for ACIS software commands. It does not appear that either the ONLS or the OFLS has the capability of generating such real-time commands.
The AXAF to AXAF-OCC ICD specifies a 28-bit header which is to precede a block of ACIS software serial commands. The OFLS does not insert these header bits, and OP-19 does not specify that they are to be inserted into the binary portion of the ACIS Tables.
The OBC stored command format not implementable outside the OFLS

Final approval for issuing critical commands to ACIS must reside withe the ACIS engineer at the ACIS EGSE. There was some confusion during the rehearsal caused by the communication between the OC and the CC when the critical command verification window came up and no "OK to proceed" was requested from the ACIS team. This was largely due to a mis-understanding between the FOT and SOT of how the procedures should work and how the ONLS commanding system works. Scripts should be written in such a way that any OKs for critical commands from the ACIS engineer are clearly identified.

There were a few minor problems identified with the ONLS displays, some of which should be fixed by future releases of the telemetry database. Missing calibration data in the telemetry database makes it difficult to interpret telemetry values. Context switching of the displays to account for whether the ACIS logic power is on would help eliminate some confusion.

ACA

The fidelity caused the scope of some of this rehearsal to be scaled back (meaningful response to some commands would not be possible). Procedures and scripts developed with some work to identify the appropriate way to command the fiducial lights. OPSIM was given a file with ACA pixel data to insert into the the correct locations in telemetry. On the day of test procedures and commanding ran smoothly. Pixel data was viewable on the ACA EGSE display.

HRC

Scripts and command sequence definitions for the original planned HRC activities were generated but not verified on the ONLS. The generation was accomplished in part by software tools that translate the HTML pages generated by M. Juda into command sequence definitions. Skeleton procedures for each of the planned activities were generated but not fleshed out into the final written versions. Before the verification was begun it was decided that preparation for OCC commanding of the HRC flight instrument would take priority. The selected commanding activity for the test was similar in content to the first of the planned activities, "Initial turn-on (HRC-I config)".

In the weeks leading to the test one of the key items that needed to be addressed was inserting missing HRC commands into the CDB. In addition to the missing commands, a few incorrect commands were identified and corrected. The CDB had to be updated before and script validation or SCS creation could be performed.

For the test OPSIM was provided with a file containing one major frame of HRC observing mode serial digital data to insert into the the correct locations in telemetry. The EGSE and ONLS displays handled this telemetry as expected.

During the rehearsal of the HRC initial turn-on, we discovered that the procedure had too many closely-spaced "wait for SOT verification". One key element that we tested was how to abort an sequence. This will be a critical capability to have during the OAC as we turn on the HRC MCP HV for the first time on orbit. It was determined that having the HRC team call for an abort from the TST room would cause the command controller to stop the executing group, and issue a pre-selected abort command twice. The abort was accomplished within 5 seconds of its being requested (albeit the operators were expecting an abort).

ONLS display pages suffered from incomplete calibration data within the telemetry database. These pages also need some reorganization of their layout to more logically group the items in the display. For clarity during OAC, activity specific displays showing the key items to monitor should be developed.

Page created: 1998 April 28

Dr. Michael Juda
Harvard-Smithsonian Center for Astrophysics
60 Garden Street, Mail Stop 70
Cambridge, MA 02138, USA
Ph.: (617) 495-7062
Fax: (617) 495-7356
E-mail: mjuda@cfa.harvard.edu< /I>