Oct_29_2003_M1560.txt Shift M1560 Start at 2003:302 19:30 UTC = 2003 Oct 29 2:30 pm EST End at 2003 Oct 29 4:30 pm EST SOT shift report 2003-10-29 M1560 See http://asc.harvard.edu/acis/SOT_reports/sot-shift.html, or /proj/ascwww/AXAF/extra/science/htdocs/acis/SOT_reports/Oct_29_2003_M1560.txt SOT Lead Robert Cameron SOT SC/ACA FOT-SI HRC-SI ACIS-SI OTHER Plan: - No load running because of high radiation. ================================ Summary of 5pm EST radiation telecon: Radiation rates remain high. Also, there was a recent X11 solar flare which has produced a full halo CME, and which has driven up the GOES and ACE proton fluxes even further. The ACIS CTI measurement for the descending leg of this orbit is now cancelled. OCT300B products, which have a maneuver for momentum control before perigee and resume science after perigee, have been produced. We will certainly execute the first SCS of OCT3003B, to perform the maneuver before perigee. We will re-assess radiation tomorrow at 9am, and probably at 5pm and 10:30pm EST, to make the final go/nogo decision about resuming science with OCT3003B products. Because of the X10 flare, FOT MP will also prepare products to resume science near apogee in the next orbit, essentially just observing obsid 3755 (although the TOO, obsid 5240 will also probably be included) in the next orbit. In addition, FOT MP will also build and release for review the NOV0303A products. To summarize: 1. If science resumes with OCT3003B, then NOV0303 products will follow. The last possible go/nogo for OCT3003B would be in the 2003:304:0345 UT comm pass. 2. If science is not resumed with OCT3003B, then NOV0103 products starting with obsid 3755, and incorporating the NOV0303 observations will be the nominal timeline. The last possible go/nogo for the NOV0103 load would be in the 2003:304:2030 UT comm pass. There was discussion of the early RADMON trigger and autonomous SCS107 run after the X17 flare. Eric Martin has released email summarizing the problem and 3 potential operational solutions to prevent this problem from recurring. It was decided today to use the "do nothing" approach. If a science load is uplinked and started which contains a RADMON enable, then a similar early RADMON trigger and SCS107 run may occur if the RADMON enable occurs during high radiation. We accept that risk, with the associated necessary replan. For details of the problem, see the SOT shift report 2003-10-29 M1560 at http://asc.harvard.edu/acis/SOT_reports/sot-shift.html =========================== Summary of early RADMON trigger and work-arounds from Eric Martin: In the latest RadMon shutdown (call it the X17-trip), RadMon tripped in the first major frame it was enabled. This circumstance was possible because the previous time RadMon had been enabled (call it the X1-trip), it also triggered a high radiation shutdown. The disable/enable commmands to RadMon act like a "pause" button. RadMon is NOT reinitialized when it is re-enabled (this will be fixed in the planned RadMon patch). As I've mentioned before, it was not expected during flight software development that RadMon would be disabled & re-enabled every orbit; this because necessary after launch. When enabled, RadMon continually adds radiation measurements in the various channels it monitors to channel-specific "accumulators". If any of the accumulators for the 3 channels used to detect high radiation have 10 consecutive values all above their respective thresholds, RadMon will set the high radiation flag. When SCS 107 disabled RadMon following the X1-trip, the E1300 accumulator was full of high values; the P4GM & P41GM accumulators were full with low values. When RadMon was enabled just prior to the X17-trip, E1300 & P41GM were both above threshold, while P4GM was still below threshold. RadMon took the most recent measured values, combined with the 9 most recent stored values in each accumulator, and checked for 10 consecutive samples above threshold. This was true for E1300 in the first major frame that RadMon was enabled, so the high radiation flag was set and SCS 107 ran (which disabled RadMon). At present, the E1300 accumulator is still full of high values, so we could have another 1-cycle RadMon trip if it is re-enabled in a high radiation environment. Also, the P41GM accumulator now has 1 high sample, so even if E1300 were low, RadMon could trip on P41GM after 9 cycles (not of much concern this time). In order to avoid a less than 10 cycle RadMon trip, we need to get low values into the most recent stored samples in each of the 3 high-radiation-detection accumulators (this is already the case for P4GM). Option 1: In the next comm pass where the radiation environment is low, enable RadMon for at least 1 cycle. Option 2: In a way similar to what was done for the 10-sample K-constant on-orbit RadMon test, build & test products to patch EPHIN with low values (zero) for E1300, P4GM & P41GM. Run a script to uplink the patch load every 30 sec and enable RadMon for at least 1 cycle after telemetry indicates the OBC has recorded the low values in the Telemetry Distribution Buffer (TDB). Currently, P4GM and P41GM (based on their GOES analogs) are above their thresholds (as, presumably, is E1300), so Option 1 may not be possible very soon. If we want to go with Option 2, we need to start work on products right away. Eric There is a 3rd option I didn't mention in the original e-mail. Here are the 3 options, with a bit more description of what they entail: Option 1: In the next comm pass where the radiation environment is low, enable RadMon for at least 1 cycle. Option 1 involves nothing more than enabling & disabling RadMon (which is ordinarily done every orbit), but requires a low radiation environment. We would leave SCS 107 disabled, so there is no risk of any safing action due to this activity. It would take ~5-10 min to complete. Steps: (1) Verify low radiation in E1300, P4GM & P41GM EPHIN channels. (2) Disable SCS 107, if necessary. (3) Enable RadMon (OORMPEN). (4) Depending on the radiation environment, leave RadMon enabled for 1 to 10 updates with fresh EPHIN data (1 - 19 major frames). (5) Verify that ACIS high radiation flag clears (1OHIRADF = FALS). (6) Disable RadMon (OORMPDS). Option 2: In a way similar to what was done for the 10-sample K-constant on-orbit RadMon test, build & test products to patch EPHIN with low values (zero) for E1300, P4GM & P41GM. Run a script to uplink the patch load every 30 sec and enable RadMon for at least 1 cycle after telemetry indicates the OBC has recorded the low values in the Telemetry Distribution Buffer (TDB). Option 2 involves straightforward product development (simply replacing the high values in a command load with zero). The EPHIN patch used is NOT "persistent", i.e., it affects a single EPHIN cycle, though we would uplink it multiple times to set the counts reported to the OBC to zero for a period of ~5-10 min. It can be used no matter what the radiation environment is. EPHIN automatically returns to a normal operating state after the patch. It is not *required* that SCS 107 be disabled, but is probably desirable. This option would take ~15-25 min to complete. Steps: (1) Disable SCS 107, if necessary. (2) Execute script to repeatedly patch E1300, P4GM & P41GM measurement data in EPHIN to zero. Script should perform this patch for at least 15 min. (3) Verify that patch has executed in EPHIN. (4) Enable RadMon (OORMPEN). (5) Depending on the radiation environment, leave RadMon enabled for 1 to 10 updates with fresh EPHIN data (1 - 19 major frames). (5) Verify that ACIS high radiation flag clears (1OHIRADF = FALS). (6) Disable RadMon (OORMPDS) before script execution ends. (7) Terminate script execution or allow to complete normally, as appropriate. Option 3: Use the patch developed for CAP 771 to patch the OFP Executive process in the OBC to cause the RadMon Initialization process to run. Option 3 involves a patch to the OBC, which we are generally more reluctant to do, but the patch is to non-checksummed memory and is not "persistent". Disabling SCS 107 is not required. It would take ~5-10 min to complete. Steps: (1) Execute SOP 6O001 to load OBCA_150_PR139.CLD. (2) Complete memory load (COMEMLD). (3) Verify the patch was successful in initializing the RadMon process (CORADMIN = TRUE). Eric ======================================= Seq # ObsID It Target start time SI Grat RA dec roll ----------------------------------------------------------------------------------------------------- ORBITAL EVENTS: 2003:302:13:59:15.020 584 EAPOGEE ORBIT APOGEE 2003:303:21:43:31.304 584 EPERIGEE ORBIT PERIGEE 2003:305:05:26:58.729 585 EAPOGEE ORBIT APOGEE 2003:306:13:10:34.451 585 EPERIGEE ORBIT PERIGEE 2003:307:20:54:45.086 586 EAPOGEE ORBIT APOGEE 2003:309:04:38:07.561 586 EPERIGEE ORBIT PERIGEE 2003:310:12:22:40.962 587 EAPOGEE ORBIT APOGEE 2003:311:20:06:32.771 587 EPERIGEE ORBIT PERIGEE 2003:313:03:50:58.858 588 EAPOGEE ORBIT APOGEE 2003:314:11:35:29.516 588 EPERIGEE ORBIT PERIGEE 2003:315:19:19:29.654 589 EAPOGEE ORBIT APOGEE EVENTS DONE OBSID -------------- ---- ----- Start Shift 302:1930 AOS DSS-54 (Madrid 34m) 302:2010 60497 (load stopped) LOS DSS-54 (Madrid 34m) 302:2110 End of shift 302:2130 NOTES/ISSUES/PROBLEMS --------------------- 1) Status at 302:2030: UTC 2003:302:20:33:02 (Oct29) f_ACE 9.46e+05 F_CRM 7.60e+08 Kp 6.0 R km 125374D OBT 2003:302:20:32:28 CTUVCDU 2123647 OBC s/w NRML FMT2_SSR CPEstat NORM OBT 183846748.05 ONLVCDU 2123632 OBC Errs 0 OBSID 60497 EPState SUN SIM TTpos -99616 HETG Angle 79.46 PCADMODE NPNT RA 10.998 Bus V 29.30 SIM FApos -715 LETG Angle 77.59 PCONTROL NPNT Dec 34.999 Bus I 34.06 AOFSTAR GUID Roll 225.730 ACA Object SSSSSSSS Dither ENAB HRC-I HV ON ACA ImFunc TTTTTTNN Dith Yang 6.94 Yaw Rate -0.07 HRC-S HV OFF ACA CCD Temp -15.6 Dith Zang 2.40 Pitch Rate 0.00 OBSMode NEXT ACA Int Time 1.698 Roll Rate 0.05 Door OP UNAC AOACSTAT OK FSS SunBeta NSUN Door CL UNAC FSS Alfa -49.65 Batt 1 SOC 100.00% Avg HRMA Temp 70.61 FSS Beta -49.78 Batt 2 SOC 100.00% ACIS Stat7-0 67 Avg OBA Temp 51.05 SA Resolv 151.12 Batt 3 SOC 100.00% Cold Rad -127.9 OBA Tavg Fault NFLT SA Sun Pres ILLM Warm Rad -82.0 OBA Trng Fault NFLT +Y SA Amps 16.44 RadMon DISA HRMA power 67.30 SCS 128 INAC -Y SA Amps 17.23 EPHIN Geom SMAL OBA power 102.50 SCS 129 INAC +Y SA Temp 114.81 E150 63219.5 SCS 130 INAC -Y SA Temp 115.85 E300 1669.0 Roll Mom. 1.135 SCS 107 DISA E1300 159.6 Pitch Mom. -10.087 UpL Cmd Acc 27553 EPH A-Leak 1.0200 P4GM 2119.2 Yaw Mom. 2.307 Cmd Rej A 250 EPH B-Leak 0.2680 P41GM 28.5 Gyro 1 Curr 1 9.60 Roll Bias -0.5238 Gyro 1 Curr 2 9.60 Pitch Bias -1.8723 Gyro 2 Curr 1 103.20 Yaw Bias -0.8782 Gyro 2 Curr 2 98.40 Current Data: UTC 2003:302:20:33:02 (Oct29) f_ACE 9.46e+05 F_CRM 7.60e+08 Kp 6.0 R km 125374D Currently scheduled FPSI, OTG : HRC-S NONE Estimated Kp : 6.00 ACE EPAM P3 Proton Flux (p/cm^2-s-sr-MeV) : 1.28e+06 GOES-10 P2 flux, in RADMON P4GM units : 4.84 GOES-12 P2 flux, in RADMON P4GM units : 3.68 GOES-10 P5 flux, in RADMON P41GM units : 2.64 GOES-12 P5 flux, in RADMON P41GM units : 2.38 Orbit Start Time : 2003:301:06:19:02 Geocentric Distance (km), Orbit Leg : 125476 D CRM Region : 1 (Solar_Wind) External Proton Flux (p/cm^2-s-sr-MeV) : 1.28000000000e+06 Attenuated Proton Flux (p/cm^2-s-sr-MeV) : 0.00000000000e+00 External Proton Orbital Fluence (p/cm^2-sr-MeV) : 5.39959748469e+10 Attenuated Proton Orbital Fluence (p/cm^2-sr-MeV) : 7.59524985000e+08 2) Radiation The ACE radiation environment is very high and variable, with EPAM P3 rates about 1.5x10e6 p/second. The GOES proton radiation flux is elevated and variable. There was an X11 solar flare with associated full halo CME at 20:54 UT, which caused a further increase in GOES and ACE proton fluxes. ACE 2 hr fluence is 6.5e9 vs. threshold 3.6e8. CRM-based orbital fluence is 7.6e8 vs. threshold 3.0e9. Chandra is in the solar wind. 3) Quick Look data review (OK unless noted): Seq # ObsID Constr. Target Start Time SI Grat Comment --------------------------------------------------------------------------------------- There were no new quick-look datasets in this shift.