What to Do When Something Goes Wrong?

The ACIS Load Review script lr runs the thermal models that are used to approve a load for flight. Occasionally, one of these model scripts might fail to run. If help is not available to determine a reason for the failure, these are some steps that may be taken, which involve running the model script manually outside of lr. Many of these scenarios invovle running the model script with additional flags, all of which are detailed in Base Command-Line Arguments, but we provide some examples here.

Running a Model Manually

Log into a HEAD workstation as acisdude and run the command setska. From the home directory (or any other directory writable by acisdude), you can run a model script like acisfp_check:

barth-v.acisdude:~[104]> setska

barth-v.acisdude:~[105]> acisfp_check --oflsdir=/data/acis/LoadReviews/2025/JAN2725/ofls --out=test_jan2725_fp

where --oflsdir is the directory where the backstop and other necessary files are located, and --out is the name of the directory into which the files will be output.

Running Predictions Only

acis_thermal_check scripts run both predictions and validations, the latter so we can check the performance of the model against actual telemetry on a regular basis. If the code is crashing during the validation phase, it is possible to run with predictions only. This can be done with the --pred-only flag:

barth-v.acisdude:~[105]> dpa_check --oflsdir=/data/acis/LoadReviews/2025/JAN2725/ofls --pred-only --out=test_jan2725_dpa

Running on Loads that are Interrupted

If the load under review is a return to science from a shutdown, or a replan due to a TOO, then you would have run lr with the --break flag. If you have to run a model manually under these circumstances, you should use the --interrupt flag:

barth-v.acisdude:~[105]> dpa_check --oflsdir=/data/acis/LoadReviews/2025/JAN2725/ofls --interrupt --out=test_jan2725_dpa

Running with the kadi State Builder

The "acis" State Builder constructs a history of states using a series of backstop files and the Non-Load Event Tracker (NLET) file. This is the default state builder used with acis_thermal_check, and is used by lr. The "kadi" state builder constructs a history of states using kadi-based commands (though even for the "acis" state builder kadi is used to get states from commands). If there is a suspicion that something has gone wrong with the ACIS continuity files that the "acis" state builder uses to construct a mission history, then the script can be run with the "kadi" state builder using the --state-builder flag:

barth-v.acisdude:~[105]> dea_check --oflsdir=/data/acis/LoadReviews/2025/JAN2725/ofls --state-builder=kadi --out=test_jan2725_dea

Changing the Run Start Time

By default, the run start time is set to the current time. If the model is crashing because of problems with telemetry or commanded states at a particular time, the run start time can be set by hand:

barth-v.acisdude:~[105]> psmc_check --oflsdir=/data/acis/LoadReviews/2025/JAN2725/ofls --run-start=2025:027:00:00:00 --out=test_jan2725_psmc

Changing the Initial Temperature

By default, the initial temperature for the model will be set using an average of telemetry in a few minutes around the run start time. If there is an issue with the telemetry at the run start time, the model can be started with a user-specified temperature value using the --T-init argument, which is assumed to be in degrees C:

barth-v.acisdude:~[105]> dpa_check --oflsdir=/data/acis/LoadReviews/2025/JAN2725/ofls --out=test_jan2725_dpa --T-init=12.0

Copying Model Outputs to the Web Page Location

Once you have run the model yourself, if you wish you can copy the model run files to the appropriate location for the web page (which is what lr would have done had it run successfully). This is done with the copy_model_outputs script:

(ska) barth-v.acisdude:~[138]> copy_model_outputs --help
usage: copy_model_outputs [-h] [--overwrite] [--dry_run] location load

positional arguments:
  location     The location of the model files to copy
  load         The load name. Must be 8 characters, e.g. 'JAN2725A'

options:
  -h, --help   show this help message and exit
  --overwrite  Overwrite existing files.
  --dry_run    Show what would have been done.

In general, after you have created the model files, you run the copy_model_outputs script like this:

barth-v.acisdude:~[106]> copy_model_outputs test_jan2725_fp JAN2725A

which, if it runs successfully, gives output like this:

Files to be copied are from a "acisfp" model run.
Copied contents of test_jan2725_fp to /proj/web-cxc/htdocs/acis/FP_thermPredic/JAN2725/oflsa.

If the location to be copied to already has files, copy_model_outputs will refuse to copy the files unless you use the --overwrite flag:

barth-v.acisdude:~[106]> copy_model_outputs test_jan2725_fp JAN2725A --overwrite

If you want to see what would be copied without actually doing anything, you can use the --dry_run flag:

barth-v.acisdude:~[106]> copy_model_outputs test_jan2725_fp JAN2725A --dry_run

which, if it runs successfully, gives output like this:

Files to be copied are from a "acisfp" model run.
Would have copied contents of test_jan2725_fp to /proj/web-cxc/htdocs/acis/FP_thermPredic/JAN2725/oflsa.

If the path to the model files does not exist, or if the directory exists but the appropriate files are not found, or if the script cannot determine the model type, then it will report these errors with the necessary specificity.