[SCHEDULER-1449]  Property DomeOpen in ACPSequencer may be wrong
Type Investigation
Priority Medium
Severity Minor
Component Dispatch Scheduler Engine
Fixed In Version [8.18.1
Versions Affected [8.08.0
Severity Closed
Resolution Complete
Reported By Bob Denny
Resources Bob Denny
Start Date 6/4/2016

Description
See this Comm Center thread by Todd Benko. Here is the log which shows the problem resulting from the weather going unsafe during the time the dome is being opened or shortly thereafter. Review this logic in ACPSequencer.cs closely!
10-Apr-2016 03:43:37.2: Acquire data for Observation ngc 2300...
10-Apr-2016 03:43:37.2: (belongs to Project ngc 2300, Plan ngc 2300-G 240s (1-7))
10-Apr-2016 03:43:37.2: ++ Observatory Startup ++
10-Apr-2016 03:43:37.2: Start ACP Sequencer's StartupObs script
10-Apr-2016 03:45:26.8: Open the observatory dome/roof
10-Apr-2016 03:45:46.9: **Weather went unsafe while dome opening.
10-Apr-2016 03:45:46.9: Special-needs building, run CloseSpecialRoof script to get it closed.
10-Apr-2016 03:46:33.5: ** Script interrupted by weather safety event.
10-Apr-2016 03:46:39.7: **EXCEPTION IN SCHEDULER:
10-Apr-2016 03:46:39.7: Property DomeOpen: Unexpected shutter opening status
10-Apr-2016 03:46:39.7: Traceback:
at DC3.Scheduler.ACPSequencer.get_DomeOpen()
at DC3.Scheduler.Engine.DoStartupIf(Boolean openDome)
at DC3.Scheduler.Engine.DoSchedulePass()
at DC3.Scheduler.Engine.Run()

Look at the logic in Property DomeOpen for the case where the dome status is Opening. Should it simply return False. Also look at the sequencing in DoStartupIf(). What should happen if the weather goes unsafe during startup. Right now the resulting sequence of events can vary depending on exactly when  the weather unsafe happened. Is there a safe and rational solution for all possible cases? Ideally, as soon as the weather goes unsafe, the ACP Weather script should run and the Scheduler should go back into "wait for safe weather" mode. Note that the raised error currently sets off a CloseSpecialRoof etc. response, clearly wrong for an unsafe weather event. The case in Property DomeOpen is the trigger for this incorrect chain of events.


Comments
6/5/2016 1:40:46 PM   Bob Denny
SVN Comment
Author rbdenny
Repository svn+ssh://rbdenny@a2_svn_dc3/home/rbdenny/svn/astro/scheduler
SVN Revision 623
Affected files /trunk/Help/RelNotes.htm (Modified)
/trunk/Scheduler/ACPSequencer.cs (Modified)
/trunk/Scheduler/Engine.cs (Modified)
/trunk/Scheduler/SimulatorSequencer.cs (Modified)
Check-in comment Make system more resilient to weather unsafe during observatory startup. GEM:1449 GEM:1442
6/5/2016 1:36:35 PM   Bob Denny
Tested as much as I can. Still unknown how dome gets into opening state.
6/4/2016 3:42:30 PM   Bob Denny
Scheduler-1442 seems to have the "Property DomeOpen: Unexpected shutter opening status" problem as well. See link.