Skip to content

fix(live): hold live triggers during stage motion; survive a busy MCU at acquisition start - #639

Open
Alpaca233 wants to merge 2 commits into
masterfrom
fix/live-hold-during-stage-motion
Open

fix(live): hold live triggers during stage motion; survive a busy MCU at acquisition start#639
Alpaca233 wants to merge 2 commits into
masterfrom
fix/live-hold-during-stage-motion

Conversation

@Alpaca233

Copy link
Copy Markdown
Collaborator

Problem

On an instrument on 2026-09-02, clicking "go to point" in the focus-map navigator while live view was streaming hardware triggers (~9.3 fps) wedged the MCU: a MOVETO_Z raced the SEND_HARDWARE_TRIGGER stream and the firmware's single command-status slot got stuck reporting IN_PROGRESS for every subsequent command — while still executing them (frames kept arriving). Every wait_till_operation_is_completed() then timed out for ~10 minutes:

  • each further goto click hung 3 s and raised TimeoutError,
  • starting an acquisition aborted inside stop_live() (illumination-off wait timed out),
  • the live button kept showing "Stop" after the failed stop (the exception skipped the setText), inviting more toggling that restarted the trigger stream and re-wedged the MCU.

Log evidence shows the wedge clears a few seconds after the trigger stream stops — the one acquisition attempt that worked ran stop_live as a no-op (live already stopped by a failed toggle), reached its first MOVETO_Z ~6 s after the last illuminated trigger, and completed it in 17.5 ms.

Fix (host-side; the firmware's single command-status slot remains the underlying defect)

  1. Hold live during stage motionLiveController.trigger_acquisition() returns False while microcontroller.is_busy(). The trigger timer already re-checks within 10 ms on False, so live pauses for the duration of a move and resumes by itself. This prevents the trigger/move race on every motion path (focus map, click-to-move, autofocus).
  2. Truthful live buttonstoggle_live in LiveControlWidget, LaserAutofocusSettingWidget, and NapariLiveWidget now sets button state in a finally block from liveController.is_live, so a failed stop can no longer leave the button lying.
  3. Patient acquisition startrun_acquisition() catches a TimeoutError from stop_live() (live is already stopped and its trigger timer cancelled by then; the command was sent) and waits for the MCU to go idle via _wait_for_microcontroller_idle() (3 × 5 s attempts, each logged) before the worker moves the stage. If the MCU stays busy it aborts through the existing failed_to_start path with an actionable "home the stage or power-cycle" error instead of an uncaught traceback.

With (1) the wedge shouldn't occur; (2) and (3) make the recovery path work if it ever does.

Testing

  • New: tests/control/core/test_live_controller_motion_hold.py (trigger gate), tests/control/test_widget_toggle_live_truthful.py (14 parametrized button-truthfulness tests), and 5 pre-flight tests in test_MultiPointController.py — including one proving an acquisition still runs to completion after a stop_live() timeout when the MCU recovers. All written failing-first.
  • Full suite (CI selection): 1810 passed, 9 skipped, 1 xfailed.

🤖 Generated with Claude Code

… at acquisition start

Sending SEND_HARDWARE_TRIGGER while a MOVETO_* is in flight can wedge the
firmware's single command-status slot: the MCU keeps executing commands but
reports IN_PROGRESS for everything, and every wait_till_operation_is_completed
call times out until the trigger stream stops (seen on an instrument on
2026-09-02 during focus-map point navigation with live view running).

- LiveController.trigger_acquisition() returns False while the
  microcontroller is busy; the trigger timer re-checks within 10 ms, so live
  holds during stage moves and resumes automatically.
- toggle_live handlers (LiveControlWidget, LaserAutofocusSettingWidget,
  NapariLiveWidget) update the button in a finally block from
  liveController.is_live, so a failed stop can't leave "Stop" showing while
  live is off.
- run_acquisition() survives a stop_live() TimeoutError and waits for the MCU
  to go idle (3 x 5 s, logged) before the worker moves the stage; if the MCU
  stays busy it aborts through the normal failed_to_start path with an
  actionable error.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Command dispatch remains race-prone, and failed preflight can leave UI and API callers reporting incorrect state.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Improves live-view coordination with MCU-controlled stage motion and acquisition startup recovery.

Changes:

  • Pauses live triggers while the MCU reports busy.
  • Synchronizes live-button labels with controller state.
  • Adds MCU-idle acquisition preflight and regression tests.
File summaries
File Description
software/control/core/live_controller.py Adds the MCU-busy trigger gate.
software/control/core/multi_point_controller.py Adds acquisition-start recovery and idle waiting.
software/control/widgets.py Updates live controls in finally blocks.
software/tests/control/core/test_live_controller_motion_hold.py Tests trigger gating.
software/tests/control/test_MultiPointController.py Tests MCU preflight recovery.
software/tests/control/test_widget_toggle_live_truthful.py Tests live-control state updates.
Review details

Suppressed comments (1)

software/control/core/multi_point_controller.py:837

  • This abort leaves the live controls untruthful in the acquisition-start recovery path. If live was on, stop_live() has already set is_live=False, but this return occurs before hardware_prepared=True, so the finally block neither resumes live nor notifies any live widget; the three new toggle_live finally blocks are not invoked because the controller stopped live directly. The checked button therefore still reads “Stop” after the failed start—the same operator-facing state this PR is intended to eliminate. Publish/synchronize the controller's live-state change before returning.
            if not self._wait_for_microcontroller_idle():
                self._log.error(
                    f"Microcontroller still busy after "
                    f"{self._MCU_IDLE_WAIT_ATTEMPTS * self._MCU_IDLE_WAIT_TIMEOUT_S} s - home the stage or "
                    f"power-cycle the controller. Aborting the acquisition start."
                )
                return
  • Files reviewed: 6/6 changed files
  • Comments generated: 2
  • Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +499 to +505
if self.microscope.low_level_drivers.microcontroller.is_busy():
# Hold live while the MCU executes a command (usually a stage move):
# triggering mid-move corrupts the firmware's single command-status
# slot and wedges it - every later wait times out until the trigger
# stream stops. The trigger timer re-checks within 10 ms when we
# return False, so live resumes as soon as the move completes.
return False
Comment on lines +831 to +837
if not self._wait_for_microcontroller_idle():
self._log.error(
f"Microcontroller still busy after "
f"{self._MCU_IDLE_WAIT_ATTEMPTS * self._MCU_IDLE_WAIT_TIMEOUT_S} s - home the stage or "
f"power-cycle the controller. Aborting the acquisition start."
)
return
- Re-check a busy-MCU trigger hold at the normal frame cadence instead of
  every 10 ms - each re-check builds a fresh Timer thread, and a stage move
  lasts hundreds of 10 ms ticks.
- Guard snap() with the same wedge-avoidance as trigger_acquisition(): wait
  for the MCU, refuse to trigger if it stays busy.
- Deduplicate the three toggle_live handlers into one _sync_live_button
  helper, which now also syncs the checked state so the next click sends the
  right `pressed`.
- Drop the _PreflightStub shadow class and redundant unit tests; the retry
  behavior is covered by one real-controller test plus the existing
  integration tests.
- Trim repeated rationale comments to point at the trigger_acquisition guard,
  and state plainly that the pre-flight loop exists for progress logging.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants