Skip to content

Stop the broker schema monitor spinning on "starting" when the agent is unreachable - #9001

Merged
andypalmi merged 3 commits into
mainfrom
8996-broker-schema-starting-timeout
Oct 21, 2026
Merged

andypalmi merged 3 commits into
mainfrom
8996-broker-schema-starting-timeout

Conversation

@cstns

@cstns cstns commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

If the schema agent can't be reached, its status keeps coming back as "still getting status", which the broker page showed as "starting" forever, with no end and no explanation.

Now that only holds for a short window. Past it the page stops waiting, drops the broker back to an idle state so the toggle can be used again, and shows a single toast saying the agent couldn't be reached. The timed-out state is UI-only and clears on a refresh, a successful status, or another attempt to start, so nothing is persisted and the backend is untouched.

While here, the schema monitor toggle and the connect toggle both fired the start request without waiting on it, so a failed start went unnoticed. Both now wait on the request and surface a failure the same way.

One thing worth knowing for review: the team broker status badge is hardcoded to connected and its toggle reads on/off from the agent's connected flag, so the visible "stuck on starting" is really on third-party brokers. The fix sits on the shared status poll, so it covers both: the third-party spinner stops, and the team-broker path times out cleanly too.

Closes #8996

…is unreachable

If the schema agent couldn't be reached, its status kept coming back as "still getting status", which the broker page showed as "starting" with no end and no explanation. Now that only holds for a bounded window: past it the page stops waiting, drops the broker back to an idle state so the toggle can be used again, and shows a single toast. The timed-out state is UI-only and clears on a refresh, a successful status, or another attempt to start.

The schema monitor toggle and the connect toggle also awaited nothing on start, so a failed start went unnoticed. Both now wait on the request and surface a failure the same way.
@cstns cstns self-assigned this Oct 20, 2026
@codecov

codecov Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.46%. Comparing base (40b10a5) to head (4f87bfc).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #9001   +/-   ##
=======================================
  Coverage   79.46%   79.46%           
=======================================
  Files         475      475           
  Lines       26018    26018           
  Branches     6934     6934           
=======================================
  Hits        20674    20674           
  Misses       5344     5344           
Flag Coverage Δ
backend 79.46% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@andypalmi andypalmi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice fix for the endless spinner, and good call awaiting the start requests. A few questions:

  1. After the timeout the page shows the broker as idle, but the backend row is still running. toggleAgent decides from the backend state, so the first click after a timeout goes down the running branch and calls suspendBroker, not startBroker. I checked this with the page's methods and mocked responses: after the timeout, one click makes 0 start calls and 1 suspend call, and the UI goes to suspending. So "retry" needs two clicks. Could toggleAgent start when the page has timed out, since the toggle is showing as off?
async toggleAgent () {
    this.loadingState = true
    try {
        const state = await brokerAPI.getBrokerStatus(this.team.id, this.$route.params.brokerId)
        if (state.state === 'running' && !this.startingTimedOut) {
            await brokerAPI.suspendBroker(this.team.id, this.$route.params.brokerId)
            this.brokerState = 'suspending'
        } else {
            // Fresh attempt to start, so give the agent the full window again
            this.resetStartingTimeout()
            await brokerAPI.startBroker(this.team.id, this.$route.params.brokerId)
            this.brokerState = 'starting'
        }
    } catch (err) {
        ...
    }
}

For a third-party broker whose row is running, the start route sends the agent a start command, so this is a real retry. A test that clicks the toggle after a timeout and asserts startBroker was called (and suspendBroker was not) would lock it in.

  1. Is 30s enough for STARTING_TIMEOUT on Kubernetes, where a cold agent pod (image pull, scheduling) can take longer? The user would see "Could not reach the broker agent" and then watch it flip to connected a moment later. A longer window might avoid the false alarm.

  2. Nit: the new const sits between two imports, and BrokerStatusBadge is imported below it. Moving the const under all the imports would read better.

After a timeout the toggle shows as off but the backend row is still running, so a click went down the suspend branch and the user needed two clicks to retry. The toggle now starts when the page has timed out, which sends the agent a start command. Two tests cover the retry and the normal suspend.

The wait window is 60s instead of 30s. On Kubernetes the driver returns as soon as the Deployment exists and doesn't wait for the pod, so a cold image pull sat entirely inside the old window and produced a false alarm.
@cstns

cstns commented Oct 21, 2026

Copy link
Copy Markdown
Contributor Author
  1. Yep, that was wrong. After a timeout the toggle shows off but the row is still running, so the click went down the suspend branch. It now starts when the page has timed out, and there's a test that clicks after a timeout and checks for one start call and no suspend, plus one that a healthy running agent still suspends.

  2. Fair point. The k8s driver returns as soon as the Deployment and Service exist, it doesn't wait for the pod, so a cold pull sits entirely inside the window. Bumped it to 60s. We'll revise if it turns out to still bite someone.

  3. Moved under the imports.

@cstns
cstns requested a review from andypalmi October 21, 2026 11:40
@andypalmi
andypalmi merged commit 23079bb into main Oct 21, 2026
29 checks passed
@andypalmi
andypalmi deleted the 8996-broker-schema-starting-timeout branch October 21, 2026 12:34

This branch was successfully deployed

1 active deployment
staging — 4f87bfcb Deployed Oct 21, 2026 by andypalmi via Remove application #12663
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Broker schema monitoring toggle spins on "starting" forever when the agent can't be reached

2 participants