Skip to content

chore(orchestrator): Move per-visit sweep logs to DEBUG and add a heartbeat - #419

Open
morgan-wowk wants to merge 1 commit into
masterfrom
orchestrator-quiet-running-sweep
Open

morgan-wowk wants to merge 1 commit into
masterfrom
orchestrator-quiet-running-sweep

Conversation

@morgan-wowk

@morgan-wowk morgan-wowk commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

Each running-sweep visit logs three INFO lines, even when nothing changed. Volume grows with active executions and sweep rate.

Changes

  • Before processing …, After processing … and … remains in … state. move to DEBUG. Turning on DEBUG still shows a Before without an After for a hung visit.
  • New INFO Slow running container execution visit when queries or processing take over 1s.
  • New INFO Orchestrator heartbeat with queued and running counts about once a minute, even when idle. A missing heartbeat means the sweep stopped, which is easy to alert on.
  • Durations are also in the container_execution.refresh.duration metric from feat(orchestrator): Add refresh duration and completion lag metrics #425.

Shares a few lines with #426 and #427; the second to merge needs a small rebase.

@morgan-wowk
morgan-wowk requested a review from a team October 30, 2026 23:18
@morgan-wowk
morgan-wowk requested a review from Ark-kun as a code owner October 30, 2026 23:18
@morgan-wowk
morgan-wowk force-pushed the orchestrator-quiet-running-sweep branch from 4ebb687 to ac47a1e Compare October 30, 2026 23:29
_logger.info(
f"Before processing running container execution. Queries duration {queries_duration_ms}ms."
)
start_timestamp = time.monotonic_ns()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is moving this start_timestamp here intentional and any beahviour changes?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI-assisted: Yes, intentional. start_timestamp used to be set before the execution-node-IDs query, so that query was counted in both queries_duration_ms and the processing duration. Setting it after the queries makes the two numbers in the slow-visit line separate, which tells a slow database apart from a slow launcher. No other behaviour changes: duration_ms only feeds that log line.

Also added in f0daa92: a once-a-minute heartbeat (Orchestrator heartbeat: processed N queued executions and M running container executions in the last 60s.). The per-visit lines we removed were the only sign the sweep was alive, so the heartbeat keeps that signal at one line a minute. If it stops appearing, the sweep has stopped.

@morgan-wowk
morgan-wowk force-pushed the orchestrator-quiet-running-sweep branch from ac47a1e to f0daa92 Compare October 30, 2026 23:42
@morgan-wowk morgan-wowk changed the title chore(orchestrator): Stop logging every running container execution visit chore(orchestrator): Replace per-visit sweep logs with a heartbeat Oct 30, 2026
@morgan-wowk
morgan-wowk force-pushed the orchestrator-quiet-running-sweep branch from f0daa92 to 4d8615d Compare November 2, 2026 22:42
@morgan-wowk
morgan-wowk changed the base branch from master to orchestrator-sweep-metrics November 2, 2026 22:42
@morgan-wowk
morgan-wowk force-pushed the orchestrator-sweep-metrics branch from 9f0aa73 to d7b523d Compare November 2, 2026 22:47
@morgan-wowk
morgan-wowk force-pushed the orchestrator-quiet-running-sweep branch from 4d8615d to 27ead6d Compare November 2, 2026 22:47
@morgan-wowk
morgan-wowk force-pushed the orchestrator-sweep-metrics branch from d7b523d to 75a55e6 Compare November 2, 2026 22:49
@morgan-wowk
morgan-wowk force-pushed the orchestrator-quiet-running-sweep branch from 27ead6d to 227f42a Compare November 2, 2026 22:49
@morgan-wowk
morgan-wowk force-pushed the orchestrator-sweep-metrics branch from 75a55e6 to 89b07cc Compare November 2, 2026 22:52
@morgan-wowk
morgan-wowk force-pushed the orchestrator-quiet-running-sweep branch from 227f42a to 6f573cb Compare November 2, 2026 22:53
@morgan-wowk
morgan-wowk force-pushed the orchestrator-sweep-metrics branch from 89b07cc to 8da2f52 Compare November 3, 2026 00:40
@morgan-wowk
morgan-wowk force-pushed the orchestrator-quiet-running-sweep branch from 6f573cb to 1dbfb6d Compare November 3, 2026 00:41
@morgan-wowk
morgan-wowk force-pushed the orchestrator-sweep-metrics branch from 8da2f52 to aa4bb4e Compare November 3, 2026 03:51
@morgan-wowk
morgan-wowk force-pushed the orchestrator-quiet-running-sweep branch 2 times, most recently from 99b4d85 to 750469f Compare November 3, 2026 04:21
@morgan-wowk
morgan-wowk changed the base branch from orchestrator-sweep-metrics to master November 3, 2026 04:21
@morgan-wowk
morgan-wowk force-pushed the orchestrator-quiet-running-sweep branch from 750469f to e580072 Compare November 16, 2026 19:55
@Ark-kun

Ark-kun commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Each running-sweep visit logs three INFO lines

Would it help to switch them to DEBUG for most cases?

A hung sweep is hard to alert on.

Can you elaborate? A hung sweep is exactly the issue that the current logging helps debug. We see an "open parenthesis" without a "closing parenthesis".

I think that this information belongs to the debug logs. Many libraries have even more verbose debug logging (e.g. SSL and HTTP debug logging). We can just limit the log level visibility if needed.

…rtbeat

The running sweep logged three INFO lines on every visit, even when nothing
changed, so the volume grew with active executions and sweep rate. Those lines
are now DEBUG, where they still pair up (Before/After) when debugging a hung
visit.

At INFO the orchestrator now logs:
- a slow-visit line when queries or processing take over 1s;
- a heartbeat about once a minute with queued and running counts, even when
  idle, so a stopped sweep shows up as a missing heartbeat.
@morgan-wowk
morgan-wowk force-pushed the orchestrator-quiet-running-sweep branch from e580072 to 193de5a Compare November 18, 2026 00:09
@morgan-wowk morgan-wowk changed the title chore(orchestrator): Replace per-visit sweep logs with a heartbeat chore(orchestrator): Move per-visit sweep logs to DEBUG and add a heartbeat Nov 18, 2026
@morgan-wowk

Copy link
Copy Markdown
Collaborator Author

🤖 AI-assisted reply to #419 (comment)

Agreed, switched them to DEBUG in 193de5a instead of removing them, so the Before/After pair is still there when debugging a hung visit.

"Hard to alert on" meant alerting, not debugging: an alert can't fire on a closing line that never shows up, but it can on a missing heartbeat. So the heartbeat (and a slow-visit line over 1s) stay at INFO.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants