Repository navigation
chore(orchestrator): Move per-visit sweep logs to DEBUG and add a heartbeat - #419
morgan-wowk wants to merge 1 commit into
Conversation
4ebb687 to
ac47a1e
Compare
| _logger.info( | ||
| f"Before processing running container execution. Queries duration {queries_duration_ms}ms." | ||
| ) | ||
| start_timestamp = time.monotonic_ns() |
There was a problem hiding this comment.
Is moving this start_timestamp here intentional and any beahviour changes?
There was a problem hiding this comment.
🤖 AI-assisted: Yes, intentional. start_timestamp used to be set before the execution-node-IDs query, so that query was counted in both queries_duration_ms and the processing duration. Setting it after the queries makes the two numbers in the slow-visit line separate, which tells a slow database apart from a slow launcher. No other behaviour changes: duration_ms only feeds that log line.
Also added in f0daa92: a once-a-minute heartbeat (Orchestrator heartbeat: processed N queued executions and M running container executions in the last 60s.). The per-visit lines we removed were the only sign the sweep was alive, so the heartbeat keeps that signal at one line a minute. If it stops appearing, the sweep has stopped.
ac47a1e to
f0daa92
Compare
f0daa92 to
4d8615d
Compare
9f0aa73 to
d7b523d
Compare
4d8615d to
27ead6d
Compare
d7b523d to
75a55e6
Compare
27ead6d to
227f42a
Compare
75a55e6 to
89b07cc
Compare
227f42a to
6f573cb
Compare
89b07cc to
8da2f52
Compare
6f573cb to
1dbfb6d
Compare
8da2f52 to
aa4bb4e
Compare
99b4d85 to
750469f
Compare
750469f to
e580072
Compare
Would it help to switch them to DEBUG for most cases?
Can you elaborate? A hung sweep is exactly the issue that the current logging helps debug. We see an "open parenthesis" without a "closing parenthesis". I think that this information belongs to the debug logs. Many libraries have even more verbose debug logging (e.g. SSL and HTTP debug logging). We can just limit the log level visibility if needed. |
…rtbeat The running sweep logged three INFO lines on every visit, even when nothing changed, so the volume grew with active executions and sweep rate. Those lines are now DEBUG, where they still pair up (Before/After) when debugging a hung visit. At INFO the orchestrator now logs: - a slow-visit line when queries or processing take over 1s; - a heartbeat about once a minute with queued and running counts, even when idle, so a stopped sweep shows up as a missing heartbeat.
e580072 to
193de5a
Compare
|
🤖 AI-assisted reply to #419 (comment) Agreed, switched them to DEBUG in 193de5a instead of removing them, so the Before/After pair is still there when debugging a hung visit. "Hard to alert on" meant alerting, not debugging: an alert can't fire on a closing line that never shows up, but it can on a missing heartbeat. So the heartbeat (and a slow-visit line over 1s) stay at INFO. |
Problem
Each running-sweep visit logs three INFO lines, even when nothing changed. Volume grows with active executions and sweep rate.
Changes
Before processing …,After processing …and… remains in … state.move to DEBUG. Turning on DEBUG still shows a Before without an After for a hung visit.Slow running container execution visitwhen queries or processing take over 1s.Orchestrator heartbeatwith queued and running counts about once a minute, even when idle. A missing heartbeat means the sweep stopped, which is easy to alert on.container_execution.refresh.durationmetric from feat(orchestrator): Add refresh duration and completion lag metrics #425.Shares a few lines with #426 and #427; the second to merge needs a small rebase.