[shawnburke]

Why

This is an audit of every log call in the gRPC tunnel agent (agent/server/grpctunnel) and server (server/tunnel, server/dispatch, server/adapters, server/broker, server/cmd).

Dispatch failures are already logged at Warn on main (failureLevel, #186). The main problem the audit found is the opposite: routine events are too loud. A server restart, a load balancer reset or the 24h connection limit ends every stream on a connection at once. Today that produces one Warn per stream on the agent and on the server, which hides the warnings that matter.

Lowered

Log line	Before	After
agent Tunnel slot stream ended; will retry	Warn	Info if the stream had been established; Warn only if it failed to open. Adds an established field.
server Tunnel stream closed (transport error)	Warn	Info (heartbeat timeout keeps its own Warn)
BROKER_SERVER request failed; will retry (per attempt)	Warn	Info (callers already warn when retries run out)
agent Initial registration aborted (only happens on shutdown)	Warn	Info
agent Registered with Cortex API	Info on every refresh	Info at startup, Debug on periodic refresh (rotation is still Info)
agent Call execution failed with cause AGENT_SHUTDOWN	Warn	Info
server Response body aborted mid-stream with side=caller	Warn	Info (side=tunnel stays Warn)
registry Registered new client / Removed client entry	Info	Debug. These duplicated Agent instance connected/disconnected, which carry more fields.

Raised

Log line	Before	After
agent Response body read failed mid-stream	Always Warn	Same rule as Call execution failed (shared callFailureLevel): Error for an upstream error, Warn for a server cancel or timeout, Info on shutdown
agent Request completed for a response that never reached CallEnd	Info	Warn, with aborted=true. The message is unchanged so existing queries still match.
server BROKER_SERVER client-connected exhausted retries	Warn forever	Error once the backoff reaches its cap (BROKER_SERVER thinks the token is offline). Adds tokenHash.

Not changed (deliberately)

Heartbeat timeouts, the stream-cap rejection, protocol violations (duplicate hello, call_id or CallStart; End before Start), invalid :status, TLS/credential errors, and the failureLevel levels for dispatch failures.

Testing

• go test ./server/grpctunnel/ (agent) and go test ./... (server) pass.
• TestManageStream_BackoffResetsAfterEstablishedStream now also checks the levels: three Warns for the failed opens, then Info for the dropped established stream.
• New TestCallFailureLevel.

🤖 Generated with Claude Code