[Medformatik]

What changed and why

An update recreated the operations agent while it was rebuilding the Valhalla traffic extract (valhalla.traffic.maintain). The journal correctly failed the job as recovery_required, but its fence valhalla/osm-pbf/.traffic-maintenance.json stayed. Valhalla stayed stopped, and data-manager's supervisor kept its worker paused without logging anything. That worker is the only component that requests the next maintenance, so nothing could recover. On openmapx.com this lasted about 8 hours (2026-10-30 15:45 UTC until a manual recovery):

• walking directions returned 502,
• driving fell back to OSRM without road conditions,
• live traffic and every other data-manager job stopped, and the data-manager health check failed.

Operations agent. At startup, right after the journal opens, recoverInterruptedTrafficMaintenance matches the fence to its job: same fingerprint, and the fence written within that job's deadline after its submission. Every rebuild shares one fingerprint, and only interrupted jobs outlive the journal's retention, so the time check keeps an old record from claiming a newer fence.

When the matched job was interrupted, the agent:

1. removes the job's openmapx-valhalla-maint-* container and confirms it is gone,
2. confirms Valhalla is stopped,
3. withdraws traffic-generations.json and traffic-generations.unverified.json,
4. keeps the fence as .traffic-maintenance.interrupted-<id>.json and removes it.

The worker then resumes, the next inspection reports not_ready, and the worker requests a fresh coordinated maintenance. This automates the manual recovery used on openmapx.com, which completed in 177 s.

A job that failed on its own keeps its fence, because retrying a broken graph would only fail again. The agent leaves the fence in place and logs why whenever it can't prove these conditions: no matching job, a container that survives removal, or Valhalla still running.

The fence lives where data-manager can write, so it is read through a new bounded, symlink-refusing AnchoredDirectory.readFile, and only plan-shaped values from it are logged.

Data manager. The supervisor logs once when it pauses its worker for maintenance (job, plan, start time), warns every 30 minutes once a pause outlasts any maintenance, and logs the resume.

Docs. The monitoring guide gains the two log lines and a "Graph maintenance pauses" section with the manual recovery for a maintenance that failed on its own. The engine verification notes describe the release.

Verification

• New tests for the recovery:
  • an interrupted job is released (tool container removed, Valhalla not started, manifests withdrawn, fence kept as evidence),
  • a job that failed on its own is kept,
  • an older interrupted record sharing the fingerprint is kept,
  • no journal job is kept,
  • a surviving container is kept,
  • a running Valhalla is kept,
  • no fence does nothing,
  • a symlinked fence is refused.
• New tests for readFile (bounded, absent, symlink, directory) and for the supervisor's pause reporting.
• The fence's operationId was checked against the production journal: it equals the interrupted job's fingerprint.
• vitest --project node: 1362 files, 18,504 tests passed. apps/ops-agent and the supervision tests passed again after the last edits.
• tsc --noEmit for @openmapx/ops-agent and @openmapx/data-manager, Biome and Prettier: clean.
• Not run: the Docker-backed verify:traffic-maintenance probe, and a real agent restart during maintenance. Linux descriptor-anchored reads are covered only by the Linux CI leg of the anchored-directory tests.

Not addressed here: updates still recreate the agent while it runs an async job. This change makes that recoverable, not avoided.

Release and upgrade impact

Product impact: ordinary