Skip to content

ref(k8s): Use arroyo.healthcheck for the pipeline liveness probe - #389

Merged
oioki merged 3 commits into
mainfrom
ref/k8s-shell-free-liveness-probe
Nov 3, 2026
Merged

oioki merged 3 commits into
mainfrom
ref/k8s-shell-free-liveness-probe

Conversation

@oioki

@oioki oioki commented Oct 15, 2026 •

Copy link
Copy Markdown
Member

Changes the PipelineStep liveness probe from rm /tmp/health.txt to python -m arroyo.healthcheck /tmp/health.txt (getsentry/arroyo#580).

rm doesn't exist on distroless images, so there the probe fails and every pipeline pod gets restarted. The new command works on both slim and distroless images.

It needs sentry-arroyo>=2.46 in the image. super-big-consumers, the only production user of sentry-streams-k8s, already runs arroyo 2.47 in every region, so this is safe to merge. After release, bump sentry-streams-k8s in ops.

Refs SEC-1096

The pipeline liveness probe ran `rm /tmp/health.txt`, which fails on
images without coreutils (distroless/DHI), so the probe would fail and
restart every pipeline pod. Remove the file with Python instead. The
container command already relies on `python` being on PATH, so this
adds no new requirement on the image.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace the inline `python -c "import os; os.remove(...)"` snippet with
`python -m arroyo.healthcheck`, which arroyo ships for this probe since
2.46.0. Arroyo's Healthcheck strategy writes /tmp/health.txt, so arroyo
should own the check, and its failure message says what went wrong
instead of printing a traceback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@oioki oioki changed the title ref(k8s): Make the liveness probe work on distroless images ref(k8s): Use arroyo.healthcheck for the pipeline liveness probe Oct 15, 2026
…veness-probe

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@linear-code

linear-code Bot commented Nov 3, 2026

Copy link
Copy Markdown

SEC-1096

@oioki
oioki marked this pull request as ready for review November 3, 2026 23:29
@oioki
oioki requested a review from a team as a code owner November 3, 2026 23:29
@oioki
oioki merged commit f8c5577 into main Nov 3, 2026
26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants