ci(k8s): deploy the ActiveJob worker from cdo-rails

[stephenliang]

The Kubernetes cluster runs one workload. That workload is the ActiveJob worker. To run it, every push to staging, test, levelbuilder and production built the 4.13 GB code-dot-org image. Skaffold built it on two native runners. A second job stitched the result. The worker needs Rails, the gem set and bin/delayed_job. It does not need the frontend bundle, the curriculum, the asset pipeline or the build toolchain. Those are most of the 4.13 GB.

cdo-rails is that image without the parts the worker never reads. It is 0.40 GB compressed. It holds the same Rails source on the same gem set. Its own workflow already builds and tests it. A pull request runs the full matrix. That matrix is smoke tests on docker and podman across amd64 and arm64. It also boots the image against real MySQL and Redis on both architectures. A publish runs the docker smoke test and the same boot. It runs them against the bytes it is about to push, natively on each architecture. Nothing was wired to it.

The change

This repository stops deploying. It builds, verifies and publishes tags, and stops there.

Kargo already knows how to do the rest. The codeai Warehouse subscribes to a registry, and all four Stages already carry promotion templates that clone k8s-gitops, update the environment values file, commit and push, then sync ArgoCD. Today it watches code-dot-org. A companion pull request points it at cdo-rails.

So cdo-rails-image.yml barely changes. k8s.yml loses its push trigger and its k8s-gitops job, which leaves it close to what it was. k8s-commit-to-kargo-warehouse.yml is deleted, because nothing calls it any more.

The publish trigger loses its paths: filter, so every push to staging builds, verifies and publishes. docker/rails/Dockerfile.dockerignore already defines the slice, and only docker matches against it. A filter in a workflow would be a second copy of that list, and the two would drift apart.

That means a push changing nothing the image contains still publishes and still deploys. This is what the pipeline being replaced already did, on a much larger image, so it is accepted rather than solved here. The worker restarts cleanly by design. Dropping the redundant deploys needs a way to tell docker-visible change from commit change, which is a change worth making on its own evidence and not as a rider on this one.

Publishing per push grows the registry. GHCR retention already prunes untagged manifests for these images, and the same lever extends to old git-<sha> tags when it is worth pulling.

A lockfile push is the one case the build cannot serve directly. It changes the content key that names the cdo-deps layer, and that layer does not exist yet. cdo-deps-image is building it at that moment. So a push with no layer ends the run with a notice and publishes nothing, and the cdo-deps chain runs this workflow again once the layer lands. Any other event still fails loudly, because nothing will re-run it.

flowchart TB
    push["push to staging"] --> img["build the image<br/>smoke + boot-verify"]
    chain["cdo-deps-image chain"] --> img
    img --> tag["publish git-sha + latest"]
    tag --> wh["Kargo Warehouse<br/>sees latest's digest"]
    wh --> fr["Freight"] --> promo["promote a Stage"]
    promo --> gitops["Kargo writes<br/>deployments/env/values.yaml"]
    gitops --> argo["ArgoCD → ActiveJob worker"]

Loading

What now triggers a deploy

event	new
any push to staging	build, smoke, boot-verify, publish, Freight
a lockfile push to staging	notice, then build through the cdo-deps chain
a push to test, levelbuilder or production	nothing
any pull request	image CI on docker/rails/ changes, skaffold on *k8s* branches

The accepted regression

test, levelbuilder and production stop following their branches for this image. Only staging publishes. Promotion between Kargo Stages replaces DTT and DTP for the worker.

This is a real change in how those environments advance, and it applies only to the ActiveJob worker. Nothing else runs on the cluster. Note that the Stage graph is not a straight line. test promotes from staging, and both levelbuilder and production promote from test.

Links

• Jira: N/A
• Companion k8s-gitops pull request: points the Warehouse and the four Stages at cdo-rails
• cdo-rails multi-arch publish: build(docker): publish cdo-deps and cdo-rails multi-arch #74592

Testing story

The publish path ran on real runners from a dispatch of this branch. Both build-push legs succeeded natively, pushing sha256:d2b0d07b… for amd64 and sha256:69c33243… for arm64, and the merge job combined them. That is the whole pipeline this change ships.

actionlint 1.7.12 reports no findings on either changed workflow. It ran shellcheck 0.11.0 over the run: blocks. Both files parse under yq and PyYAML. tools/hooks/pre-commit is clean. Nothing references the deleted workflow.

helm template runs on the unchanged chart. The inputs are the live k8s-gitops staging values and the image ghcr.io/code-dot-org/cdo-rails:git-TEST. It renders the active-job-worker Deployment on that image. The command and args stay unchanged. No other image appears anywhere in the render.

ghcr.io/code-dot-org/cdo-rails:latest boots RAILS_ENV=staging against MySQL and Redis. That is the environment the staging worker actually runs. rails runner prints staging. The result was reproduced in two independent runs. The boot needs 24 config values that the adhoc boot does not. In the cluster the External Secrets Operator syncs staging/cdo/* into the pod as CDO_*. That is the same source the local boot tried and could not authenticate to.

The chart-deployed worker was run on a disposable kind cluster, against the image this branch publishes. That found the two chart values this pull request changes.

check	result
worker on the chart's 2Gi limit	OOMKilled in a loop, about six seconds per attempt
worker memory once given room	rises 2365Mi, 3046Mi, 3053Mi, then flat at 3059Mi, no restarts
worker pod with 4Gi	1/1 Running, 0 restarts
a real SampleJob enqueued from a separate exec	the running daemon logs RUNNING then COMPLETED after 5.0110
the delayed_jobs row	0, then 1 locked by host:cdo-active-job-worker-... pid:14, then 0
CDO_active_job_queue_adapter in the pod	:delayed_job, from the chart's locals ConfigMap through envFrom

SKIP_SCRIPT_PRELOAD was the other finding. Without it the boot runs the curriculum preload, which reads a table the slice does not carry, and on an empty database it aborts before db:schema:load can create that table.

A hypothesis worth recording rather than asserting. The memory the old image needed and the memory this one needs may differ because cdo-rails ships every locale on the i18n load path, where the image it replaces was English-only. That is untested.

The subscription was proven end to end on a disposable local cluster running the same Kargo the cluster runs. The warehouse file from the companion pull request was applied byte for byte, and the Warehouse reported Ready=True with Successfully discovered artifacts from 1 subscriptions about two seconds later. The Freight it created recorded ghcr.io/code-dot-org/cdo-rails at sha256:8b85efa7da0dcee1771f4f40725562e27b0a3bdc62d3df4ed88cca6abf32b9da, which is what the registry reports for the latest index. A promotion against a local git server then wrote image: ghcr.io/code-dot-org/cdo-rails@sha256:8b85efa7… into the staging values file and committed Promote staging to sha256:8b85efa7… [skip ci].

One path only CI can exercise, which is the notice a push takes when the cdo-deps layer is missing. It is shaped so that a wrong answer fails the run rather than publishing on a stale dependency layer.

Deployment notes

k8s/experimental.yaml names Seth Nickell as the owner of this area. Merging waits on his review.

The companion pull request also moves the subscription to the Digest strategy watching latest, so Freight is the image that tag resolves to and promotion pins by digest. NewestBuild would work too. Digest is preferred because it ties Freight to a tag rather than to timestamp behaviour this side makes no promise about. Every staging publish becomes new Freight, staging takes it automatically, and the other stages promote on demand.

Merge this repository first. Between the two merges nothing moves. Each environment stays on the code-dot-org ref already pinned in its values file, and the worker keeps running. The cutover happens when the k8s-gitops pull request merges and the first cdo-rails Freight promotes.

Rollback is a revert of the k8s-gitops pull request. The old code-dot-org tags stay in the registry.

The chart changes in exactly two worker values, both found by running it. activeJobWorker memory goes from 2Gi to 4Gi, and SKIP_SCRIPT_PRELOAD=1 joins its extraEnv. The kustomize twin carries the same two. Everything else stays as it is. Per-component image values, cdo-web, cdo-migrate, and the worker's command are untouched. Scope is the worker, and nothing else runs on the cluster today.

🤖 Generated with Claude Code

[github-actions]

🖼️ Storybook Visual Comparison Report

✅ No Storybook eyes differences detected!

[carl-codeorg]

LGTM, just a couple minor comments

[carl-codeorg]

Can probably truncate or remove this comment. This system is still pre-production so there's no need to references how things used to work with it.

[stephenliang]

Cleaned up the comment.

[carl-codeorg]

Comment seems like unnecessary explanation here

[stephenliang]

Cleaned up the comment.

[carl-codeorg]

Does it rarely exceed 3.2Gi? If so we could separate the memory out into a 3Gi request and 4Gi limit to make more efficient use of memory.

[stephenliang]

3.2 was the steady state on my local Kind k8s cluster.