Skip to content

fix(security): repair the flux-operator post-renderer so its security fields apply - #3580

Merged
devantler merged 2 commits into
mainfrom
claude/flux-operator-postrender-parent-path
Sep 4, 2026
Merged

fix(security): repair the flux-operator post-renderer so its security fields apply#3580
devantler merged 2 commits into
mainfrom
claude/flux-operator-postrender-parent-path

Conversation

@devantler

Copy link
Copy Markdown
Contributor

🤖 Generated by the Agentic Engineer

Why

The change that shipped in #3577 does not work, and it left the Flux Operator's release unable to
reconcile. The helm-controller has been reporting the operator as healthy the whole time, so nothing
surfaced it.

The intent was right: give the Flux Operator two security fields it was missing. But one of the two
was written in a form that only works if the chart already provides a place to put it, and this chart
does not. Because that style of edit is all-or-nothing, the one field that could not be placed took
the other one down with it — so neither landed, and every attempt to update the operator since has
failed the same way.

Left alone this gets worse rather than staying still: the next routine version bump of the operator
would fail to apply for the same reason.

What

Splits the two fields into the two forms the chart actually needs — the one that has a place to go
keeps the precise form, and the one that does not now uses a form that creates its own place. That
also makes it safe if a future chart version starts setting these itself, which the previous approach
would have silently overwritten.

Confirmed by reproducing the exact failure, applying the fix, and checking the result against the
real chart rather than against the running cluster — reading the cluster is what produced the wrong
conclusion the first time.

Part of #3239

…lds apply

#3577 placed both fields in one JSON-6902 patch. The pod-level op targets
/spec/template/spec/securityContext/fsGroupChangePolicy, but flux-operator
0.50.0 renders no pod-level securityContext at all, so the op fails
`doc is missing path` — and a JSON patch is all-or-nothing, so it took the
container ops down with it. Neither field landed, and the HelmRelease has been
unable to run an upgrade since (helm-controller 2026-09-04T10:41:27.775Z),
while latching observedPostRenderersDigest and reporting Ready=True.

The `{}` the previous comment cited was read from the live Deployment, where
the API server serialises the absent object; the rendered chart is the surface
a post-renderer patches, and there it is absent.

Splits the pod-level field into a strategic-merge patch, which creates the
missing parent and merges into a chart-set one — so a future chart that starts
rendering it keeps its fields, which a whole-object `add` would have discarded.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…post-renderer repair

The repair changes the flux-operator HelmRelease's rendered text, and that
HelmRelease is a selected authorization-capable document, so the aggregate
surface digest moves — the tripwire working as designed, not a privilege change.

Conservation re-proven with one renderer on both sides: all five overlays from
main d7613b0 and from this branch, compared over the full
apiVersion|kind|namespace|name identity in both directions. 552 documents each
side, zero added, zero removed; a negative control that drops one document
reports removed=1 and names it. Four of five roots are byte-identical and the
whole delta is 3 lines removed / 13 added inside one postRenderers block.

Digest taken from CI run 33866934697, which pins kubectl v1.36.2; that run
reported this fingerprint as its only error class.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

@coderabbitai review

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

Readiness record.

Programmatically tested. RED: the pre-fix patch reproduces helm-controller's exact failure through
kubectl kustomizeadd operation does not apply: doc is missing path: "/spec/template/spec/securityContext/fsGroupChangePolicy": missing value, byte-identical to the
controller's 10:41:27.775Z log line. GREEN: driving the kustomization built from this file's own
postRenderers block against the chart's real rendered Deployment yields
fsGroupChangePolicy: OnRootMismatch at pod level and seLinuxOptions.level: s0 on the container,
with every chart-set container field (allowPrivilegeEscalation, capabilities.drop,
readOnlyRootFilesystem, runAsNonRoot, seccompProfile) preserved.

Control on the design choice, not just the outcome. Against a fixture representing a future chart
that does render a pod-level securityContext carrying runAsNonRoot: true: the strategic-merge
form keeps that sibling, and the whole-object add form silently destroys it. That is why this is a
merge rather than an object add — it repairs the failure and keeps the property the previous
comment was trying to protect.

Evidence for the diagnosis itself. helm template flux-operator 0.50.0 with this HelmRelease's own
values renders no pod-level securityContext — the {} the previous comment cited was the API
server's serialisation of an absent object on the live Deployment, which is not the surface a
post-renderer patches.

Blast radius measured, not assumed. The four FluxInstance controllers from #3544 do carry both
fields live (OnRootMismatch / s0), so that pattern is unaffected; only flux-operator broke
(tofu-controller is the retiring one, #3480). Strategic-merge post-renderer patches are already the
established form in 9 of this repo's 17 HelmReleases carrying postRenderers.

Validation. ksail workload validate and ksail --config ksail.prod.yaml workload validate both
pass — 602 files each. gofmt clean, go vet clean, go test ./scripts/validate-eks-ci-role-policy
passes.

Authorization digest. Conservation proven with one renderer on both sides across all five overlays:
552 documents each side, zero added, zero removed, both directions; a negative control dropping one
document reports removed=1 and names it. Four of five roots render byte-identical; the entire delta
is 3 lines removed / 13 added inside one postRenderers block, touching no rule, verb, resource,
subject or role reference. The digest itself is CI's own (run 33866934697, pinned kubectl v1.36.2),
which reported that fingerprint as its only error class.

Tried and evaluated as a user — with a caveat I want to be explicit about. The failure this fixes
was observed on the live cluster, and the fix's rendering behaviour is exercised directly above. What
cannot be exercised before merge is helm-controller actually consuming it, because that only happens
once this reaches the cluster. So the post-merge check is part of the work, not a formality: after this
reconciles I will re-read the live Deployment and confirm both fields are present and the Helm release
has a new revision — the precise check whose absence let #3577 look successful for 20 minutes.

@coderabbitai

coderabbitai Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

@devantler I will review the changes in #3580.

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai

coderabbitai Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: ASSERTIVE

Plan: Team

Run ID: 68a7b12a-5cad-40f5-a18e-a7ba515d4c69

📥 Commits

Reviewing files that changed from the base of the PR and between d7613b0 and d9d4075.

📒 Files selected for processing (2)
  • k8s/bases/infrastructure/controllers/flux-operator/helm-release.yaml
  • scripts/validate-eks-ci-role-policy/main.go

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

📜 Recent review details
🧰 Additional context used
📓 Path-based instructions (1)
Never run a cluster Put a change in the layer that matches its scope

📄 CodeRabbit inference engine (AGENTS.md)

Files:

  • k8s/bases/infrastructure/controllers/flux-operator/helm-release.yaml
🧠 Learnings (1)
📚 Learning: 2026-08-11T12:41:28.242Z
Learnt from: devantler
Repo: devantler-tech/platform PR: 3082
File: k8s/bases/infrastructure/controllers/coroot/cron-job-cnpg-degraded-alert.yaml:113-120
Timestamp: 2026-08-11T12:41:28.242Z
Learning: When changing behavior in Kubernetes manifests or related documentation, review comments and documentation in YAML/YML and Markdown files for statements describing the previous behavior. Update every stale statement in the same change so the repository’s explanatory text remains consistent with the implementation.

Applied to files:

  • k8s/bases/infrastructure/controllers/flux-operator/helm-release.yaml
🔇 Additional comments (3)
k8s/bases/infrastructure/controllers/flux-operator/helm-release.yaml (2)

76-92: LGTM!


113-125: LGTM!

scripts/validate-eks-ci-role-policy/main.go (1)

1687-1720: LGTM!


📝 Walkthrough

Walkthrough

The Flux operator HelmRelease post-renderer now applies the pod-level securityContext through a strategic-merge patch that creates the missing parent object. The container-level securityContext remains a JSON-6902 patch. The validation record documents the correction and updates the expected rendered authorization-surface fingerprint.

Merge Risk: ⚪ Minimal · up to d9d40

This corrects Flux Operator rendering so the pod security setting is applied without preventing container security settings from rendering. The validated change preserves existing security fields and does not alter the authorization resource surface.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 1…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly identifies the security fix and the affected Flux Operator post-renderer.
Description check ✅ Passed The description directly explains the failed JSON-6902 patch, the strategic-merge fix, and the validation performed.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@devantler
devantler marked this pull request as ready for review September 4, 2026 11:48
@devantler
devantler added this pull request to the merge queue Sep 4, 2026
Merged via the queue into main with commit 7529ec4 Sep 4, 2026
27 checks passed
@devantler
devantler deleted the claude/flux-operator-postrender-parent-path branch September 4, 2026 12:02
@github-project-automation github-project-automation Bot moved this from 🫴 Ready to ✅ Done in 🌊 Project Board Sep 4, 2026
@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

Post-merge verification on the live cluster — the check whose absence let #3577 look successful.

Merged 12:02:12Z; headRefOid d9d4075e… is the head that was evaluated and reviewed (the merge
commit differs, as a queue merge always does).

Read from PROD at 12:03:03Z:

before after
Helm release revision .v10, last deployed 2026-08-30T12:20:34Z .v11, 2026-09-04T11:57:46Z
Ready condition Ready=True — quoting the previous upgrade Ready=True TestSucceeded for .v11
Deployment generation 7 8 (observed 8, 1/1 ready)
pod securityContext {} {"fsGroupChangePolicy":"OnRootMismatch"}
container seLinuxOptions absent {"level":"s0"}

Both fields are present, a genuinely new Helm revision exists, and the pod rebuilt and became ready —
so the guardrail #3239 records (a securityContext injection that stops the sandbox building) did not
fire.

Worth stating plainly, because it is the reason this PR exists: the previous attempt merged green on
every GitHub surface and changed nothing.
The distinguishing signal was never the PR state — it was
the Helm revision timestamp still sitting five days in the past while the release reported healthy. A
readiness check could not have caught it either, because the failure latched
observedPostRenderersDigest and went back to reporting Ready=True. #3581 tracks the CI gate that
would have caught it before merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: ✅ Done

Development

Successfully merging this pull request may close these issues.

1 participant