Skip to content

nan-vm-operator dev deploy never takes effect: CI bumps a values file the dev ArgoCD app does not read #205

Open

Description

@sre-helmcode

Finding (2026-10-20)

The nan-vm-operator deploy workflow (deploy-dev job) commits the new dev-<sha> tag to helm/nan-vm-operator/values-dev.yaml in nan-devops.

The ArgoCD app prod-nan-vm-operator-dev renders with kubernetes/apps/prod/default/nan-system-dev/nan-vm-operator-dev/values.yaml, which pins:

image:
  tag: dev-42ed1632782093d657d20021af8557335090e3dc

The comment next to that pin says dev CI rewrites it, but nothing does. As a result, the dev operator still runs the image from nan-vm-operator#42, while prod runs v0.0.53-alpha (lease + leader-epoch fencing). Every dev deploy since then has been a no-op.

The app is also sync-manual. On 2026-10-20 it was synced, but only labels changed and the image stayed at dev-42ed163.

Fix options

  1. Point the deploy-dev commit-back at kubernetes/apps/prod/default/nan-system-dev/nan-vm-operator-dev/values.yaml (image.tag).
  2. Or make the dev app also read helm/nan-vm-operator/values-dev.yaml and drop the pin.

After the fix, bump dev and sync it: argocd --core app diff/sync prod-nan-vm-operator-dev. Then verify the pod image and that operator_leader_epoch in nan_platform_dev gets an epoch, since migration 124 is already applied there.

Context: the 2026-10-19 operator incident, postmortem in nan-devops#909.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions