Repository navigation
fix(argocd): report a Degraded rollout as a failed deployment - #9354
Merged
Merged
Conversation
A sync can succeed while the rollout it starts never becomes healthy, and the plugin only looks at the sync phase, so such a deployment was stored with result SUCCESS. Record the application's health on the newest history entry when it is collected and turn a Succeeded sync whose rollout is Degraded into result FAILURE. Signed-off-by: Zihan Dai <dzh1436286758@gmail.com>
klesh
approved these changes
Nov 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
An Argo CD sync can succeed while the rollout it starts fails: the manifests apply, the new pods then crash-loop and the application goes Degraded. The plugin only reads the sync phase, so those deployments were saved in
cicd_deploymentsandcicd_deployment_commitswith result SUCCESS and counted as good deployments._tool_argocd_sync_operations.health_statushas been in the schema from the start, but nothing ever filled it.Argo CD keeps health on the application, not per sync, and
status.history[]entries carry no health. So it can only be tied to a sync while that sync is the newest one. The collector now writes the application's health onto the newest history entry when it reads the application. IfoperationStateshows a newer sync that failed or is still running, nothing is written, because that health does not belong to the last history entry. The next collection returns the same entry without the value, so the extractor keeps the last health it saw for that sync and stores it inhealth_status. The convertor reports a Succeeded sync whose health is Degraded as FAILURE, and setsoriginal_resultto Degraded so the reason sits next tooriginal_statusSucceeded.Only Degraded changes the result. Progressing and the other states are left as they are: Progressing is normal right after every sync, and a run that lands mid-rollout should not take the deployment out of the DORA numbers. The issue also says failed operations get dropped, but
isRedundantOperationStateonly skips Succeeded, so Failed and Error already come through as FAILURE and I did not touch that. History entries that were never the newest one during a collection, and everything after a full resync (which clears the raw rows), have no health recorded and stay SUCCESS. Argo CD offers nothing to recover it from.How I tested it. I ran the real collect, extract and convert subtasks in Docker against MySQL 8.4 and a stub Argo CD API that returns an application with two history entries and health Degraded. Unmodified main stored SUCCESS for both deployments. With the change, the newest one is FAILURE and the older one SUCCESS. After a second incremental run where a newer sync landed and the application was Healthy again, the Degraded deployment stayed FAILURE and the new one was SUCCESS. An application whose operation is Failed leaves the last good deployment SUCCESS. A Healthy application is unchanged.
TestSyncOperationDataFlowis a new e2e test that feeds raw rows from two collections through the extractor and convertor. With the production changes reverted it fails withexpected: "FAILURE" actual: "SUCCESS"on the Degraded deployment, and without the carry-over in the extractor it fails the same way. The unit tests cover which entry gets tagged and the result mapping.go test ./plugins/argocd/..., gofmt and golangci-lint v2.15.0 pass.I did not run this against a real Argo CD server: the stub uses the field names from Argo CD's API types. I did not run it on PostgreSQL, and I did not run the e2e suites of other plugins.
Does this close any open issues?
Closes #9329
Screenshots
None.
Other Information
None.