Repository navigation
fix(revoke): make the Azure in-flight flip an exclusive claim so only one revoke posts a Return - #1520
Conversation
…00059 re-applicable after a rollback FlipPurchaseRevocationInFlight was an unconditional UPDATE, so a stale concurrent Azure revoke reaching it after another request's MarkPurchaseRevoked set revocation_in_flight on a revoked row, where nothing clears it. The flip now only matches revoked_at IS NULL; a revoked row returns ErrPurchaseAlreadyRevoked (a missing row keeps its not-found error), and the handler answers already_revoked without posting a second Azure Return. Rolling back through 000059 and re-applying it failed: 000064's down is a no-op, so Purchaser stays at ...0007 and 000059's guard refused it as a foreign "Purchaser". 000059 now yields to that row (and restores its system_managed flag), and its admin backfill only targets ...0005 when that row really is Purchaser, so a re-apply no longer adds admins to Standard Users. Adds a 000059 down/up round-trip test and a down-to-version-1 round trip. No new migration. Refs LeanerCloud#316 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… one revoke posts a Return FlipPurchaseRevocationInFlight let a row that was already in flight be flipped again, so two first-time revokes of one Azure purchase could both pass it and both post a Return. Azure refused the second, and that request's error path cleared the flag while the first request's refund was still in flight; if the first request's MarkPurchaseRevoked then failed, the row was left neither revoked nor flagged, out of reach of the finalize_revocations sweep. - The flip is now a compare-and-set on revoked_at IS NULL AND revocation_in_flight = false. The loser gets the new config.ErrRevocationInFlight. - callAzureReturn answers RECONCILE_PENDING (the up-front in-flight response) to the loser and posts no Return. Any other flip error now stops before the Return with a 500 instead of posting one unclaimed. So only the claim holder reaches the Return and its ClearRevocationInFlight only clears its own flag. - OpenAPI: RECONCILE_PENDING also covers a concurrent revoke posting the return. Tests: real-Postgres race of two first-time revokes through the handler (exactly one Return, the other answers RECONCILE_PENDING, the flag survives while the winner's Return is in flight); store race of 6 concurrent flips (exactly one claims); handler unit tests for the loser and the flip-error path. Closes LeanerCloud#1518 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Independent review (IAmJSD's agent, fresh context, given only the diff, the issue and the PR text): VERDICT: APPROVE Blocking:
Non-blocking:
Attacked:
Body-stale:
Reviewer: rF9. Bar while the maintainers are away (set by IAmJSD as acting owner): one approving independent review on the head, two for money, auth and security paths. |
LeanerCloud#1516 was squash-merged, so this branch's copies of its commits conflicted with main in the six files both PRs touch. Resolved to main plus this PR's own commit (a81a3c3): the merged tree is identical to cherry-picking a81a3c3 onto origin/main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Independent review (IAmJSD's agent, fresh context, given only the diff, the issue and the PR text): VERDICT: APPROVE Blocking:
Non-blocking:
Attacked:
Body-stale:
Reviewer: rF10. Bar while the maintainers are away (set by IAmJSD as acting owner): one approving independent review on the head, two for money, auth and security paths. |
|
Independent review (IAmJSD's agent, fresh context, given only the diff, the issue and the PR text): VERDICT: APPROVE Blocking:
Non-blocking:
Attacked:
Body-stale:
Reviewer: rF11. Bar while the maintainers are away (set by IAmJSD as acting owner): one approving independent review on the head, two for money, auth and security paths. |
|
Independent review (IAmJSD's agent, fresh context, given only the diff, the issue and the PR text): VERDICT: APPROVE Blocking:
Non-blocking:
Attacked:
Body-stale:
Reviewer: rF12. Bar while the maintainers are away (set by IAmJSD as acting owner): one approving independent review on the head, two for money, auth and security paths. |
|
Local verification at |
|
Merged by IAmJSD's agent at IAmJSD's request (IAmJSD is acting owner while the maintainers are away), under the bar IAmJSD set: 2 approving independent review(s) on |
Picks up LeanerCloud#1520 (exclusive in-flight claim). No conflicts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Follow-up to #1516 for the concurrency gap that PR left to the owner.
Builds on #1516 (merged). #1516 and this PR both change
FlipPurchaseRevocationInFlightandflipRevocationInFlight. Main was merged into this branch after #1516's squash merge, so the PR diff is just this change (commita81a3c35). Follow-ups: #1522 (an in-progress answer for the losing request) and #1523 (the sweep marks rows revoked without a Return).Closes #1518
Root cause
FlipPurchaseRevocationInFlightwasUPDATE … SET revocation_in_flight = true WHERE purchase_id = $1 [AND revoked_at IS NULL]. A row that was already in flight could be flipped again, and the flip reported success. So two first-time revokes that both loaded the row unrevoked and unflagged could both pass the flip and both post an Azure Return. Azure refused the second one, and that request'shandleAzureReturnErrorcalledClearRevocationInFlightwhile the first request's refund was still in flight. If the first request'sMarkPurchaseRevokedthen failed after all its retries, the row was left neither revoked nor flagged. Thefinalize_revocationssweep never picked it up, even though that user had been told RECONCILE_PENDING.Fix
internal/config/store_postgres.go: the flip is now a claim. The UPDATE also requiresrevocation_in_flight = false. When nothing matches, one follow-up count separates three cases. A missing row keeps thenot founderror. A revoked row returnsErrPurchaseAlreadyRevoked, as in fix(config/db): compare-and-set the revocation in-flight flip; make 000059 re-applicable after a rollback #1516. An unrevoked row that another request has claimed returns the newconfig.ErrRevocationInFlight. The row lock serializes concurrent flips, and under READ COMMITTED a waiting UPDATE re-checks both predicates against the committed row, so exactly one flip of a free row succeeds. If the holder clears the flag between the loser's UPDATE and its count, the loser still gets in-flight. It never posts a Return it didn't claim.internal/config/errors.go,internal/config/interfaces.go: the new sentinel and the updated contracts for the flip and forClearRevocationInFlight(only the claim holder calls it).internal/api/handler_purchases_revoke.go:flipRevocationInFlightbecomesclaimRevocationInFlight:ErrRevocationInFlight: no Return. Answer RECONCILE_PENDING, the same body the handler's up-frontrecord.RevocationInFlightcheck already returns (now shared throughrevocationInFlightResult());ErrPurchaseAlreadyRevoked: unchanged from fix(config/db): compare-and-set the revocation in-flight flip; make 000059 re-applicable after a rollback #1516 (already_revoked, no Return);So only the claim holder ever reaches the Return. The
ClearRevocationInFlightinhandleAzureReturnErrortherefore only ever clears the caller's own flag, which is the issue's second criterion.ClearRevocationInFlightitself is unchanged. Without a migration there is no owner token to key it on, and the handler now guarantees the precondition.internal/api/openapi.yaml: the descriptions of the revoke endpoint and ofRevokePurchaseResponsenow say RECONCILE_PENDING also covers a return that a concurrent revoke of the same purchase is posting, and that only one revoke at a time posts the return. The status codes and response shapes don't change.No migration.
Judgment call: a failed flip no longer posts the Return
Before, the flip was best-effort ("a safety net, not a hard precondition"), and #1516 kept that. Now the flip is what keeps two Returns apart, and a request that posts without it can still race the holder. There's a second risk too. A flip that failed on a DB error is often followed by a
MarkPurchaseRevokedthat also fails, and a refund then goes out with no flag pointing the sweep at it, which is exactly the state this issue is about. Failing closed only removes Returns: it never posts one that main wouldn't post. The cost is that a revoke during a DB blip gets a 500 instead of going through. No Return has been posted at that point, so a retry is safe and the 7-day window leaves room for it. If the owner prefers best-effort, the alternative is to keep posting on a flip error but skipClearRevocationInFlightfor a request that didn't claim the row. That still allows two Returns in that narrow case.Verification
Local Linux VM, Go 1.26.9, every command run with
AWS_CONFIG_FILE=/dev/null AWS_SHARED_CREDENTIALS_FILE=/dev/null AWS_EC2_METADATA_DISABLED=true. "Integration" means real PostgreSQL in Docker (testcontainers, the packages' shared containers). The handler tests use stub Azure CalculateRefund/Return clients injected throughazureRevokeFactory. Nothing calls a cloud API.New and changed tests:
TestRevokeAzure_ConcurrentFirstRevokes_OneReturn(integration,internal/api, 5 rounds). TwoloadAndRevokePurchaseHistorycalls on one Azure row in real Postgres. A CalculateRefund barrier holds both until each has loaded the row and got its quote, so they reach the flip together. The stub Return succeeds once, but only after the other request has finished. Any later Return gets a 400, like Azure. The test asserts exactly one Return. One request answersrevokedand the other RECONCILE_PENDING. The flag is still set at the moment the other request has finished while the winner's Return is in flight (criterion 2). Finally the row is revoked with the flag clear.TestFlipPurchaseRevocationInFlight_ExclusiveClaim(integration,internal/config). 20 rounds of 6 concurrent flips of a fresh row: exactly one succeeds and the rest getErrRevocationInFlight. A second subtest checks that after the holder's clear, the row can be claimed again.TestFlipPurchaseRevocationInFlight_CompareAndSetis updated for the new contract. A second flip of an in-flight row is nowErrRevocationInFlight, not a no-op. In the stale-flips-vs-mark race, every flip now reports either in-flight or already-revoked; none succeeds.TestCallAzureReturn_FlipFindsRowRevoked(unit, mock store). Newin flightsubtest: no Return, no clear, no mark, and RECONCILE_PENDING. fix(config/db): compare-and-set the revocation in-flight flip; make 000059 re-applicable after a rollback #1516'sother flip error: still returns and marksbecomesother flip error: no Return, no clear, 500.Fail-before evidence. I swapped in this branch's merge-base code for
store_postgres.goandhandler_purchases_revoke.go(the #1516 versions), kept the new tests, ran them, then restored:TestRevokeAzure_ConcurrentFirstRevokes_OneReturn(integration)-race)TestFlipPurchaseRevocationInFlight_ExclusiveClaim/concurrent_first_flips…(integration)…/the_holder's_clear_releases_the_claim(integration)TestFlipPurchaseRevocationInFlight_CompareAndSet/an_unrevoked_row_flips_once;…,…/stale_flips_racing_a_mark(integration)…/a_revoked_row_is_refused…,…/a_missing_row_is_still_not_found(integration, preservation)TestCallAzureReturn_FlipFindsRowRevoked/other_flip_error:_no_Return,_no_clear,_500(unit)Mutation probes on this branch's handler (each reverted afterwards):
ErrRevocationInFlightinstead of answering RECONCILE_PENDING: the unitin flightsubtest fails ("the request that loses the claim must not post a Return"), and the integration race fails all 5 rounds (2 Returns, the flag cleared during the winner's Return).other flip errorsubtest fails.I didn't run the other integration tests in the touched packages (whole packages,
-race ./...) or git-secrets. Per the queue process, those run on the combined tree.Combined-tree verification at local queue commit
ccab8379= origin/mainec78d85c+fix/issue-1488(#1509),fix/issue-316-next(#1516),fix/issue-1517andfix/issue-1518merged in that order. Passed:go build ./...;go vetwith and without-tags integration;go test -race -short -count=1 ./...; golangci-lint v2.10.1 with and without the integration tag; gocyclo -over 10;go test -race -tags integrationoninternal/api,internal/commitmentopts,internal/configandinternal/database/postgres/migrations; the git-secrets scan andscripts/test-git-secrets-allowlist.sh.Merge order: #1516 has merged (squash). This branch then conflicted with main in the six files both PRs touch, so main was merged in at
33ab2725and resolved to main plus this PR's own commita81a3c35. The resulting tree is identical to cherry-pickinga81a3c35onto origin/main2faf67fc, and the PR diff against main is now just that change.Out of scope (noted, not changed)
record.RevocationInFlightcheck already answers the same thing in that same window, so this PR stays consistent with it rather than adding a new response. A more precise "revocation in progress" answer would be an API change for the owner.finalize_revocationscallsMarkPurchaseRevokedon every flagged, unrevoked row. It can't tell "Return succeeded, mark failed" from "Return in progress" or "the process died between the flip and the Return". This is pre-existing and unchanged; this PR doesn't widen that window.purchase_idalone. The flip, the clear andMarkPurchaseRevokedstill key onpurchase_idalone, not on the unique key(provider, account_id, purchase_id). This is pre-existing (also noted on fix(config/db): compare-and-set the revocation in-flight flip; make 000059 re-applicable after a rollback #1516).Frontend follow-up for the UI owner
None required for correctness. RECONCILE_PENDING is an existing, documented 2xx response. Today
frontend/src/history.tsdoesn't read itscodeand shows the generic "Purchase revocation submitted" success toast for it, and that is a fine answer for the losing request too. The flip-error path returns a plain 500, which the existing "Failed to revoke: …" toast shows. An optional polish for the UI owner: readcode === "RECONCILE_PENDING"and show a "refund in progress, don't retry" message, as the handler'srevokeReconcilePendingResultcomment intends.Maintainer: please mirror the issue's triage labels onto this PR:
priority/p2,type/bug. The issue has noseverity/*,urgency/*,impact/*oreffort/*labels and isn'ttriaged.🤖 Generated with Claude Code