Skip to content

fix(core): validate replay operation identity - #698

Open
zhongkechen wants to merge 3 commits into
mainfrom
fix/replay-operation-identity
Open

fix(core): validate replay operation identity#698
zhongkechen wants to merge 3 commits into
mainfrom
fix/replay-operation-identity

Conversation

@zhongkechen

Copy link
Copy Markdown
Contributor

Summary

  • validate checkpoint type, subtype, and name before operation-specific replay handling
  • fail mismatches with NonDeterministicExecutionError instead of consuming another operation's checkpoint
  • cover type, subtype, and name drift through unit and end-to-end regression tests

Testing

  • 1,587 core non-e2e tests passed
  • 46 core e2e tests passed
  • hatch run dev-core:typecheck
  • hatch fmt --check

Closes #692

@zhongkechen
zhongkechen deployed to ai-pr-review-runtime August 31, 2026 18:58 — with GitHub Actions Active
@zhongkechen
zhongkechen had a problem deploying to ai-pr-review-runtime August 31, 2026 19:11 — with GitHub Actions Failure
@zhongkechen
zhongkechen had a problem deploying to ai-pr-review-runtime August 31, 2026 19:11 — with GitHub Actions Failure
@zhongkechen
zhongkechen had a problem deploying to ai-pr-review-runtime August 31, 2026 19:31 — with GitHub Actions Failure
@zhongkechen
zhongkechen deployed to ai-pr-review-runtime August 31, 2026 19:31 — with GitHub Actions Active
@github-actions

This comment has been minimized.

@zhongkechen
zhongkechen had a problem deploying to ai-pr-review-runtime August 31, 2026 22:50 — with GitHub Actions Failure
@zhongkechen
zhongkechen deployed to ai-pr-review-runtime August 31, 2026 22:50 — with GitHub Actions Active
except ExecutionError as e:
# Execution-terminal SDK errors (including nondeterminism) must
# bypass branch failure tolerance and custom completion policies.
events.put(BranchEvent.fatal(executable.index, e))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex AI review · Finding arf_v1_dgw6ii4qxklkvjhea4ieegv2i6

[P1] Preserve fatal errors reported after early completion

Once a completion policy ends the coordinator loop, the queue is drained only once and the pool shuts down with wait=False. A running branch reaching this catch afterward leaves an unconsumed fatal event; ExecutionState.close() joins the pool without inspecting its futures, so map/parallel can checkpoint success despite nondeterministic history. Retain fatal state that is checked before the parent terminal checkpoint, and add a barrier-based early-completion regression test.

branch_checkpoint = child_context.state.get_checkpoint_result(
operation_identifier.operation_id
)
operation_identifier.validate_checkpoint(branch_checkpoint.operation)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex AI review · Finding arf_v1_7pptyptccbf3r2g5kacbf3pjp6

[P1] Reject NESTED branch checkpoints when replaying in FLAT mode

This validation runs only for non-virtual branches. When a STARTED or ReplayChildren parent is replayed after changing NESTED to FLAT, the old successful branch checkpoint has the same id, type, subtype, name, and parent; child_handler consumes its cached result without executing the virtual branch, so the changed inner hierarchy is never validated. In FLAT mode, reject any existing branch-container checkpoint before calling child_handler, and add a real map/parallel NESTED→FLAT replay test.

@github-actions

Copy link
Copy Markdown
Contributor

Codex AI review

Two P1 replay-safety gaps remain in concurrent map/parallel handling. Tests do not cover early-completion races or actual NESTED→FLAT replay.

Reviewed commit d8f9c9846eb9aef8367324e72bf72f55f170293b. Workflow run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Replay mismatch silently consumes a checkpoint from a different operation

1 participant