Skip to content

fix(ontogeny): serialize concurrent same-key action replays on SQLite - #439

Open
ZhYQ1 wants to merge 1 commit into
apache:ontogenyfrom
ZhYQ1:fix/sqlite-idempotency-concurrency
Open

ZhYQ1 wants to merge 1 commit into
apache:ontogenyfrom
ZhYQ1:fix/sqlite-idempotency-concurrency

Conversation

@ZhYQ1

@ZhYQ1 ZhYQ1 commented Nov 21, 2026

Copy link
Copy Markdown

Location

ontogeny/server/ontogeny/action/runtime.py — ActionRuntime.execute idempotency block;
ontogeny/tests/test_action.py — regression coverage.

Problem (Closes #437)

Bug: on SQLite, concurrent requests with the same
(action, principal, idempotency_key) are not serialized (SQLite has no
advisory lock), so the idempotency contract is broken.

Expected

Same-key concurrent requests execute the side effects once and all return the
same committed revision.

Actual

All callers can observe "revision not found" before any caller commits, then each
executes the effects and inserts a revision, hitting the unique index:

  • UNIQUE constraint failed: ontogeny_revision.idempotency_key -> uncaught
    IntegrityError -> HTTP 500
  • effect writes may also collide, e.g.
    UNIQUE constraint failed: ontogeny_obj_production_order.order_id, _valid_from
  • under multi-connection writes SQLite can additionally return SQLITE_BUSY

Impact

Idempotent retries both fail (500) and may duplicate side effects, so clients
cannot safely retry actions.

Fix

  • Add _key_locks: dict[tuple[str, str, str], asyncio.Lock] to ActionRuntime,
    keyed by (action, principal, idempotency_key).
  • In execute, when a key is supplied, acquire the per-key lock (lazily created)
    before the replay lookup; wrap the body in try/finally to always release.
  • Calls without a key are unaffected; the unique index stays as the final
    backstop for multi-process writers.

Concurrent requests sharing (action, principal, idempotency_key) on SQLite
could all pass the replay check before another caller committed, executing
the action effects more than once and racing into the unique index, which
surfaced as an uncaught IntegrityError (HTTP 500).

SQLite has no advisory lock, so serialize same-key executions with a
per-key asyncio lock on the runtime; the (action, principal,
idempotency_key) unique index remains the backstop for writers the lock
cannot cover (multi-process deployments).

Refs: apache#437

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: yes. Summary: The per-key lock is released before the caller commits, so concurrent same-key requests on a file-backed SQLite database still fail with the unique-index IntegrityError, and the lock map grows by one entry per idempotency key without bound.

return rev
finally:
if key_lock is not None:
key_lock.release()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important: the lock is released when execute() returns, before the route's await s.commit(), so the next waiter's replay lookup runs on another pooled connection, misses the uncommitted revision and raises the same UNIQUE constraint failed: ontogeny_revision.idempotency_key. Hold the lock until the revision commits, for example with a per-key guard the route holds across execute() and commit().

The new tests pass only because sqlite+aiosqlite:///:memory: shares one StaticPool connection; on a file DSN with a warm pool, 4 concurrent same-key creates still fail once per round.

RevisionRow.action == action_name,
RevisionRow.principal == str(principal.get("id", "anonymous")),
key = (action_name, str(principal.get("id", "anonymous")), idempotency_key)
key_lock = self._key_locks.setdefault(key, asyncio.Lock())

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important: _key_locks gains one entry per distinct (action, principal, idempotency_key) and never drops it, so a long-running server leaks about 290 bytes per keyed request. Drop the entry once no caller holds or waits on it, for example by making _key_locks a weakref.WeakValueDictionary.

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Score: 5.0/10 · Confidence: high

Blocking: yes

Summary

The per-runtime registry also permits concurrent same-key actions across a service rebuild; two previously reported lock and memory issues remain.

Evidence

Service rebuild creates a fresh ActionRuntime and lock registry; the lock also releases before the route commits and the dictionary retains every key. An existing review reports a file-backed SQLite reproduction of the commit race.

Validation notes

Source inspection only in this review; the file-backed reproduction was reported by the existing reviewer and was not independently rerun. Multi-process and service-rebuild behavior were not tested.

# committed revision. The (action, principal, idempotency_key) unique
# index stays as the final backstop for writers this lock cannot cover
# (multi-process deployments, dialect-specific paths).
self._key_locks: dict[tuple[str, str, str], asyncio.Lock] = {}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Preserve the lock registry across service rebuilds

ServiceContext.publish() and reload() rebuild services and construct a new ActionRuntime. If a same-key request is still executing on the old runtime when that happens, a request on the new runtime acquires a different lock and can enter the effects concurrently. The unique index can then reject the retry rather than replay it, even if the lock is extended through commit. Share the registry across runtime replacements and cover an in-flight action during reload. This race follows from source inspection; I did not reproduce it at runtime.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants