Repository navigation
Conversation
Concurrent requests sharing (action, principal, idempotency_key) on SQLite could all pass the replay check before another caller committed, executing the action effects more than once and racing into the unique index, which surfaced as an uncaught IntegrityError (HTTP 500). SQLite has no advisory lock, so serialize same-key executions with a per-key asyncio lock on the runtime; the (action, principal, idempotency_key) unique index remains the backstop for writers the lock cannot cover (multi-process deployments). Refs: apache#437
bitflicker64
left a comment
There was a problem hiding this comment.
Blocking: yes. Summary: The per-key lock is released before the caller commits, so concurrent same-key requests on a file-backed SQLite database still fail with the unique-index IntegrityError, and the lock map grows by one entry per idempotency key without bound.
| return rev | ||
| finally: | ||
| if key_lock is not None: | ||
| key_lock.release() |
There was a problem hiding this comment.
Important: the lock is released when execute() returns, before the route's await s.commit(), so the next waiter's replay lookup runs on another pooled connection, misses the uncommitted revision and raises the same UNIQUE constraint failed: ontogeny_revision.idempotency_key. Hold the lock until the revision commits, for example with a per-key guard the route holds across execute() and commit().
The new tests pass only because sqlite+aiosqlite:///:memory: shares one StaticPool connection; on a file DSN with a warm pool, 4 concurrent same-key creates still fail once per round.
| RevisionRow.action == action_name, | ||
| RevisionRow.principal == str(principal.get("id", "anonymous")), | ||
| key = (action_name, str(principal.get("id", "anonymous")), idempotency_key) | ||
| key_lock = self._key_locks.setdefault(key, asyncio.Lock()) |
There was a problem hiding this comment.
Important: _key_locks gains one entry per distinct (action, principal, idempotency_key) and never drops it, so a long-running server leaks about 290 bytes per keyed request. Drop the entry once no caller holds or waits on it, for example by making _key_locks a weakref.WeakValueDictionary.
imbajin
left a comment
There was a problem hiding this comment.
Score: 5.0/10 · Confidence: high
Blocking: yes
Summary
The per-runtime registry also permits concurrent same-key actions across a service rebuild; two previously reported lock and memory issues remain.
Evidence
Service rebuild creates a fresh ActionRuntime and lock registry; the lock also releases before the route commits and the dictionary retains every key. An existing review reports a file-backed SQLite reproduction of the commit race.
Validation notes
Source inspection only in this review; the file-backed reproduction was reported by the existing reviewer and was not independently rerun. Multi-process and service-rebuild behavior were not tested.
| # committed revision. The (action, principal, idempotency_key) unique | ||
| # index stays as the final backstop for writers this lock cannot cover | ||
| # (multi-process deployments, dialect-specific paths). | ||
| self._key_locks: dict[tuple[str, str, str], asyncio.Lock] = {} |
There was a problem hiding this comment.
ServiceContext.publish() and reload() rebuild services and construct a new ActionRuntime. If a same-key request is still executing on the old runtime when that happens, a request on the new runtime acquires a different lock and can enter the effects concurrently. The unique index can then reject the retry rather than replay it, even if the lock is extended through commit. Share the registry across runtime replacements and cover an in-flight action during reload. This race follows from source inspection; I did not reproduce it at runtime.
Location
ontogeny/server/ontogeny/action/runtime.py—ActionRuntime.executeidempotency block;ontogeny/tests/test_action.py— regression coverage.Problem (Closes #437)
Bug: on SQLite, concurrent requests with the same
(action, principal, idempotency_key)are not serialized (SQLite has noadvisory lock), so the idempotency contract is broken.
Expected
Same-key concurrent requests execute the side effects once and all return the
same committed revision.
Actual
All callers can observe "revision not found" before any caller commits, then each
executes the effects and inserts a revision, hitting the unique index:
UNIQUE constraint failed: ontogeny_revision.idempotency_key-> uncaughtIntegrityError-> HTTP 500UNIQUE constraint failed: ontogeny_obj_production_order.order_id, _valid_fromSQLITE_BUSYImpact
Idempotent retries both fail (500) and may duplicate side effects, so clients
cannot safely retry actions.
Fix
_key_locks: dict[tuple[str, str, str], asyncio.Lock]toActionRuntime,keyed by
(action, principal, idempotency_key).execute, when a key is supplied, acquire the per-key lock (lazily created)before the replay lookup; wrap the body in
try/finallyto always release.backstop for multi-process writers.