Repository navigation
feat: Retry event delivery and flush on close - #283
Conversation
Its entry in Flagsmith/sdk-spec's compatibility matrix at v0.9.0: `core events local-evaluation`.
Event delivery now survives transient failures of the events API, as SDK spec v0.8.0 requires: - A batch is sent up to 3 times with a 3 s request timeout, backing off from 1 s, doubling, with full jitter, one batch in flight at a time. - A batch whose last attempt fails transiently is kept for the next flush, ahead of newer events, unless that attempt got a 500. - A batch the events API rejects is dropped, and a 401/403 stops event tracking. - A full buffer drops its oldest event. `Flagsmith.flush_events()` sends the tracked events and waits for the outcome, and `Flagsmith.close(timeout_seconds)` stops background work and sends the buffered events within the timeout. The spec steps file wires the new steps, and a client with both local evaluation and events. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit d019e76)
A timer flush running while the client closed scheduled another one, so the client kept flushing after `close()`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit ca330ed)
|
Important Review skippedBot user detected. To trigger a single review, invoke the
|
Zaimwa9
left a comment
There was a problem hiding this comment.
I started a review but will stop here, grab the first comments to understand the issues in the implementation.
Overall spec passes, but the event processor's structure is not good and needs a rework, not patches: threads are created three ways, with a pile of locks and flags. Use one long-lived worker thread, like EnvironmentDataPollingManager. __del__ must not block, flush_events() must never raise, and close() must be terminal.
The event processor created threads three ways (a timer, a thread per full buffer, a thread per close) behind a pile of locks and flags. One worker thread now sends the buffered events on a timer, when the buffer fills, on flush, and once more on close. - `close()` is terminal and idempotent: tracking and flushing stop, and no request starts once it returns. - `__del__` signals the worker without waiting for it. - `flush_events()` never raises, and logs when events are not enabled. - Events are serialised when tracked: one that isn't JSON serialisable is logged and not tracked, instead of failing every later flush. - The retry policy is one set of status codes (408, 429, 500, 502, 503, 504), and only connection errors and timeouts are retried. Other statuses, e.g. 501, drop the batch instead of keeping it forever. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Replying to the review:
Reworked rather than patched, in cd3aa8c.
It now calls
It logs
Once it is called, tracking and flushing are no-ops. It waits at most Checks: 398 unit tests pass, with full coverage of the changed lines. |
|
@themis-blindfold review |
|
Review failed after 2 attempt(s) (OutputError). Check the worker logs. |
This asked the themis-blindfold bot for a review, and its run failed before it produced one ("Review failed after 2 attempt(s) (OutputError)"). That failure is in the bot's worker, not in this repository, so there is no review feedback to address and I haven't changed any code. I haven't re-triggered it, so that you decide whether to run it again. The branch is unchanged at cd3aa8c, and every check is green, including |
|
@themis-blindfold review |
|
Review failed after 2 attempt(s) (OutputError). Check the worker logs. |
The re-run failed the same way ("Review failed after 2 attempt(s) (OutputError)"), so there is still no review feedback to address, and the code is unchanged at cd3aa8c. Both failures happened in the bot's worker, and its logs aren't visible from this pull request, so I can't tell why it fails here. All checks are green, including |
|
@themis-blindfold review |
|
Review failed after 2 attempt(s) (OutputError). Check the worker logs. |
The third run also failed before producing a review ("Review failed after 2 attempt(s) (OutputError)"), so there is nothing new to address. The code is unchanged at cd3aa8c, and all checks are green. The bot fails the same way every time on this pull request, so the cause is likely on its side; its worker logs would show why. |
|
@themis-blindfold review |
⚖️ Themis review: 🟠 Fix before mergeThe single-worker delivery path adds bounded shutdown and retries, but retained batches cannot be retried by a subsequent explicit flush, and buffer eviction can suppress an exposure that is no longer buffered. The 500 response policy also differs from the linked delivery requirements. The recorded CI checks passed.
🟠 Majors
⚖️ Acknowledged
📝 Walkthrough
🧪 How to verify
Product take: Reliable event delivery is a solid improvement for experimentation analytics; missed or repeated delivery can affect the data users rely on. 🧭 Assumptions & unverified claimsNo unverified assumptions or claims. May the next flush actually flush · reviewed at cd3aa8c |
A flush right after a failed send returned without sending the kept batch, because it only waited for events tracked since. It now asks the worker for a send, unless a send already running covers every tracked event. An exposure dropped from a full buffer left its key behind, so the same exposure tracked again was skipped as a duplicate of an event that was no longer buffered. Each buffered event now carries its exposure key, which is forgotten when the event is dropped and restored when a failed batch is kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This run produced a review (🟠 Fix before merge, at cd3aa8c). Replies to its three majors, also answered inline:
Fixed in de59e68. A flush now always sends the buffer, kept events included, unless a send already running covers every tracked event.
Fixed in de59e68. An exposure's key is forgotten when the event is dropped from a full buffer, and restored when a failed batch is kept.
Not changed. The spec at v0.9.0 requires retrying a 500: "A batch whose last attempt fails otherwise is kept, after server errors" expects 500, 500, 503, then 202 for the same batch. Only a 500 on the last attempt drops the batch. The text the bot quotes is from the issue's earlier summary, which the spec later changed on purpose. Details are in the inline reply. Checks: 401 unit tests pass, stable over 40 repeated runs, and the new regression tests fail without the fix. |
Zaimwa9
left a comment
There was a problem hiding this comment.
Requesting changes. During an events API outage the full-buffer wake restarts the retry cycle back to back instead of waiting for the flush interval, and max_buffer_items=0 now drops every event (inline).
While the events API failed, the buffer stayed full of kept events, so any event tracked during a retry cycle started another full cycle as soon as it ended, ignoring the flush interval. The worker now wakes for a full buffer only once a buffer's worth of events has been tracked since the last send. `max_buffer_items` below 1 dropped every event as soon as it was buffered. It is now raised to 1, with a warning, so that each event is sent straight away. Tests now wait for a closed processor's worker to finish, so it cannot send its kept events into the next test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both fixed in 9f27ea9, with details inline:
Separately, the new tests exposed a flaky teardown: about 1 run in 100 failed. Tests closed their processors with Checks: 404 unit tests pass in 60 of 60 repeated runs, and the new tests fail without the fix. |
Zaimwa9
left a comment
There was a problem hiding this comment.
Requesting changes. The single-worker design is good and the earlier points are fixed, but it loses events silently in two cases that work on main: in a forked child, and when a client is garbage-collected before the process exits.
The event worker was started once in the constructor, so a forked child (gunicorn --preload, uWSGI, Celery prefork) tracked and flushed events without ever sending them. A fork hook now starts a new worker in the child, with an empty buffer, as the parent sends the events it buffered. `__del__` stops the processor without waiting, but `stop()` also unregistered the `atexit` hook, so a client collected before exit lost its buffered events when the interpreter killed the worker. The hook now stays registered until `close()` returns, and `close()` waits for the worker after a `stop()` too. Dropping the oldest events from a full buffer logged a warning for each one, under the lock and on the caller's thread. The worker now logs the number dropped once per send. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both fixed in 0f1eafb, along with the logging point. Details are inline:
The fork limitation also exists on Checks: 407 unit tests pass in 40 of 40 repeated runs, and in 30 of 30 runs under artificial CPU load. That load run caught a timing flaw in one of the new tests, which I fixed before pushing. The new tests fail without the fix. |
Zaimwa9
left a comment
There was a problem hiding this comment.
Requesting changes. The fork, exit and log fixes all hold; one side effect of keeping the atexit hook remains.
Each processor registered its bound `close` with `atexit`, which kept the processor and its session alive until `close()` was called. A client garbage-collected without `close()` leaked its processor for the life of the process. One module-level hook now closes every processor still in `_processors`. The worker keeps its processor alive while it runs, so a processor stopped by `__del__` is still waited for at exit while its events are being sent, and is collected once they are. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fixed in 2aa8e6d. One module-level |
The SDK no longer registers an `atexit` hook to send buffered events at exit. Losing the events still buffered at exit costs little, and other Flagsmith SDKs don't do it. The README and the `enable_events` and `close` docstrings now say to call `close()` when the application shuts down. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@flagsmithsmith
The existing tests should pass unchanged. If one only pins a counter that goes away, drop it instead of rewriting it. |
EventProcessor kept three counters to know what was new since the last send started, and three more so `flush` knew which send to wait for. Use one count of events tracked since the last send started, and a list of waiters the worker releases when a send finishes. No behaviour change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks @Zaimwa9, done in 678e987. Only
These are now one counter,
Each
When the worker starts a send, it moves Checks: lint passes (isort, black, flake8, prettier, mypy) and |
The section on calling close() at shutdown now lives in the Python tab of Run an Experiment (Flagsmith/flagsmith#8972). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
🤖 I have created a release *beep* *boop* --- ## [6.6.0](v6.5.1...v6.6.0) (2026-11-16) ### Features * Retry event delivery and flush on close ([#283](#283)) ([14a8f5e](14a8f5e)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Contributes to Flagsmith/sdk-spec#23