Skip to content

feat: Retry event delivery and flush on close - #283

Merged
Zaimwa9 merged 12 commits into
mainfrom
openhands/sdk-spec-v0.9.0
Nov 16, 2026
Merged

Zaimwa9 merged 12 commits into
mainfrom
openhands/sdk-spec-v0.9.0

Conversation

@flagsmithsmith

Copy link
Copy Markdown
Contributor

Contributes to Flagsmith/sdk-spec#23

renovate Bot and others added 4 commits November 11, 2026 15:32
Its entry in Flagsmith/sdk-spec's compatibility matrix at v0.9.0: `core events local-evaluation`.
Event delivery now survives transient failures of the events API, as
SDK spec v0.8.0 requires:

- A batch is sent up to 3 times with a 3 s request timeout, backing off
  from 1 s, doubling, with full jitter, one batch in flight at a time.
- A batch whose last attempt fails transiently is kept for the next
  flush, ahead of newer events, unless that attempt got a 500.
- A batch the events API rejects is dropped, and a 401/403 stops event
  tracking.
- A full buffer drops its oldest event.

`Flagsmith.flush_events()` sends the tracked events and waits for the
outcome, and `Flagsmith.close(timeout_seconds)` stops background work
and sends the buffered events within the timeout.

The spec steps file wires the new steps, and a client with both local
evaluation and events.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit d019e76)
A timer flush running while the client closed scheduled another one, so
the client kept flushing after `close()`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit ca330ed)
@flagsmithsmith flagsmithsmith Bot added the sdk-spec Bumps of the Flagsmith SDK behavioural spec label Nov 11, 2026
@coderabbitai

coderabbitai Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 4c9e8dbf-5b80-4479-a312-babdd5e28741

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@codspeed

codspeed Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 15 untouched benchmarks


Comparing openhands/sdk-spec-v0.9.0 (6fdb424) with main (b480743)

Open in CodSpeed

@flagsmithsmith
flagsmithsmith Bot marked this pull request as ready for review November 11, 2026 15:44
@flagsmithsmith
flagsmithsmith Bot requested a review from a team as a code owner November 11, 2026 15:44
@flagsmithsmith
flagsmithsmith Bot requested review from khvn26 and removed request for a team November 11, 2026 15:44

@Zaimwa9 Zaimwa9 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I started a review but will stop here, grab the first comments to understand the issues in the implementation.
Overall spec passes, but the event processor's structure is not good and needs a rework, not patches: threads are created three ways, with a pile of locks and flags. Use one long-lived worker thread, like EnvironmentDataPollingManager. __del__ must not block, flush_events() must never raise, and close() must be terminal.

Comment thread flagsmith/analytics.py Resolved
Comment thread flagsmith/flagsmith.py Resolved
Comment thread flagsmith/analytics.py Resolved
Comment thread flagsmith/analytics.py Resolved
Comment thread flagsmith/analytics.py Resolved
The event processor created threads three ways (a timer, a thread per
full buffer, a thread per close) behind a pile of locks and flags. One
worker thread now sends the buffered events on a timer, when the buffer
fills, on flush, and once more on close.

- `close()` is terminal and idempotent: tracking and flushing stop, and
  no request starts once it returns.
- `__del__` signals the worker without waiting for it.
- `flush_events()` never raises, and logs when events are not enabled.
- Events are serialised when tracked: one that isn't JSON serialisable
  is logged and not tracked, instead of failing every later flush.
- The retry policy is one set of status codes (408, 429, 500, 502, 503,
  504), and only connection errors and timeouts are retried. Other
  statuses, e.g. 501, drop the batch instead of keeping it forever.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@flagsmithsmith

Copy link
Copy Markdown
Contributor Author

Replying to the review:

threads are created three ways, with a pile of locks and flags. Use one long-lived worker thread, like EnvironmentDataPollingManager.

Reworked rather than patched, in cd3aa8c. EventProcessor starts one long-lived daemon worker in its constructor. Like EnvironmentDataPollingManager, the worker waits on its stop signal between runs. It sends the buffered events when the flush interval elapses, when the buffer fills, when flush() asks, and once more on close. That replaces the timer, the thread per full buffer and the thread per close. The state is now one threading.Condition, three counters (tracked, flush requested, sent), plus closed, refused and an _abandoned event set when close() gives up. I kept the processor as the thread's owner rather than a Thread subclass, so that track_event/flush/close stay the processor's public interface. Happy to switch to a subclass if you prefer.

__del__ must not block

It now calls EventProcessor.stop(), which only marks the processor closed and wakes the worker. It never joins.

flush_events() must never raise

It logs Not flushing events: events are not enabled. and returns, instead of raising. It does nothing once the client is closed. The worker catches unexpected errors around a send, so a flush can't be left waiting on a dead thread.

close() must be terminal.

Once it is called, tracking and flushing are no-ops. It waits at most timeout_seconds for the final send, and no attempt starts after it returns. Calling it again does nothing. A request already in flight at that point can't be cancelled by requests, and ends within the 3 s request timeout (details in the inline reply).

Checks: 398 unit tests pass, with full coverage of the changed lines. mypy --strict . and pre-commit pass. Spec v0.9.0 passes locally with TAGS="core events local-evaluation" (24 scenarios).

@flagsmithsmith
flagsmithsmith Bot requested a review from Zaimwa9 November 12, 2026 16:21
@Zaimwa9

Zaimwa9 commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

@themis-blindfold review

@themis-blindfold

Copy link
Copy Markdown

Review failed after 2 attempt(s) (OutputError). Check the worker logs.

@flagsmithsmith

Copy link
Copy Markdown
Contributor Author

@themis-blindfold review

This asked the themis-blindfold bot for a review, and its run failed before it produced one ("Review failed after 2 attempt(s) (OutputError)"). That failure is in the bot's worker, not in this repository, so there is no review feedback to address and I haven't changed any code. I haven't re-triggered it, so that you decide whether to run it again.

The branch is unchanged at cd3aa8c, and every check is green, including sdk-spec / Spec v0.9.0. The open choices from my previous replies are still yours: whether the atexit hook should wait up to the close timeout or not wait at all, and whether the worker should be a threading.Thread subclass.

@Zaimwa9

Zaimwa9 commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

@themis-blindfold review

@themis-blindfold

Copy link
Copy Markdown

Review failed after 2 attempt(s) (OutputError). Check the worker logs.

@flagsmithsmith

Copy link
Copy Markdown
Contributor Author

@themis-blindfold review

The re-run failed the same way ("Review failed after 2 attempt(s) (OutputError)"), so there is still no review feedback to address, and the code is unchanged at cd3aa8c. Both failures happened in the bot's worker, and its logs aren't visible from this pull request, so I can't tell why it fails here. All checks are green, including sdk-spec / Spec v0.9.0.

@Zaimwa9

Zaimwa9 commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

@themis-blindfold review

@themis-blindfold

Copy link
Copy Markdown

Review failed after 2 attempt(s) (OutputError). Check the worker logs.

@flagsmithsmith

Copy link
Copy Markdown
Contributor Author

@themis-blindfold review

The third run also failed before producing a review ("Review failed after 2 attempt(s) (OutputError)"), so there is nothing new to address. The code is unchanged at cd3aa8c, and all checks are green. The bot fails the same way every time on this pull request, so the cause is likely on its side; its worker logs would show why.

@Zaimwa9

Zaimwa9 commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

@themis-blindfold review

Comment thread flagsmith/analytics.py Resolved
Comment thread flagsmith/analytics.py Resolved
Comment thread flagsmith/analytics.py Resolved
@themis-blindfold

Copy link
Copy Markdown

⚖️ Themis review: 🟠 Fix before merge

The single-worker delivery path adds bounded shutdown and retries, but retained batches cannot be retried by a subsequent explicit flush, and buffer eviction can suppress an exposure that is no longer buffered. The 500 response policy also differs from the linked delivery requirements. The recorded CI checks passed.

Area Score
🎯 Correctness 2/5
🧪 Test coverage 3/5
📐 Code quality 3/5
🚀 Product impact 3/5

🟠 Majors

  • flagsmith/analytics.py — A second explicit flush does not retry a retained batch.
  • flagsmith/analytics.py — Evicted exposures can suppress subsequent tracking.
  • flagsmith/analytics.py — HTTP 500 is retried despite the stated delivery requirements.

⚖️ Acknowledged

  • Unbounded send threads under load — thread resolved by @Zaimwa9.
  • Blocking destructor and shutdown thread creation — thread resolved by @Zaimwa9.
  • Non-serialisable metadata breaking delivery — thread resolved by @Zaimwa9.
  • Inconsistent retry classification and retention of 501/505 responses — thread resolved by @Zaimwa9.
  • Tracking and requests continuing after close — thread resolved by @Zaimwa9.
📝 Walkthrough
  • Event delivery — replaces timer-driven sends with a single worker, retry policy and bounded buffer.
  • Client lifecycle — adds explicit event flushing and a timeout-based close.
  • Specification — updates the shared SDK specification and adds event lifecycle steps.
🧪 How to verify
  1. Return three 503 responses, then 202; flush twice without tracking another event and check the second flush sends the retained batch.
  2. Fill a small buffer, evict an exposure, then track that exposure again and check it is delivered.
  3. Return 500 on the first attempt and check the batch is dropped without another request.
  4. Close while a request is in flight and verify the timeout and remaining buffered events.
  5. Automate: add regression tests for the retained-batch, eviction and 500 cases.

Product take: Reliable event delivery is a solid improvement for experimentation analytics; missed or repeated delivery can affect the data users rely on.

🧭 Assumptions & unverified claims

No unverified assumptions or claims.

May the next flush actually flush · reviewed at cd3aa8c

A flush right after a failed send returned without sending the kept
batch, because it only waited for events tracked since. It now asks the
worker for a send, unless a send already running covers every tracked
event.

An exposure dropped from a full buffer left its key behind, so the same
exposure tracked again was skipped as a duplicate of an event that was
no longer buffered. Each buffered event now carries its exposure key,
which is forgotten when the event is dropped and restored when a failed
batch is kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@flagsmithsmith

Copy link
Copy Markdown
Contributor Author

@themis-blindfold review

This run produced a review (🟠 Fix before merge, at cd3aa8c). Replies to its three majors, also answered inline:

A second explicit flush does not retry a retained batch.

Fixed in de59e68. A flush now always sends the buffer, kept events included, unless a send already running covers every tracked event.

Evicted exposures can suppress subsequent tracking.

Fixed in de59e68. An exposure's key is forgotten when the event is dropped from a full buffer, and restored when a failed batch is kept.

HTTP 500 is retried despite the stated delivery requirements.

Not changed. The spec at v0.9.0 requires retrying a 500: "A batch whose last attempt fails otherwise is kept, after server errors" expects 500, 500, 503, then 202 for the same batch. Only a 500 on the last attempt drops the batch. The text the bot quotes is from the issue's earlier summary, which the spec later changed on purpose. Details are in the inline reply.

Checks: 401 unit tests pass, stable over 40 repeated runs, and the new regression tests fail without the fix. mypy --strict . and pre-commit pass. Spec v0.9.0 passes locally with TAGS="core events local-evaluation" (24 scenarios).

@Zaimwa9 Zaimwa9 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes. During an events API outage the full-buffer wake restarts the retry cycle back to back instead of waiting for the flush interval, and max_buffer_items=0 now drops every event (inline).

Comment thread flagsmith/analytics.py Resolved
Comment thread flagsmith/analytics.py Resolved
While the events API failed, the buffer stayed full of kept events, so
any event tracked during a retry cycle started another full cycle as
soon as it ended, ignoring the flush interval. The worker now wakes for
a full buffer only once a buffer's worth of events has been tracked
since the last send.

`max_buffer_items` below 1 dropped every event as soon as it was
buffered. It is now raised to 1, with a warning, so that each event is
sent straight away.

Tests now wait for a closed processor's worker to finish, so it cannot
send its kept events into the next test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@flagsmithsmith

Copy link
Copy Markdown
Contributor Author

Requesting changes. During an events API outage the full-buffer wake restarts the retry cycle back to back instead of waiting for the flush interval, and max_buffer_items=0 now drops every event (inline).

Both fixed in 9f27ea9, with details inline:

  • The worker wakes for a full buffer only once a buffer's worth of new events has been tracked since the last send, so retries during an outage follow the flush interval.
  • max_buffer_items below 1 is raised to 1, with a warning, so each event is sent straight away.

Separately, the new tests exposed a flaky teardown: about 1 run in 100 failed. Tests closed their processors with close(timeout_seconds=0) without waiting for the worker, so a worker holding kept events could send them through the next test's mocked events API. Removing the waits reproduced it (2 of 100 runs), and every teardown now joins the worker after closing it (0 of 100).

Checks: 404 unit tests pass in 60 of 60 repeated runs, and the new tests fail without the fix. mypy --strict . and pre-commit pass. Spec v0.9.0 passes locally with TAGS="core events local-evaluation" (24 scenarios).

@flagsmithsmith
flagsmithsmith Bot requested a review from Zaimwa9 November 13, 2026 15:11

@Zaimwa9 Zaimwa9 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes. The single-worker design is good and the earlier points are fixed, but it loses events silently in two cases that work on main: in a forked child, and when a client is garbage-collected before the process exits.

Comment thread flagsmith/analytics.py Resolved
Comment thread flagsmith/analytics.py Resolved
Comment thread flagsmith/analytics.py Resolved
The event worker was started once in the constructor, so a forked child
(gunicorn --preload, uWSGI, Celery prefork) tracked and flushed events
without ever sending them. A fork hook now starts a new worker in the
child, with an empty buffer, as the parent sends the events it buffered.

`__del__` stops the processor without waiting, but `stop()` also
unregistered the `atexit` hook, so a client collected before exit lost
its buffered events when the interpreter killed the worker. The hook now
stays registered until `close()` returns, and `close()` waits for the
worker after a `stop()` too.

Dropping the oldest events from a full buffer logged a warning for each
one, under the lock and on the caller's thread. The worker now logs the
number dropped once per send.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@flagsmithsmith

Copy link
Copy Markdown
Contributor Author

Requesting changes. The single-worker design is good and the earlier points are fixed, but it loses events silently in two cases that work on main: in a forked child, and when a client is garbage-collected before the process exits.

Both fixed in 0f1eafb, along with the logging point. Details are inline:

  • Forked child: a fork hook starts a new worker in the child, with fresh locks and a fresh session. The child sends only events tracked after the fork, and the parent still sends its own.
  • Garbage-collected client: stop() keeps the atexit hook registered, and close() waits for the worker even after a stop(), so events buffered by a collected client are sent before exit, within the close timeout.
  • Dropped events: logged once per send with a count, outside the lock.

The fork limitation also exists on main for local evaluation's EnvironmentDataPollingManager. I left that out of this PR (details inline).

Checks: 407 unit tests pass in 40 of 40 repeated runs, and in 30 of 30 runs under artificial CPU load. That load run caught a timing flaw in one of the new tests, which I fixed before pushing. The new tests fail without the fix. mypy --strict . and pre-commit pass. Spec v0.9.0 passes locally with TAGS="core events local-evaluation" (24 scenarios).

@flagsmithsmith
flagsmithsmith Bot requested a review from Zaimwa9 November 13, 2026 16:22

@Zaimwa9 Zaimwa9 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes. The fork, exit and log fixes all hold; one side effect of keeping the atexit hook remains.

Comment thread flagsmith/analytics.py Resolved
Each processor registered its bound `close` with `atexit`, which kept the
processor and its session alive until `close()` was called. A client
garbage-collected without `close()` leaked its processor for the life of
the process.

One module-level hook now closes every processor still in `_processors`.
The worker keeps its processor alive while it runs, so a processor stopped
by `__del__` is still waited for at exit while its events are being sent,
and is collected once they are.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@flagsmithsmith

Copy link
Copy Markdown
Contributor Author

Requesting changes. The fork, exit and log fixes all hold; one side effect of keeping the atexit hook remains.

Fixed in 2aa8e6d. One module-level atexit hook now closes every processor in _processors, so a processor no longer stays alive until exit. Details are in the inline reply. Whether to keep an exit hook at all (raised by @khvn26) is still open in that thread.

@flagsmithsmith
flagsmithsmith Bot requested a review from Zaimwa9 November 13, 2026 19:41
The SDK no longer registers an `atexit` hook to send buffered events at
exit. Losing the events still buffered at exit costs little, and other
Flagsmith SDKs don't do it. The README and the `enable_events` and
`close` docstrings now say to call `close()` when the application shuts
down.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Zaimwa9

Zaimwa9 commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

@flagsmithsmith
One last pass hopefully EventProcessor now carries too much states and counters. No behaviour change please, just less state.

  • _tracked_count, _sent_count and _sending_count all answer "what's new since the last send started". One counter should be enough.
  • _sends_started, _sends_completed and _flush_requested only exist so flush() knows which send to wait for. A list of waiters that the worker releases when a send finishes would be easier to follow.

The existing tests should pass unchanged. If one only pins a counter that goes away, drop it instead of rewriting it.

EventProcessor kept three counters to know what was new since the last
send started, and three more so `flush` knew which send to wait for. Use
one count of events tracked since the last send started, and a list of
waiters the worker releases when a send finishes. No behaviour change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@flagsmithsmith

Copy link
Copy Markdown
Contributor Author

Thanks @Zaimwa9, done in 678e987. Only flagsmith/analytics.py changed. No tests needed changing or dropping, because none of them read these counters.

_tracked_count, _sent_count and _sending_count all answer "what's new since the last send started". One counter should be enough.

These are now one counter, _new_count. Tracking an event adds one to it, and the worker sets it to 0 when it takes a send. The behaviour is the same:

  • Full buffer. _has_full_buffer_of_new_events() is now _new_count >= max_buffer. The old check was tracked - sent, where sent became the count at send start once the send finished. So once a send finishes, both checks give the same number. While a send runs, the check only decides whether track_event calls notify_all. The worker isn't waiting then, so that call does nothing either way.
  • Flush during a send. The old test, "a send is running and _sending_count >= _tracked_count", is now "a send is running and _new_count == 0".

_sends_started, _sends_completed and _flush_requested only exist so flush() knows which send to wait for. A list of waiters that the worker releases when a send finishes would be easier to follow.

Each flush() now adds a threading.Event to one of two lists:

  • _sending_waiters if a running send already covers every tracked event. This list is None when no send is running.
  • _flush_waiters otherwise. This list also wakes the worker, so it replaces _flush_requested.

When the worker starts a send, it moves _flush_waiters into _sending_waiters. When the send finishes, it sets each waiter and notifies. flush() still stops waiting if the worker thread dies, as before.

Checks: lint passes (isort, black, flake8, prettier, mypy) and tests/test_event_processor.py passes on 5 runs in a row. Locally, 6 test_flagsmith__init__expected_headers_sent tests fail with and without this change, so that comes from my local environment.

Comment thread flagsmith/analytics.py Resolved
Comment thread flagsmith/analytics.py Resolved
Comment thread README.md Resolved
The section on calling close() at shutdown now lives in the Python tab of
Run an Experiment (Flagsmith/flagsmith#8972).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@khvn26 khvn26 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can ship this.

@Zaimwa9
Zaimwa9 merged commit 14a8f5e into main Nov 16, 2026
9 checks passed
@Zaimwa9
Zaimwa9 deleted the openhands/sdk-spec-v0.9.0 branch November 16, 2026 17:20
Zaimwa9 pushed a commit that referenced this pull request Nov 16, 2026
🤖 I have created a release *beep* *boop*
---


##
[6.6.0](v6.5.1...v6.6.0)
(2026-11-16)


### Features

* Retry event delivery and flush on close
([#283](#283))
([14a8f5e](14a8f5e))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

sdk-spec Bumps of the Flagsmith SDK behavioural spec

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants