Skip to content

[TH-9515] test: prove the flush-window bound without timing the runner - #286

Merged
khushalsonawat merged 3 commits into
feat/TH-8804-source-subfolderfrom
feat/TH-9515-flush-window-test
Nov 12, 2026
Merged

khushalsonawat merged 3 commits into
feat/TH-8804-source-subfolderfrom
feat/TH-9515-flush-window-test

Conversation

@khushalsonawat

@khushalsonawat khushalsonawat commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

Description

test_post_terminal_wire_block_is_bounded_by_the_remaining_flush_window failed once on #273's CI. run_job returned 4, EXIT_TERMINAL_UNDELIVERED, because the terminal's own delivery didn't fit in the test's 0.2 s flush window on a slow runner. The test also asserted the whole run took under 1.5 s, which locally used about 1.3 s of it. Both numbers measured how fast the runner was, not whether the bound holds.

This is a test-only change. The flush-window code in hosted_entrypoint.py is unchanged.

Type of Change

  • Tests

Changes

  • The window is 2.0 s, not 0.2 s, so the terminal's delivery has room.
  • emit_skipped_receipts is replaced by a call that waits on an event that is never set, so it can only end when the window's wait_for cancels it. The stall stands in for the receipt push.
  • The test checks the bound the code sets, not elapsed time (62b316b4, after Dileep's review). A spy on wait_for records the timeout passed with the stalled call, and the test asserts 0 < timeout < window. The code passes the remaining window, which is always under the full one, so a late cut and a full-window timeout both fail.
  • The run sits under an outer asyncio.wait_for of 30 s, so a broken bound fails with a clear message instead of hanging the suite.
  • The 1.5 s wall-clock assertion is gone. So is the check that the second scenario sent no receipt, which can't fail now that the push is never reached. Exit 0, one canceled terminal, the first scenario passed and the pool closed are all still asserted.

The test now always takes about 2 s.

Testing

  • tests/harness/test_hosted_entrypoint.py:
    • Python 3.12.14: 139 passed. The changed test takes 2.03 s.
    • Python 3.12.3: 133 passed. The 6 failures are the voice tests, which need livekit; it isn't installed in that venv, and they fail the same way on the base.
  • Planted bugs in hosted_entrypoint.py, each run on this test and then restored:
plant what's broken result
w1 the wait_for around emit_skipped_receipts removed fails at the 30 s guard
w2 timeout=None fails at the 30 s guard
d1 the cut comes 1.8 s late (Dileep's case) fails at the bound (3.76 against 2.0)
d2 the full window instead of the remaining one fails at the bound (2.0 against 2.0)

Known limits

  • The window still depends on headroom. The window starts when the cancel is seen, before the terminal is sent. If the steps before the terminal take more than 2.0 s, run_job returns 4 again. A planted 2.5 s delay there fails the test; a planted 1.5 s delay passes. Locally those steps take under 0.07 s, so the 2.0 s window is about 30 times what's needed. The source reads time.monotonic() rather than deps.clock, so a test-only change can't control that clock.
  • No slow-runner simulation. I didn't run the test under cpulimit or beside a busy loop, to keep the laptop cool. The reviewer measured the same margin directly with the two delay plants above.
  • Reviews. One cold review (Lane Q); its Medium and a Low were fixed in 5e35bf80. A fix-only check of 5e35bf80 and 62b316b4, one reviewer: clear, 0 Medium (results in the comments).
  • A constant just under the full window would pass the bound check; the code computes the remaining window, and nothing in it would produce such a constant.

Related Issues

Refs: TH-9515

The test gave the run a 0.2-second flush window and asserted it ended
in under 1.5 seconds. Both numbers measured the runner, not the bound:
on a slow CI runner the terminal's own delivery missed the tiny window
and the run exited 4.

The window is now 2.0 seconds, and emit_skipped_receipts waits on an
event that is never set, so the run can end only because the window's
wait_for cuts it off. An outer 30-second guard turns a broken bound into
a failure rather than a hang. The other assertions are kept.

TH-9515

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
The change review (Lane Q, round 1) showed the 30 s guard was the only time
check left, so a cut-off 10 s or 25 s late still passed. The stalled receipts
call now records when it starts and when the window's cancel reaches it, re-raises
the cancel, and the test asserts that gap is under twice the window. Both late
plants now fail at that assertion. The never-reached receipt push and its
assertion are dropped.

TH-9515

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
@khushalsonawat khushalsonawat self-assigned this Nov 11, 2026

@cdileep23 cdileep23 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check the comments.

Comment thread tests/harness/test_hosted_entrypoint.py Resolved
…me it took

Dileep found the test still timed the cut, with enough slack that a cut
1.8 s late on a 2.0 s window passed, and that it couldn't tell the
remaining window from the full one. The test now records the timeout
passed to wait_for around the stalled emit_skipped_receipts and asserts
0 < timeout < window: the code passes the remaining window, which is
always under the full one. The outer 30 s guard stays, so a hang still
fails. The +1.8 plant and timeout=None both fail it.

TH-9515

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
@khushalsonawat

Copy link
Copy Markdown
Member Author

Fix-only check of 5e35bf8 (the first review's fix, never checked until now) and 62b316b (Dileep's P2), one reviewer: clear, with 0 High, 0 Medium and 3 Low.

  • 5e35bf8: a cut 10 s late fails the old test, but a cut 1.8 s late passes. That is Dileep's P2, and 62b316b replaces the assertion.
  • 62b316b: these wrong builds each fail it:
    • a cut 1.8 s late (3.758 against 2.0);
    • the full window instead of the remaining one (2.0 against 2.0);
    • timeout=None, through the 30 s guard.
  • The strict < is safe: the timeout came out 42 to 57 ms under the window, far above clock resolution.
  • Lows:
    • A constant just under the full window would pass.
    • The comment's wording about None is slightly off.
    • The test could fail on correct code if the whole 2 s window were used before the timeout is computed. About 1.94 s of headroom was measured.
  • The test file: 133 passed. The 6 failures are the voice tests that need livekit, which isn't installed on the review machine.

@khushalsonawat

Copy link
Copy Markdown
Member Author

CI at 62b316b: SDK smoke's clean-install failed one test, and it comes from the base, not this PR.

  • This PR's run: test_contract_model.py::test_a_crafted_file_of_block_comments_does_not_hold_up_the_check[a-file-of-openers] took 0.675 s against a 0.5 s bound. Everything else passed: 5957 passed. run
  • The base's own run: feat/TH-8804-source-subfolder at 084bca2 fails the same case on its push run. run
  • This PR changes only test_hosted_entrypoint.py. The failed job is re-running.

@khushalsonawat
khushalsonawat merged commit 02334dc into feat/TH-8804-source-subfolder Nov 12, 2026
2 of 3 checks passed
@khushalsonawat
khushalsonawat deleted the feat/TH-9515-flush-window-test branch November 12, 2026 08:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants