Skip to content

fix(hotblocks): give each shadow run its own thread - #120

Merged
mo4islona merged 1 commit into
masterfrom
fix/shadow-run-own-thread
Oct 16, 2026
Merged

mo4islona merged 1 commit into
masterfrom
fix/shadow-run-own-thread

Conversation

@mo4islona

Copy link
Copy Markdown
Contributor

Problem

The shadow's slow-run count makes the new engine look slower than it is, for two reasons.

  • A run's time can hold other runs. Every shadow run is a job on one rayon pool. While a thread waits in join for a run's parallel work, rayon gives it any other job of the pool, and in this pool every job is a whole run. The waiting run's time then includes the other run. Both engines use rayon on the current pool, and the legacy engine's polars steps wait the same way.
  • Only one side is counted. A run is counted and logged when the new engine is much slower. A stall in the legacy engine leaves no trace, so random stalls show up only as "the new engine is slow".

The slow-run log of one busy dataset shows both effects. Its runs on a one-block chunk took the new engine 76 ms at the median and the legacy one 2 ms. Neither engine spends 76 ms on one block, and over all runs the new engine's time tail is lighter than the legacy one's.

Change

  • Each run goes to one of --shadow-max-in-flight single-thread workers, the TaskQueue that query tasks use with --query-thread-per-task. A run's par_iter and join stay on its thread, and no run waits for another. The number of shadow threads does not change.
  • hotblocks_shadow_legacy_slow_runs_total counts the runs where the legacy engine is slow by the same rule, so that the two counts can be compared. Only runs where the new engine is slow are logged, as before.

Trade-off

A run's rayon work now runs on one thread for both engines; the legacy engine's polars steps stay on the shared pool. A heavy run takes longer than before, when idle shadow threads could help it, and the same as it would with a thread per task. Slow-run counts from before and after this change do not compare.

Tests

  • a_run_never_runs_another_inside_its_time: a run waits in join while a second run is queued. It fails on the old shared pool 10 times in 10, and passes 20 times in 20.
  • either_engine_can_be_the_slower checks the rule in both directions, under the margin and under the ratio.
  • Unit tests, shadow_engine and query_task_metrics pass, and clippy reports nothing new.

🤖 Generated with Claude Code

A shadow thread that waits for a run's parallel work runs other jobs of its
pool meanwhile, and in the shared shadow pool those jobs are whole other
runs, so one run's time could hold several. Runs now go to single-thread
workers, as query tasks do with a thread per task. Runs where the legacy
engine is the slow one are counted too: the slow-run count alone cannot
tell a slow engine from a slow host.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@mo4islona
mo4islona merged commit 179411e into master Oct 16, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant