feat(hotblocks): run each query task on its own thread behind a flag

[mo4islona]

Part of #118.

Problem

Every query task runs on the shared rayon pool. The par_iter and join inside a task spread over every worker, and rayon takes injected jobs (new tasks) last, so one heavy query holds the whole pool. A worker that waits in join also runs whole tasks from the injector, and its own task ends only after them.

Change

• --query-thread-per-task, also PATCH /settings {"query": {"threadPerTask": true}}, off by default. Each query task runs on one of --query-threads one-thread rayon pools that take tasks from one FIFO queue, so its par_iter and join stay on its thread. No change to the engine code.
• Polars keeps its global pool of the same size, so the polars part of a query stays parallel and shared.
• The mode is chosen when a task takes its slot. hotblocks_query_thread_per_task shows it per replica.
• A separate commit adds benches/query_pool.rs: an open-loop model of the pool that compares the shared pool, thread groups and a thread per task.

Results (model)

With rare heavy tasks, the light-task latency mean / p99 / p99.9 is 61 / 1322 / 3726 ms on the shared pool and 11 / 187 / 608 ms with a thread per task, the same as with no heavy tasks at all. Heavy tasks need 3.6× more own time; their mean latency grows 1.7×. Details: #118 (comment)

benches/query_pool.rs in this PR gives the same picture (8 threads, 280 light tasks/s, 0.3 heavy tasks/s, 120 s per run): 39 / 912 / 2749 ms on the shared pool, 9.9 / 154 / 665 ms with a thread per task, and 9.8 / 154 / 660 ms with no heavy tasks. Heavy own time grows 4.2× and heavy mean latency 2.3× in this run.

Results (production)

One replica in this mode against one on the shared pool, about an hour, with 1.4 times the tasks on the switched one: tasks that waited in the queue over 0.4 s fell from 2,070 to 29, tasks that ran over 3.3 s from 66 to 10, and streams over 80 ms from 3.8% to 1.7%. No client retries or errors. Details: #118 (comment)

Risk

A heavy step loses its parallelism and can get several times slower. Hence the flag and the per-replica switch at runtime.

Tests

• parallel_work_inside_a_task_stays_on_its_thread and a_task_runs_while_another_holds_its_worker. Each fails on the defect it guards: the shared pool, and a queue lock held while a task runs.
• Harness: a query answers the same in both modes; the flag and PATCH /settings switch the mode.

🤖 Generated with Claude Code