Repository navigation
feat: recover local workers on macOS and Windows - #1305
Merged
Merged
Conversation
Adds a change-selected Verify job that runs the control store, scheduler and worker supervision tests on macOS and Windows runners against SQLite and a native PostgreSQL server, and counts its result in the aggregate.
Work in progress. Adds sqlbuild-process-ownership, the only crate allowed unsafe code, with a safe API to prepare, start, recover, observe and stop an owned worker. Linux keeps its inherited lock, /proc and pidfd proof. macOS checks the lock through libproc and rechecks the start time just before signalling. Windows runs each worker in a private named Job object owned by, and open only to, the server account. The CLI's local launcher and recovery now use it on all three platforms. Validated so far: the crate's recovery cases pass on Linux, and clippy is clean for the crate on Linux, Windows and macOS targets. The Verify job also carries a temporary diagnostics step for the Windows lease failures, to be removed before review.
The process-ownership crate follows the workspace's fixed 0.0.0 crate version, like every other unreleased crate.
The Windows recovery job failed six orchestration tests with Fenced. Run serially on the same runner, five of them passed: four store-heavy tests at once on the four-core runner starved each other past the 30-second scheduler lease. The job now runs two at a time there. The sixth failed serially as well. The tick-retention test inserts a backlog of a thousand ticks one at a time after taking the lease, and on that runner the inserts outlasted it. The scheduler renews its lease between passes, and the test now renews before pruning in the same way. The temporary latency and serial-timing diagnostics are removed, and so is the CLI step whose filter no longer matched any test.
Store transactions on the Windows runner ran about 20 times slower than on Linux, so tests holding one scheduler lease across their setup were fenced regardless of test thread count.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Scheduler recovery is only proven on Linux. After a scheduler restart on macOS or Windows, local-worker claims are skipped, so a surviving worker keeps its run and write claims with nothing able to stop it. This PR makes recovery work on both platforms and tests it on their runners.
Changes
sqlbuild-process-ownership, the only crate allowed unsafe code (item-level allows with SAFETY comments behind a safe prepare/start/recover/observe/stop API). Every other crate still forbids it./procscan, pidfd).kill. Only a PID reused within those microseconds could be signalled.Local\name with location and claim, owner and DACL limited to the server account), joined while suspended; recovery opens it by name and never adopts a job another account owns.Verification
Work in progress. The crate's recovery cases pass on Linux, and mutating out the marker or descriptor check fails them. Clippy is clean for the crate on Linux, Windows and macOS targets;
make check-ciand the public-repository scan pass. The first runner results are pending. A temporary diagnostics step for the Windows lease failures is included and will be removed before review. No DuckShop case: this is process supervision, not a project shape.