You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Make existing Engine tests fail when the code they run is broken - #212
Existing tests only. No test is added, and no production code changes.
Why
A test that runs a line does not always fail when the line is wrong. Mutation testing finds those cases: it makes one small change to the production code at a time (a swapped operator, a deleted match arm, a function body replaced by a default value) and reports each change that every test still passes. Per-test coverage then names the tests that run the changed line and pass anyway.
Outside runvil-vfs, 89 such changes sit on a line that an existing Engine test runs. For each one, this strengthens the test whose subject is that code: it adds a check on a value the test already produces, or an input case the test did not try.
Crate
Run by a test and passing
Now fail a test
Alter no behavior
No fitting test
runvil-lua
42
24
14
4
runvil-runtime
23
20
3
0
runvil-prompt
14
13
1
0
runvil-types
7
4
1
2
runvil-plugin
2
1
1
0
runvil-model-client
1
1
0
0
Total
89
63
20
6
What changed
45 existing tests gain checks or input cases, in 37 files. The Engine has 2,638 tests before and after. Three examples:
a_duplicate_role_key_is_rejected checked only that an error came back. It now checks that the error names the role, so replacing ContractKey::shown with an empty string fails it.
one_exact_fence_is_the_prologue_and_two_can_surround_empty_prose tried two lua fences with a blank line between them. It now also tries them with none, which opening < pos as <= in split_section_blocks turns into an internal error.
the_records_span_the_complete_runtime_lifecycle now calls every store operation, so a deleted arm in vfs_observations fails it.
Hunks worth a closer look
Most hunks are one or two assertions. These change more of a test's setup:
an_end_to_end_run_threads_one_vfs_ref_through_every_section now builds its own watched store, to read the origin line that first_chunk_line gives the walk.
a_prelude_write_across_two_arms_ends_the_run_with_determinism_caught_or_not is wrapped in a loop over two parent bodies, which re-indents its existing assertions unchanged.
status_reports_a_parked_task_and_then_a_finished_one gains two fixture sections, so that a task at depth 2 exists.
a_canceled_tasks_partial_write_is_readable_after_the_join gains a second trace.
a_message_to_one_task_does_not_withdraw_another_chains_request now ends with done, to read the unread messages.
The 26 left
20 alter no behavior, so no test can tell them apart. Examples: Vec::new() as vec![], | as ^ on two values with no bit in common, and the library flags of a temporary VM that only parses a chunk.
6 are reached only by tests about something else, where a check would make the test's name untrue. They are missing coverage, not weak tests.
The plan, vibe/2026-11-12-1-strengthen-engine-tests.md, names all 26 with the reason for each.
Checks run
cargo nextest run --locked --all-features over the nine Engine packages: 2,638 passed, twice.
cargo clippy --locked --all-targets --all-features over the six changed packages, with CARGO_BUILD_WARNINGS=deny.
cargo fmt --all --check, cargo test -p build-xtask, and node --test crates/workshop/ui/test/docs-claims.mjs.
A script over the diff: no file added, no #[test] added or removed, no line touched outside test code.
The 89 changes applied again to this tree: 63 fail a test and 26 pass every test, the 26 above.
vibelight on the six crates gives the same output with and without the change.
Not run locally: cargo dylint, which is not installed on my machine, and the whole-workspace test, clippy, and doc commands. CI covers them.
Not in this change
runvil-vfs.
54 changes on lines that no test runs. Those need new tests, which is separate work.
runvil-markdown, which is newer than the measurement. The list of changes comes from fa9936f64 and was tested again at 2d40bafa2.
Expanded coverage across Lua, prompt handling, runtime execution, storage, model selection, diagnostics, and error handling. New cases verify boundary conditions, invalid inputs, cancellation behavior, task ordering, and event reporting.
Documentation
Added a testing plan summarizing coverage gaps, measurement results, and execution criteria.
No actionable comments were generated in the recent review.
Walkthrough
This pull request adds edge-case assertions to existing tests across Lua, prompt, runtime, shared types, model-client, and plugin code. It also adds a plan that records mutation-testing findings and the scope and execution of the test-strengthening work.
Changes: Engine test coverage
Layer / File(s)
Summary
Mutation-testing findings and scope vibe/2026-11-12-1-strengthen-engine-tests.md
The plan documents 89 mutation cases, their disposition, verification criteria, exclusions, and a completed execution step.
Lua API, model calls, and cancellation tests crates/runvil-internal/lua/src/coro.rs, crates/runvil-internal/lua/src/error-tests.rs, crates/runvil-internal/lua/src/messages-tests-list.rs, crates/runvil-internal/lua/src/models*, crates/runvil-internal/lua/src/program.rs, crates/runvil-internal/lua/src/protocol/tests/parse_tasks.rs, crates/runvil-internal/lua/src/tests/{cancellation,diagnostics,sys_and_var}.rs
Tests cover handle validation, error mapping, model-call batches, integral float limits, cancellation, diagnostics, and global reassignment.
Lua serialization, storage, and tool tests crates/runvil-internal/lua/src/tests/json/*, crates/runvil-internal/lua/src/tests/{store,store_errors}.rs, crates/runvil-internal/lua/src/tools/tests-*
Tests cover JSON depth and key limits, numbered reads, conflict errors, offer validation, and tool-call statistics.
Prompt contracts and section parsing crates/runvil-internal/prompt/src/body_text.rs, crates/runvil-internal/prompt/src/contract/tests.rs, crates/runvil-internal/prompt/src/tests/section_fences.rs
Tests cover body equality, integer defaults, duplicate-role errors, Lua fence parsing, and compile-error attribution.
Runtime task scheduling and store behavior crates/runvil-internal/runtime/src/execute/run/tests-{drops,inbox,steer}.rs, crates/runvil-internal/runtime/src/execute/tests/{happens_before,preludes,waits}.rs
Tests cover answer ordering, inbox delivery, steering, cancellation and store visibility, fanout conflicts, task selection, and nested task depth.
Runtime VFS and diagnostic observations crates/runvil-internal/runtime/src/execute/tests/{observations,run_inputs}.rs, crates/runvil-internal/runtime/src/execute/tests/suite/vfs.rs, crates/runvil-internal/runtime/src/subst-tests.rs
Tests cover VFS lifecycle events and operation origins, run-parameter formatting, and substitution diagnostic output.
Shared types and identifier behavior crates/runvil-internal/model-client/src/client-tests.rs, crates/runvil-internal/types/src/{error-tests.rs,models.rs,names-tests.rs,untrusted_inventory.rs}, crates/runvil-plugin/src/service.rs
Tests cover iterator sizing, error causes, model catalog membership, name and marker validation, and service identifier hashing.
This PR strengthens existing tests and adds a planning document, with no production behavior change. No actionable merge-blocking risk remains.
Pre-merge checks: 4
Check name
Status
Explanation
Description Check
Check skipped - CodeRabbit’s high-level summary is enabled.
Title check
The title states the main change: existing Engine tests were strengthened so they fail when the code they exercise is broken. The raw summary shows many test files gaining new checks and assertions, a…
Linked Issues check
Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check
Check skipped because no linked issues were found for this pull request.
Full details: Title check
Explanation
The title states the main change: existing Engine tests were strengthened so they fail when the code they exercise is broken. The raw summary shows many test files gaining new checks and assertions, and the PR objectives confirm 63 mutation cases now cause test failures. The title is specific and matches the changeset.
Finishing Touches
Generate unit tests (beta)
Create a new PR
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info
Run configuration
Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: defaults
Review profile: CHILL
Plan: Advanced
Run ID: 1cdf59ff-4cdd-4c16-98f1-ef7bfeed58fa
Commits
Reviewing files that changed from the base of the PR and between 2d40baf and f2b18aa.
Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.
Mutation testing makes one small change to the production code at a time and reports each change that every test still passes. Outside `runvil-vfs`, 89 such changes sit on a line that an existing test runs: the test passes with the code it runs broken. Checks and input cases added to 45 existing tests now make 63 of the 89 fail a test. No test is added, and no production code changes.
- `runvil-lua`: 24 of 42, in 19 tests.
- `runvil-runtime`: 20 of 23, in 12 tests.
- `runvil-prompt`: 13 of 14, in 8 tests.
- `runvil-types`: 4 of 7, in 4 tests.
- `runvil-plugin` and `runvil-model-client`: 1 each, in one test each.
- Of the 26 left, 20 alter no behavior, so no test can tell them apart, and 6 are reached only by tests about something else. The plan names each one.
- The Engine has 2,638 tests before and after.
- The plan is `vibe/2026-11-12-1-strengthen-engine-tests.md`. `cargo dylint` did not run here, because it is not installed on this machine.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Laserwolve
changed the title
Test the runvil-prompt lines that could break without failing a test
Make existing Engine tests fail when the code they run is broken
Nov 12, 2026
[Low impact] The PR appears safe to merge; no actionable issue was found in the test changes.
Summary
Strengthens 45 existing Engine tests across six crates without changing production code or adding test functions.
Existing Engine tests catch more faults in the code they exercise.
Laserwolve explicitly deferred the 54 mutations on untested lines and six without a fitting existing test to separate coverage work. Laserwolve also excluded 20 behavior-equivalent mutations, runvil-vfs, and runvil-markdown from this change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Existing tests only. No test is added, and no production code changes.
Why
A test that runs a line does not always fail when the line is wrong. Mutation testing finds those cases: it makes one small change to the production code at a time (a swapped operator, a deleted
matcharm, a function body replaced by a default value) and reports each change that every test still passes. Per-test coverage then names the tests that run the changed line and pass anyway.Outside
runvil-vfs, 89 such changes sit on a line that an existing Engine test runs. For each one, this strengthens the test whose subject is that code: it adds a check on a value the test already produces, or an input case the test did not try.runvil-luarunvil-runtimerunvil-promptrunvil-typesrunvil-pluginrunvil-model-clientWhat changed
45 existing tests gain checks or input cases, in 37 files. The Engine has 2,638 tests before and after. Three examples:
a_duplicate_role_key_is_rejectedchecked only that an error came back. It now checks that the error names the role, so replacingContractKey::shownwith an empty string fails it.one_exact_fence_is_the_prologue_and_two_can_surround_empty_prosetried twoluafences with a blank line between them. It now also tries them with none, whichopening < posas<=insplit_section_blocksturns into an internal error.the_records_span_the_complete_runtime_lifecyclenow calls every store operation, so a deleted arm invfs_observationsfails it.Hunks worth a closer look
Most hunks are one or two assertions. These change more of a test's setup:
an_end_to_end_run_threads_one_vfs_ref_through_every_sectionnow builds its own watched store, to read the origin line thatfirst_chunk_linegives the walk.a_prelude_write_across_two_arms_ends_the_run_with_determinism_caught_or_notis wrapped in a loop over two parent bodies, which re-indents its existing assertions unchanged.status_reports_a_parked_task_and_then_a_finished_onegains two fixture sections, so that a task at depth 2 exists.a_canceled_tasks_partial_write_is_readable_after_the_joingains a second trace.a_message_to_one_task_does_not_withdraw_another_chains_requestnow ends withdone, to read the unread messages.The 26 left
Vec::new()asvec![],|as^on two values with no bit in common, and the library flags of a temporary VM that only parses a chunk.The plan,
vibe/2026-11-12-1-strengthen-engine-tests.md, names all 26 with the reason for each.Checks run
cargo nextest run --locked --all-featuresover the nine Engine packages: 2,638 passed, twice.cargo clippy --locked --all-targets --all-featuresover the six changed packages, withCARGO_BUILD_WARNINGS=deny.cargo fmt --all --check,cargo test -p build-xtask, andnode --test crates/workshop/ui/test/docs-claims.mjs.#[test]added or removed, no line touched outside test code.cargo dylint, which is not installed on my machine, and the whole-workspace test, clippy, and doc commands. CI covers them.Not in this change
runvil-vfs.runvil-markdown, which is newer than the measurement. The list of changes comes fromfa9936f64and was tested again at2d40bafa2.🤖 Generated with Claude Code
Summary by CodeRabbit
No production behavior changes are included.