You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Make existing Engine tests fail when the code they run is broken - #1
Replaces cppalliance/runvil#212, which was opened against the repository the Engine has since left. The test changes are the same, line for line.
Existing tests only. No test is added, and no production code changes.
Why
A test that runs a line does not always fail when the line is wrong. Mutation testing finds those cases: it makes one small change to the production code at a time (a swapped operator, a deleted match arm, a function body replaced by a default value) and reports each change that every test still passes. Per-test coverage then names the tests that run the changed line and pass anyway.
Outside runvil-vfs, 89 such changes sit on a line that an existing Engine test runs. For each one, this strengthens the test whose subject is that code: it adds a check on a value the test already produces, or an input case the test did not try.
Crate
Run by a test and passing
Now fail a test
Alter no behavior
No fitting test
runvil-lua
42
24
14
4
runvil-runtime
23
20
3
0
runvil-prompt
14
13
1
0
runvil-types
7
4
1
2
runvil-plugin
2
1
1
0
runvil-model-client
1
1
0
0
Total
89
63
20
6
What changed
45 existing tests gain checks or input cases, in 37 files. The workspace has 3,577 tests before and after, 2,638 of them the Engine's. Three examples:
a_duplicate_role_key_is_rejected checked only that an error came back. It now checks that the error names the role, so replacing ContractKey::shown with an empty string fails it.
one_exact_fence_is_the_prologue_and_two_can_surround_empty_prose tried two lua fences with a blank line between them. It now also tries them with none, which opening < pos as <= in split_section_blocks turns into an internal error.
the_records_span_the_complete_runtime_lifecycle now calls every store operation, so a deleted arm in vfs_observations fails it.
Hunks worth a closer look
Most hunks are one or two assertions. These change more of a test's setup:
an_end_to_end_run_threads_one_vfs_ref_through_every_section now builds its own watched store, to read the origin line that first_chunk_line gives the walk.
a_prelude_write_across_two_arms_ends_the_run_with_determinism_caught_or_not is wrapped in a loop over two parent bodies, which re-indents its existing assertions unchanged.
status_reports_a_parked_task_and_then_a_finished_one gains two fixture sections, so that a task at depth 2 exists.
a_canceled_tasks_partial_write_is_readable_after_the_join gains a second trace.
a_message_to_one_task_does_not_withdraw_another_chains_request now ends with done, to read the unread messages.
The 26 left
20 alter no behavior, so no test can tell them apart. Examples: Vec::new() as vec![], | as ^ on two values with no bit in common, and the library flags of a temporary VM that only parses a chunk.
6 are reached only by tests about something else, where a check would make the test's name untrue. They are missing coverage, not weak tests.
The plan, vibe/2026-11-12-1-strengthen-engine-tests.md, names all 26 with the reason for each.
Checks run
cargo nextest run --locked --workspace --all-features: 3,577 passed.
cargo clippy --locked --workspace --all-targets --all-features with CARGO_BUILD_WARNINGS=deny.
cargo fmt --all --check, cargo test -p build-xtask, and node --test tools/docs-claims.mjs.
A script over the diff: no #[test] added or removed, and no line touched outside test code.
The 89 changes applied again to this tree: 63 fail a test and 26 pass every test, the 26 above.
vibelight gives the same output with and without the change.
Not run locally: cargo dylint, cargo deny, the doc builds, the facade surface check, and the Windows tests. CI covers them.
Where the numbers come from
The Engine was measured in cppalliance/runvil with cargo-mutants 27.1.0, at fa9936f64 and again at 2d40bafa2. This repository at cc51d454 holds the same Engine crates, byte for byte, as that second commit.
Not in this change
runvil-vfs.
54 changes on lines that no test runs. Those need new tests, which is separate work.
runvil-markdown, which is newer than the measurement.
No actionable comments were generated in the recent review.
Walkthrough
This pull request expands existing tests across Lua, prompt, runtime, types, model-client, and plugin code. Added cases cover edge inputs, error mapping, limits, cancellation, task scheduling, VFS observations, and value contracts. A plan records mutation-testing results and scope. No production changes are reported.
Changes: Engine test strengthening
Layer / File(s)
Summary
Lua API and control-flow test cases crates/runvil-internal/lua/src/coro.rs, crates/runvil-internal/lua/src/messages-tests-list.rs, crates/runvil-internal/lua/src/models-tests.rs, crates/runvil-internal/lua/src/models_loop-tests.rs, crates/runvil-internal/lua/src/models_loop/machine-tests.rs, crates/runvil-internal/lua/src/protocol/tests/parse_tasks.rs, crates/runvil-internal/lua/src/tests/cancellation.rs, crates/runvil-internal/lua/src/program.rs, crates/runvil-internal/lua/src/error-tests.rs, crates/runvil-internal/lua/src/tests/diagnostics.rs
Tests add cases for rejected Lua value types, model options and loop responses, integral concurrency limits, cancellation outcomes, error mapping, and source markers.
Lua JSON, store, and tool test cases crates/runvil-internal/lua/src/tests/json/*, crates/runvil-internal/lua/src/tests/store.rs, crates/runvil-internal/lua/src/tests/store_errors.rs, crates/runvil-internal/lua/src/tests/sys_and_var.rs, crates/runvil-internal/lua/src/tools/tests-offering.rs, crates/runvil-internal/lua/src/tools/tests-stats.rs
Tests add JSON depth and cumulative-key boundary cases, store read and conflict cases, global reassignment cases, and tool-offer and statistics cases.
Prompt parsing and contract test cases crates/runvil-internal/prompt/src/body_text.rs, crates/runvil-internal/prompt/src/contract/tests.rs, crates/runvil-internal/prompt/src/tests/section_fences.rs
Tests add borrowed BodyText equality checks, argument-default and duplicate-role cases, and Lua-fence parsing and error cases.
Runtime task, wait, and cancellation test cases crates/runvil-internal/runtime/src/execute/run/tests-drops.rs, crates/runvil-internal/runtime/src/execute/run/tests-inbox.rs, crates/runvil-internal/runtime/src/execute/run/tests-steer.rs, crates/runvil-internal/runtime/src/execute/tests/happens_before.rs, crates/runvil-internal/runtime/src/execute/tests/preludes.rs, crates/runvil-internal/runtime/src/execute/tests/waits.rs
Tests add cases for timed waits, inbox delivery, message steering, nested cancellation, fanout determinism, join ordering, and task depth.
Runtime VFS and execution observation cases crates/runvil-internal/runtime/src/execute/tests/observations.rs, crates/runvil-internal/runtime/src/execute/tests/suite/vfs.rs, crates/runvil-internal/runtime/src/execute/tests/run_inputs.rs, crates/runvil-internal/runtime/src/subst-tests.rs
Tests add VFS lifecycle observations, verify read and write origins during a section handoff, and extend checks for debug output and substitution previews.
Type, identity, and mutation-test records crates/runvil-internal/types/src/error-tests.rs, crates/runvil-internal/types/src/models.rs, crates/runvil-internal/types/src/names-tests.rs, crates/runvil-internal/types/src/untrusted_inventory.rs, crates/runvil-plugin/src/service.rs, crates/runvil-internal/model-client/src/client-tests.rs, vibe/2026-11-12-1-strengthen-engine-tests.md
Tests add cases for error causes, catalog membership, name validation, inventory matching, service-ID hashing, and iterator size hints. The plan records mutation-testing findings, verification details, and scope.
This change only strengthens existing tests and adds a planning document, so it has no runtime impact. It carries no actionable merge risk beyond normal CI checks.
Pre-merge checks: 4
Check name
Status
Explanation
Description Check
Check skipped - CodeRabbit’s high-level summary is enabled.
Title check
The title says the change makes existing Engine tests fail when the code they run is broken. The PR strengthens 45 existing tests with added checks and input cases so they catch production-code mutati…
Linked Issues check
Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check
Check skipped because no linked issues were found for this pull request.
Full details: Title check
Explanation
The title says the change makes existing Engine tests fail when the code they run is broken. The PR strengthens 45 existing tests with added checks and input cases so they catch production-code mutations, which matches the title. The title is specific and describes the main change. It does not name the mutation-testing method, but it does not need to cover every detail.
Finishing Touches
Generate unit tests (beta)
Create a new PR
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info
Run configuration
Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: defaults
Review profile: CHILL
Plan: Advanced
Run ID: a656600d-77be-4627-9a50-0d09d7147d30
Commits
Reviewing files that changed from the base of the PR and between cc51d45 and 4a9117a.
Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.
Mutation testing makes one small change to the production code at a time and reports each change that every test still passes. Outside `runvil-vfs`, 89 such changes sit on a line that an existing test runs: the test passes with the code it runs broken. Checks and input cases added to 45 existing tests now make 63 of the 89 fail a test. No test is added, and no production code changes.
- `runvil-lua`: 24 of 42, in 19 tests.
- `runvil-runtime`: 20 of 23, in 12 tests.
- `runvil-prompt`: 13 of 14, in 8 tests.
- `runvil-types`: 4 of 7, in 4 tests.
- `runvil-plugin` and `runvil-model-client`: 1 each, in one test each.
- Of the 26 left, 20 alter no behavior, so no test can tell them apart, and 6 are reached only by tests about something else. The plan names each one.
- The Engine has 2,638 tests before and after.
- The plan is `vibe/2026-11-12-1-strengthen-engine-tests.md`. `cargo dylint` did not run here, because it is not installed on this machine.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces cppalliance/runvil#212, which was opened against the repository the Engine has since left. The test changes are the same, line for line.
Existing tests only. No test is added, and no production code changes.
Why
A test that runs a line does not always fail when the line is wrong. Mutation testing finds those cases: it makes one small change to the production code at a time (a swapped operator, a deleted
matcharm, a function body replaced by a default value) and reports each change that every test still passes. Per-test coverage then names the tests that run the changed line and pass anyway.Outside
runvil-vfs, 89 such changes sit on a line that an existing Engine test runs. For each one, this strengthens the test whose subject is that code: it adds a check on a value the test already produces, or an input case the test did not try.runvil-luarunvil-runtimerunvil-promptrunvil-typesrunvil-pluginrunvil-model-clientWhat changed
45 existing tests gain checks or input cases, in 37 files. The workspace has 3,577 tests before and after, 2,638 of them the Engine's. Three examples:
a_duplicate_role_key_is_rejectedchecked only that an error came back. It now checks that the error names the role, so replacingContractKey::shownwith an empty string fails it.one_exact_fence_is_the_prologue_and_two_can_surround_empty_prosetried twoluafences with a blank line between them. It now also tries them with none, whichopening < posas<=insplit_section_blocksturns into an internal error.the_records_span_the_complete_runtime_lifecyclenow calls every store operation, so a deleted arm invfs_observationsfails it.Hunks worth a closer look
Most hunks are one or two assertions. These change more of a test's setup:
an_end_to_end_run_threads_one_vfs_ref_through_every_sectionnow builds its own watched store, to read the origin line thatfirst_chunk_linegives the walk.a_prelude_write_across_two_arms_ends_the_run_with_determinism_caught_or_notis wrapped in a loop over two parent bodies, which re-indents its existing assertions unchanged.status_reports_a_parked_task_and_then_a_finished_onegains two fixture sections, so that a task at depth 2 exists.a_canceled_tasks_partial_write_is_readable_after_the_joingains a second trace.a_message_to_one_task_does_not_withdraw_another_chains_requestnow ends withdone, to read the unread messages.The 26 left
Vec::new()asvec![],|as^on two values with no bit in common, and the library flags of a temporary VM that only parses a chunk.The plan,
vibe/2026-11-12-1-strengthen-engine-tests.md, names all 26 with the reason for each.Checks run
cargo nextest run --locked --workspace --all-features: 3,577 passed.cargo clippy --locked --workspace --all-targets --all-featureswithCARGO_BUILD_WARNINGS=deny.cargo fmt --all --check,cargo test -p build-xtask, andnode --test tools/docs-claims.mjs.#[test]added or removed, and no line touched outside test code.cargo dylint,cargo deny, the doc builds, the facade surface check, and the Windows tests. CI covers them.Where the numbers come from
The Engine was measured in
cppalliance/runvilwithcargo-mutants27.1.0, atfa9936f64and again at2d40bafa2. This repository atcc51d454holds the same Engine crates, byte for byte, as that second commit.Not in this change
runvil-vfs.runvil-markdown, which is newer than the measurement.🤖 Generated with Claude Code
Summary by CodeRabbit