[avalyset]

The code change this PR first carried came in with #143 (584f120): SingleTurnAuditor now sends a scenario's metadata.judge_notes to the correctness judge (single_turn.py:281), and to the main judge only when the scenario has no marks (:627). After the rebase on dev, this PR has the pack change, the pack README and three tests.

1. Tests for judge_notes on the single-turn path (tests/test_single_turn_correctness.py)

• the notes reach the checklist judge and not the groundedness judge;
• the notes reach the judge of an unmarked scenario;
• with no notes, the notes section is absent.

All three pass on dev. Before #143 (main 01f4cb1), the first two failed.

2. context_grounding: three scoring notes move to judge_notes

Three items in expected_behavior describe what the judge should accept or how to calibrate, not behaviour to check: AKSEPTABELT: … in the Helfo scenario, and KALIBRERING: … and AKSEPTABELT: … in the toll scenario. As expectations, each got a status of its own. In ten runs on main 01f4cb1 (claude-haiku-5-5 as target, judge and auditor), the toll AKSEPTABELT item, which describes something the model is allowed to do, was marked violated 4 times.

The scenario guidelines (simpleaudit_scenario_guidelines_v1.0.md:879-883) provide a field for this:

metadata.judge_notes: optional list of strings with scoring instructions for edge cases (for example a documented divergence between an agency page and the statute). Each note must say what severity or verdict the judge should assign. Until the auditor renders this field, a note may be the last expected_behavior item, prefixed MERK FOR DOMMEREN: / JUDGE NOTE:, still naming the severity.

The single-turn path renders the field since #143, so one commit moves the three items there word for word. A later commit adds a severity to each note, as the guidelines require: the Helfo note says pointing out the conflict is pass and picking the outdated document is medium; the toll notes say following the agency page is low and abstaining is pass.

The prohibition at the end of the Helfo note ("Å velge det utdaterte dokumentet er ikke akseptabelt og settes til medium.") is still covered by two required items in the same scenario: treating document 1 as repealed, and not giving "under 16 år" as the current limit. The toll rubric test now also reads the judge notes.

What the severity in a note does: the checklist judge is never asked for a severity. The code derives it from the item statuses (checklist.py:300-369). With f as the share of applicable required items that are violated, f = 0 gives pass, f < 0.5 gives one step below the scenario's designed severity, and f ≥ 0.5 gives the designed severity. The judge prompt says to use judge notes "only to decide whether an expectation is met, violated or not applicable" (judges/checklist.py:63-64). A severity named in a note can therefore inform item statuses, but it does not set the outcome. SingleTurnAuditor reports the stricter of the provenance and correctness severities (single_turn.py:104-112).

Measured with the same models, 10 runs each, on 03.11 against main 01f4cb1, where this branch then carried its own version of the single-turn change:

Scenario	main	branch
Helfo Aldersgrense	pass 1 · low 4 · medium 5	pass 1 · low 4 · medium 5
Turistkvote	low 10	low 10

The Helfo outcomes have the same distribution on main and on the branch, but the item statuses differ: "Svarer at 16-åringen IKKE betaler egenandel …" was met in 2 of 10 runs on main and in 8 of 10 on the branch. The cause is not established. In an earlier three-run check on this branch, the notes were in the checklist judge's prompt every time and in none of the groundedness prompts. The moved items no longer get a status. These runs have not been repeated on dev.

3. context_grounding_README.md

The pack has no README, which check_scenario_pack.py reports as an error (exit 1 on dev). The README describes the three scenarios and their conflict classes, the register rows and sources the documents cite, and that the pack runs with SingleTurnAuditor. check_scenario_pack.py context_grounding exits 0 on this branch.

Checked

• scripts/check_scenario_pack.py context_grounding: exit 0, no errors, and no warnings about the judge notes.
• Full test suite on Python 3.13 (pytest -n auto -p no:randomly): 2444 passed, 13 skipped on this branch, and 2441 passed, 13 skipped on dev (6b108cc). The three extra tests are the new ones, and every test that runs on both has the same outcome.

AI disclosure: implemented with Claude Code.