Bind the translation figure to its obligations (D272)

coverage/translation/README.md

@@ -15,6 +15,17 @@ separately from syntax coverage, and every figure comes from the checker.
 - `not-translatable.toml`: forms with no meaningful DuckDB translation, in groups, each with its
   reason. A group leaves the denominator only once the auditor signs it off in
   `reviews/auditor-results.toml`, which binds its reason and forms by SHA-256.
+- `obligations/<family>.toml`: what each form must be shown to do (decision D272). Each
+  `[[obligation]]` has `id` (`<form>.t<NNN>`), `form`, `kind` (`type_combination`,
+  `null_argument`, `boundary` with a `subkind`, `error_parity`), `target`, `note`, `catches`,
+  `reading`, `cases` (references `cases/translation/<family>/<file>.toml#<case>`) and `required`.
+  Two shapes have no case:
+  - held: `required = true`, `cases = []`, `held = "<reason>"`, for a case held back (a parse gap,
+    a case waiting on another PR or a receipt). It is never met, so its form cannot match.
+  - unmeasured: `required = false`, `cases = []`, `reason = "<reason>"`, for a documented
+    alternative no case can measure. It is never met and does not block its form.
+- `retired-obligations.toml`: withdrawn obligation ids, each `[[retired]]` with `id`, `form`,
+  `reason` and `replaced_by` (live obligation ids). A retired id is never reused.
 - `results-baseline.json`: every statement's outcome and every form's state, as the checker last
   measured them.
 
@@ -41,10 +52,17 @@ forms when it names none. Any other statement counts for every form it links to.
 | `not_translatable` | Listed in a signed-off group. Outside the denominator. |
 | `differs` | Any linked statement differs. |
 | `refused` | Any linked statement is refused for it, raised a guard for it, or cannot be judged. |
-| `matching` | Otherwise, at least one linked statement returns Snowflake's rows. |
+| `no_obligations` | Otherwise, a linked statement returns Snowflake's rows, but the form declares no required obligation, so nothing shows what it must do. |
+| `unmet_obligations` | Otherwise, a linked statement returns Snowflake's rows, but a required obligation is unmet or held. |
+| `matching` | Otherwise, at least one linked statement returns Snowflake's rows, and it declares required obligations, every one met. |
 | `syntax_only` | Otherwise, linked statements exist but none returns compared rows: only types or matching errors. |
 | `unmeasured` | No recorded statement links to it. |
 
+An obligation is `met` when it cites at least one case and every cited case is `matching` or
+`matching_error`; otherwise it is `unmet`, or `held` or `unmeasured` by its shape. The report lists
+each obligation's state, the `held` and `unmeasured` obligations with their reasons, and the
+`uncited_cases`: recorded translation cases no obligation cites, which meet no obligation.
+
 ## Figures
 
 - Forms: `matching` forms out of all catalogued forms less the signed-off not-translatable ones.
@@ -63,10 +81,17 @@ syntax-coverage results target/translation-statements.json --report target/trans
 syntax-coverage results target/translation-statements.json --write-baseline
 ```
 
-The check fails on a construct that neither links nor names no form, a link to a form the catalogue
+The check fails on obligations it cannot read whole: a missing `obligations/` directory, an
+unknown key, a form the catalogue does not hold or that is absent from its source, an id that is
+not the form's or is declared or retired twice, a case outside its family's folder, cited twice,
+not recorded, or in a case file whose `construct` is another form, a boundary without a known
+subkind, a required obligation with neither cases nor a `held` reason, a blank reason, and
+a retired id that is reused or replaced by an id that is not a live obligation. It also fails on a
+construct that neither links nor names no form, a link to a form the catalogue
 does not hold, a signed-off not-translatable form with a matching statement, and any difference
 from `results-baseline.json`. Against the target branch's baseline it prints the diff and fails if a
-statement or form drops out of matching, or becomes `differs`.
+statement or form drops out of matching, a met obligation stops being met, an obligation is
+dropped without being listed in `retired-obligations.toml`, or anything becomes `differs`.
 
 The report's worklist lists differences first, then refusals by how many forms each keeps from
 matching, then forms that need a recorded result.

crates/sqlquack-syntax-coverage/src/cli/_helpers/results_command.rs

@@ -44,13 +44,16 @@ fn percent(part: usize, whole: usize) -> String {
 
 fn tally_line(name: &str, tally: &FormTally) -> String {
     format!(
-        "{name}: {} of {} translatable forms match ({}); {} syntax only, {} refused, {} differ, \
-         {} unmeasured; {} of {} catalogued forms not translatable",
+        "{name}: {} of {} translatable forms match ({}); {} syntax only, {} refused, {} with unmet \
+         obligations, {} with none declared, {} differ, {} unmeasured; {} of {} catalogued forms \
+         not translatable",
         tally.matching,
         tally.translatable,
         percent(tally.matching, tally.translatable),
         tally.syntax_only,
         tally.refused,
+        tally.unmet_obligations,
+        tally.no_obligations,
         tally.differs,
         tally.unmeasured,
         tally.not_translatable,
@@ -92,6 +95,16 @@ fn report_lines(report: &ResultsReport) -> Vec<String> {
                 .collect::<Vec<_>>()
                 .join(", ")
         ),
+        format!(
+            "obligations: {} declared, {} met, {} unmet, {} held, {} unmeasured; a form matches only \
+             when every required obligation is met; {} recorded translation cases cite none",
+            report.obligations.declared,
+            report.obligations.met,
+            report.obligations.unmet,
+            report.obligations.held,
+            report.obligations.unmeasured,
+            report.uncited_cases.len(),
+        ),
         tally_line("total", &report.forms),
     ];
     lines.extend(
@@ -187,12 +200,11 @@ pub(crate) fn results(
         statements.display()
     ));
     let coverage = load_coverage(&common.root).map_err(|problems| fail(&problems))?;
-    let directory: PathBuf = common
-        .root
-        .parent()
-        .unwrap_or(Path::new("."))
-        .join(TRANSLATION_DIR);
-    let files = load_results_files(&directory, &coverage).map_err(|problems| fail(&problems))?;
+    let coverage_root: &Path = common.root.parent().unwrap_or(Path::new("."));
+    let directory: PathBuf = coverage_root.join(TRANSLATION_DIR);
+    let repository: &Path = coverage_root.parent().unwrap_or(Path::new("."));
+    let files = load_results_files((&directory, repository), &coverage)
+        .map_err(|problems| fail(&problems))?;
     let recorded: StatementsFile = read_json(statements).map_err(|problem| fail(&[problem]))?;
     let report =
         measure_results(&coverage, &files, &recorded).map_err(|problems| fail(&problems))?;

crates/sqlquack-syntax-coverage/src/results/_helpers/diff.rs

@@ -3,7 +3,7 @@
 use std::collections::{BTreeMap, BTreeSet};
 use std::fmt::Debug;
 
-use crate::results::models::{ResultState, ResultsDiff, StatementOutcome};
+use crate::results::models::{ObligationState, ResultState, ResultsDiff, StatementOutcome};
 
 /// Whether a change from the old value or none to the new or none regresses.
 pub(crate) type Regresses<T> = fn(Option<T>, Option<T>) -> bool;
@@ -25,6 +25,34 @@ pub(crate) fn form_regresses(base: Option<ResultState>, head: Option<ResultState
     (was_matching && head != Some(ResultState::Matching)) || differs
 }
 
+/// Obligation changes: a met one stops being met, or any is dropped that `retired` does not list.
+pub(crate) fn compare_obligations(
+    (base, head): (
+        &BTreeMap<String, ObligationState>,
+        &BTreeMap<String, ObligationState>,
+    ),
+    retired: &BTreeSet<String>,
+) -> ResultsDiff {
+    let mut diff = ResultsDiff::default();
+    let keys: BTreeSet<&String> = base.keys().chain(head.keys()).collect();
+    for key in keys {
+        let (before, after) = (base.get(key).copied(), head.get(key).copied());
+        if before == after {
+            continue;
+        }
+        let line = format!("obligation {key}: {before:?} -> {after:?}");
+        let regresses = match after {
+            None => !retired.contains(key.as_str()),
+            Some(after) => before == Some(ObligationState::Met) && after != ObligationState::Met,
+        };
+        match regresses {
+            true => diff.regressions.push(line),
+            false => diff.changes.push(line),
+        }
+    }
+    diff
+}
+
 /// Each changed key of one kind, as a regression or another change.
 pub(crate) fn compare<T: Copy + PartialEq + Debug>(
     label: &str,

crates/sqlquack-syntax-coverage/src/results/_helpers/files.rs

@@ -7,6 +7,7 @@ use serde::de::DeserializeOwned;
 
 use crate::formats::main::content_sha256::content_sha256;
 use crate::formats::main::is_utc_timestamp::is_utc_timestamp;
+use crate::results::_helpers::obligations::load_obligations;
 use crate::results::constants::{
     FORM_LINKS_SUBJECT, FUNCTION_LINKS_SUBJECT, LINKS_FILE, NOT_TRANSLATABLE_FILE,
     NOT_TRANSLATABLE_SUBJECT, RESULTS_REVIEW_FILE,
@@ -146,11 +147,12 @@ fn check_review(review: &ResultsReviewFile) -> Vec<String> {
     problems
 }
 
-/// The files under `directory`, or every problem with them; ids must be catalogued forms.
+/// The files under `directory`, cases under `repository`, or every problem; ids must be catalogued.
 pub(crate) fn load_files(
-    directory: &Path,
-    catalogued: &BTreeSet<&str>,
+    (directory, repository): (&Path, &Path),
+    families: &BTreeMap<&str, &str>,
 ) -> Result<ResultsFiles, Vec<String>> {
+    let catalogued: &BTreeSet<&str> = &families.keys().copied().collect();
     let read = |name: &str| {
         let path = directory.join(name);
         read_text(&path).map(|text| (path, text))
@@ -182,11 +184,14 @@ pub(crate) fn load_files(
     if !problems.is_empty() {
         return Err(problems);
     }
+    let (obligations, retired) = load_obligations((directory, repository), families)?;
     Ok(ResultsFiles {
         links,
         not_translatable,
         review,
         links_sha256: content_sha256(links_text.as_bytes()),
+        obligations,
+        retired,
     })
 }
 

crates/sqlquack-syntax-coverage/src/results/_helpers/forms.rs

@@ -6,8 +6,12 @@ use crate::results::models::{
     FormResult, FormTally, RecordedStatement, ResultState, StatementOutcome, StatementTally,
 };
 
-/// A form's state: listed, then any statement that differs, then any refused, then matching rows.
-pub(crate) fn form_state(listed: bool, outcomes: &[StatementOutcome]) -> ResultState {
+/// A form's state: listed, differs, refused, then matching only with every obligation met.
+pub(crate) fn form_state(
+    listed: bool,
+    outcomes: &[StatementOutcome],
+    obligations_met: Option<bool>,
+) -> ResultState {
     if listed {
         return ResultState::NotTranslatable;
     }
@@ -20,9 +24,14 @@ pub(crate) fn form_state(listed: bool, outcomes: &[StatementOutcome]) -> ResultS
     if outcomes.iter().any(|outcome| outcome.refused()) {
         return ResultState::Refused;
     }
-    match outcomes.contains(&StatementOutcome::Matching) {
-        true => ResultState::Matching,
-        false => ResultState::SyntaxOnly,
+    match (
+        outcomes.contains(&StatementOutcome::Matching),
+        obligations_met,
+    ) {
+        (true, Some(true)) => ResultState::Matching,
+        (true, Some(false)) => ResultState::UnmetObligations,
+        (true, None) => ResultState::NoObligations,
+        (false, _) => ResultState::SyntaxOnly,
     }
 }
 
@@ -31,6 +40,7 @@ pub(crate) fn form_result(
     (id, family): (&str, &str),
     listed: bool,
     linked: &[&RecordedStatement],
+    (unmet_obligations, declared): (Vec<String>, bool),
 ) -> FormResult {
     let outcomes: Vec<StatementOutcome> =
         linked.iter().map(|statement| statement.outcome).collect();
@@ -42,9 +52,14 @@ pub(crate) fn form_result(
     FormResult {
         id: id.to_owned(),
         family: family.to_owned(),
-        state: form_state(listed, &outcomes),
+        state: form_state(
+            listed,
+            &outcomes,
+            declared.then_some(unmet_obligations.is_empty()),
+        ),
         statements: linked.len(),
         matching_statements: outcomes.iter().filter(|outcome| outcome.matches()).count(),
+        unmet_obligations,
         codes: codes.into_iter().collect(),
     }
 }
@@ -59,6 +74,8 @@ pub(crate) fn tally<'a>(forms: impl Iterator<Item = &'a FormResult>) -> FormTall
             ResultState::Matching => &mut tally.matching,
             ResultState::SyntaxOnly => &mut tally.syntax_only,
             ResultState::Refused => &mut tally.refused,
+            ResultState::UnmetObligations => &mut tally.unmet_obligations,
+            ResultState::NoObligations => &mut tally.no_obligations,
             ResultState::Differs => &mut tally.differs,
             ResultState::Unmeasured => &mut tally.unmeasured,
         };

crates/sqlquack-syntax-coverage/src/results/_helpers/mod.rs

@@ -2,5 +2,6 @@ pub(crate) mod diff;
 pub(crate) mod files;
 pub(crate) mod forms;
 pub(crate) mod links;
+pub(crate) mod obligations;
 pub(crate) mod report;
 pub(crate) mod worklist;