fix: score the vPlan whatever the regression concluded

Review fixes for the vPlan back-annotation series.

The plan could not be scored in the regression it most needs to describe.
`needs_all_dependencies_passing` had two states and this job needs a third,
so it becomes `DependencyPolicy`. `ALL_PASSING` is the default and
`ANY_PASSING` is what CovMerge always did, so neither changes behaviour.
CovVPlan takes `ALWAYS` and runs once its dependencies are terminal,
whatever they concluded. Under either existing policy a regression where
nothing passed was killed rather than scored, and with --cov the job has a
single dependency, so anything that stopped the coverage report also
stopped the plan.

dvsim now defines the evidence format rather than deferring to dvplan.
doc/dv_evidence.md specifies it and the pydantic models are normative.
dvsim produces the file and is the public repo, so a consumer can be
written against a spec rather than against whichever tool was built first.

Whether dvplan is installed is decided by the job's own script, so it reads
the PATH of the machine the job lands on rather than that of the host dvsim
was launched from, which on a compute farm need not be the same.

A `dvplan_inspect` pattern matching nothing is now a config error. The
command is built while the jobs are, so it stops the run in seconds rather
than failing inside dvplan once the regression has already gone.

The vPlan report page is linked only once it exists, since a killed job or
a machine without dvplan otherwise left a dead link in the HTML report.

AI-assisted (Claude Code) — reviewed and approved by author

Author: martin-velay

README.md

@@ -86,6 +86,7 @@ You can access it [online at opentitan.org/book/](https://opentitan.org/book/).
 
 * [Testplanner tool](./doc/testplanner.md)
 * [Design document](./doc/design_doc.md)
+* [The `lowrisc-dv-evidence` format](./doc/dv_evidence.md)
 * [Glossary](./doc/glossary.md)
 
 ## How to contribute

doc/dv_evidence.md

@@ -0,0 +1,92 @@
+<!--
+# Copyright lowRISC contributors (OpenTitan project).
+# Licensed under the Apache License, Version 2.0, see LICENSE for details.
+# SPDX-License-Identifier: Apache-2.0
+-->
+# The `lowrisc-dv-evidence` format
+
+A regression tells you which tests passed.
+A verification plan asks a different question: of everything we said we would verify, how much is now backed by something that ran?
+Answering it needs the regression's own outcomes in a form a planning tool can read, rather than a log directory and a human.
+
+This is that form.
+DVSim writes one of these files per simulation flow, and it is the definition of the format rather than a description of one tool's output.
+Anything that can produce it can be scored against a verification plan, whether or not it is DVSim.
+
+## Where DVSim writes it
+
+`<scratch_path>/cov_vplan/dv_evidence.json`, produced by the `cov_vplan` job, and only when the sim config names a `vplan`.
+It is written before the annotation step runs and is archived alongside the reports, so it outlives the scratch area it describes.
+
+## Shape
+
+```json
+{
+  "schema": "lowrisc-dv-evidence",
+  "dut": "hmac",
+  "tool": "xcelium",
+  "produced_by": "dvsim 1.50.1",
+  "revision": "https://github.com/lowRISC/opentitan/tree/a1b2c3d (dirty)",
+  "timestamp": "2026-08-18T09:00:00+00:00",
+  "testcase": {
+    "hmac_smoke": [
+      { "status": "passed", "seed": 1234, "log": "/scratch/hmac/1234.hmac_smoke/run.log" },
+      { "status": "failed", "seed": 5678, "log": "...", "message": "UVM_ERROR", "line": 812 }
+    ],
+    "hmac_stress_all": [
+      { "status": "not_run" }
+    ]
+  }
+}
+```
+
+Fields are omitted when they have no value rather than written as `null`.
+
+### Top level
+
+| Key | Meaning |
+| --- | --- |
+| `schema` | Always `lowrisc-dv-evidence`. Identifies the format to whatever reads the file. |
+| `testcase` | Test name to the list of runs of that test. The only required key. |
+| `dut` | The design the results are about, named as a verification plan addresses it. |
+| `tool` | The simulator that produced them. |
+| `produced_by` | What wrote the file, with its version. |
+| `revision` | The tree the results were produced against, suffixed ` (dirty)` when it was not clean. |
+| `timestamp` | When the run started, as an ISO 8601 datetime with an offset. |
+
+### A run
+
+Every entry under `testcase` is keyed by the test name, because that is the name a plan refers to.
+Reseeds of one test share the key and are told apart by `seed`.
+
+| Key | Meaning |
+| --- | --- |
+| `status` | One of `passed`, `failed`, `killed`, `not_run`. Required. |
+| `seed` | The seed the run used, where the flow randomises. |
+| `log` | Path to the run's log. |
+| `message` | Why it ended that way. Present only on a run that did not pass. |
+| `line` | The log line the failure was first reported at. |
+
+`killed` and `not_run` are separate on purpose.
+A killed test started and was terminated, so the design was exercised and something went wrong.
+A `not_run` test never started, because the scheduler cancelled it once a dependency failed or the run was shut down.
+The two are different answers to "did we verify this", and collapsing them would let a build failure read as a passing plan item.
+
+There is no `waived` status.
+A waiver needs an owner and a date, and a regression can supply neither, so a known failure is recorded as an inspection instead.
+
+## Inspections
+
+The format also carries an `inspection` key, for claims no simulation can measure, such as a parameterisation or a structural fact.
+Those records are written by hand and live in the tree next to the plan they support.
+DVSim never produces them; it only passes their path through to whatever consumes this format, so they are out of scope for this document.
+
+## Consumers
+
+[DVPlan](https://github.com/lowRISC/dvplan) reads it to back-annotate a verification plan.
+It is not the only thing that could: the format carries no DVPlan concepts, and a dashboard or a CI job wanting machine-readable regression results can read the same file.
+
+## Changing it
+
+The pydantic models in `src/dvsim/report/dv_evidence.py` are the normative definition, and this document describes them.
+A change to either is a change to the format, so change both, and bear in mind that a consumer may be reading files this repo wrote months ago.

src/dvsim/job/data.py

@@ -10,6 +10,7 @@
 """
 
 from collections.abc import Callable, Mapping, Sequence
+from enum import Enum
 from pathlib import Path
 from typing import TypeAlias
 
@@ -20,12 +21,30 @@
 
 __all__ = (
     "CompletedJobStatus",
+    "DependencyPolicy",
     "JobSpec",
     "JobStatusInfo",
     "WorkspaceConfig",
 )
 
 
+class DependencyPolicy(Enum):
+    """When a job may start, given how the jobs it depends on ended."""
+
+    ALL_PASSING = "all_passing"
+    """Start only if every dependency passed, which is right for a job consuming their output."""
+
+    ANY_PASSING = "any_passing"
+    """Start if at least one dependency passed, for a job gathering whatever results exist."""
+
+    ALWAYS = "always"
+    """Start once every dependency is terminal, whatever they concluded.
+
+    For a job whose input is the outcome itself rather than an artefact a dependency produced, so
+    a regression where nothing passed is still the thing it has to report on.
+    """
+
+
 class WorkspaceConfig(BaseModel):
     """Workspace configuration."""
 
@@ -92,8 +111,8 @@ class JobSpec(BaseModel):
 
     dependencies: list[str]
     """Full names of the other Jobs that this one depends on."""
-    needs_all_dependencies_passing: bool
-    """Wait for dependent jobs to pass before scheduling."""
+    dependency_policy: DependencyPolicy
+    """What the jobs this one depends on must have concluded before it may be scheduled."""
     weight: int
     """Weight to apply to the scheduling priority."""
     timeout_mins: float | None

src/dvsim/job/deploy.py

@@ -12,7 +12,7 @@
 from typing import TYPE_CHECKING, ClassVar
 
 from dvsim.flow.base import FlowCfg
-from dvsim.job.data import JobSpec
+from dvsim.job.data import DependencyPolicy, JobSpec
 from dvsim.job.status import JobStatus
 from dvsim.job.time import JobTime
 from dvsim.logging import log
@@ -94,10 +94,9 @@ def __init__(self, sim_cfg: "FlowCfg") -> None:
         # A list of jobs on which this job depends.
         self.dependencies = []
 
-        # Indicates whether running this job requires all dependencies to pass.
-        # If this flag is set to False, any passing dependency will trigger
-        # this current job to run
-        self.needs_all_dependencies_passing = True
+        # What the jobs this one depends on must have concluded before it may run. The default
+        # suits anything consuming a dependency's output, which is most jobs
+        self.dependency_policy = DependencyPolicy.ALL_PASSING
 
         # These variables will be extracted from the hjson file by _set_attrs,
         # and then _check_attrs checks that they were indeed extracted. Define
@@ -175,7 +174,7 @@ def get_job_spec(self) -> "JobSpec":
             ),
             workspace_cfg=self.sim_cfg.workspace_cfg,
             dependencies=[d.full_name for d in self.dependencies],
-            needs_all_dependencies_passing=self.needs_all_dependencies_passing,
+            dependency_policy=self.dependency_policy,
             weight=self.weight,
             timeout_mins=(None if self.gui else self.get_timeout_mins()),
             cmd=self.cmd,
@@ -911,8 +910,8 @@ def __init__(self, run_items: Iterable[RunTest], sim_cfg: FlowCfg) -> None:
 
         super().__init__(sim_cfg)
         self.dependencies.extend(run_items)
-        # Run coverage merge even if one test passes.
-        self.needs_all_dependencies_passing = False
+        # Merge whatever coverage exists, so one passing test is enough to be worth merging.
+        self.dependency_policy = DependencyPolicy.ANY_PASSING
 
         # Append cov_db_dirs to the list of exports.
         self.merged_exports["cov_db_dirs"] = shlex.quote(" ".join(self.cov_db_dirs))
@@ -1074,8 +1073,9 @@ def __init__(self, dependencies: "Iterable[Deploy]", sim_cfg: "SimCfg") -> None:
         super().__init__(sim_cfg)
         # Every run it scores has to be terminal first, so the collector's evidence is complete
         self.dependencies.extend(dependencies)
-        # A failed or killed run is still evidence, so score what happened rather than skipping
-        self.needs_all_dependencies_passing = False
+        # A failed or killed run is still evidence, and a regression where nothing passed is the
+        # case the plan most needs to describe, so this is scored whatever the dependencies did
+        self.dependency_policy = DependencyPolicy.ALWAYS
 
     def _define_attrs(self) -> None:
         super()._define_attrs()

src/dvsim/report/dv_evidence.py

@@ -4,8 +4,12 @@
 
 """Regression results in the tool-neutral `lowrisc-dv-evidence` format.
 
-dvplan defines the format, so a vPlan can be back-annotated from any regression flow and a person
-can write one by hand. What dvsim writes here is a plain serialisation of what it already knows.
+These models are the format's definition, and `doc/dv_evidence.md` describes them. It lives here
+because dvsim is what produces the file, so anything reading one can be written against a public
+spec rather than against whichever consumer happened to be built first.
+
+The format carries no planning-tool concepts, so a verification plan can be scored from any
+regression flow that emits it, and a person can write one by hand.
 
 Built from what the scheduler concludes about each job, through its completion hook. That is the
 same state the JSON report is derived from, so the two cannot disagree about a run, and it is
@@ -50,8 +54,8 @@
 class Outcome(Enum):
     """How one run of a test ended, in the neutral format's vocabulary.
 
-    There is no waived outcome: dvplan requires an owner and a date on a waiver, and a regression
-    can supply neither. A known failure is accepted there by recording an inspection instead.
+    There is no waived outcome. A waiver needs an owner and a date, and a regression can supply
+    neither, so the format only allows one on an inspection, which is written by hand.
     """
 
     PASSED = "passed"
@@ -96,8 +100,8 @@ class TestRun(BaseModel):
 class EvidenceFile(BaseModel):
     """A regression's results, in the tool-neutral evidence format.
 
-    dvsim only ever fills the `testcase` half. The format also carries manual inspections, which a
-    person writes by hand.
+    dvsim only ever fills the `testcase` half. The format also has an `inspection` key, for claims
+    no simulation can measure, and those records are written by hand.
     """
 
     model_config = ConfigDict(frozen=True, extra="forbid", populate_by_name=True)

src/dvsim/report/vplan.py

@@ -12,7 +12,6 @@
 
 import glob
 import shlex
-import shutil
 from collections.abc import Sequence
 from dataclasses import dataclass, field
 from pathlib import Path
@@ -21,7 +20,7 @@
 
 from dvsim.logging import log
 
-__all__ = ("VPlanInputs", "overall_coverage", "shell_command")
+__all__ = ("SKIP_WITHOUT_DVPLAN", "VPlanInputs", "overall_coverage", "shell_command")
 
 # Scratch subdirectory the annotated plan and its report are written to. Unchanged, so an existing
 # link to the report still resolves
@@ -31,6 +30,14 @@
 ANNOTATED_HTML = "vplan_annotated.html"
 EVIDENCE_JSON = "dv_evidence.json"
 
+# Whether dvplan is installed is decided by the script, on the machine the job lands on, rather
+# than by dvsim on whichever host the run was launched from. Exits 0 so that a checkout without
+# dvplan does not fail every regression that names a vPlan
+SKIP_WITHOUT_DVPLAN = (
+    "if ! command -v dvplan >/dev/null 2>&1; then "
+    "echo 'WARNING: dvplan is not installed on PATH. Skipping vPlan annotation.'; exit 0; fi;"
+)
+
 
 @dataclass(frozen=True)
 class VPlanInputs:
@@ -74,12 +81,10 @@ def shell_command(inputs: VPlanInputs) -> str:
 
     Returned as one `bash -c` string because a scheduled job runs a shell command. `set -e` and the
     `&&` mean a broken annotation shows as a failed job rather than a silently missing score.
-    """
-    if shutil.which("dvplan") is None:
-        # Warn and pass, so a checkout without dvplan does not fail every regression naming a vPlan
-        warning = "WARNING: dvplan is not installed on PATH. Skipping vPlan annotation."
-        return f"/usr/bin/env bash -c {shlex.quote(f'echo {shlex.quote(warning)}')}"
 
+    The command is the same whether or not dvplan is installed here, because here is not where it
+    runs. See `SKIP_WITHOUT_DVPLAN`.
+    """
     # The vPlan sits at <ip_root>/<something>/<vplan>, so its grandparent is the IP root that
     # `prepare_vplan` traces specifications against.
     ip_root = inputs.vplan.parent.parent
@@ -94,7 +99,8 @@ def shell_command(inputs: VPlanInputs) -> str:
     process = _process_command(inputs)
 
     script = (
-        f"set -e; mkdir -p {shlex.quote(str(inputs.out_dir))}; "
+        f"set -e; {SKIP_WITHOUT_DVPLAN} "
+        f"mkdir -p {shlex.quote(str(inputs.out_dir))}; "
         f"{shlex.join(prepare)} && {shlex.join(process)}"
     )
     return f"/usr/bin/env bash -c {shlex.quote(script)}"
@@ -146,11 +152,17 @@ def _expand(pattern: str) -> list[str]:
 
     A cfg naming inspections through `{proj_root}` always produces an absolute pattern, which
     `Path.glob` refuses, so this is one of the places the pathlib rule does not apply.
+
+    A pattern matching nothing raises, because both other answers are worse: passing it through
+    fails the job with dvplan's own message once the regression has already run, and dropping it
+    scores the plan as though the cfg had never named inspections at all. The command is built
+    while the jobs are, so this lands before a single test starts.
     """
     matches = sorted(glob.glob(pattern))  # noqa: PTH207 (Path.glob rejects an absolute pattern)
     if not matches:
-        log.warning("No inspection records matched '%s', so none were annotated from.", pattern)
-    return matches or [pattern]
+        msg = f"No inspection records matched 'dvplan_inspect' pattern '{pattern}'."
+        raise ValueError(msg)
+    return matches
 
 
 def overall_coverage(annotated: Path) -> float | None:

src/dvsim/scheduler/core.py

@@ -13,7 +13,7 @@
 from types import FrameType
 from typing import Any, TypeAlias
 
-from dvsim.job.data import CompletedJobStatus, JobSpec, JobStatusInfo
+from dvsim.job.data import CompletedJobStatus, DependencyPolicy, JobSpec, JobStatusInfo
 from dvsim.job.status import JobStatus
 from dvsim.logging import log
 from dvsim.runtime.backend import RuntimeBackend
@@ -365,7 +365,10 @@ def _update_completed_job_deps(self, job: JobRecord) -> None:
 
             # Handle dependency management and marking dependents as ready
             if dep.remaining_deps == 0 and dep.status == JobStatus.SCHEDULED:
-                if dep.spec.needs_all_dependencies_passing:
+                policy = dep.spec.dependency_policy
+                if policy is DependencyPolicy.ALWAYS:
+                    self._mark_job_ready(dep)
+                elif policy is DependencyPolicy.ALL_PASSING:
                     if dep.passing_deps == len(dep.spec.dependencies):
                         self._mark_job_ready(dep)
                     else:

src/dvsim/sim/flow.py

@@ -884,10 +884,13 @@ def make_test_result(tr) -> TestResult | None:
             cov_report_dir = self.cov_report_dir or "cov_report"
             cov_report_page = Path(cov_report_dir, self.cov_report_page)
 
+        # Linked only once the page is actually there. The job can be killed, and it exits without
+        # annotating anything where dvplan is not installed, so its output directory is not proof
         vplan_report_page = None
         vplan_coverage = None
         if self.cov_vplan_deploy is not None:
-            vplan_report_page = self.cov_vplan_deploy.report_page
+            page = self.cov_vplan_deploy.report_page
+            vplan_report_page = page if page.is_file() else None
             vplan_coverage = self.cov_vplan_deploy.vplan_coverage
 
         failures = BucketedFailures.from_job_status(results=run_results)