Version the report payload with bmf_version

A report payload can now say what format it is written in. `bmf_version` is a
top level key of the payload, next to `results`, and it accepts exactly 0 and 1.
An absent key is version 0, so every payload that ingests today ingests
unchanged. Any other value is refused with a message that names the accepted
versions.

At version 1 the `json` adapter node tries its `json_v1` leaf first and falls
back to `json_v0`; at version 0 it tries them in today's order. That reorders
the attempts rather than filtering the payload, so version 1 refuses no v0 shape
and version 0 still ingests a v1 shape through the second attempt. `magic`
inherits the preference through the node, an explicitly named leaf is unaffected
because it is already an exact statement, and no non-JSON adapter reads the key.

An empty payload is now foldable at any version. It is the one payload both
leaves claim, so at version 1 it parses as v1, and since fold is all or nothing
across the array it would otherwise disable fold for every v0 iteration beside
it, changing the number of metric rows a report writes. The fold refusal exists
because a pooled statistic cannot be recomputed from per iteration values, and a
payload that reported nothing has none, so it has nothing to refuse over.

`BmfVersion` is a validated newtype in `bencher_valid` that replaces the enum of
the same name in `bencher_adapter`. The payload's declared version and the
version a leaf parsed are two values of one type, and they have to be able to
differ: a payload that declares version 1 may still hold v0 results.

`/v0/run` carries the key because `JsonNewRun` gains the field and the
conversion into the report payload forwards it. A job based run does not: its
results are the runner's own output rather than the submitted payload.

The v1 thresholds shape and the project gate are later layers, and the CLI gains
no flag here.

Author: epompeii

lib/api_projects/tests/bmf_version.rs

@@ -131,6 +131,103 @@ async fn run_post_unauthenticated() {
 
 // --- Job creation integration tests (Plus only) ---
 
+/// POST /v0/run - the run payload carries `bmf_version` through to the adapter.
+///
+/// `/v0/run` takes its own payload and converts it into a report payload, so the
+/// key reaches ingest only because `JsonNewRun` carries it and the conversion
+/// forwards it. A BMF v1 payload declared as version 1 ingests through the default
+/// Magic adapter, which reaches the JSON leaves through the `json` node.
+#[tokio::test]
+async fn run_post_bmf_version_1() {
+    let server = TestServer::new().await;
+    let user = server.signup("Test User", "runbmf@example.com").await;
+    let org = server.create_org(&user, "Run Bmf Org").await;
+    let project = server.create_project(&user, &org, "Run Bmf Project").await;
+
+    let project_slug: &str = project.slug.as_ref();
+    let body = serde_json::json!({
+        "project": project_slug,
+        "branch": "main",
+        "testbed": "localhost",
+        "start_time": "2024-01-01T00:00:00Z",
+        "end_time": "2024-01-01T00:01:00Z",
+        "results": [bmf_v1_results().to_string()],
+        "bmf_version": 1,
+    });
+
+    let resp = server
+        .client
+        .post(server.api_url("/v0/run"))
+        .header(
+            bencher_json::AUTHORIZATION,
+            bencher_json::bearer_header(&user.token),
+        )
+        .json(&body)
+        .send()
+        .await
+        .expect("Request failed");
+
+    assert_eq!(resp.status(), StatusCode::CREATED);
+    let report: JsonReport = resp.json().await.expect("Failed to parse response");
+    // The grid point is the proof the v1 shape was read: a v0 payload has no
+    // parameters to report.
+    let results = report.results.expect("Report results");
+    let iteration = results.first().expect("Report iteration");
+    let result = iteration.first().expect("Report result");
+    assert!(!result.parameter.set.is_empty());
+}
+
+/// POST /v0/run - an unknown `bmf_version` is rejected before anything is created.
+#[tokio::test]
+async fn run_post_unknown_bmf_version() {
+    let server = TestServer::new().await;
+    let user = server.signup("Test User", "runbmfbad@example.com").await;
+    let org = server.create_org(&user, "Run Bad Bmf Org").await;
+    let project = server
+        .create_project(&user, &org, "Run Bad Bmf Project")
+        .await;
+
+    let project_slug: &str = project.slug.as_ref();
+    let body = serde_json::json!({
+        "project": project_slug,
+        "branch": "main",
+        "testbed": "localhost",
+        "start_time": "2024-01-01T00:00:00Z",
+        "end_time": "2024-01-01T00:01:00Z",
+        "results": [bmf_results().to_string()],
+        "bmf_version": 2,
+    });
+
+    let resp = server
+        .client
+        .post(server.api_url("/v0/run"))
+        .header(
+            bencher_json::AUTHORIZATION,
+            bencher_json::bearer_header(&user.token),
+        )
+        .json(&body)
+        .send()
+        .await
+        .expect("Request failed");
+
+    assert_eq!(resp.status(), StatusCode::BAD_REQUEST);
+    let body = resp.text().await.expect("Failed to read the response");
+    assert!(
+        body.contains("0 or 1"),
+        "expected the rejection to name the accepted versions: {body}"
+    );
+}
+
+/// A BMF v1 payload: a benchmark maps to an array of grid points.
+fn bmf_v1_results() -> serde_json::Value {
+    serde_json::json!({
+        "benchmark_name": [{
+            "parameters": { "size_mb": 16 },
+            "measures": { "latency": { "value": 100.0 } },
+        }]
+    })
+}
+
 fn bmf_results() -> serde_json::Value {
     serde_json::json!({
         "benchmark_name": {

lib/api_run/tests/run.rs

@@ -172,6 +172,7 @@ impl DotNet {
 
 #[cfg(test)]
 pub(crate) mod test_c_sharp_dot_net {
+    use bencher_json::BmfVersion;
     use bencher_json::project::report::JsonAverage;
     use ordered_float::OrderedFloat;
     use pretty_assertions::assert_eq;
@@ -209,9 +210,7 @@ pub(crate) mod test_c_sharp_dot_net {
 
         let results = opt_convert_file_path::<AdapterCSharpDotNet>(
             &file_path,
-            Settings {
-                average: Some(JsonAverage::Mean),
-            },
+            Settings::new(Some(JsonAverage::Mean), BmfVersion::default()),
         )
         .unwrap();
 
@@ -233,9 +232,7 @@ pub(crate) mod test_c_sharp_dot_net {
 
         let results = opt_convert_file_path::<AdapterCSharpDotNet>(
             &file_path,
-            Settings {
-                average: Some(JsonAverage::Mean),
-            },
+            Settings::new(Some(JsonAverage::Mean), BmfVersion::default()),
         )
         .unwrap();
         validate_adapter_c_sharp_dot_net(&results);

lib/bencher_adapter/src/adapters/c_sharp/dot_net.rs

@@ -196,6 +196,7 @@ fn parse_catch2_duration(input: &str) -> IResult<&str, OrderedFloat<f64>> {
 
 #[cfg(test)]
 pub(crate) mod test_cpp_catch2 {
+    use bencher_json::BmfVersion;
     use bencher_json::project::report::JsonAverage;
     use pretty_assertions::assert_eq;
 
@@ -247,9 +248,7 @@ pub(crate) mod test_cpp_catch2 {
         let file_path = "./tool_output/cpp/catch2/four.txt";
         let results = opt_convert_file_path::<AdapterCppCatch2>(
             file_path,
-            Settings {
-                average: Some(JsonAverage::Mean),
-            },
+            Settings::new(Some(JsonAverage::Mean), BmfVersion::default()),
         )
         .unwrap();
         validate_adapter_cpp_catch2(&results);
@@ -258,9 +257,7 @@ pub(crate) mod test_cpp_catch2 {
             None,
             opt_convert_file_path::<AdapterCppCatch2>(
                 file_path,
-                Settings {
-                    average: Some(JsonAverage::Median)
-                }
+                Settings::new(Some(JsonAverage::Median), BmfVersion::default())
             )
         );
     }

lib/bencher_adapter/src/adapters/cpp/catch2.rs

@@ -70,6 +70,7 @@ impl TryFrom<Google> for Option<AdapterResults> {
 
 #[cfg(test)]
 pub(crate) mod test_cpp_google {
+    use bencher_json::BmfVersion;
     use bencher_json::project::report::JsonAverage;
     use pretty_assertions::assert_eq;
 
@@ -90,9 +91,7 @@ pub(crate) mod test_cpp_google {
         let file_path = "./tool_output/cpp/google/two.txt";
         let results = opt_convert_file_path::<AdapterCppGoogle>(
             file_path,
-            Settings {
-                average: Some(JsonAverage::Mean),
-            },
+            Settings::new(Some(JsonAverage::Mean), BmfVersion::default()),
         )
         .unwrap();
         validate_adapter_cpp_google(&results);
@@ -101,9 +100,7 @@ pub(crate) mod test_cpp_google {
             None,
             opt_convert_file_path::<AdapterCppGoogle>(
                 file_path,
-                Settings {
-                    average: Some(JsonAverage::Median)
-                }
+                Settings::new(Some(JsonAverage::Median), BmfVersion::default())
             )
         );
     }

lib/bencher_adapter/src/adapters/cpp/google.rs

@@ -70,6 +70,7 @@ fn parse_dart_line(input: &str) -> IResult<&str, (BenchmarkName, JsonNewMetric)>
 
 #[cfg(test)]
 pub(crate) mod test_dart_benchmark_harness {
+    use bencher_json::BmfVersion;
     use bencher_json::project::report::JsonAverage;
     use pretty_assertions::assert_eq;
 
@@ -92,9 +93,7 @@ pub(crate) mod test_dart_benchmark_harness {
             None,
             opt_convert_file_path::<AdapterDartBenchmarkHarness>(
                 file_path,
-                Settings {
-                    average: Some(JsonAverage::Median)
-                }
+                Settings::new(Some(JsonAverage::Median), BmfVersion::default())
             )
         );
     }

lib/bencher_adapter/src/adapters/dart/benchmark_harness.rs

@@ -77,6 +77,7 @@ fn parse_go_bench(input: &str) -> IResult<&str, JsonNewMetric> {
 
 #[cfg(test)]
 pub(crate) mod test_go_bench {
+    use bencher_json::BmfVersion;
     use bencher_json::{JsonNewMetric, project::report::JsonAverage};
     use pretty_assertions::assert_eq;
 
@@ -192,9 +193,7 @@ pub(crate) mod test_go_bench {
         let file_path = "./tool_output/go/bench/five.txt";
         let results = opt_convert_file_path::<AdapterGoBench>(
             file_path,
-            Settings {
-                average: Some(JsonAverage::Mean),
-            },
+            Settings::new(Some(JsonAverage::Mean), BmfVersion::default()),
         )
         .unwrap();
         validate_adapter_go_bench(&results);
@@ -203,9 +202,7 @@ pub(crate) mod test_go_bench {
             None,
             opt_convert_file_path::<AdapterGoBench>(
                 file_path,
-                Settings {
-                    average: Some(JsonAverage::Median)
-                }
+                Settings::new(Some(JsonAverage::Median), BmfVersion::default())
             )
         );
     }

lib/bencher_adapter/src/adapters/go/bench.rs

@@ -109,6 +109,7 @@ impl TryFrom<Jmh> for Option<AdapterResults> {
 
 #[cfg(test)]
 pub(crate) mod test_java_jmh {
+    use bencher_json::BmfVersion;
     use bencher_json::project::report::JsonAverage;
     use pretty_assertions::assert_eq;
 
@@ -163,9 +164,7 @@ pub(crate) mod test_java_jmh {
         let file_path = "./tool_output/java/jmh/six.json";
         let results = opt_convert_file_path::<AdapterJavaJmh>(
             file_path,
-            Settings {
-                average: Some(JsonAverage::Mean),
-            },
+            Settings::new(Some(JsonAverage::Mean), BmfVersion::default()),
         )
         .unwrap();
         validate_adapter_java_jmh(&results);
@@ -174,9 +173,7 @@ pub(crate) mod test_java_jmh {
             None,
             opt_convert_file_path::<AdapterJavaJmh>(
                 file_path,
-                Settings {
-                    average: Some(JsonAverage::Median)
-                }
+                Settings::new(Some(JsonAverage::Median), BmfVersion::default())
             )
         );
     }

lib/bencher_adapter/src/adapters/java/jmh.rs

@@ -88,6 +88,7 @@ fn parse_benchmark_time(input: &str) -> IResult<&str, JsonNewMetric> {
 
 #[cfg(test)]
 pub(crate) mod test_js_benchmark {
+    use bencher_json::BmfVersion;
     use bencher_json::project::report::JsonAverage;
     use pretty_assertions::assert_eq;
 
@@ -110,17 +111,13 @@ pub(crate) mod test_js_benchmark {
             None,
             opt_convert_file_path::<AdapterJsBenchmark>(
                 file_path,
-                Settings {
-                    average: Some(JsonAverage::Mean)
-                }
+                Settings::new(Some(JsonAverage::Mean), BmfVersion::default())
             )
         );
 
         let results = opt_convert_file_path::<AdapterJsBenchmark>(
             file_path,
-            Settings {
-                average: Some(JsonAverage::Median),
-            },
+            Settings::new(Some(JsonAverage::Median), BmfVersion::default()),
         )
         .unwrap();
         validate_adapter_js_benchmark(&results);

lib/bencher_adapter/src/adapters/js/benchmark.rs

@@ -82,6 +82,7 @@ fn parse_time_time(input: &str) -> IResult<&str, JsonNewMetric> {
 
 #[cfg(test)]
 pub(crate) mod test_js_time {
+    use bencher_json::BmfVersion;
     use bencher_json::project::report::JsonAverage;
     use pretty_assertions::assert_eq;
 
@@ -104,19 +105,15 @@ pub(crate) mod test_js_time {
             None,
             opt_convert_file_path::<AdapterJsTime>(
                 file_path,
-                Settings {
-                    average: Some(JsonAverage::Mean)
-                }
+                Settings::new(Some(JsonAverage::Mean), BmfVersion::default())
             )
         );
 
         assert_eq!(
             None,
             opt_convert_file_path::<AdapterJsTime>(
                 file_path,
-                Settings {
-                    average: Some(JsonAverage::Median)
-                }
+                Settings::new(Some(JsonAverage::Median), BmfVersion::default())
             )
         );
     }