Skip to content

Label the grid in comments, the CLI table, and the BMF v1 reference - #1022

Draft
epompeii wants to merge 1 commit into
u/ep/parameters-api/plot-parametersfrom
u/ep/parameters-api/tail
Draft

Label the grid in comments, the CLI table, and the BMF v1 reference#1022
epompeii wants to merge 1 commit into
u/ep/parameters-api/plot-parametersfrom
u/ep/parameters-api/tail

Conversation

@epompeii

@epompeii epompeii commented Aug 26, 2026

Copy link
Copy Markdown
Member

The tail of the benchmark parameters stack: the places a grid point still had no name.

The pull request comment

A result row names its parameter set by the same rule the perf image uses. A
benchmark whose every row in the comment carries the empty parameter set keeps the
bare benchmark name it has always had. One non-empty set among its rows names them
all, the empty set among them included, which reads {}. The set is spelled in its
canonical form and follows the benchmark name. Results and alerts are counted
together when the decision is made, so the two tables of one comment never disagree
about how a benchmark is named. Every BMF v0 comment renders exactly as it did:
without a grid point there is no set to spell.

A measure that carries named values beyond the conventional trio renders one column
per name, headed Measure (name), after the measure's existing columns. A measure
that named no point estimate renders its named columns instead of nothing, which is
the one behavior this changes for a payload that has names at all. The per measure
name cap bounds the width. A named column shows what a threshold computed for that
name inside the cell, the way the point estimate column shows its baseline, so a
name is one column and never three.

Finding the alert for a cell now matches four of the things an alert fires on: the
benchmark, the grid point, the measure, and the name. Matching on the benchmark and
the measure alone attached one alert to every grid point of that benchmark and
bolded the wrong row. The iteration stays deliberately unmatched, as it always has
been, because matching it would change what an existing multi-iteration comment
renders. The alert table reads the flat value it already read, labels the grid
point from alert.parameter, and names the metric from alert.threshold.metric
when that is not value.

The cell's threshold badge, the no threshold warning, and --ci-only-thresholds ask
one predicate, which reads the deprecated bare gate first and falls back to the
boundaries on the metric rows. Every threshold a BMF v0 payload can create is bare,
so the v0 cell is the cell it always was. A measure gated only by a threshold that
names a metric or filters grid points used to render a cell shouting NO THRESHOLD
under a warning that stayed silent about it. It now links the threshold that gated
it.

The CLI perf table

The perf table gains a Parameters column between the benchmark and the measure,
where the parameter set sits in a perf result. The column is removed entirely unless
some line of the query plots a non-empty set, so a project that never reported a
parameter set prints the table it always printed, column for column, which a test
pins byte for byte. Every cell spells its set canonically, the empty set among them
as {}. Named value display in the table is deferred.

The BMF v1 reference

The Bencher Metric Format reference gains a v1 section beside the v0 material, in
all nine locales: the array of entries, parameters as JSON scalars with their
caps and their canonical form, named metric values with the conventional trio and
the per measure cap, bmf_version on the report and the project gate that has to be
raised for any of it, the v1 thresholds.models list with its optional metric and
its parameters filter, every matching threshold firing, what reset can reach at
each version, and fold being warned about and ingested unfolded.

The warning the format owes its authors is called out on its own: a v1 payload
written without its array wrapper is a valid v0 payload, so the generic json
adapter reads parameters and every named value as measures and the intended
results are silently lost. The json_v1 adapter accepts nothing but v1, so it is
what turns that mistake into a rejected report rather than a quiet downgrade.

A v1/bmf.json schema artifact is published beside v0/bmf.json. The unversioned
bmf.json is unchanged and stays the default version, v0, so nothing an existing
author validates against moves.

@github-actions

github-actions Bot commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

🐰 Bencher Report

ProjectBencher
Branchu/ep/parameters-api/tail
Testbedintel-v1
Click to view all benchmark results
BenchmarkLatencyBenchmark Result
microseconds (µs)
(Result Δ%)
Upper Boundary
microseconds (µs)
(Limit %)
Adapter::Json📈 view plot
🚷 view threshold
5.27 µs
(+9.86%)Baseline: 4.79 µs
5.73 µs
(91.96%)
Adapter::Magic (JSON)📈 view plot
🚷 view threshold
5.07 µs
(+9.00%)Baseline: 4.65 µs
5.49 µs
(92.27%)
Adapter::Magic (Rust)📈 view plot
🚷 view threshold
27.90 µs
(+6.42%)Baseline: 26.22 µs
29.04 µs
(96.08%)
Adapter::Rust📈 view plot
🚷 view threshold
4.65 µs
(+23.72%)Baseline: 3.76 µs
5.62 µs
(82.86%)
Adapter::RustBench📈 view plot
🚷 view threshold
4.65 µs
(+23.84%)Baseline: 3.76 µs
5.60 µs
(83.13%)
🐰 View full continuous benchmarking report in Bencher

@epompeii
epompeii force-pushed the u/ep/parameters-api/tail branch from 185d60f to d370875 Compare August 26, 2026 23:56
@epompeii
epompeii force-pushed the u/ep/parameters-api/tail branch from d370875 to dda7dcf Compare August 27, 2026 03:46
@epompeii
epompeii force-pushed the u/ep/parameters-api/tail branch from dda7dcf to 54555d1 Compare August 27, 2026 04:22
A result row of a pull request comment names its parameter set by the rule the perf
image already uses. A benchmark whose every row in the comment carries the empty
parameter set keeps the bare benchmark name it has always had. One non-empty set
among its rows names them all, the empty set among them included, which reads `{}`.
The set is spelled in its canonical form and follows the benchmark name. Results and
alerts decide together, so the two tables of one comment never disagree about how a
benchmark is named, and a BMF v0 comment renders exactly as it did because there is
no set to spell.

A measure that carries names beyond the conventional trio renders one column per
name, headed `Measure (name)`, after the columns it already had. A measure that
named no point estimate renders its named columns instead of nothing. The per
measure name cap bounds the width. What a threshold computed for a named value rides
inside that name's cell, the way a baseline rides inside the point estimate's, so a
name is one column and never three.

Finding the alert for a cell matches all four of the things an alert fires on: the
benchmark, the grid point, the measure, and the name. Matching on the benchmark and
the measure alone attached one alert to every grid point of that benchmark and
bolded the wrong row. The alert table labels the grid point from the alert's own
parameter set and names the metric its threshold gates when that is not `value`.

The cell's threshold badge, the no threshold warning, and `--ci-only-thresholds` now
ask one predicate, which reads the deprecated bare gate first and falls back to the
boundaries on the metric rows. Every threshold a BMF v0 payload can create is bare,
so the v0 cell is the cell it always was. A measure gated only by a threshold that
names a metric or filters grid points used to render a cell shouting NO THRESHOLD
under a warning that stayed silent about it, which is one comment contradicting
itself. It now links the threshold that gated it.

The CLI perf table gains a Parameters column between the benchmark and the measure,
where the parameter set sits in a perf result. The column is removed entirely unless
some line of the query plots a non-empty set, so a project that never reported a
parameter set prints the table it always printed, column for column, which a test
pins byte for byte. Every cell spells its set canonically. Named value display in
the table is deferred.

The Bencher Metric Format reference gains a v1 section beside the v0 material, in
every locale: the array of entries, `parameters` as JSON scalars with their caps and
their canonical form, named metric values with the conventional trio and the per
measure cap, `bmf_version` on the report and the project gate that has to be raised
for any of it, the v1 threshold list with its optional `metric` and its `parameters`
filter, every matching threshold firing, what `reset` can reach at each version, and
fold being warned about and ingested unfolded.

The warning the format owes its authors gets a section of its own: a v1 payload
written without its array wrapper is a valid v0 payload, so the generic `json`
adapter reads `parameters` and every named value as measures and the intended
results are silently lost. The `json_v1` adapter accepts nothing but v1, so it is
what turns that mistake into a rejected report rather than a quiet downgrade, and
`bmf_version` is what states the version and orders the attempts.

A `v1/bmf.json` schema is published beside `v0/bmf.json`. The unversioned `bmf.json`
is unchanged and stays the default version, v0, so nothing an existing author
validates against moves.
@epompeii
epompeii force-pushed the u/ep/parameters-api/tail branch from 54555d1 to e93a204 Compare August 27, 2026 05:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant