Label the grid in comments, the CLI table, and the BMF v1 reference - #1022
Draft
epompeii wants to merge 1 commit into
Draft
Label the grid in comments, the CLI table, and the BMF v1 reference#1022epompeii wants to merge 1 commit into
epompeii wants to merge 1 commit into
Conversation
Contributor
|
| Project | Bencher |
| Branch | u/ep/parameters-api/tail |
| Testbed | intel-v1 |
Click to view all benchmark results
| Benchmark | Latency | Benchmark Result microseconds (µs) (Result Δ%) | Upper Boundary microseconds (µs) (Limit %) |
|---|---|---|---|
| Adapter::Json | 📈 view plot 🚷 view threshold | 5.27 µs(+9.86%)Baseline: 4.79 µs | 5.73 µs (91.96%) |
| Adapter::Magic (JSON) | 📈 view plot 🚷 view threshold | 5.07 µs(+9.00%)Baseline: 4.65 µs | 5.49 µs (92.27%) |
| Adapter::Magic (Rust) | 📈 view plot 🚷 view threshold | 27.90 µs(+6.42%)Baseline: 26.22 µs | 29.04 µs (96.08%) |
| Adapter::Rust | 📈 view plot 🚷 view threshold | 4.65 µs(+23.72%)Baseline: 3.76 µs | 5.62 µs (82.86%) |
| Adapter::RustBench | 📈 view plot 🚷 view threshold | 4.65 µs(+23.84%)Baseline: 3.76 µs | 5.60 µs (83.13%) |
epompeii
force-pushed
the
u/ep/parameters-api/tail
branch
from
August 26, 2026 23:56
185d60f to
d370875
Compare
epompeii
force-pushed
the
u/ep/parameters-api/tail
branch
from
August 27, 2026 03:46
d370875 to
dda7dcf
Compare
epompeii
force-pushed
the
u/ep/parameters-api/tail
branch
from
August 27, 2026 04:22
dda7dcf to
54555d1
Compare
A result row of a pull request comment names its parameter set by the rule the perf
image already uses. A benchmark whose every row in the comment carries the empty
parameter set keeps the bare benchmark name it has always had. One non-empty set
among its rows names them all, the empty set among them included, which reads `{}`.
The set is spelled in its canonical form and follows the benchmark name. Results and
alerts decide together, so the two tables of one comment never disagree about how a
benchmark is named, and a BMF v0 comment renders exactly as it did because there is
no set to spell.
A measure that carries names beyond the conventional trio renders one column per
name, headed `Measure (name)`, after the columns it already had. A measure that
named no point estimate renders its named columns instead of nothing. The per
measure name cap bounds the width. What a threshold computed for a named value rides
inside that name's cell, the way a baseline rides inside the point estimate's, so a
name is one column and never three.
Finding the alert for a cell matches all four of the things an alert fires on: the
benchmark, the grid point, the measure, and the name. Matching on the benchmark and
the measure alone attached one alert to every grid point of that benchmark and
bolded the wrong row. The alert table labels the grid point from the alert's own
parameter set and names the metric its threshold gates when that is not `value`.
The cell's threshold badge, the no threshold warning, and `--ci-only-thresholds` now
ask one predicate, which reads the deprecated bare gate first and falls back to the
boundaries on the metric rows. Every threshold a BMF v0 payload can create is bare,
so the v0 cell is the cell it always was. A measure gated only by a threshold that
names a metric or filters grid points used to render a cell shouting NO THRESHOLD
under a warning that stayed silent about it, which is one comment contradicting
itself. It now links the threshold that gated it.
The CLI perf table gains a Parameters column between the benchmark and the measure,
where the parameter set sits in a perf result. The column is removed entirely unless
some line of the query plots a non-empty set, so a project that never reported a
parameter set prints the table it always printed, column for column, which a test
pins byte for byte. Every cell spells its set canonically. Named value display in
the table is deferred.
The Bencher Metric Format reference gains a v1 section beside the v0 material, in
every locale: the array of entries, `parameters` as JSON scalars with their caps and
their canonical form, named metric values with the conventional trio and the per
measure cap, `bmf_version` on the report and the project gate that has to be raised
for any of it, the v1 threshold list with its optional `metric` and its `parameters`
filter, every matching threshold firing, what `reset` can reach at each version, and
fold being warned about and ingested unfolded.
The warning the format owes its authors gets a section of its own: a v1 payload
written without its array wrapper is a valid v0 payload, so the generic `json`
adapter reads `parameters` and every named value as measures and the intended
results are silently lost. The `json_v1` adapter accepts nothing but v1, so it is
what turns that mistake into a rejected report rather than a quiet downgrade, and
`bmf_version` is what states the version and orders the attempts.
A `v1/bmf.json` schema is published beside `v0/bmf.json`. The unversioned `bmf.json`
is unchanged and stays the default version, v0, so nothing an existing author
validates against moves.
epompeii
force-pushed
the
u/ep/parameters-api/tail
branch
from
August 27, 2026 05:11
54555d1 to
e93a204
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The tail of the benchmark parameters stack: the places a grid point still had no name.
The pull request comment
A result row names its parameter set by the same rule the perf image uses. A
benchmark whose every row in the comment carries the empty parameter set keeps the
bare benchmark name it has always had. One non-empty set among its rows names them
all, the empty set among them included, which reads
{}. The set is spelled in itscanonical form and follows the benchmark name. Results and alerts are counted
together when the decision is made, so the two tables of one comment never disagree
about how a benchmark is named. Every BMF v0 comment renders exactly as it did:
without a grid point there is no set to spell.
A measure that carries named values beyond the conventional trio renders one column
per name, headed
Measure (name), after the measure's existing columns. A measurethat named no point estimate renders its named columns instead of nothing, which is
the one behavior this changes for a payload that has names at all. The per measure
name cap bounds the width. A named column shows what a threshold computed for that
name inside the cell, the way the point estimate column shows its baseline, so a
name is one column and never three.
Finding the alert for a cell now matches four of the things an alert fires on: the
benchmark, the grid point, the measure, and the name. Matching on the benchmark and
the measure alone attached one alert to every grid point of that benchmark and
bolded the wrong row. The iteration stays deliberately unmatched, as it always has
been, because matching it would change what an existing multi-iteration comment
renders. The alert table reads the flat
valueit already read, labels the gridpoint from
alert.parameter, and names the metric fromalert.threshold.metricwhen that is not
value.The cell's threshold badge, the no threshold warning, and
--ci-only-thresholdsaskone predicate, which reads the deprecated bare gate first and falls back to the
boundaries on the metric rows. Every threshold a BMF v0 payload can create is bare,
so the v0 cell is the cell it always was. A measure gated only by a threshold that
names a metric or filters grid points used to render a cell shouting NO THRESHOLD
under a warning that stayed silent about it. It now links the threshold that gated
it.
The CLI perf table
The perf table gains a
Parameterscolumn between the benchmark and the measure,where the parameter set sits in a perf result. The column is removed entirely unless
some line of the query plots a non-empty set, so a project that never reported a
parameter set prints the table it always printed, column for column, which a test
pins byte for byte. Every cell spells its set canonically, the empty set among them
as
{}. Named value display in the table is deferred.The BMF v1 reference
The Bencher Metric Format reference gains a v1 section beside the v0 material, in
all nine locales: the array of entries,
parametersas JSON scalars with theircaps and their canonical form, named metric values with the conventional trio and
the per measure cap,
bmf_versionon the report and the project gate that has to beraised for any of it, the v1
thresholds.modelslist with its optionalmetricandits
parametersfilter, every matching threshold firing, whatresetcan reach ateach version, and fold being warned about and ingested unfolded.
The warning the format owes its authors is called out on its own: a v1 payload
written without its array wrapper is a valid v0 payload, so the generic
jsonadapter reads
parametersand every named value as measures and the intendedresults are silently lost. The
json_v1adapter accepts nothing but v1, so it iswhat turns that mistake into a rejected report rather than a quiet downgrade.
A
v1/bmf.jsonschema artifact is published besidev0/bmf.json. The unversionedbmf.jsonis unchanged and stays the default version, v0, so nothing an existingauthor validates against moves.