Declare gated thresholds in the v1 report payload - #1019
Draft
epompeii wants to merge 1 commit into
Draft
Conversation
Contributor
|
| Project | Bencher |
| Branch | u/ep/parameters-api/thresholds-payload |
| Testbed | intel-v1 |
Click to view all benchmark results
| Benchmark | Latency | Benchmark Result microseconds (µs) (Result Δ%) | Upper Boundary microseconds (µs) (Limit %) |
|---|---|---|---|
| Adapter::Json | 📈 view plot 🚷 view threshold | 5.27 µs(+10.41%)Baseline: 4.77 µs | 5.64 µs (93.38%) |
| Adapter::Magic (JSON) | 📈 view plot 🚷 view threshold | 5.07 µs(+9.47%)Baseline: 4.63 µs | 5.42 µs (93.56%) |
| Adapter::Magic (Rust) | 📈 view plot 🚷 view threshold | 28.07 µs(+7.36%)Baseline: 26.15 µs | 28.68 µs (97.89%) |
| Adapter::Rust | 📈 view plot 🚷 view threshold | 4.67 µs(+25.51%)Baseline: 3.72 µs | 5.47 µs (85.46%) |
| Adapter::RustBench | 📈 view plot 🚷 view threshold | 4.65 µs(+25.09%)Baseline: 3.72 µs | 5.45 µs (85.42%) |
epompeii
force-pushed
the
u/ep/parameters-api/thresholds-payload
branch
2 times, most recently
from
August 26, 2026 23:05
4a5328c to
2ad64f8
Compare
epompeii
force-pushed
the
u/ep/parameters-api/thresholds-payload
branch
from
August 27, 2026 03:46
2ad64f8 to
75903f9
Compare
A report has always carried its thresholds as a map of measure to model, and a
map key names a measure and nothing else. Every threshold a report could declare
was therefore the bare one: the conventional `value` name of every grid point. A
pipeline that wanted `p99` watched, or wanted one corner of its grid watched, had
to reach for the thresholds endpoint and then keep it in step with the run by
hand.
At BMF version 1 `thresholds.models` is a list. An entry is `parameters`,
`measure`, `metric`, and `model`: the dimensions a threshold hangs off that the
report does not already state, in their canonical order, and the model to gate
with. `measure` is the same name, slug, or UUID the map key is today, created if
the project has never seen it. An absent `metric` is the conventional `value`
name and an absent `parameters` gates every grid point, so an entry naming only a
measure and a model is the map pair written out longhand. One measure may carry
several entries, and each one creates or updates the threshold with that identity
under the report's branch and testbed, through the same null collapse and
canonical filter storage every other writer goes through.
A threshold the report declares gates the very report that declared it. Nothing
moved to make that true: thresholds are resolved before results are parsed, which
is where they already were. It is worth saying out loud because it is what makes
the list worth having, and it is pinned by a test whose history is five ungated
reports and whose alerts all belong to thresholds that did not exist when the
request arrived.
The declared version says which shape to expect and the shape is checked rather
than guessed at. A list at version 0 and a map at version 1 are both a 400 naming
the version that was declared and the shape it calls for. A payload that declares
a version has stated something exact and gets exact validation, and the project's
version gate still fires first, so a project that does not accept version 1 turns
the payload away for that reason and never reads the list at all.
`reset` reaches exactly as far as the shape can address. A version 0 map can only
name bare thresholds, so it takes a model away from bare thresholds and nothing
else, which is what it already did. A legacy run cannot strip a named or filtered
threshold it has no way to spell, and the pipelines sending `reset` today are
exactly the ones that cannot spell them. A version 1 list can name every identity,
so `reset` reaches every threshold on the branch and testbed that the entries did
not name, including a payload that names none at all.
One identity declared twice in one payload is one threshold: the position is
where it was first written and the model is what it was last told, resolved
without an error. Two spellings of one filter are one identity, because a filter
canonicalizes before it is compared.
Two shapes behind one key is a place where error quality quietly dies. The obvious
spelling, an untagged enum, buffers the input, tries each variant, and on failure
says only that nothing matched, so a misspelled model test in a version 0 map would
come back as "data did not match any variant" rather than as the field and the
variants it could have been. The shape is known from the first token, so it is
decided by looking rather than by trying: every error a version 0 client got before
this layer it gets after it, byte for byte, and a malformed version 1 entry is named
by its position and its field. The one message that moves is the one that has to,
where `models` is neither shape and what it expects now names both.
Deleting a parameter set learns about thresholds. A threshold whose filter names
a set is a reference to that row, so the set cannot go out from under it, and the
refusal says which threshold to delete first. Naming is canonical equality and
nothing looser: a filter of `{"a":1}` matches the grid point `{"a":1,"b":2}`
without naming it, because a filter names only the keys it cares about and is a
predicate over values rather than a pointer at a row. That set can be deleted and
the filter still says what it said. The comparison runs in Rust over the
project's filtered thresholds, because canonical equality is what the canonical
form defines and that form is written in Rust. Nothing caps that read: it is small
because a filtered threshold is a rare thing to write and deleting a parameter set
is a rare thing to ask for, not because a limit says so. The delete and the check
share one transaction with the delete first, so a set that a report still
references is refused for that reason and the client is sent to the results rather
than to a threshold it would have to delete anyway.
The CLI is unchanged: it declares bare thresholds, which is what the version 0
map spells, and naming a metric or a grid point from the command line is a
separate piece of work.
epompeii
force-pushed
the
u/ep/parameters-api/thresholds-payload
branch
from
August 27, 2026 05:11
75903f9 to
296d96c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The list shape
A report has always carried its thresholds as a map of measure to model, and a map key names a measure and nothing else. Every threshold a report could declare was therefore the bare one: the conventional
valuename of every grid point.At BMF version 1
thresholds.modelsis a list:{ "bmf_version": 1, "thresholds": { "models": [ { "measure": "latency", "metric": "p99", "model": { "test": "t_test", "upper_boundary": 0.98 } }, { "parameters": [{ "size_mb": 16 }], "measure": "latency", "model": { "test": "percentage", "upper_boundary": 0.25 } } ] } }An entry is
parameters,measure,metric,model: the dimensions a threshold hangs off that the report does not already state, in their canonical order, and the model to gate with.measureis the same name, slug, or UUID the map key is today, created if the project has never seen it. An absentmetricis the conventionalvaluename and an absentparametersgates every grid point, so an entry naming only a measure and a model is the map pair written out longhand. One measure may carry several entries, and each creates or updates the threshold with that identity under the report's branch and testbed, through the same null collapse and canonical filter storage every other writer goes through.A threshold the report declares gates the very report that declared it. Nothing moved to make that true: thresholds are resolved before results are parsed, which is where they already were. It is pinned by a test whose history is five ungated reports and whose alerts all belong to thresholds that did not exist when the request arrived.
The declared version says which shape to expect and the shape is checked rather than guessed at: a list at version 0 and a map at version 1 are both a 400 naming the version declared and the shape it calls for, an empty map at version 1 included. The project's version gate still fires first, so a project that does not accept version 1 turns the payload away for that reason and never reads the list.
One identity declared twice in one payload is one threshold: the position is where it was first written and the model is what it was last told, resolved without an error. Two spellings of one filter are one identity, because a filter canonicalizes before it is compared.
Reset scoping (adjudicated)
resetreaches exactly as far as the payload's shape can address.resettakes a model away from bare thresholds and nothing else, which is what it already did. A named or filtered threshold is not expressible in that shape, and the pipelines sendingresettoday are exactly the ones that cannot spell it. A legacy run must not strip a threshold it has no way to name.resetreaches every threshold on the branch and testbed that the entries did not name, including a payload that names none at all. A shape that can address everything is a shape whose silence means something.Both scopes are pinned: a version 0 reset leaves the named and filtered thresholds standing, a version 1 reset with entries strips the unnamed filtered threshold and keeps the named one, and a version 1 reset with no entries strips them all.
resetstays on the report's own branch and testbed either way.The parameter delete learns about thresholds
DELETE /v0/projects/{project}/benchmarks/{benchmark}/parameters/{parameter}gains a second refusal. A threshold whoseparametersfilter names a set is a reference to that row, so the set cannot go out from under it, and the refusal says which threshold to delete first.Naming is canonical equality and nothing looser. A filter of
{"a":1}matches the grid point{"a":1,"b":2}without naming it: a filter names only the keys it cares about, which makes it a predicate over values rather than a pointer at a row. Delete that grid point and the filter still says exactly what it said, so the delete goes through. Only a filter carrying a set canonically equal to this one blocks.The comparison runs in Rust over the project's non-null filters, because canonical equality is what the canonical form defines and that form is written in Rust. Nothing caps that read: it is small because a filtered threshold is a rare thing to write, and a project that accumulates very many of them is what would move this comparison into SQL. The delete and the check share one transaction with the delete first, so a set that a report still references is refused for that reason: the foreign key fires on the delete itself and the client is sent to the results rather than to a threshold it would have to delete anyway.
Adjudicated calls worth a second opinion
modelrather than flattening it.JsonNewThresholdflattens itsModel, which is the sibling precedent. The entry is the map pair unrolled into fields, and the map's value is the model, somodelis a key here. It also keeps the entry legible besidemetricandparameters.serde(untagged). An untagged enum buffers the input and, on failure, discards every inner error for one opaque sentence. The two shapes are distinguished by the first JSON token, so the visitor dispatches a map to the map's own deserializer and a list to the list's, and their field-level errors survive verbatim, path prefixes included. One message changes deliberately: amodelsvalue that is neither shape is now told the field expects a map of measure to threshold model (BMF version 0) or a list of threshold entries (BMF version 1), because that is what the field accepts; the old text named only the map. A test pins the new sentence so the change stays visible.oneOf, not a derivedanyOf. A derived untagged enum emitsanyOf, and the client generator turns ananyOfof two subschemas into one struct of flattened optional members. A list cannot be flattened into a struct.oneOfis also what the shapes are: mutually exclusive. The generated client gets a clean untagged enum withMapandListvariants, named from the schema titles.models: models.map(JsonReportThresholdModels::Map)) and nothing else. The CLI declares bare thresholds, which is what the version 0 map spells. Naming a metric or a grid point from the command line is deferred to its own piece of work.Byte compatibility
Version 0 compatibility is capture-proven, not argued: the map ingest, map update, and map-with-reset responses, and the malformed-map 400 bodies with their field paths, were captured on the base branch and on this head and diff empty. The one deliberate exception is the neither-shape message described above, pinned by its own test. Three new tests pin the 400 bodies (a malformed map field, a malformed entry field with its index, and the neither-shape sentence) so the diagnostics cannot regress silently.
Gates
cargo fmt -- --check,./scripts/clippy.sh(both legs),cargo nextest run --all-features --profile ci(2,255 passed), doc tests, andcargo gen-typeswith clean porcelain. No migration in this layer.