Skip to content

feat(compare): add a Chart view of the summary rows - #218

Merged
harenome merged 2 commits into
ThinkFlowLab:mainfrom
harenome:feat/compare-chart
Nov 19, 2026
Merged

harenome merged 2 commits into
ThinkFlowLab:mainfrom
harenome:feat/compare-chart

Conversation

@harenome

@harenome harenome commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #217

What

Compare gets a third view, Chart (?view=chart), beside Side by side and Task by task. It has the same columns, cohort and controls row. The column rows (header, Workspace, Run) stay above the charts, so you change the runs as in the other views.

Cards. Each chart is a card. Its header is one row of toolkit buttons, all --tool-h high, with Remove at its right end:

  • Style, with the chart icon. Its menu lists Bars, Dots and Scatter from the registry of styles, and a check marks the current one.
  • For Bars and Dots, a Data button with the sliders icon and the chosen metric's name. Its menu lists the side table's summary rows, in the table's groups, and a check marks the current one.
  • For Scatter, an X button and a Y button. Each has its letter in the icon slot and its metric's name, and opens the same menu for its axis.
  • Remove, the same × as a column's Remove, with the name "Remove this chart".

Every menu item is a link, and NulliusTable.menu opens the menus, so the change adds no other script. A button never breaks its name over two lines. When a row is too narrow, whole buttons move to a second row, and Remove stays at the end of the first. At 1280 px, a Scatter card's Style, X and Y fit on one row, also with long metric names. A style change keeps the card's first metric. For Scatter, Y becomes the headline rate, or the cost when the rate is X. Add chart, in the actions slot beside Add a run, appends a Bars card of the headline rate.

URL. Each card is one chart= value, in order: chart=bars:adjusted_rate&chart=scatter:cost,adjusted_rate. The metric keys are stable names, not labels. Every link on the page keeps them, with the view and the cohort. Without chart=, the page shows three cards: Bars of the headline rate, Bars of the cost, and a Scatter of the cost against the rate. The headline rate is the adjusted rate, or the mean score for a benchmark that only scores. chart= alone shows one Add chart button. A card whose metric this comparison does not have keeps its controls and says "Not in this comparison."

Styles.

  • Bars: one bar for each run, from zero, on the metric's own axis and unit. A dashed mark in each row shows the baseline's value.
  • Dots: the same layout with dots. The axis covers the values, not zero, so close values stay apart.
  • Scatter: one dot for each run, numbered by its column, with a key under the plot. Runs at the same point sit in a ring around it, each with its own number. A number goes on the first free side of its dot inside the plot.

Values. Each value is the side table's own cell: the number its difference uses, and the text it shows. A missing value has no mark, only its dash. A check is hollow and named "(check)". A check's $0.00 or 0 tokens is a real zero, as in the table, so it has a mark. Nothing is ranked. The tick labels use the row's unit: % for rates, $ for cost, k and M for tokens, seconds or minutes for time. A tick label that would touch the one before it is left out, but its gridline stays.

Colours. Each run has one colour in every chart: its column's place on the page picks --series-1 to --series-8, and the ninth column starts again at the first. The drawing data puts the index on each mark as a class (s1 to s8), and the stylesheet maps each class to its token, so the template has no colour logic. A check stays hollow, with its outline in the run's colour. The baseline mark stays a dashed grey line. The key under a Scatter gives each run a small swatch beside its number and name, so a colour is never the only cue.

The series palette. Eight new tokens, --series-1 to --series-8, in style.css. They are muted blues, two teals, a brown and a plum, in the app's existing lightness range. None of them reads as an outcome colour for a pass (sage), a failure (red) or a warning (amber). The nearest pair is the dark brown and the warning amber, at ΔE 9.5, and every other pair is at 12.5 or more. The trace categories keep their own --cat-* colours, so a colour has one meaning on each page. The dark values serve the default dark theme and the charcoal, graphite and midnight palettes, and they pass on the surface of each.

Slot Light Dark
1 #215da5 #3a75bf
2 #179696 #16a8a8
3 #7d3f17 #ad6a44
4 #657dd1 #748ee3
5 #7e2e59 #aa5d83
6 #1587ba #3aa0d4
7 #5e5197 #7768be
8 #2695ac #38a3ba

The data-viz palette validator checks the colours against the card surface (--surface-2). All checks pass in both modes:

Light (#edf0f4) Dark (#292b2f, #202024, #191f29, #18263c)
Lowest contrast 3.07:1 3.02:1 (default dark); 3.24:1 or more on the others
Adjacent slots, protan and deutan ΔE (target 8) 10.7 or more 10.1 or more
Adjacent slots, normal vision ΔE (floor 15) 15.2 or more 15.2 or more
First 4 slots, every pair, CVD ΔE 10.4 or more 9.5 or more
First 5 slots, every pair, CVD ΔE 9.4 or more 8.1 or more
Lightness band and chroma floor pass pass

ΔE is the OKLab distance ×100, with protanopia and deuteranopia simulated by Machado-Oliveira-Fernandes at full severity. Slots 6 to 8 repeat the blue and teal families, so a scatter of more than five runs also relies on the numbers and the key. Another view can use the palette with the class s1 to s8 on a mark, or var(--series-n) directly, for any set of items with a stable order.

Chart module. nullius/charts.py computes the geometry in SVG units. templates/_charts.html draws the marks and texts as it gets them, with no arithmetic. Each chart is an SVG with role="img" and a title. Each mark has a tooltip, and a table for a screen reader gives every value. A registry, STYLES, gives each style its label, its drawing function and the names of the metrics it takes (("Metric",) or ("X", "Y")). The card header reads it. A new style, for example Lines for pass@k after #146, is one function and one entry. The trace distributions (traces/charts.py) put durations in bins on a log axis. They share no scale with these charts, so they keep their own code.

Docs: the results guide names the Chart view. AGENTS.md lists the module.

Touches

backend (nullius/charts.py, new) · routes (web/compare.py) · templates and static (_charts.html new, compare/compare.html, base.html (an i-chart icon), compare/columns.js, compare/comparison.css, style.css) · tests (test_categorical_charts.py new, test_comparison_routes.py, field_contract.json) · docs (docs/results.md, AGENTS.md)

Before and after

Seeded data: a showcase profile of Terminal-Bench 2.0 runs, seven compared (six agents and the oracle check). The view is new, so there is no before. The nine-run shots add the nop check and a copy of the first run.

After
The default cards, 1280 px, light default-light-1280
The default cards, 390 px, dark default-dark-390
Dots and a scatter, 1280 px, light styles-light-1280
Dots and a scatter, 390 px, dark styles-dark-390
Two scatters with long metric names, 1280 px, light long-names-light-1280
Two scatters with long metric names, 390 px, dark long-names-dark-390
Seven runs at one point, 1280 px, light ties-light-1280
Seven runs at one point, 390 px, dark ties-dark-390
A card's Style menu, 1280 px, light style-menu-light-1280
After Add chart, Remove and a style change, 1280 px, light edited-light-1280
After Add chart, Remove and a style change, 390 px, dark edited-dark-390
A Y menu opened with the keyboard, 1280 px, light keyboard-menu-light-1280
A Y menu opened with the keyboard, 390 px, dark keyboard-menu-dark-390
Nine runs: eight colours, and the ninth back at the first, 1280 px, light nine-light-1280
Nine runs, 390 px, dark nine-dark-390
No chart, 1280 px, light empty-light-1280
The trial page's distributions, unchanged, 1280 px, light trial-distributions-light-1280

Verified

In a throwaway rootless python:3.12 container with Node.js and uv, on a copy of the branch:

  • python3 app/tests/parallel.py: 3004 tests in 154 modules, OK.
  • pip install -r app/requirements.txt, then python3 app/tests/parallel.py app/ui_tests: 655 tests in 54 modules, OK.
  • New: test_categorical_charts.py covers the ticks and tick labels that would touch, a gap for a missing value, a zero and a negative value, a check, the baseline mark, a single run, a long name that wraps, very different magnitudes, a scatter of runs at one point, the numbers inside the plot, the registry, and each mark's series class, which cycles after eight. test_comparison_routes.py covers the default cards, the URL shape, Add chart, Remove, a style change that keeps the metric, the Data, X and Y menus and the names on their buttons, no chart, a metric the cohort does not have, a check run, one run and an empty column, the column colours of eleven runs in every style, and the --series-* tokens in the dark and the light theme.

Browser check, Chromium (Playwright 1.49.1), on the showcase data with seven runs, at 1280 px light and 390 px dark:

  • Every non-check mark of a card has its run's colour: six fills for the six agents, with the oracle and nop hollow. With nine runs, the ninth has the first run's colour.
  • Every Scatter card shows a dot and a number for each run with both values, and no number is outside its plot. That includes Trials against Broke tests, where the seven runs share one point.
  • Each card's header buttons are all 34 px high. At 1280 px every header is one row, the scatters with long metric names too. At 390 px a Scatter header takes two rows, with whole buttons on each.
  • Add chart gives four cards. Remove on the second card gives three, and the others stay in order. Dots in the first card's Style menu changes only that card.
  • Keyboard: Enter on the Y button opens its menu with the focus on the first metric, and Tab and Enter pick the next one. An open menu does not widen the page.
  • No page JavaScript error. No sideways scroll on the Compare page.
  • A trial page with Distributions: its HTML on main and on this branch is the same, except the form tokens, the interface hash and the new icon in the sprite.

Not checked: other browsers, a screen reader, touch.

Risk

  • The view toggle has a third link, and ?view=chart is new. Every other value of view still opens Side by side.
  • summary_rows now gives each row a key, a group, its number and its unit. The side table reads none of these, so its cells do not change.
  • Every menu item is a full link, so a page with many cards carries many long URLs (one for each metric of each axis).
  • A menu choice reloads the page, so the focus goes back to the top.
  • A value's text is the table's text. Where the table shows a dash for a real 0 (Infra errors, Broke tests), the chart draws a zero mark beside the dash.
  • To undo, revert the two commits. No stored data changes.

Related

A backend module places a few named values, one per item, in SVG user
units, and a template macro draws the marks and texts as given, with a
table of every value for a screen reader. A missing value is a gap, a
check is hollow and named, and nothing is ranked. Styles are a registry
of drawing functions, so a new style is one function and one entry.
An item's series gives it one colour in every chart, from eight
--series-* tokens set for the light and the dark themes.
Compare gains a third view beside Side by side and Task by task. Each
chart is a card. Its header holds a Style menu (Bars, Dots or Scatter),
a Data menu of one metric, or X and Y menus for a scatter, each button
naming its metric, and a Remove button. Add chart adds a card. The
charts live in the URL, one chart= value each, and every link keeps
them. The values are the side table's own cells, and a run's colour
is its column's place, the same in every chart.
@harenome
harenome merged commit 3e25413 into ThinkFlowLab:main Nov 19, 2026
2 checks passed
@harenome
harenome deleted the feat/compare-chart branch November 19, 2026 10:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Compare: a Chart view of the summary rows, a card for each chart

1 participant