From d01340e856ef152cfa68df422b4a5befa591e09b Mon Sep 17 00:00:00 2001 From: Cirdes Date: Sun, 1 Nov 2026 14:06:01 -0300 Subject: [PATCH 01/15] [Documentation] shadcn-audit: design for deterministic browser collection Co-Authored-By: Claude Opus 5.5 --- audit/docs/2026-11-01-collect-design.md | 196 ++++++++++++++++++++++++ 1 file changed, 196 insertions(+) create mode 100644 audit/docs/2026-11-01-collect-design.md diff --git a/audit/docs/2026-11-01-collect-design.md b/audit/docs/2026-11-01-collect-design.md new file mode 100644 index 00000000..8b00ab2b --- /dev/null +++ b/audit/docs/2026-11-01-collect-design.md @@ -0,0 +1,196 @@ +# shadcn-audit `collect`: deterministic browser collection + +Status: design, approved for planning once reviewed. +Date: 2026-11-01. + +## Intent + +The shadcn-audit skill compares a RubyUI component with its shadcn +(Base UI + vega + neutral) counterpart. Today the agent does every browser +step by hand through the chrome-devtools MCP: opens both fixtures, clicks the +trigger, reads classes and ARIA, takes screenshots, and grades each line. + +Nothing is broken. The goal is **determinism**: running the audit twice on the +same commit should produce the same collected facts and the same starting +grades, line by line. Today that depends on the agent collecting the same +things the same way every run (which selector it derives for a RubyUI part, +which properties it compares, which states it screenshots). + +## Goals + +1. A new CLI subcommand, `collect`, in `audit/` (same one-JSON-object + contract as every other subcommand, run from `audit/`), that drives a + browser through **playwright-ruby-client** and collects, for both sides: + - **Setup**: fixed 1280×800 viewport, light and dark, and the environment + probe from `interactions.md` (viewport, DPR, fonts, body colors, form + control defaults, dark class). Differences go into `environment_diffs`. + - **States** common to every `kind`: closed, open (click on + `[data-audit-trigger]`), keyboard focus, after Escape. A state that does + not apply to the kind, or can't be reached, is recorded as unreachable + with the reason (→ ⏸). + - **Per state and per mapped part**: computed styles, the accessibility + tree (Playwright `ariaSnapshot`), plus a screenshot per state and side. + - **Pixel diff** per state (same viewport, both sides). +2. `collect` grades the **objective** lines itself: + - computed style equal on both sides → `pass` (even when class tokens + differ: this settles look-alikes such as `ring-1` vs `border`); + - computed style differs on a visual property → `fail`, both values named; + - ARIA difference (role, accessible name, state, or one of the checked + attributes present on one side only) → `warn`, as `grading.md` rule 4 + says (ARIA is ⚠️, not ❌; see "Builder-check decisions"); + - difference registered in `divergences.yml` → `pass`, basis + `intentional`; + - state unreachable / trigger missing / page error → `inconclusive` with + the reason. + Every graded line carries a `source` the way `report.json` requires. +3. The skill (`.claude/skills/shadcn-audit/SKILL.md` and + `references/interactions.md`) is updated so the agent runs `collect` + first and builds `report.json` on top of its output. +4. The agent keeps: Spec, Native, Animation, the kind-specific behavior + sequences (focus-trap order, scroll lock, outside click, arrow keys, + typeahead, Enter/Space toggling), fixture-originated wrappers, and + exploration of anything `collect` didn't cover. The agent **may override** + a `collect` grade, but only with a written reason that ends up in the + report line's `note`. + +## Non-goals + +- Running `collect` in CI. Local only for now (the user's call). The `audit` + CI job keeps running unit tests only. +- Kind-specific behavior sequences in v1. They move into `collect` later, one + kind at a time. +- Pixel diff as a grade. It is **evidence** for the agent (number of + differing pixels / ratio, plus a diff image), never a status on its own. + Font and antialiasing noise make it unreliable as an automatic verdict. +- Booting the servers. Sandbox (5179) and preview (3179) keep being started + as today (Herdr panes or background processes); `collect` receives the URLs. +- Changing `report.json`'s schema in a breaking way, or the renderers. +- Cross-browser runs (Firefox/WebKit). Chromium only, even though Playwright + would allow more later. + +## Anti-goals + +- `collect` producing grades that need judgment (spec conformance, "is this + JS replaceable by native HTML", "is this a fixture wrapper"). A rule-based + guess there is a false positive generator. +- Output that differs between two runs on the same SHAs for reasons other + than timestamps, absolute paths, or pixel noise under the tolerance. +- Running anything from `.shadcn-audit/ui` (still untrusted), or booting fork + code without the existing cross-repository confirmation step. + +## Constraints and decisions already made + +- **Library: playwright-ruby-client** (user's call). It needs the Node + `playwright` package and a Chromium installed locally; the README and the + skill's prerequisites must say how (the audit already requires Node and + pnpm). +- **Local only** (user's call). +- **Grading split** as in Goals 2 and 4 (user agreed). +- **Pixel diff = evidence only** (proposed, user agreed). +- **v1 scope** = setup + common states + styles/ARIA/screenshots/pixel diff + for every kind; behavior sequences stay with the agent (user agreed). +- **RubyUI part selectors become fixed data.** RubyUI parts have no + `data-slot`; today the agent derives a selector from the Ruby source on + every run, which is a source of non-determinism. `component_map.yml` gains + a selector per RubyUI part, next to the existing class → `data-slot` + mapping. `ComponentMap` validation and `map` output include it. + `collect` refuses (⏸ for that part) when a mapped part has no selector. +- Existing conventions hold: Ruby 3.4 via mise, StandardRB, Minitest, one JSON + object per subcommand, `{"error": …}` + exit 1 on failure, errors listed + in `CLI::ERRORS`. + +## Divergences become machine-matchable (user's call) + +`divergences.yml` entries today are free text (`rubyui`, `shadcn`, +`reason`) keyed by component + dimension, and `grading.md` says an entry +covers only the difference its text describes. Reading that prose is +judgment, so: + +- Entries gain **optional structured match keys**: the part, the computed + property or ARIA attribute/role, and the value on each side (exact shape + left to the builder). `Divergences` validates them. +- `collect` marks a difference `pass` / basis `intentional` **only** when a + structured key matches it exactly. It never matches on component + + dimension alone. +- Entries without match keys stay valid; the agent applies them as overrides, + as today. +- The four existing entries are migrated to include match keys where the + difference is expressible that way. + +## Builder-check decisions (designer's calls, check these) + +- **ARIA grade and basis.** ARIA differences are `warn`, per `grading.md` + rule 4, so the old flow's verdicts are preserved. The comparison is on the + computed accessibility tree (`ariaSnapshot`: role, name, states), so a native + `` vs `div[role=dialog]` is not a difference. On top of that, an + attribute probe checks the APG attributes the old flow read + (`aria-expanded`, `aria-controls`, `aria-haspopup`, `aria-modal`, + `aria-labelledby`/`aria-describedby` resolved to their text). ID-valued + attributes are compared by what they resolve to, never by the raw ID. +- **Line granularity follows `grading.md`.** One Look line per differing + part, with all matching parts grouped into a single `pass` line; same for + ARIA. State and theme where the difference shows up go into `note` + (e.g. "open, light+dark"). No per-property or per-state lines, so a re-run + can be compared line by line with the old report. +- **Theme and environment explain before they fail.** A differing computed + value that equals a `theme` diff pair (RubyUI value vs shadcn value of the + same token) is not graded as a component `fail`; it becomes `inconclusive` + with a note naming the token (theme diffs are already reported on their + own). Same for typography properties when `environment_diffs` has a font + difference. Only what's left is graded. +- **Selectors come from the audited head.** `collect` loads + `component_map.yml` from the `preview` checkout (the audited SHA), falling + back to the working tree only when the head has no selectors yet, and says + which one it used. Each selector must match exactly the expected number of + elements (one, or N for repeated parts such as accordion items); otherwise + that part is `inconclusive` ("selector matched 0/3 elements"). A PR that + changes a part's markup updates its selector in the same PR. + +## Shape of the output (guidance, not a contract) + +`collect` writes its artifacts into the run folder and prints one JSON +object, roughly: + +- `viewport`, `environment` (per side) and `environment_diffs`; +- `states`: for each state × theme, per side: reached or not (+ reason), + screenshot path (relative to the run folder), per-part computed styles, + `ariaSnapshot` text; plus the pixel diff (ratio + diff image path); +- `lines`: the objectively graded lines, already in `report.json` line shape + (`dimension`, `status`, `basis`, `rubyui`, `shadcn`, `source`, `note`), so + the agent can append them to the component's `lines`; +- `screenshots`: entries already in `report.json`'s screenshot shape. + +The agent then adds its own lines and overrides, and writes `report.json` as +today; `render` and `comment` are unchanged. + +## Success criteria + +- Two `collect` runs on the same `head_sha`/`shadcn_sha`, with servers + booted the same way, produce identical JSON after removing timestamps and + absolute paths, and identical `lines`. +- Re-running the audit of an already-mapped component (e.g. Dialog) with the + new flow yields every Look/ARIA line the old flow had, with the same + verdicts or a documented reason for the change. +- Unit tests cover grading rules and map validation without a browser; the + browser-driving code is exercised by a local smoke test that is skipped + when Playwright/Chromium is not installed. + +## Left to the builder + +- Subcommand arguments and flags (how URLs, run folder, folder/component and + states are passed), and whether light/dark runs in one call. +- Which computed-style properties count as "visual" and how they are + normalized (color formats, sub-pixel rounding). +- How "keyboard focus" is reached generically (e.g. Tab to the trigger) and + how a state is declared not applicable per kind. +- Pixel diff implementation and the noise tolerance used to report it. +- The exact YAML shape of the RubyUI selectors in `component_map.yml`, and + filling them in for the five mapped components. +- How Playwright is installed/located and how its absence is reported. +- How `source` is filled for `collect` lines (e.g. `style-vega.css:` + from `reference` effective classes, or the fixture URL + state). + +## Not in this project + +- The accordion audit itself (no map entry or fixture yet). It runs with the + current flow or after this lands, separately. From bfcc10a6970dac13e331dab5a34333c1040797f7 Mon Sep 17 00:00:00 2001 From: Cirdes Date: Sun, 1 Nov 2026 14:09:14 -0300 Subject: [PATCH 02/15] [Documentation] shadcn-audit: implementation plan for collect Co-Authored-By: Claude Opus 5.5 --- audit/docs/2026-11-01-collect-plan.md | 251 ++++++++++++++++++++++++++ 1 file changed, 251 insertions(+) create mode 100644 audit/docs/2026-11-01-collect-plan.md diff --git a/audit/docs/2026-11-01-collect-plan.md b/audit/docs/2026-11-01-collect-plan.md new file mode 100644 index 00000000..becd29f7 --- /dev/null +++ b/audit/docs/2026-11-01-collect-plan.md @@ -0,0 +1,251 @@ +# shadcn-audit `collect` Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** A `collect` subcommand that drives both fixtures in Chromium through playwright-ruby-client, collects computed styles, accessibility facts, screenshots and pixel diffs per state, and grades the objective Look/ARIA lines deterministically. + +**Architecture:** A browser-free core (`Collect::Normalize`, `Collect::Grader`, `Collect::PixelDiff`) fed by a thin Playwright layer (`Collect::Driver`) and glued by `Collect::Runner`. The core is unit-tested with plain hashes; the driver gets a smoke test against local HTML files that skips when Chromium isn't installed. `ComponentMap` gains `selectors`, `Divergences` gains `match`. + +**Tech Stack:** Ruby 3.4 (mise), Minitest, StandardRB, playwright-ruby-client 1.64.0 + Node `playwright` 1.64.1 (pnpm, in `audit/`), chunky_png 1.4.0. + +**Spec:** `audit/docs/2026-11-01-collect-design.md` + +## Global Constraints + +- Every subcommand prints exactly one JSON object; failures are `{"error": …}` + exit 1 via `CLI::ERRORS`. +- Run from `audit/`: `mise exec -- bundle exec …`. `bundle exec rake` (tests + standardrb) must pass after every task. +- Viewport fixed at **1280×800**, DPR 1, Chromium headless only. +- Pixel diff is evidence only: it never sets a line's `status`. +- ARIA differences grade `warn`; Look differences `fail`; registered (structured `match`) → `pass` basis `intentional`; theme/environment-explained or unreachable → `inconclusive`. Nothing unverified is `pass`. +- Lines use `report.json` line shape: `dimension, status, basis, rubyui, shadcn, source, note`. Every `pass`/`warn`/`fail` has a non-empty `source`. +- Line granularity: one line per part per outcome; all fully matching parts grouped into one `pass` line per dimension. State/theme go in `note`. +- Screenshot names: `screens/---.png`; diff images `screens/--diff-.png`. Paths in output are relative to the run folder. +- Local only: CI keeps running unit tests; nothing browser-bound runs there. +- Never run anything from `.shadcn-audit/ui`. + +## Review Focus + +1. **Repeated parts with mismatched counts** (accordion items: 3 on shadcn, 2 visible on RubyUI): the part is `inconclusive` "selector matched 2/3 visible elements in open/light", never compared index-by-index. → Task 3 test. +2. **Hidden-but-present elements** (RubyUI keeps closed content in the DOM, Base UI unmounts it): only visible elements (`checkVisibility()`) are counted and collected. → Task 6 smoke test. +3. **Missing trigger, page error, navigation timeout**: the state is `reached: false` with the reason and the run continues; lines become `inconclusive`. → Task 6 smoke test + Task 7 runner test. +4. **Playwright/Chromium not installed**: `collect` returns `{"error": "…run `pnpm install && pnpm exec playwright install chromium` in audit/"}`, not a backtrace. → Task 7 CLI test. +5. **Order stability**: the same observations in a different hash/array order produce byte-identical `lines`. → Task 3 test. + +--- + +### Task 1: Dependencies and `selectors` in the component map + +**Files:** +- Modify: `audit/Gemfile` (+ `Gemfile.lock`), `audit/.gitignore`, `audit/lib/shadcn_audit/component_map.rb` +- Create: `audit/package.json`, `audit/pnpm-lock.yaml` +- Test: `audit/test/component_map_test.rb` + +**Interfaces:** +- Produces: `entry(folder)["selectors"]` → `Hash{String part => String css}` (may be absent). `ComponentMap#selectors(folder) -> Hash` (empty hash when absent). + +- [ ] **Step 1: Write failing tests** in `component_map_test.rb`: + - `test_selectors_are_returned`: entry with `parts: {DialogContent: dialog-content}`, `selectors: {DialogContent: "dialog > div"}` → `map.selectors("dialog") == {"DialogContent" => "dialog > div"}`. + - `test_selectors_default_to_empty`: no `selectors` key → `{}`. + - `test_selector_for_unknown_part_is_rejected`: `selectors: {Nope: "div"}` → `ConfigError` matching `/selectors.Nope is not a mapped part/`. + - `test_blank_selector_is_rejected`: `selectors: {DialogContent: " "}` → `ConfigError` matching `/selectors.DialogContent must be a non-empty CSS selector/`. +- [ ] **Step 2: Run** `mise exec -- bundle exec rake test TEST=test/component_map_test.rb` → the four fail. +- [ ] **Step 3: Implement** validation in `ComponentMap#validate!` and `#selectors(folder)`. Update the header comment of `component_map.yml` with one line: `# selectors: RubyUI class => CSS selector of that part in the rendered fixture (fixed, so collect is deterministic).` +- [ ] **Step 4: Add dependencies**: Gemfile `gem "playwright-ruby-client", "1.64.0"` and `gem "chunky_png", "1.4.0"`; `bundle install`. `audit/package.json` `{"private": true, "devDependencies": {"playwright": "1.64.1"}}`; `pnpm install`. `.gitignore` += `/node_modules/`. +- [ ] **Step 5: Run** `mise exec -- bundle exec rake` → all pass, standardrb clean. +- [ ] **Step 6: Commit** `[Feature] shadcn-audit: selectors in component map + playwright deps`. + +### Task 2: Structured `match` keys in divergences + +**Files:** +- Modify: `audit/lib/shadcn_audit/divergences.rb`, `audit/divergences.yml` (header comment only) +- Test: `audit/test/divergences_test.rb` + +**Interfaces:** +- Produces: `Divergences#match(component:, dimension:, part:, key:, rubyui:, shadcn:) -> Hash | nil` — returns the entry whose `match` equals `{part, property|attribute: key, rubyui, shadcn}` after `Collect::Normalize.value` on both values (Task 3 defines it; until then compare `strip` — Task 3 swaps it in). Never matches entries without `match`. + +Entry shape: +```yaml +- component: dialog + dimension: Look # Look → property; ARIA → attribute + rubyui: "border 1px" + shadcn: "ring 1px" + reason: "…" + since: "2026-11-01" + match: {part: DialogContent, property: border-top-width, rubyui: "1px", shadcn: "0px"} +``` +`attribute` may be `snapshot` to match the whole accessibility snapshot of the part. + +- [ ] **Step 1: Write failing tests**: + - `test_match_finds_structured_entry`: entry above → `match(component: "dialog", dimension: "Look", part: "DialogContent", key: "border-top-width", rubyui: "1px", shadcn: "0px")` returns it. + - `test_match_ignores_prose_only_entries`: same component/dimension without `match` → `nil`. + - `test_match_requires_exact_values`: `rubyui: "2px"` → `nil`. + - `test_match_validation`: `match` on dimension `HTML` → `ConfigError` `/match is only for Look or ARIA/`; Look with `attribute` → `/Look match needs property/`; missing `rubyui` → `/match needs part, property\|attribute, rubyui and shadcn/`. + - `test_existing_registry_still_loads`: `Divergences.load(ShadcnAudit.root.join("audit/divergences.yml"))` doesn't raise. +- [ ] **Step 2: Run** `rake test TEST=test/divergences_test.rb` → new tests fail. +- [ ] **Step 3: Implement** validation + `#match`. Header comment in `divergences.yml`: one line explaining `match`. The four existing entries are HTML/Behavior differences, not computed Look/ARIA values, so none gets a `match` (say so in the commit message). +- [ ] **Step 4: Run** `rake` → pass. +- [ ] **Step 5: Commit** `[Feature] shadcn-audit: machine-matchable divergences`. + +### Task 3: Normalize + Look grading + +**Files:** +- Create: `audit/lib/shadcn_audit/collect/normalize.rb`, `audit/lib/shadcn_audit/collect/grader.rb` +- Modify: `audit/lib/shadcn_audit.rb` (requires), `divergences.rb` (use `Normalize.value`) +- Test: `audit/test/collect/normalize_test.rb`, `audit/test/collect/grader_test.rb` + +**Interfaces:** +- Consumes: `Divergences#match` (Task 2). +- Produces: + - `Collect::Normalize.value(String) -> String`: downcase, collapse whitespace, every decimal number rounded to 2 places with trailing zeros dropped (`"0.6459"→"0.65"`, `"1.50px"→"1.5px"`). + - `Collect::Grader.new(folder:, parts:, divergences:, theme_diffs:, environment_diff_keys:)` where `parts` is the ordered RubyUI part list from the map (`Array`), `theme_diffs` is `theme["diffs"]`, `environment_diff_keys` is e.g. `["font"]`. + - `Grader#lines(observations) -> Array` in report line shape. `observations`: `Array<{"state"=>, "theme"=>, "rubyui"=>SideFacts, "shadcn"=>SideFacts}>`. + - `SideFacts = {"reached"=>Bool, "reason"=>String|nil, "screenshot"=>String, "parts"=>{part=>[{"styles"=>{prop=>val}, "aria"=>String, "attrs"=>{name=>val|nil}}]}}`. + - Constants: `Grader::FONT_PROPS = %w[font-family font-size font-weight line-height letter-spacing width height]`; `Grader::LOOK_SOURCE = "computed styles (shadcn-audit collect)"`; `Grader::ARIA_SOURCE = "accessibility tree (shadcn-audit collect)"`. + +Look rules, per part, over every observation where both sides were reached: +1. Element counts differ → one `inconclusive` Look line for the part: note `"selector matched / visible elements in /"` (first mismatch, then `; …` for others). Skip property comparison for that observation. +2. Per element index and property, values equal after `Normalize.value` → no diff. +3. Diff explained by theme: some `theme_diffs` entry with same `mode` as the observation's theme and normalized `rubyui`/`shadcn` equal to the observed values → bucket `theme` (note names the tokens). +4. Diff explained by environment: `environment_diff_keys` includes `"font"` and property ∈ `FONT_PROPS` → bucket `environment`. +5. `divergences.match(dimension: "Look", part:, key: property, …)` → bucket `intentional`. +6. Otherwise → bucket `diff`. + +Output per part (in `parts` order), at most one line per non-empty bucket, in this order: `diff` → `fail` (`rubyui`/`shadcn` = `"prop: value"` joined by `"; "`, unique, sorted by property; note = the `state/theme` pairs where seen, joined `", "`); `intentional` → `pass` basis `intentional`, source `audit/divergences.yml`, note = reason; `theme`/`environment` → `inconclusive`, note `"explained by theme token(s) "` / `"explained by environment diff: font"`. Parts with no lines at all → collected into one `pass` Look line: `rubyui` = `", : computed styles match"`, `shadcn` = `"same"`, source `LOOK_SOURCE`. + +- [ ] **Step 1: Write failing tests** (`normalize_test.rb`): the two examples above, `" RGB(0, 0, 0) "→"rgb(0, 0, 0)"`, `"10px"` unchanged. +- [ ] **Step 2: Write failing tests** (`grader_test.rb`), each building observations by hand: + - `test_matching_parts_group_into_one_pass_line`: two parts identical → exactly one line `{dimension: "Look", status: "pass", rubyui: "DialogContent, DialogTitle: computed styles match"}`. + - `test_style_difference_is_fail_with_both_values`: `padding-top` `24px` vs `16px` in `open/light` and `open/dark` → one `fail` line, `rubyui: "padding-top: 24px"`, `shadcn: "padding-top: 16px"`, `note: "open/light, open/dark"`, source `LOOK_SOURCE`. + - `test_count_mismatch_is_inconclusive`: 2 vs 3 elements → `inconclusive`, note `"selector matched 2/3 visible elements in open/light"`, and no `fail` for that part. + - `test_theme_token_explains_difference`: `color` `oklch(0.646 0.222 41.116)` vs `oklch(0.809 0.105 251.813)` with theme diff `chart-1` light → `inconclusive`, note includes `chart-1`. + - `test_font_environment_explains_typography_only`: env keys `["font"]`; `font-size` differs → `inconclusive`; `padding-top` differs → `fail`. + - `test_structured_divergence_passes_as_intentional`: matching `match` entry → `pass` basis `intentional`, source `audit/divergences.yml`. + - `test_output_is_order_independent` (Review Focus 5): shuffle observations and the styles hash key order → `lines` equal. + - `test_unreached_observations_are_skipped_for_comparison`: rubyui `reached: false` → no Look lines from that observation (Task 4 adds the inconclusive line). +- [ ] **Step 3: Run** `rake test TEST=test/collect/grader_test.rb` → fail. +- [ ] **Step 4: Implement** `Normalize.value` and `Grader#lines` (Look only); switch `Divergences#match` to `Normalize.value`. +- [ ] **Step 5: Run** `rake` → pass. +- [ ] **Step 6: Commit** `[Feature] shadcn-audit: deterministic Look grading for collect`. + +### Task 4: ARIA grading and unreachable states + +**Files:** +- Modify: `audit/lib/shadcn_audit/collect/grader.rb` +- Test: `audit/test/collect/grader_test.rb` + +**Interfaces:** +- Produces: `Grader#lines` also emits ARIA lines and unreachable lines. `Grader::ARIA_ATTRS = %w[aria-expanded aria-controls aria-haspopup aria-describedby]` (the driver fills `attrs` with exactly these keys; `aria-controls` is reported as `"resolves"`/`"dangling"`/nil, `aria-describedby` as the referenced text or nil). + +ARIA rules, per part, same count/skip logic as Look: +- `aria` (Playwright aria snapshot of the element) differs after `Normalize.value` → diff with key `snapshot`. +- Each `ARIA_ATTRS` value differs (incl. present vs nil) → diff with key = attribute. +- `divergences.match(dimension: "ARIA", part:, key:, …)` → `intentional`; theme/environment never apply to ARIA. +- Output like Look but `diff` bucket → `warn`; `rubyui`/`shadcn` = `": "`, snapshots trimmed to 200 chars; grouped `pass` line `": accessibility tree matches"`, source `ARIA_SOURCE`. + +Unreachable: for each observation where a side has `reached: false` → one `inconclusive` line per dimension (Look, ARIA), deduplicated by reason, note `"/ not reached on : "`, all observations sharing the same `state`+`side`+`reason` merged into one note. + +- [ ] **Step 1: Write failing tests**: + - `test_aria_snapshot_difference_is_warn`: snapshots `- button "Open" [expanded]` vs `- button "Open"` → `warn`, `rubyui` starts with `"snapshot: - button"`, source `ARIA_SOURCE`. + - `test_missing_attribute_is_warn`: `aria-controls` `"resolves"` vs nil → `warn` with `shadcn: "aria-controls: (none)"`. + - `test_matching_aria_groups_into_pass`. + - `test_aria_divergence_by_attribute_is_intentional`. + - `test_unreached_state_yields_inconclusive_look_and_aria`: rubyui `open/light` and `open/dark` unreached with reason `"no [data-audit-trigger]"` → exactly two lines (Look, ARIA), note `"open/light, open/dark not reached on rubyui: no [data-audit-trigger]"`. +- [ ] **Step 2: Run** → fail. **Step 3: Implement.** **Step 4: Run** `rake` → pass. +- [ ] **Step 5: Commit** `[Feature] shadcn-audit: ARIA and unreachable-state grading for collect`. + +### Task 5: Pixel diff + +**Files:** +- Create: `audit/lib/shadcn_audit/collect/pixel_diff.rb` +- Test: `audit/test/collect/pixel_diff_test.rb` + +**Interfaces:** +- Produces: `Collect::PixelDiff.compare(rubyui_png:, shadcn_png:, diff_png:) -> {"pixels"=>Integer, "ratio"=>Float (6 decimals), "diff"=>String path}` or `{"error"=>"size 1280x800 vs 1280x720"}` when dimensions differ (no diff image written). `PixelDiff::CHANNEL_TOLERANCE = 16`: a pixel differs when any RGBA channel differs by more than 16. Diff image: differing pixels red (`#ff0000`), others the shadcn pixel at 25% alpha. + +- [ ] **Step 1: Write failing tests** generating PNGs with `ChunkyPNG::Image.new(4, 4, …)` in `Dir.mktmpdir`: identical → `pixels: 0, ratio: 0.0`; one pixel off by 10 → `0`; one pixel off by 40 → `pixels: 1, ratio: 0.0625`, diff file exists; 4×4 vs 4×3 → `{"error" => "size 4x4 vs 4x3"}`. +- [ ] **Step 2: Run** → fail. **Step 3: Implement.** **Step 4: Run** `rake` → pass. +- [ ] **Step 5: Commit** `[Feature] shadcn-audit: pixel diff evidence for collect`. + +### Task 6: Playwright driver + +**Files:** +- Create: `audit/lib/shadcn_audit/collect/driver.rb`, `audit/lib/shadcn_audit/collect/probes.js` +- Test: `audit/test/collect/driver_smoke_test.rb`, `audit/test/fixtures/collect/{a,b}.html` + +**Interfaces:** +- Produces: + - `Collect::BrowserError < StandardError` (message used verbatim by the CLI). + - `Collect::Driver.open(audit_dir:) { |driver| … }`: launches headless Chromium via `Playwright.create(playwright_cli_executable_path: /node_modules/.bin/playwright)`; raises `BrowserError` with the install hint (Review Focus 4) when the CLI or browser is missing. + - `Driver#environment(url) -> Hash` keys `w h dpr font fonts body forms dark` (the probe in `interactions.md`, `fonts` sorted). + - `Driver#observe(url:, state:, selectors:, screenshot_path:) -> SideFacts` (shape in Task 3). `selectors` is `{part => css}`; the runner passes `[data-slot=""]` for shadcn. + - `Driver::STATES = %w[closed open focus after-esc]`. + +Behavior (fresh `page` per call, viewport 1280×800, `reduced_motion: "reduce"`): +- Navigate, wait for `load` (timeout 15 s); page error / timeout → `reached: false`, reason = message's first line. +- `closed`: nothing more. `open`: real `click` on `[data-audit-trigger]` (missing → `reached: false`, reason `"no [data-audit-trigger]"`). `focus`: press `Tab` up to 20 times until `document.activeElement` is or contains the trigger (else unreached `"trigger never received keyboard focus"`). `after-esc`: `open` then press `Escape`. +- Settle: await all `document.getAnimations()` `finished` (cap 2 s), then two `requestAnimationFrame`s. +- Per part: elements matching the selector **and** `checkVisibility()` (Review Focus 2); for each: computed styles for the property list in `Driver::STYLE_PROPS` (color, background-color, border-{top,right,bottom,left}-{width,style,color}, border-{top-left,top-right,bottom-right,bottom-left}-radius, box-shadow, outline-width, outline-style, outline-color, outline-offset, padding-{top,right,bottom,left}, gap, row-gap, column-gap, width, height, font-family, font-size, font-weight, line-height, letter-spacing, text-align, text-decoration-line, opacity, display, transform), `locator.aria_snapshot`, `attrs` per `Grader::ARIA_ATTRS` rules. +- Screenshot `page.screenshot(path:, animations: "disabled", caret: "hide")`; `screenshot` in facts is the path relative to the run folder (runner passes both). + +- [ ] **Step 1: Write the smoke test** (skips with `skip "chromium not installed"` when `node_modules/.bin/playwright` is absent or `Driver.open` raises `BrowserError`), loading `file://` URLs of two fixtures: `a.html` has ` + + + + diff --git a/audit/test/fixtures/collect/b.html b/audit/test/fixtures/collect/b.html new file mode 100644 index 00000000..36804022 --- /dev/null +++ b/audit/test/fixtures/collect/b.html @@ -0,0 +1,2 @@ + +

No trigger here

From b6b3a292b8a93555d9625513204bed037d6a013a Mon Sep 17 00:00:00 2001 From: Cirdes Date: Sun, 1 Nov 2026 14:23:42 -0300 Subject: [PATCH 09/15] [Feature] shadcn-audit: collect subcommand Co-Authored-By: Claude Opus 5.5 --- audit/lib/shadcn_audit.rb | 1 + audit/lib/shadcn_audit/cli.rb | 28 ++++- audit/lib/shadcn_audit/collect/runner.rb | 140 +++++++++++++++++++++++ audit/test/cli_collect_test.rb | 27 +++++ audit/test/collect/runner_test.rb | 107 +++++++++++++++++ 5 files changed, 297 insertions(+), 6 deletions(-) create mode 100644 audit/lib/shadcn_audit/collect/runner.rb create mode 100644 audit/test/cli_collect_test.rb create mode 100644 audit/test/collect/runner_test.rb diff --git a/audit/lib/shadcn_audit.rb b/audit/lib/shadcn_audit.rb index 72de5515..f33c414b 100644 --- a/audit/lib/shadcn_audit.rb +++ b/audit/lib/shadcn_audit.rb @@ -24,6 +24,7 @@ def self.workspace_dir = root.join(".shadcn-audit") require "shadcn_audit/collect/grader" require "shadcn_audit/collect/pixel_diff" require "shadcn_audit/collect/driver" +require "shadcn_audit/collect/runner" require "shadcn_audit/renderers/terminal" require "shadcn_audit/renderers/markdown" require "shadcn_audit/renderers/html" diff --git a/audit/lib/shadcn_audit/cli.rb b/audit/lib/shadcn_audit/cli.rb index a9df64d1..a766e969 100644 --- a/audit/lib/shadcn_audit/cli.rb +++ b/audit/lib/shadcn_audit/cli.rb @@ -16,27 +16,29 @@ class UnknownCommand < StandardError; end "sandbox" => ->(args, out:) { sandbox(args, out: out) }, "preview" => ->(args, out:) { preview(args, out: out) }, "render" => ->(args, out:) { render(args, out: out) }, - "comment" => ->(args, out:) { comment(args, out: out) } + "comment" => ->(args, out:) { comment(args, out: out) }, + "collect" => ->(args, out:, **deps) { collect(args, out: out, **deps) } } # Failures the CLI reports as {"error": ...} with exit status 1 instead # of a backtrace, so callers always get one JSON object. - ERRORS = [UnknownCommand, Shell::Error, ConfigError, ReportError, ArgumentError, + ERRORS = [UnknownCommand, Collect::BrowserError, Shell::Error, ConfigError, ReportError, ArgumentError, Errno::ENOENT, JSON::ParserError, Psych::SyntaxError, OptionParser::ParseError].freeze # Entry point for exe/shadcn-audit; returns the process exit status. - def self.run(argv, out: $stdout) - start(argv, out: out) + # `deps` injects collaborators in tests (e.g. collect's driver_factory). + def self.run(argv, out: $stdout, **deps) + start(argv, out: out, **deps) 0 rescue *ERRORS => e out.puts JSON.generate("error" => e.message) 1 end - def self.start(argv, out: $stdout) + def self.start(argv, out: $stdout, **deps) name, *args = argv handler = COMMANDS[name] or raise UnknownCommand, "unknown subcommand: #{name.inspect}" - out.puts JSON.generate(handler.call(args, out: out)) + out.puts JSON.generate(handler.call(args, out: out, **deps)) end def self.map(args, out:) @@ -110,6 +112,20 @@ def self.comment(args, out:, shell: Shell.new) result.merge("posted" => result["action"] != "refused") end + # Drives both fixtures in Chromium and grades the objective lines. + def self.collect(args, out:, driver_factory: Collect::Driver.method(:open), root: ShadcnAudit.root) + run_dir = preview = nil + parser = OptionParser.new do |o| + o.on("--run DIR") { |v| run_dir = v } + o.on("--preview DIR") { |v| preview = v } + end + folder = parser.parse!(args).first + raise ArgumentError, "usage: collect --run [--preview ]" unless folder && run_dir + + Collect::Runner.new(folder: folder, run_dir: run_dir, preview: preview, root: root, + driver_factory: driver_factory).call + end + def self.workspace(args, out:) sha = nil OptionParser.new { |o| o.on("--sha SHA") { |v| sha = v } }.parse!(args) diff --git a/audit/lib/shadcn_audit/collect/runner.rb b/audit/lib/shadcn_audit/collect/runner.rb new file mode 100644 index 00000000..f015af3e --- /dev/null +++ b/audit/lib/shadcn_audit/collect/runner.rb @@ -0,0 +1,140 @@ +module ShadcnAudit + module Collect + # Runs the driver over every state × theme × side for one component, + # grades the objective lines and writes /collect.json. + class Runner + THEMES = %w[light dark].freeze + SIDES = %w[rubyui shadcn].freeze + STATES_BY_KIND = Hash.new(Driver::STATES).merge( + "form_control" => %w[closed focus], "static" => %w[closed] + ).freeze + FONT_KEYS = %w[font fonts].freeze + + def initialize(folder:, run_dir:, preview: nil, root: ShadcnAudit.root, theme: nil, + driver_factory: Driver.method(:open)) + @folder = folder.to_s + @run_dir = Pathname.new(run_dir) + @preview = preview && Pathname.new(preview) + @root = root + @theme = theme + @driver_factory = driver_factory + end + + def call + map_source, map, divergences = load_config + entry = map.entry(@folder) + selectors = map.selectors(@folder) + raise ArgumentError, "#{@folder} has no selectors in audit/component_map.yml" if entry.nil? || selectors.empty? + + slots = (entry["parts"] || {}).compact + compared = slots.keys.select { |part| selectors.key?(part) } + unselected = slots.keys - compared + FileUtils.mkdir_p(@run_dir.join("screens")) + + observations, environment = @driver_factory.call(audit_dir: @root.join("audit")) do |driver| + [observe_all(driver, STATES_BY_KIND[entry["kind"]], selectors.slice(*compared), slots), probe_environment(driver)] + end + environment_diffs, diff_keys = environment_diffs(environment) + grader = Grader.new(folder: @folder, parts: compared, divergences: divergences, + theme_diffs: theme["diffs"], environment_diff_keys: diff_keys) + lines = unselected.map { |part| missing_selector_line(part) } + grader.lines(observations) + + result = { + "folder" => @folder, "map_source" => map_source, + "viewport" => {"w" => Driver::VIEWPORT[:width], "h" => Driver::VIEWPORT[:height]}, + "environment" => environment, "environment_diffs" => environment_diffs, + "states" => observations, "pixel_diffs" => pixel_diffs(observations), + "lines" => lines, "screenshots" => screenshots(observations) + } + File.write(@run_dir.join("collect.json"), JSON.pretty_generate(result)) + result + end + + private + + # The audited head's map wins when it already has selectors for the folder. + def load_config + if @preview + preview_map = ComponentMap.load(@preview.join("audit/component_map.yml")) + unless preview_map.selectors(@folder).empty? + return ["preview", preview_map, Divergences.load(@preview.join("audit/divergences.yml"))] + end + end + ["working tree", ComponentMap.load(@root.join("audit/component_map.yml")), Divergences.load(@root.join("audit/divergences.yml"))] + end + + def theme + @theme ||= Theme.compare( + themes_ts: ShadcnAudit.workspace_dir.join("ui/apps/v4/registry/themes.ts"), + rubyui_css: (@preview || @root).join("docs/app/assets/stylesheets/application.tailwind.css") + ) + end + + def url(side, theme) + template = (side == "rubyui") ? RubyuiPreview::URL_TEMPLATE : Sandbox::URL_TEMPLATE + template.sub("{folder}", @folder).sub("{light|dark}", theme) + end + + def observe_all(driver, states, selectors, slots) + shadcn_selectors = selectors.keys.to_h { |part| [part, %([data-slot="#{slots[part]}"])] } + THEMES.flat_map do |theme| + states.map do |state| + sides = SIDES.to_h do |side| + facts = driver.observe(url: url(side, theme), state: state, + selectors: (side == "rubyui") ? selectors : shadcn_selectors, + screenshot_path: @run_dir.join(shot(state, side, theme))) + facts = facts.merge("screenshot" => shot(state, side, theme)) if facts["reached"] + [side, facts] + end + {"state" => state, "theme" => theme}.merge(sides) + end + end + end + + def shot(state, side, theme) = "screens/#{@folder}-#{state}-#{side}-#{theme}.png" + + def probe_environment(driver) + SIDES.to_h { |side| [side, THEMES.to_h { |theme| [theme, driver.environment(url(side, theme))] }] } + end + + def environment_diffs(environment) + keys = [] + diffs = THEMES.flat_map do |theme| + r = environment["rubyui"][theme] + s = environment["shadcn"][theme] + (r.keys | s.keys).reject { |key| r[key] == s[key] }.map do |key| + keys << (FONT_KEYS.include?(key) ? "font" : key) + "#{key}: #{r[key]} vs #{s[key]} (#{theme})" + end + end + [diffs, keys.uniq] + end + + def pixel_diffs(observations) + observations.select { |o| o["rubyui"]["reached"] && o["shadcn"]["reached"] }.to_h do |o| + ["#{o["state"]}/#{o["theme"]}", pixel_diff(o)] + end + end + + def pixel_diff(observation) + diff = "screens/#{@folder}-#{observation["state"]}-diff-#{observation["theme"]}.png" + result = PixelDiff.compare(rubyui_png: @run_dir.join(observation["rubyui"]["screenshot"]).to_s, + shadcn_png: @run_dir.join(observation["shadcn"]["screenshot"]).to_s, diff_png: @run_dir.join(diff).to_s) + result.key?("diff") ? result.merge("diff" => diff) : result + rescue ChunkyPNG::Exception, Errno::ENOENT => e + {"error" => e.message} + end + + def screenshots(observations) + observations.select { |o| o["rubyui"]["reached"] && o["shadcn"]["reached"] }.map do |o| + {"state" => "#{o["state"]} (#{o["theme"]})", "rubyui" => o["rubyui"]["screenshot"], "shadcn" => o["shadcn"]["screenshot"]} + end + end + + def missing_selector_line(part) + {"dimension" => "Look", "status" => "inconclusive", "basis" => nil, "rubyui" => "", "shadcn" => "", + "source" => "", "note" => "#{part}: no selector in component_map.yml"} + end + end + end +end diff --git a/audit/test/cli_collect_test.rb b/audit/test/cli_collect_test.rb new file mode 100644 index 00000000..671d151e --- /dev/null +++ b/audit/test/cli_collect_test.rb @@ -0,0 +1,27 @@ +require "test_helper" + +class CliCollectTest < Minitest::Test + def run_cli(argv, **opts) + out = StringIO.new + status = ShadcnAudit::CLI.run(argv, out: out, **opts) + [status, JSON.parse(out.string)] + end + + def test_usage_without_run + status, result = run_cli(%w[collect dialog]) + assert_equal 1, status + assert_match(/usage: collect --run /, result["error"]) + end + + def test_missing_browser_is_one_error_object + factory = ->(audit_dir:, &block) { raise ShadcnAudit::Collect::BrowserError, "playwright not found: run `pnpm install && pnpm exec playwright install chromium` in audit/" } + Dir.mktmpdir do |dir| + FileUtils.mkdir_p(File.join(dir, "audit")) + File.write(File.join(dir, "audit/component_map.yml"), + "dialog:\n kind: modal\n parts: {DialogContent: dialog-content}\n selectors: {DialogContent: div}\n") + status, result = run_cli(["collect", "dialog", "--run", File.join(dir, "run")], driver_factory: factory, root: Pathname.new(dir)) + assert_equal 1, status + assert_includes result["error"], "pnpm exec playwright install chromium" + end + end +end diff --git a/audit/test/collect/runner_test.rb b/audit/test/collect/runner_test.rb new file mode 100644 index 00000000..c88ee454 --- /dev/null +++ b/audit/test/collect/runner_test.rb @@ -0,0 +1,107 @@ +require "test_helper" + +class CollectRunnerTest < Minitest::Test + Runner = ShadcnAudit::Collect::Runner + + # Canned driver: every observation reaches the state with one identical + # element per selector, except the states listed in `unreached`. + class FakeDriver + attr_reader :observed + + def initialize(unreached: []) + @unreached = unreached + @observed = [] + end + + def environment(url) = {"w" => 1280, "h" => 800, "font" => url.include?("3179") ? "Inter" : "Geist"} + + def observe(url:, state:, selectors:, screenshot_path:) + @observed << [url, state, selectors] + return {"reached" => false, "reason" => "boom", "screenshot" => nil, "parts" => {}} if @unreached.include?(state) + + ChunkyPNG::Image.new(2, 2, ChunkyPNG::Color::WHITE).save(screenshot_path.to_s) + parts = selectors.keys.to_h { |part| [part, [{"styles" => {"padding-top" => "8px"}, "aria" => "- button", "attrs" => {}}]] } + {"reached" => true, "reason" => nil, "screenshot" => screenshot_path.to_s, "parts" => parts} + end + end + + def setup + @tmp = Dir.mktmpdir + @root = File.join(@tmp, "root") + @run = File.join(@tmp, "run") + write_map(@root, "kind: modal\n parts: {Dialog: dialog, DialogContent: dialog-content, DialogMiddle: null}\n selectors: {DialogContent: \"dialog > div\"}") + @driver = FakeDriver.new + end + + def teardown = FileUtils.remove_entry(@tmp) + + def write_map(root, body) + FileUtils.mkdir_p(File.join(root, "audit")) + File.write(File.join(root, "audit/component_map.yml"), "dialog:\n shadcn: dialog\n #{body}\n") + File.write(File.join(root, "audit/divergences.yml"), "[]\n") + end + + def run_collect(preview: nil, driver: @driver) + factory = ->(audit_dir:, &block) { block.call(driver) } + Runner.new(folder: "dialog", run_dir: @run, preview: preview, root: Pathname.new(@root), + theme: {"diffs" => []}, driver_factory: factory).call + end + + def test_writes_collect_json_and_returns_same_object + result = run_collect + assert_equal JSON.parse(JSON.generate(result)), JSON.parse(File.read(File.join(@run, "collect.json"))) + assert_equal({"w" => 1280, "h" => 800}, result["viewport"]) + assert_includes result["screenshots"], {"state" => "open (light)", "rubyui" => "screens/dialog-open-rubyui-light.png", "shadcn" => "screens/dialog-open-shadcn-light.png"} + assert_equal ["font: Inter vs Geist (light)", "font: Inter vs Geist (dark)"], result["environment_diffs"] + assert(result["pixel_diffs"].key?("open/light")) + shadcn_selectors = @driver.observed.find { |url, _, _| url.include?("5179") }[2] + assert_equal({"DialogContent" => %([data-slot="dialog-content"])}, shadcn_selectors) + end + + def test_states_follow_kind + write_map(@root, "kind: form_control\n parts: {DialogContent: dialog-content}\n selectors: {DialogContent: \"div\"}") + assert_equal %w[closed focus], run_collect["states"].map { |o| o["state"] }.uniq + end + + def test_missing_selector_part_is_inconclusive + line = run_collect["lines"].find { |l| l["note"].include?("no selector") } + assert_equal ["Look", "inconclusive", "Dialog: no selector in component_map.yml"], line.values_at("dimension", "status", "note") + end + + def test_preview_map_wins_when_it_has_selectors + preview = File.join(@tmp, "preview") + write_map(preview, "kind: static\n parts: {DialogContent: dialog-content}\n selectors: {DialogContent: \"section\"}") + result = run_collect(preview: preview) + assert_equal "preview", result["map_source"] + assert_equal %w[closed], result["states"].map { |o| o["state"] }.uniq + end + + def test_falls_back_to_working_tree_map + preview = File.join(@tmp, "preview") + write_map(preview, "kind: static\n parts: {DialogContent: dialog-content}") + assert_equal "working tree", run_collect(preview: preview)["map_source"] + end + + def test_folder_without_selectors_is_an_error + write_map(@root, "kind: modal\n parts: {DialogContent: dialog-content}") + error = assert_raises(ArgumentError) { run_collect } + assert_equal "dialog has no selectors in audit/component_map.yml", error.message + end + + def test_unreadable_screenshot_only_fails_its_own_diff + driver = FakeDriver.new + def driver.observe(**kwargs) + super.tap { |facts| File.write(kwargs[:screenshot_path], "not a png") if kwargs[:state] == "open" } + end + diffs = run_collect(driver: driver)["pixel_diffs"] + assert diffs["open/light"].key?("error") + assert_equal 0, diffs["closed/light"]["pixels"] + end + + def test_driver_error_on_one_state_does_not_abort + result = run_collect(driver: FakeDriver.new(unreached: %w[open])) + assert File.exist?(File.join(@run, "collect.json")) + notes = result["lines"].select { |l| l["status"] == "inconclusive" }.map { |l| l["note"] } + assert_includes notes, "open/light, open/dark not reached on rubyui: boom" + end +end From 5494af59b6091362286cb713072375236d9d6e3d Mon Sep 17 00:00:00 2001 From: Cirdes Date: Sun, 1 Nov 2026 14:30:58 -0300 Subject: [PATCH 10/15] [Feature] shadcn-audit: selectors for mapped components Selectors for dialog, alert_dialog, sheet, command and form, checked against the rendered fixtures. A null selector marks a part shadcn renders no element for (Base UI roots), so it isn't compared. Verified on the live sandbox/preview: two collect runs on dialog give identical collect.json. Also: snapshot diffs show only the differing lines, and environment-explained differences share one line. Co-Authored-By: Claude Opus 5.5 --- audit/component_map.yml | 49 +++++++++++++++++++++++- audit/lib/shadcn_audit/collect/grader.rb | 44 +++++++++++++++++---- audit/lib/shadcn_audit/collect/runner.rb | 5 ++- audit/lib/shadcn_audit/component_map.rb | 3 +- audit/test/collect/grader_test.rb | 28 +++++++++++++- audit/test/collect/runner_test.rb | 7 ++++ audit/test/component_map_test.rb | 4 ++ 7 files changed, 127 insertions(+), 13 deletions(-) diff --git a/audit/component_map.yml b/audit/component_map.yml index f1321bea..c96a195c 100644 --- a/audit/component_map.yml +++ b/audit/component_map.yml @@ -2,7 +2,8 @@ # parts: RubyUI class => shadcn data-slot (null = no shadcn part). # props: RubyUI class => {shadcn prop => RubyUI kwarg (null = none)}; shadcn props only. # rubyui_only: RubyUI class => [kwargs with no shadcn counterpart]. className is never listed (all parts take class via attrs). -# selectors: RubyUI class => CSS selector of that part in the rendered fixture (fixed, so collect is deterministic). +# selectors: RubyUI class => CSS selector of that part in the rendered fixture (fixed, so collect is deterministic); +# null = not compared (shadcn renders no element for it, e.g. a Base UI root). # fixture_shadcn: other shadcn components the fixture renders (added to the sandbox, not audited here). dialog: shadcn: dialog @@ -17,6 +18,14 @@ dialog: DialogTitle: dialog-title DialogDescription: dialog-description DialogMiddle: null + selectors: + Dialog: null + DialogTrigger: '[data-ruby-ui--dialog-target="trigger"] > button' + DialogContent: 'dialog[data-ruby-ui--dialog-target="dialog"]' + DialogHeader: 'dialog[data-ruby-ui--dialog-target="dialog"] > div:first-child' + DialogFooter: 'dialog[data-ruby-ui--dialog-target="dialog"] > div:last-of-type' + DialogTitle: '[data-ruby-ui--dialog-target="title"]' + DialogDescription: '[data-ruby-ui--dialog-target="description"]' props: Dialog: {open: open} DialogContent: {showCloseButton: show_close_button} @@ -38,6 +47,16 @@ alert_dialog: AlertDialogDescription: alert-dialog-description AlertDialogAction: alert-dialog-action AlertDialogCancel: alert-dialog-cancel + selectors: + AlertDialog: null + AlertDialogTrigger: '[data-controller="ruby-ui--alert-dialog"] > [data-action] > button' + AlertDialogContent: '[role="alertdialog"]' + AlertDialogHeader: '[role="alertdialog"] > div:first-child' + AlertDialogFooter: '[role="alertdialog"] > div:last-child' + AlertDialogTitle: '[role="alertdialog"] > div:first-child > h2' + AlertDialogDescription: '[role="alertdialog"] > div:first-child > p' + AlertDialogAction: '[role="alertdialog"] > div:last-child > button:last-child' + AlertDialogCancel: '[role="alertdialog"] > div:last-child > button:first-child' props: AlertDialogContent: {size: null} AlertDialogCancel: {variant: variant, size: size} @@ -55,6 +74,14 @@ sheet: SheetTitle: sheet-title SheetDescription: sheet-description SheetMiddle: null + selectors: + Sheet: null + SheetTrigger: '[data-controller="ruby-ui--sheet"] > [data-action] > button' + SheetContent: '[data-ruby-ui--sheet-content-target="panel"]' + SheetHeader: '[data-ruby-ui--sheet-content-target="panel"] > div:first-child' + SheetFooter: '[data-ruby-ui--sheet-content-target="panel"] > div:last-of-type' + SheetTitle: '[data-ruby-ui--sheet-content-target="panel"] > div:first-child > h3' + SheetDescription: '[data-ruby-ui--sheet-content-target="panel"] > div:first-child > p' props: SheetContent: {side: side, showCloseButton: null} @@ -72,6 +99,15 @@ command: CommandDialog: dialog CommandDialogTrigger: null CommandDialogContent: dialog-content + selectors: + Command: '[data-ruby-ui--command-target="panel"] > div' + CommandInput: 'input[data-ruby-ui--command-target="input"]' + CommandList: '[data-ruby-ui--command-target="panel"] > div > div:has([data-ruby-ui--command-target="group"])' + CommandEmpty: '[data-ruby-ui--command-target="empty"]' + CommandGroup: '[data-ruby-ui--command-target="group"]' + CommandItem: '[data-ruby-ui--command-target="item"]' + CommandDialog: null + CommandDialogContent: '[data-ruby-ui--command-target="panel"]' props: CommandInput: {placeholder: placeholder} CommandGroup: {heading: title} @@ -98,6 +134,17 @@ form: FormFieldContent: field-content FormFieldTitle: field-label FormFieldSeparator: field-separator + selectors: + FormField: '[data-slot="field"]' + FormFieldLabel: '[data-slot="field-label"]' + FormFieldHint: '[data-slot="field-description"]' + FormFieldError: '[data-slot="field-error"]' + FormFieldSet: '[data-slot="field-set"]' + FormFieldLegend: '[data-slot="field-legend"]' + FormFieldGroup: '[data-slot="field-group"]' + FormFieldContent: '[data-slot="field-content"]' + FormFieldTitle: null + FormFieldSeparator: '[data-slot="field-separator"]' props: FormField: {orientation: orientation} FormFieldLegend: {variant: variant} diff --git a/audit/lib/shadcn_audit/collect/grader.rb b/audit/lib/shadcn_audit/collect/grader.rb index aedc7fec..08cd0856 100644 --- a/audit/lib/shadcn_audit/collect/grader.rb +++ b/audit/lib/shadcn_audit/collect/grader.rb @@ -51,17 +51,28 @@ def dimension_lines(dimension, observations) return [] if observations.empty? matching = [] + environment = {} lines = @parts.flat_map do |part| - part_lines = compare_part(dimension, part, observations) - matching << part if part_lines.empty? + part_lines, env_keys = compare_part(dimension, part, observations) + environment[part] = env_keys unless env_keys.empty? + matching << part if part_lines.empty? && env_keys.empty? part_lines end + lines << environment_line(dimension, environment) unless environment.empty? return lines if matching.empty? config = DIMENSIONS[dimension] lines << line(dimension, "pass", "#{matching.join(", ")}: #{config[:match_text]}", "same", config[:source]) end + # One line for every difference the environment explains, whatever the + # part: the cause is shared and already in environment_diffs. + def environment_line(dimension, environment) + keys = environment.values.flatten.uniq.sort + line(dimension, "inconclusive", "properties: #{keys.join(", ")}", "", "", + "#{environment.keys.join(", ")} — explained by environment diff: font") + end + def compare_part(dimension, part, observations) buckets = Hash.new { |h, k| h[k] = [] } mismatches = [] @@ -79,11 +90,13 @@ def compare_part(dimension, part, observations) sv = sf[key].to_s next if Normalize.value(rv) == Normalize.value(sv) - buckets[classify(dimension, part, key, rv, sv, o["theme"])] << {key: key, rubyui: rv, shadcn: sv, at: at} + shown = (key == "snapshot") ? snapshot_delta(rv, sv) : [rv, sv] + buckets[classify(dimension, part, key, rv, sv, o["theme"])] << {key: key, rubyui: shown[0], shadcn: shown[1], at: at} end end end - part_lines(dimension, part, buckets, mismatches) + env_keys = buckets.select { |(kind, _), _| kind == :environment }.values.flatten.map { |f| f[:key] }.uniq + [part_lines(dimension, part, buckets.reject { |(kind, _), _| kind == :environment }, mismatches), env_keys] end def facts(dimension, element) @@ -121,10 +134,10 @@ def part_lines(dimension, part, buckets, mismatches) buckets.select { |(kind, _), _| kind == :intentional }.sort.each do |(_, reason), found| lines << values_line(dimension, part, "pass", found, basis: "intentional", source: "audit/divergences.yml", note: reason) end - explained = buckets.select { |(kind, _), _| %i[theme environment].include?(kind) } + explained = buckets.select { |(kind, _), _| kind == :theme } unless explained.empty? - causes = explained.keys.sort.map { |kind, what| (kind == :theme) ? "theme token #{what}" : "environment diff: #{what}" }.uniq - lines << values_line(dimension, part, "inconclusive", explained.values.flatten, source: "", note: "explained by #{causes.join(", ")}") + tokens = explained.keys.map(&:last).uniq.sort + lines << values_line(dimension, part, "inconclusive", explained.values.flatten, source: "", note: "explained by theme token #{tokens.join(", ")}") end lines end @@ -132,7 +145,22 @@ def part_lines(dimension, part, buckets, mismatches) def values_line(dimension, part, status, found, source:, basis: nil, note: nil) side = ->(which) { found.map { |f| "#{f[:key]}: #{display(f[which])}" }.uniq.sort.join("; ") } at = found.map { |f| f[:at] }.uniq - line(dimension, status, side.call(:rubyui), side.call(:shadcn), source, note || "#{part} — #{at.join(", ")}", basis: basis) + line(dimension, status, side.call(:rubyui), side.call(:shadcn), source, "#{part} — #{note || at.join(", ")}", basis: basis) + end + + # Snapshot lines found on one side only, so the difference is visible + # instead of buried in a truncated tree. + def snapshot_delta(rv, sv) + r = rv.lines.map(&:strip).reject(&:empty?) + s = sv.lines.map(&:strip).reject(&:empty?) + [only_here(r, s), only_here(s, r)] + end + + def only_here(lines, other) + extra = lines - other + return "only here: #{extra.join(" | ")}" unless extra.empty? + + lines.empty? ? "" : "same lines, different order" end def display(value) diff --git a/audit/lib/shadcn_audit/collect/runner.rb b/audit/lib/shadcn_audit/collect/runner.rb index f015af3e..29e65bc3 100644 --- a/audit/lib/shadcn_audit/collect/runner.rb +++ b/audit/lib/shadcn_audit/collect/runner.rb @@ -24,11 +24,12 @@ def call map_source, map, divergences = load_config entry = map.entry(@folder) selectors = map.selectors(@folder) - raise ArgumentError, "#{@folder} has no selectors in audit/component_map.yml" if entry.nil? || selectors.empty? + raise ArgumentError, "#{@folder} has no selectors in audit/component_map.yml" if entry.nil? || selectors.compact.empty? slots = (entry["parts"] || {}).compact compared = slots.keys.select { |part| selectors.key?(part) } - unselected = slots.keys - compared + compared.select! { |part| selectors[part] } + unselected = slots.keys - selectors.keys FileUtils.mkdir_p(@run_dir.join("screens")) observations, environment = @driver_factory.call(audit_dir: @root.join("audit")) do |driver| diff --git a/audit/lib/shadcn_audit/component_map.rb b/audit/lib/shadcn_audit/component_map.rb index 1a38ecfc..3dd7329f 100644 --- a/audit/lib/shadcn_audit/component_map.rb +++ b/audit/lib/shadcn_audit/component_map.rb @@ -27,6 +27,7 @@ def entry(folder) = @entries[folder.to_s] def all_folders = @entries.keys # RubyUI part => CSS selector in the rendered fixture ({} when unset). + # A null selector means "not compared" (e.g. a Base UI root with no DOM). def selectors(folder) = (entry(folder) || {})["selectors"] || {} # Folder for a component path, only when it is in the map. @@ -81,7 +82,7 @@ def validate_selectors!(folder, entry) parts = entry["parts"] || {} selectors.each do |part, css| raise ConfigError, "component_map.yml: #{folder}: selectors.#{part} is not a mapped part" unless parts.key?(part) - next if css.is_a?(String) && !css.strip.empty? + next if css.nil? || (css.is_a?(String) && !css.strip.empty?) raise ConfigError, "component_map.yml: #{folder}: selectors.#{part} must be a non-empty CSS selector" end diff --git a/audit/test/collect/grader_test.rb b/audit/test/collect/grader_test.rb index 4be3a25f..bd78290e 100644 --- a/audit/test/collect/grader_test.rb +++ b/audit/test/collect/grader_test.rb @@ -109,7 +109,7 @@ def test_aria_snapshot_difference_is_warn s = side("Trigger" => [el({}, %(- button "Open"))]) lines = aria(grader(parts: %w[Trigger]).lines([obs("open", "light", r, s)])) assert_equal ["warn"], lines.map { |l| l["status"] } - assert lines[0]["rubyui"].start_with?("snapshot: - button") + assert_equal %(snapshot: only here: - button "Open" [expanded]), lines[0]["rubyui"] assert_equal Grader::ARIA_SOURCE, lines[0]["source"] end @@ -143,4 +143,30 @@ def test_unreached_state_yields_inconclusive_look_and_aria assert_equal [%w[Look inconclusive], %w[ARIA inconclusive]], lines.map { |l| [l["dimension"], l["status"]] } assert_equal "open/light, open/dark not reached on rubyui: no [data-audit-trigger]", lines[0]["note"] end + + def test_snapshot_difference_shows_only_differing_lines + common = %(- dialog "Edit":\n - heading "Edit" [level=2]\n - button "Close") + r = side("DialogContent" => [el({}, "#{common}:\n - img\n - text: Close")]) + s = side("DialogContent" => [el({}, common)]) + line = aria(grader(parts: %w[DialogContent]).lines([obs("open", "light", r, s)])).first + assert_equal %(snapshot: only here: - button "Close": | - img | - text: Close), line["rubyui"] + assert_equal %(snapshot: only here: - button "Close"), line["shadcn"] + end + + def test_theme_explained_note_names_the_part + diff = {"mode" => "light", "token" => "chart-1", "rubyui" => "red", "shadcn" => "blue"} + r = side("DialogContent" => [el("color" => "red")]) + s = side("DialogContent" => [el("color" => "blue")]) + line = look(grader(parts: %w[DialogContent], theme_diffs: [diff]).lines([obs("open", "light", r, s)])).first + assert_equal "DialogContent — explained by theme token chart-1", line["note"] + end + + def test_environment_explained_parts_share_one_line + r = side("DialogContent" => [el("font-size" => "14px")], "DialogTitle" => [el("width" => "10px")]) + s = side("DialogContent" => [el("font-size" => "16px")], "DialogTitle" => [el("width" => "12px")]) + lines = look(grader(env: ["font"]).lines([obs("open", "light", r, s)])) + assert_equal 1, lines.size + assert_equal ["inconclusive", "properties: font-size, width", "DialogContent, DialogTitle — explained by environment diff: font"], + lines[0].values_at("status", "rubyui", "note") + end end diff --git a/audit/test/collect/runner_test.rb b/audit/test/collect/runner_test.rb index c88ee454..aa97ea41 100644 --- a/audit/test/collect/runner_test.rb +++ b/audit/test/collect/runner_test.rb @@ -68,6 +68,13 @@ def test_missing_selector_part_is_inconclusive assert_equal ["Look", "inconclusive", "Dialog: no selector in component_map.yml"], line.values_at("dimension", "status", "note") end + def test_null_selector_part_is_skipped_silently + write_map(@root, "kind: modal\n parts: {Dialog: dialog, DialogContent: dialog-content}\n selectors: {Dialog: null, DialogContent: \"dialog > div\"}") + result = run_collect + refute(result["lines"].any? { |l| l["note"].include?("no selector") }) + refute(@driver.observed.any? { |_, _, selectors| selectors.key?("Dialog") }) + end + def test_preview_map_wins_when_it_has_selectors preview = File.join(@tmp, "preview") write_map(preview, "kind: static\n parts: {DialogContent: dialog-content}\n selectors: {DialogContent: \"section\"}") diff --git a/audit/test/component_map_test.rb b/audit/test/component_map_test.rb index 941d2fa6..ff3747d3 100644 --- a/audit/test/component_map_test.rb +++ b/audit/test/component_map_test.rb @@ -159,6 +159,10 @@ def test_selector_for_unknown_part_is_rejected assert_match(/selectors.Nope is not a mapped part/, error.message) end + def test_null_selector_means_not_compared + assert_equal({"DialogContent" => nil}, selector_map("DialogContent" => nil).selectors("dialog")) + end + def test_blank_selector_is_rejected error = assert_raises(ShadcnAudit::ConfigError) { selector_map("DialogContent" => " ") } assert_match(/selectors.DialogContent must be a non-empty CSS selector/, error.message) From 62b9a55a774f597cfce232c1a27cedb35d4241c2 Mon Sep 17 00:00:00 2001 From: Cirdes Date: Sun, 1 Nov 2026 14:31:38 -0300 Subject: [PATCH 11/15] [Feature] shadcn-audit: collect takes --shadcn-url/--rubyui-url for non-default ports Co-Authored-By: Claude Opus 5.5 --- audit/lib/shadcn_audit/cli.rb | 7 +++++-- audit/lib/shadcn_audit/collect/runner.rb | 6 +++--- audit/test/collect/runner_test.rb | 12 ++++++++++-- 3 files changed, 18 insertions(+), 7 deletions(-) diff --git a/audit/lib/shadcn_audit/cli.rb b/audit/lib/shadcn_audit/cli.rb index a766e969..c530ad8a 100644 --- a/audit/lib/shadcn_audit/cli.rb +++ b/audit/lib/shadcn_audit/cli.rb @@ -115,15 +115,18 @@ def self.comment(args, out:, shell: Shell.new) # Drives both fixtures in Chromium and grades the objective lines. def self.collect(args, out:, driver_factory: Collect::Driver.method(:open), root: ShadcnAudit.root) run_dir = preview = nil + urls = {} parser = OptionParser.new do |o| o.on("--run DIR") { |v| run_dir = v } o.on("--preview DIR") { |v| preview = v } + o.on("--shadcn-url TEMPLATE") { |v| urls[:shadcn_url] = v } + o.on("--rubyui-url TEMPLATE") { |v| urls[:rubyui_url] = v } end folder = parser.parse!(args).first - raise ArgumentError, "usage: collect --run [--preview ]" unless folder && run_dir + raise ArgumentError, "usage: collect --run [--preview ] [--shadcn-url T] [--rubyui-url T]" unless folder && run_dir Collect::Runner.new(folder: folder, run_dir: run_dir, preview: preview, root: root, - driver_factory: driver_factory).call + driver_factory: driver_factory, **urls).call end def self.workspace(args, out:) diff --git a/audit/lib/shadcn_audit/collect/runner.rb b/audit/lib/shadcn_audit/collect/runner.rb index 29e65bc3..8c9c2970 100644 --- a/audit/lib/shadcn_audit/collect/runner.rb +++ b/audit/lib/shadcn_audit/collect/runner.rb @@ -11,13 +11,14 @@ class Runner FONT_KEYS = %w[font fonts].freeze def initialize(folder:, run_dir:, preview: nil, root: ShadcnAudit.root, theme: nil, - driver_factory: Driver.method(:open)) + driver_factory: Driver.method(:open), shadcn_url: Sandbox::URL_TEMPLATE, rubyui_url: RubyuiPreview::URL_TEMPLATE) @folder = folder.to_s @run_dir = Pathname.new(run_dir) @preview = preview && Pathname.new(preview) @root = root @theme = theme @driver_factory = driver_factory + @templates = {"shadcn" => shadcn_url, "rubyui" => rubyui_url} end def call @@ -72,8 +73,7 @@ def theme end def url(side, theme) - template = (side == "rubyui") ? RubyuiPreview::URL_TEMPLATE : Sandbox::URL_TEMPLATE - template.sub("{folder}", @folder).sub("{light|dark}", theme) + @templates.fetch(side).sub("{folder}", @folder).sub("{light|dark}", theme) end def observe_all(driver, states, selectors, slots) diff --git a/audit/test/collect/runner_test.rb b/audit/test/collect/runner_test.rb index aa97ea41..9c793885 100644 --- a/audit/test/collect/runner_test.rb +++ b/audit/test/collect/runner_test.rb @@ -41,10 +41,18 @@ def write_map(root, body) File.write(File.join(root, "audit/divergences.yml"), "[]\n") end - def run_collect(preview: nil, driver: @driver) + def run_collect(preview: nil, driver: @driver, **urls) factory = ->(audit_dir:, &block) { block.call(driver) } Runner.new(folder: "dialog", run_dir: @run, preview: preview, root: Pathname.new(@root), - theme: {"diffs" => []}, driver_factory: factory).call + theme: {"diffs" => []}, driver_factory: factory, **urls).call + end + + def test_url_templates_can_point_at_other_ports + run_collect(shadcn_url: "http://localhost:5180/?c={folder}&theme={light|dark}", + rubyui_url: "http://localhost:3180/__audit/{folder}?theme={light|dark}") + urls = @driver.observed.map(&:first).uniq + assert_includes urls, "http://localhost:5180/?c=dialog&theme=light" + assert_includes urls, "http://localhost:3180/__audit/dialog?theme=dark" end def test_writes_collect_json_and_returns_same_object From 2fbdb929ea7235ac82fd943a35ce0cf8d510aed6 Mon Sep 17 00:00:00 2001 From: Cirdes Date: Sun, 1 Nov 2026 14:32:08 -0300 Subject: [PATCH 12/15] [Documentation] shadcn-audit: drive collect from the skill Co-Authored-By: Claude Opus 5.5 --- .claude/skills/shadcn-audit/SKILL.md | 48 +++++++++++++++---- .../skills/shadcn-audit/references/grading.md | 9 +++- .../shadcn-audit/references/interactions.md | 7 +++ audit/README.md | 4 +- 4 files changed, 58 insertions(+), 10 deletions(-) diff --git a/.claude/skills/shadcn-audit/SKILL.md b/.claude/skills/shadcn-audit/SKILL.md index 49fd2083..ddeb4fb8 100644 --- a/.claude/skills/shadcn-audit/SKILL.md +++ b/.claude/skills/shadcn-audit/SKILL.md @@ -42,7 +42,10 @@ an error as an empty result. Talk to the user in their language; - `gh` authenticated (`gh auth status`) with read access to `ruby-ui/ruby_ui`; posting the comment needs write access. - Node and pnpm (the shadcn sandbox and the docs assets). -- The chrome-devtools MCP server, for every browser step. +- Playwright's Chromium for `collect`: `cd audit && pnpm install && pnpm exec + playwright install chromium` (once per machine). +- The chrome-devtools MCP server, for the browser steps `collect` doesn't + cover (the kind-specific behavior sequences, animation, exploration). - Optional: Herdr. When `HERDR_ENV=1` the servers run in Herdr panes; otherwise as background shell processes (step 2.6). @@ -168,24 +171,53 @@ For each folder in `components`: list is ⏸ "map gap". - `divergences ` → registered differences (`dimension`, `rubyui`, `shadcn`, `reason`). A matching difference is ✅ basis `intentional`. + Entries with a structured `match` are applied by `collect` on its own; + prose-only entries you apply by hand. +- `collect --run --preview ` (add + `--shadcn-url`/`--rubyui-url` with the `url_template`s when step 2.6 picked + other ports). It drives both fixtures in headless Chromium at 1280×800, + light and dark, through the states the `kind` has (closed, open, focus, + after-esc; `form_control`: closed, focus; `static`: closed), and grades the + objective lines: computed styles per part (Look ❌ / ✅ grouped), the + accessibility snapshot plus `aria-expanded`/`aria-controls`/`aria-haspopup`/ + `aria-describedby` per part (ARIA ⚠️ / ✅ grouped), structured divergences + (✅ intentional), differences explained by a theme token or an environment + diff (⏸), unreachable states and selector count mismatches (⏸). It writes + `/collect.json` and `/screens/`, with a pixel diff per state as + evidence only. Append its `lines` and `screenshots` to the component and + copy its `environment_diffs` into the report. Running it twice on the same + SHAs gives the same `collect.json`. + - `{"error": "… has no selectors …"}` → the component has no `selectors` + in `component_map.yml`: tell the user to add them (RubyUI part → CSS + selector in the rendered fixture; `null` = shadcn renders no element for + that part) and fall back to the manual steps 4–5 for it. + - `{"error": "playwright not found …"}` / `"chromium failed to launch"` → + install as in Prerequisites and re-run. ### 4. Look -Light and dark. For each mapped part: shadcn classes from the DOM -(`[data-slot=]`) checked against `effective`; RubyUI classes from the -preview DOM via `evaluate_script`. RubyUI parts have no `data-slot`: find the -element from the part's Ruby source at `/gem/lib/ruby_ui//` -(tag, Stimulus target, class string). List the tokens missing and extra on each -side; settle look-alikes with computed styles. Variant props (size, side) are +`collect` already graded Look from computed styles. Review its lines instead +of re-collecting: explain each ❌ with the class tokens behind it (shadcn +`effective` from `reference` vs the RubyUI class string) in the `note`, and +settle the ⏸ it left (count mismatches, theme/environment explanations, +reference `warnings`) when the DOM lets you. Variant props (size, side) are compared against `variants`. +**Overrides.** You may change a `collect` grade only with a reason written in +that line's `note` (e.g. `text-align: left` vs `start` in LTR computes the same +→ ✅ with the note). Never drop a `collect` line. + Fixture-originated wrappers aren't findings (`references/grading.md`): Sheet body (shadcn `div.px-4` vs `SheetMiddle`), Dialog body (bare `

` vs `DialogMiddle`). ### 5. HTML, ARIA, Behavior, Animation -Run the set for the component's `kind` from `references/interactions.md` on +`collect` already covered the setup, the environment probe, the states +closed/open/focus/after-esc, the ARIA snapshot and attributes per part, and +the screenshots. Run the rest of the `kind`'s set from +`references/interactions.md` (focus trap order, Esc and focus return, outside +click, scroll lock, arrow keys, typeahead, Enter/Space, animation) on both URLs (`url_template`s; `{folder}` = folder, `{light|dark}` = theme), in the same viewport on both sides (aim for 1280×800; record the actual size from the environment probe), with chrome-devtools MCP. Open by clicking `[data-audit-trigger]`. diff --git a/.claude/skills/shadcn-audit/references/grading.md b/.claude/skills/shadcn-audit/references/grading.md index 9002c130..1b3b89ef 100644 --- a/.claude/skills/shadcn-audit/references/grading.md +++ b/.claude/skills/shadcn-audit/references/grading.md @@ -21,7 +21,7 @@ Behavior or Animation. When unsure, use ⏸ and say what's missing. | `basis` | Shows as | Rule | | --- | --- | --- | -| `intentional` | ✅ … — intentional | The difference matches an entry from `divergences ` (same `dimension`). `source` = `audit/divergences.yml`, `note` = its `reason`. | +| `intentional` | ✅ … — intentional | The difference matches an entry from `divergences ` (same `dimension`). `source` = `audit/divergences.yml`, `note` = its `reason`. Entries with a structured `match` are applied by `collect` automatically; prose-only entries by hand. | | `native` | ✅ … — native | RubyUI swaps shadcn's JS for a native feature **and keeps** what shadcn delivers: focus in and back, Esc, dismiss, inertness of the page behind. Labelling is markup, graded on the ARIA line, not here. Tag differences go in `note`. No registration needed. | | `spec` | ✅ … — follows spec | shadcn breaks a spec (stage 2) and RubyUI follows it. `source` = the spec URL#anchor. No registration needed. | @@ -52,6 +52,13 @@ still graded on its own and is still a finding. `native-html.md` does → ❌ "candidate for ", citing MDN. Native already used and nothing lost → ✅ basis `native`. +## Lines from `collect` + +`collect` grades Look (computed styles: ❌ / ✅) and ARIA (snapshot + attributes: +⚠️ / ✅) per part, by the same rules as below. Its ⏸ lines say why (state not +reached, selector count mismatch, theme token, environment diff). Keep every +`collect` line; to change a grade, write the reason in its `note`. + ## Per-dimension notes - **Look**: compare per part, light and dark. Class tokens that differ but diff --git a/.claude/skills/shadcn-audit/references/interactions.md b/.claude/skills/shadcn-audit/references/interactions.md index a7758a1e..9dd05f37 100644 --- a/.claude/skills/shadcn-audit/references/interactions.md +++ b/.claude/skills/shadcn-audit/references/interactions.md @@ -6,6 +6,13 @@ from the `.tsx` or the Ruby source. Run the same set on both URLs: - shadcn: `http://localhost:5179/?c=&theme=` (sandbox `url_template`) - RubyUI: `http://localhost:3179/__audit/?theme=` (preview `url_template`) +`collect` (SKILL.md step 3) already runs the setup below, the environment +probe, the states closed/open/focus/after-esc, the computed-style and ARIA +probes per part, and the screenshots, deterministically. Use this file for +what it doesn't cover: the per-`kind` sequences (focus trap order, Esc and +focus return, outside click, scroll lock, arrows, typeahead, Enter/Space), +animations, and any state `collect` reported as not reached. + ## Setup (once per page) 1. `new_page` (one per side), then `resize_page` both to the **same viewport**, diff --git a/audit/README.md b/audit/README.md index 24429ddc..ca5d399a 100644 --- a/audit/README.md +++ b/audit/README.md @@ -10,11 +10,13 @@ mise exec -- bundle exec exe/shadcn-audit workspace [--sha SHA] `workspace` keeps a sparse clone of `shadcn-ui/ui` in `.shadcn-audit/ui` (git-ignored) and checks out `main`, or the given SHA. +`collect --run

[--preview ] [--shadcn-url T] [--rubyui-url T]` drives both fixtures in headless Chromium (playwright-ruby-client) with the sandbox and preview servers running, and writes `/collect.json`: computed styles, accessibility snapshots, screenshots and pixel diffs per state, plus the objectively graded Look/ARIA lines. It needs `pnpm install && pnpm exec playwright install chromium` here once. Part selectors come from `selectors` in `component_map.yml`; a `match` key in `divergences.yml` lets `collect` apply that divergence on its own. + Tests use `ShadcnAudit::FakeShell` (`test/test_helper.rb`): stub canned stdout by argv or matcher, then assert on recorded calls. ## Prerequisites -Ruby 3.4 (via mise, as below, or an equivalent version manager; without mise, drop `mise exec --`), `gh` authenticated (`gh auth status`), Node and pnpm, and, for the browser steps, the chrome-devtools MCP. The audit runs its two dev servers in Herdr panes when `HERDR_ENV=1`, otherwise as background shell processes; either way it stops them at the end. `preview` boot commands use `mise exec --` only when `mise` is on PATH. +Ruby 3.4 (via mise, as below, or an equivalent version manager; without mise, drop `mise exec --`), `gh` authenticated (`gh auth status`), Node and pnpm, Playwright's Chromium for `collect` (above), and, for the remaining browser steps, the chrome-devtools MCP. The audit runs its two dev servers in Herdr panes when `HERDR_ENV=1`, otherwise as background shell processes; either way it stops them at the end. `preview` boot commands use `mise exec --` only when `mise` is on PATH. A component added to `component_map.yml` also needs fixtures in `fixtures//` (`content.yml`, `shadcn.tsx`, `rubyui.rb`). From 11d8758251d511dcf7a8d07207a45acb7ef0f2df Mon Sep 17 00:00:00 2001 From: Cirdes Date: Sun, 1 Nov 2026 14:57:49 -0300 Subject: [PATCH 13/15] [Bug Fix] shadcn-audit: collect review fixes (Codex) - A part never visible in any reached state is inconclusive, not pass. - ARIA snapshots and ARIA divergence matches compare exactly (CSS normalization erased hierarchy, case and numbers). - A JS error during click, Escape or collection voids the observation. - A failed environment probe is recorded instead of raising. - Computed `modal` fact (:modal or aria-modal) on every part. - checkVisibility honors visibility and opacity. - Animations still running after settle are fixed before styles are read: finite ones finish, infinite ones pause at their start. Co-Authored-By: Claude Opus 5.5 --- audit/lib/shadcn_audit/collect/driver.rb | 20 ++++++++++++----- audit/lib/shadcn_audit/collect/grader.rb | 18 +++++++++++++-- audit/lib/shadcn_audit/collect/probes.js | 14 ++++++++++-- audit/lib/shadcn_audit/collect/runner.rb | 3 +++ audit/lib/shadcn_audit/divergences.rb | 9 ++++++-- audit/test/collect/driver_smoke_test.rb | 24 ++++++++++++++++++++ audit/test/collect/grader_test.rb | 28 ++++++++++++++++++++++++ audit/test/collect/runner_test.rb | 8 +++++++ audit/test/divergences_test.rb | 9 ++++++++ audit/test/fixtures/collect/c.html | 4 ++++ audit/test/fixtures/collect/d.html | 19 ++++++++++++++++ 11 files changed, 145 insertions(+), 11 deletions(-) create mode 100644 audit/test/fixtures/collect/c.html create mode 100644 audit/test/fixtures/collect/d.html diff --git a/audit/lib/shadcn_audit/collect/driver.rb b/audit/lib/shadcn_audit/collect/driver.rb index e6e8b766..36ba85c9 100644 --- a/audit/lib/shadcn_audit/collect/driver.rb +++ b/audit/lib/shadcn_audit/collect/driver.rb @@ -45,21 +45,28 @@ def initialize(browser) @browser = browser end + # The probe hash, or {"error" => reason} when the page can't be read. def environment(url) - with_page(url) { |page| probe(page, "environment") } + result = with_page(url) { |page, _errors| probe(page, "environment") } + result.key?("reached") ? {"error" => result["reason"]} : result + rescue Playwright::Error => e + {"error" => e.message.lines.first.to_s.strip} end # SideFacts: {"reached", "reason", "screenshot", "parts" => {part => [element]}}. def observe(url:, state:, selectors:, screenshot_path:) raise ArgumentError, "unknown state #{state.inspect}" unless STATES.include?(state) - with_page(url) do |page| + with_page(url) do |page, errors| reason = reach(page, state) next unreached(reason) if reason probe(page, "settle") - page.screenshot(path: screenshot_path.to_s, animations: "disabled", caret: "hide") + probe(page, "freeze") parts = selectors.sort.to_h { |part, css| [part, elements(page, css)] } + page.screenshot(path: screenshot_path.to_s, animations: "disabled", caret: "hide") + next page_error(errors) unless errors.empty? + {"reached" => true, "reason" => nil, "screenshot" => screenshot_path.to_s, "parts" => parts} end rescue Playwright::Error => e @@ -74,9 +81,9 @@ def with_page(url) errors = [] page.on("pageerror", ->(error) { errors << error.message }) page.goto(url, waitUntil: "load", timeout: TIMEOUT_MS) - return unreached("page error: #{errors.first.lines.first.strip}") unless errors.empty? + return page_error(errors) unless errors.empty? - yield page + yield page, errors ensure context&.close end @@ -116,6 +123,9 @@ def probe(page, name, arg = nil) page.evaluate("(arg) => (#{PROBES})[#{name.to_json}](arg)", arg: arg) end + # A JS error at any point (load, click, Escape, collection) voids the observation. + def page_error(errors) = unreached("page error: #{errors.first.lines.first.to_s.strip}") + def unreached(reason) {"reached" => false, "reason" => reason, "screenshot" => nil, "parts" => {}} end diff --git a/audit/lib/shadcn_audit/collect/grader.rb b/audit/lib/shadcn_audit/collect/grader.rb index 08cd0856..8ae4fadd 100644 --- a/audit/lib/shadcn_audit/collect/grader.rb +++ b/audit/lib/shadcn_audit/collect/grader.rb @@ -5,7 +5,9 @@ module Collect class Grader LOOK_SOURCE = "computed styles (shadcn-audit collect)" ARIA_SOURCE = "accessibility tree (shadcn-audit collect)" - ARIA_ATTRS = %w[aria-expanded aria-controls aria-haspopup aria-describedby].freeze + # `modal` is computed (`:modal` or aria-modal="true"): the snapshot doesn't + # show modality, and a native is modal without the attribute. + ARIA_ATTRS = %w[aria-expanded aria-controls aria-haspopup aria-describedby modal].freeze STATE_ORDER = %w[closed open focus after-esc].freeze THEME_ORDER = %w[light dark].freeze FONT_PROPS = %w[font-family font-size font-weight line-height letter-spacing width height].freeze @@ -76,29 +78,41 @@ def environment_line(dimension, environment) def compare_part(dimension, part, observations) buckets = Hash.new { |h, k| h[k] = [] } mismatches = [] + compared = 0 observations.each do |o| at = "#{o["state"]}/#{o["theme"]}" r = Array(o["rubyui"]["parts"][part]) s = Array(o["shadcn"]["parts"][part]) next mismatches << "#{r.size}/#{s.size} visible elements in #{at}" if r.size != s.size + compared += r.size r.zip(s).each do |re, se| rf = facts(dimension, re) sf = facts(dimension, se) (rf.keys | sf.keys).sort.each do |key| rv = rf[key].to_s sv = sf[key].to_s - next if Normalize.value(rv) == Normalize.value(sv) + next if same?(dimension, rv, sv) shown = (key == "snapshot") ? snapshot_delta(rv, sv) : [rv, sv] buckets[classify(dimension, part, key, rv, sv, o["theme"])] << {key: key, rubyui: shown[0], shadcn: shown[1], at: at} end end end + if compared.zero? && mismatches.empty? + return [[line(dimension, "inconclusive", "", "", "", "#{part}: no visible element in any reached state")], []] + end + env_keys = buckets.select { |(kind, _), _| kind == :environment }.values.flatten.map { |f| f[:key] }.uniq [part_lines(dimension, part, buckets.reject { |(kind, _), _| kind == :environment }, mismatches), env_keys] end + # Look values are CSS: formatting noise isn't a difference. ARIA + # snapshots keep hierarchy, case and numbers, so they compare exactly. + def same?(dimension, rv, sv) + (dimension == "Look") ? Normalize.value(rv) == Normalize.value(sv) : rv == sv + end + def facts(dimension, element) return element["styles"] || {} if dimension == "Look" diff --git a/audit/lib/shadcn_audit/collect/probes.js b/audit/lib/shadcn_audit/collect/probes.js index 0e9430ce..04402180 100644 --- a/audit/lib/shadcn_audit/collect/probes.js +++ b/audit/lib/shadcn_audit/collect/probes.js @@ -12,6 +12,15 @@ dark: document.documentElement.classList.contains("dark"), }), + // Fixes what is still running after settle, so the styles read next don't + // depend on the instant they're read: finite animations/transitions jump to + // their end, infinite ones pause at their start. + freeze: () => { + document.getAnimations().filter(a => a.playState === "running").forEach(a => { + if (a.effect?.getComputedTiming().iterations === Infinity) { a.pause(); a.currentTime = 0 } else a.finish() + }) + }, + // Waits for running animations (capped) and two frames. settle: async () => { const done = Promise.all(document.getAnimations().map(a => a.finished.catch(() => null))) @@ -30,13 +39,14 @@ part: ({ selector, props, attrs }) => { const text = (id) => { const n = id && document.getElementById(id); return n ? n.textContent.trim().replace(/\s+/g, " ") : null } return [...document.querySelectorAll(selector)].map((e, index) => ({ e, index })) - .filter(({ e }) => e.checkVisibility()) + .filter(({ e }) => e.checkVisibility({ visibilityProperty: true, opacityProperty: true })) .map(({ e, index }) => { const s = getComputedStyle(e) const a = {} for (const name of attrs) { const v = e.getAttribute(name) - if (v === null) a[name] = null + if (name === "modal") a[name] = (e.matches(":modal") || v === "true" || e.getAttribute("aria-modal") === "true") ? "true" : null + else if (v === null) a[name] = null else if (name === "aria-controls") a[name] = v.split(/\s+/).every(id => document.getElementById(id)) ? "resolves" : "dangling" else if (name === "aria-describedby") a[name] = v.split(/\s+/).map(text).join(" ") else a[name] = v diff --git a/audit/lib/shadcn_audit/collect/runner.rb b/audit/lib/shadcn_audit/collect/runner.rb index 8c9c2970..0ec8ccee 100644 --- a/audit/lib/shadcn_audit/collect/runner.rb +++ b/audit/lib/shadcn_audit/collect/runner.rb @@ -103,6 +103,9 @@ def environment_diffs(environment) diffs = THEMES.flat_map do |theme| r = environment["rubyui"][theme] s = environment["shadcn"][theme] + failed = {"rubyui" => r, "shadcn" => s}.select { |_, probe| probe.key?("error") } + next failed.map { |side, probe| "environment probe failed on #{side} (#{theme}): #{probe["error"]}" } unless failed.empty? + (r.keys | s.keys).reject { |key| r[key] == s[key] }.map do |key| keys << (FONT_KEYS.include?(key) ? "font" : key) "#{key}: #{r[key]} vs #{s[key]} (#{theme})" diff --git a/audit/lib/shadcn_audit/divergences.rb b/audit/lib/shadcn_audit/divergences.rb index 8f6d26f8..bdf9ea5e 100644 --- a/audit/lib/shadcn_audit/divergences.rb +++ b/audit/lib/shadcn_audit/divergences.rb @@ -28,7 +28,7 @@ def match(component:, dimension:, part:, key:, rubyui:, shadcn:) m = e["match"] next false unless m && e["component"] == component.to_s && e["dimension"] == dimension.to_s m["part"] == part.to_s && (m["property"] || m["attribute"]) == key.to_s && - same?(m["rubyui"], rubyui) && same?(m["shadcn"], shadcn) + same?(dimension, m["rubyui"], rubyui) && same?(dimension, m["shadcn"], shadcn) end end @@ -52,7 +52,12 @@ def suggest(component:, dimension:, rubyui:, shadcn:) MATCH_KEYS = {"Look" => "property", "ARIA" => "attribute"}.freeze - def same?(expected, actual) = Collect::Normalize.value(expected) == Collect::Normalize.value(actual) + # Look values are normalized CSS; ARIA values (snapshots, attributes) are exact. + def same?(dimension, expected, actual) + return expected.to_s == actual.to_s unless dimension.to_s == "Look" + + Collect::Normalize.value(expected) == Collect::Normalize.value(actual) + end def validate_match!(label, entry) m = entry["match"] diff --git a/audit/test/collect/driver_smoke_test.rb b/audit/test/collect/driver_smoke_test.rb index cc05e562..b19dec5f 100644 --- a/audit/test/collect/driver_smoke_test.rb +++ b/audit/test/collect/driver_smoke_test.rb @@ -59,4 +59,28 @@ def test_states_against_local_fixtures assert_equal [1280, 800, 1], env.values_at("w", "h", "dpr") end end + + def test_errors_visibility_modality_and_infinite_animations + selectors = {"Dialog" => "dialog", "Ghost" => ".ghost", "Pulse" => ".pulse", "Slow" => ".slow"} + with_driver do |driver| + run = ->(name, state) { driver.observe(url: url(name), state: state, selectors: selectors, screenshot_path: File.join(@tmp, "#{name}-#{state}.png")) } + + thrown = run.call("c.html", "open") + refute thrown["reached"] + assert_includes thrown["reason"], "boom on click" + + open = run.call("d.html", "open") + assert open["reached"], open["reason"] + assert_equal "true", open["parts"]["Dialog"][0]["attrs"]["modal"] + assert_equal [], open["parts"]["Ghost"] + + closed = 2.times.map { run.call("d.html", "closed")["parts"] } + assert_equal ["1", "1"], closed.map { |parts| parts["Pulse"][0]["styles"]["opacity"] } + # A finite animation longer than the settle cap ends at its final frame, not its first. + assert_equal ["1", "1"], closed.map { |parts| parts["Slow"].dig(0, "styles", "opacity") } + + env = driver.environment(url("nope.html")) + assert env["error"], env.inspect + end + end end diff --git a/audit/test/collect/grader_test.rb b/audit/test/collect/grader_test.rb index bd78290e..92c20a87 100644 --- a/audit/test/collect/grader_test.rb +++ b/audit/test/collect/grader_test.rb @@ -169,4 +169,32 @@ def test_environment_explained_parts_share_one_line assert_equal ["inconclusive", "properties: font-size, width", "DialogContent, DialogTitle — explained by environment diff: font"], lines[0].values_at("status", "rubyui", "note") end + + def test_part_never_visible_is_inconclusive_not_pass + empty = side("DialogContent" => [], "DialogTitle" => [el]) + lines = grader.lines([obs("closed", "light", empty, empty)]) + notes = lines.map { |l| [l["dimension"], l["status"], l["note"]] } + assert_includes notes, ["Look", "inconclusive", "DialogContent: no visible element in any reached state"] + assert_includes notes, ["ARIA", "inconclusive", "DialogContent: no visible element in any reached state"] + refute(lines.any? { |l| l["status"] == "pass" && l["rubyui"].include?("DialogContent") }) + end + + def test_aria_snapshot_keeps_hierarchy + r = side("List" => [el({}, %(- list:\n - button "Two"))]) + s = side("List" => [el({}, %(- list:\n- button "Two"))]) + assert_equal ["warn"], aria(grader(parts: %w[List]).lines([obs("open", "light", r, s)])).map { |l| l["status"] } + end + + def test_aria_snapshot_keeps_case_and_numbers + r = side("Item" => [el({}, %(- button "Page 1.50"))]) + s = side("Item" => [el({}, %(- button "page 1.5"))]) + assert_equal ["warn"], aria(grader(parts: %w[Item]).lines([obs("open", "light", r, s)])).map { |l| l["status"] } + end + + def test_modal_fact_difference_is_warn + r = side("Content" => [el({}, "- dialog", {"modal" => "true"})]) + s = side("Content" => [el({}, "- dialog", {"modal" => nil})]) + line = aria(grader(parts: %w[Content]).lines([obs("open", "light", r, s)])).first + assert_equal ["warn", "modal: true", "modal: (none)"], line.values_at("status", "rubyui", "shadcn") + end end diff --git a/audit/test/collect/runner_test.rb b/audit/test/collect/runner_test.rb index 9c793885..ce0b82a5 100644 --- a/audit/test/collect/runner_test.rb +++ b/audit/test/collect/runner_test.rb @@ -119,4 +119,12 @@ def test_driver_error_on_one_state_does_not_abort notes = result["lines"].select { |l| l["status"] == "inconclusive" }.map { |l| l["note"] } assert_includes notes, "open/light, open/dark not reached on rubyui: boom" end + + def test_failed_environment_probe_is_recorded_not_raised + driver = FakeDriver.new + def driver.environment(url) = url.include?("3179") ? {"error" => "net::ERR_CONNECTION_REFUSED"} : super + result = run_collect(driver: driver) + assert_includes result["environment_diffs"], "environment probe failed on rubyui (light): net::ERR_CONNECTION_REFUSED" + assert File.exist?(File.join(@run, "collect.json")) + end end diff --git a/audit/test/divergences_test.rb b/audit/test/divergences_test.rb index 44fb363a..1809aa34 100644 --- a/audit/test/divergences_test.rb +++ b/audit/test/divergences_test.rb @@ -138,4 +138,13 @@ def test_match_needs_all_keys def test_existing_registry_still_loads ShadcnAudit::Divergences.load(ShadcnAudit.root.join("audit/divergences.yml")) end + + def test_aria_match_is_exact + entry = match_entry({"part" => "Item", "attribute" => "snapshot", "rubyui" => %(- button "Two"), "shadcn" => "- button"}) + .map { |e| e.merge("dimension" => "ARIA") } + divergences = ShadcnAudit::Divergences.new(entry) + found = ->(rubyui) { divergences.match(component: "dialog", dimension: "ARIA", part: "Item", key: "snapshot", rubyui: rubyui, shadcn: "- button") } + refute_nil found.call(%(- button "Two")) + assert_nil found.call(%(- button "two")) + end end diff --git a/audit/test/fixtures/collect/c.html b/audit/test/fixtures/collect/c.html new file mode 100644 index 00000000..dd4dc3c5 --- /dev/null +++ b/audit/test/fixtures/collect/c.html @@ -0,0 +1,4 @@ + + + + diff --git a/audit/test/fixtures/collect/d.html b/audit/test/fixtures/collect/d.html new file mode 100644 index 00000000..135a4481 --- /dev/null +++ b/audit/test/fixtures/collect/d.html @@ -0,0 +1,19 @@ + + + + + + + +

Modal

 + +
hidden by opacity
+
pulsing
+
slow finite fade-in
+ + From 18f537b88777f4687fb68ef71e665b9a0f2212bc Mon Sep 17 00:00:00 2001 From: Cirdes Date: Sun, 1 Nov 2026 15:20:12 -0300 Subject: [PATCH 14/15] [Bug Fix] shadcn-audit: address Cubic review on collect - Wait for [data-audit-fixture] before reaching a state (client-mounted fixtures could render after load) and for document.fonts.ready. - Snapshot deltas keep indentation and repeated nodes. - aria-controls/aria-describedby trim their IDs; a missing describedby target is "dangling". - Theme is computed before the browser opens. - Normalize maps -0 to 0; selectors only on parts with a shadcn slot; a divergence match uses only its dimension's key. - Tests pin the pixel tolerance boundary, alpha, and non-mapping selectors. - Docs: pnpm install per checkout, pixel diffs only for reached states, manual fallback without selectors, stricter override rule, expected count and ARIA contract in the design and plan. Co-Authored-By: Claude Opus 5.5 --- .claude/skills/shadcn-audit/SKILL.md | 25 +++++++++++++------ .../skills/shadcn-audit/references/grading.md | 5 +++- audit/docs/2026-11-01-collect-design.md | 19 ++++++++------ audit/docs/2026-11-01-collect-plan.md | 4 +-- audit/lib/shadcn_audit/cli.rb | 4 +-- audit/lib/shadcn_audit/collect/driver.rb | 8 ++++++ audit/lib/shadcn_audit/collect/grader.rb | 9 ++++--- audit/lib/shadcn_audit/collect/normalize.rb | 2 +- audit/lib/shadcn_audit/collect/probes.js | 7 +++--- audit/lib/shadcn_audit/collect/runner.rb | 3 ++- audit/lib/shadcn_audit/component_map.rb | 2 +- audit/lib/shadcn_audit/divergences.rb | 4 ++- audit/test/cli_collect_test.rb | 3 ++- audit/test/collect/driver_smoke_test.rb | 10 ++++++++ audit/test/collect/grader_test.rb | 19 ++++++++++++-- audit/test/collect/normalize_test.rb | 4 +++ audit/test/collect/pixel_diff_test.rb | 11 ++++++++ audit/test/collect/runner_test.rb | 7 ++++++ audit/test/component_map_test.rb | 11 ++++++++ audit/test/divergences_test.rb | 6 +++++ audit/test/fixtures/collect/a.html | 2 +- audit/test/fixtures/collect/b.html | 2 +- audit/test/fixtures/collect/c.html | 2 +- audit/test/fixtures/collect/d.html | 2 +- audit/test/fixtures/collect/e.html | 11 ++++++++ 25 files changed, 146 insertions(+), 36 deletions(-) create mode 100644 audit/test/fixtures/collect/e.html diff --git a/.claude/skills/shadcn-audit/SKILL.md b/.claude/skills/shadcn-audit/SKILL.md index ddeb4fb8..214a6a94 100644 --- a/.claude/skills/shadcn-audit/SKILL.md +++ b/.claude/skills/shadcn-audit/SKILL.md @@ -42,8 +42,10 @@ an error as an empty result. Talk to the user in their language; - `gh` authenticated (`gh auth status`) with read access to `ruby-ui/ruby_ui`; posting the comment needs write access. - Node and pnpm (the shadcn sandbox and the docs assets). -- Playwright's Chromium for `collect`: `cd audit && pnpm install && pnpm exec - playwright install chromium` (once per machine). +- Playwright for `collect`: `cd audit && pnpm install` in every checkout or + worktree (it installs `audit/node_modules`), then `pnpm exec playwright + install chromium` (the browser download is shared by every checkout on the + machine). - The chrome-devtools MCP server, for the browser steps `collect` doesn't cover (the kind-specific behavior sequences, animation, exploration). - Optional: Herdr. When `HERDR_ENV=1` the servers run in Herdr panes; @@ -184,13 +186,20 @@ For each folder in `components`: (✅ intentional), differences explained by a theme token or an environment diff (⏸), unreachable states and selector count mismatches (⏸). It writes `/collect.json` and `/screens/`, with a pixel diff per state as - evidence only. Append its `lines` and `screenshots` to the component and + evidence only, and only for states reached on both sides. Append its `lines` and `screenshots` to the component and copy its `environment_diffs` into the report. Running it twice on the same SHAs gives the same `collect.json`. - `{"error": "… has no selectors …"}` → the component has no `selectors` in `component_map.yml`: tell the user to add them (RubyUI part → CSS selector in the rendered fixture; `null` = shadcn renders no element for - that part) and fall back to the manual steps 4–5 for it. + that part). Until they exist, collect that component by hand with the + chrome-devtools MCP: for each mapped part, the shadcn classes from + `[data-slot=]` against `effective`, the RubyUI element found from + its Ruby source at `/gem/lib/ruby_ui//` (tag, + Stimulus target, class string), look-alikes settled with + `getComputedStyle`; then the whole set for its `kind` in + `references/interactions.md`, including the Setup, the states and the + screenshots. - `{"error": "playwright not found …"}` / `"chromium failed to launch"` → install as in Prerequisites and re-run. @@ -203,9 +212,11 @@ settle the ⏸ it left (count mismatches, theme/environment explanations, reference `warnings`) when the DOM lets you. Variant props (size, side) are compared against `variants`. -**Overrides.** You may change a `collect` grade only with a reason written in -that line's `note` (e.g. `text-align: left` vs `start` in LTR computes the same -→ ✅ with the note). Never drop a `collect` line. +**Overrides.** Change a `collect` grade only with a verified reason that the +rules in `references/grading.md` allow, written in that line's `note` (e.g. +`text-align: left` vs `start` in LTR renders the same → ✅ with the note). A ✅ +with a basis still needs its usual route: a registered divergence, a native +replacement or the spec. Never drop a `collect` line. Fixture-originated wrappers aren't findings (`references/grading.md`): Sheet body (shadcn `div.px-4` vs `SheetMiddle`), Dialog body (bare `

` vs diff --git a/.claude/skills/shadcn-audit/references/grading.md b/.claude/skills/shadcn-audit/references/grading.md index 1b3b89ef..1b69047e 100644 --- a/.claude/skills/shadcn-audit/references/grading.md +++ b/.claude/skills/shadcn-audit/references/grading.md @@ -57,7 +57,10 @@ still graded on its own and is still a finding. `collect` grades Look (computed styles: ❌ / ✅) and ARIA (snapshot + attributes: ⚠️ / ✅) per part, by the same rules as below. Its ⏸ lines say why (state not reached, selector count mismatch, theme token, environment diff). Keep every -`collect` line; to change a grade, write the reason in its `note`. +`collect` line. Change a grade only with a verified reason that the rules +below allow (e.g. values that render the same), written in its `note`; a ✅ +with a basis still needs its usual route (registered divergence, native, spec). +Otherwise keep the grade. ## Per-dimension notes diff --git a/audit/docs/2026-11-01-collect-design.md b/audit/docs/2026-11-01-collect-design.md index 8b00ab2b..d9435af4 100644 --- a/audit/docs/2026-11-01-collect-design.md +++ b/audit/docs/2026-11-01-collect-design.md @@ -123,10 +123,13 @@ judgment, so: rule 4, so the old flow's verdicts are preserved. The comparison is on the computed accessibility tree (`ariaSnapshot`: role, name, states), so a native `

` vs `div[role=dialog]` is not a difference. On top of that, an - attribute probe checks the APG attributes the old flow read - (`aria-expanded`, `aria-controls`, `aria-haspopup`, `aria-modal`, - `aria-labelledby`/`aria-describedby` resolved to their text). ID-valued - attributes are compared by what they resolve to, never by the raw ID. + attribute probe checks `aria-expanded`, `aria-controls` (resolves or + dangling), `aria-haspopup`, `aria-describedby` (resolved text) and a + computed `modal` fact (`:modal` or `aria-modal="true"`, so a native + `` counts). `aria-labelledby` isn't read raw: the accessible name it + produces is already in the snapshot. ID-valued attributes are compared by + what they resolve to, never by the raw ID. Snapshots compare exactly + (hierarchy, case, numbers); only Look values are normalized. - **Line granularity follows `grading.md`.** One Look line per differing part, with all matching parts grouped into a single `pass` line; same for ARIA. State and theme where the difference shows up go into `note` @@ -141,9 +144,11 @@ judgment, so: - **Selectors come from the audited head.** `collect` loads `component_map.yml` from the `preview` checkout (the audited SHA), falling back to the working tree only when the head has no selectors yet, and says - which one it used. Each selector must match exactly the expected number of - elements (one, or N for repeated parts such as accordion items); otherwise - that part is `inconclusive` ("selector matched 0/3 elements"). A PR that + which one it used. The expected count is the shadcn side's: the RubyUI + selector must match as many visible elements as `[data-slot=]` does + in the same state (one, or N for repeated parts such as accordion items); + otherwise that part is `inconclusive` ("selector matched 0/3 visible + elements"). A part never visible on either side is `inconclusive` too. A PR that changes a part's markup updates its selector in the same PR. ## Shape of the output (guidance, not a contract) diff --git a/audit/docs/2026-11-01-collect-plan.md b/audit/docs/2026-11-01-collect-plan.md index becd29f7..4fa9ddd5 100644 --- a/audit/docs/2026-11-01-collect-plan.md +++ b/audit/docs/2026-11-01-collect-plan.md @@ -134,10 +134,10 @@ Output per part (in `parts` order), at most one line per non-empty bucket, in th - Test: `audit/test/collect/grader_test.rb` **Interfaces:** -- Produces: `Grader#lines` also emits ARIA lines and unreachable lines. `Grader::ARIA_ATTRS = %w[aria-expanded aria-controls aria-haspopup aria-describedby]` (the driver fills `attrs` with exactly these keys; `aria-controls` is reported as `"resolves"`/`"dangling"`/nil, `aria-describedby` as the referenced text or nil). +- Produces: `Grader#lines` also emits ARIA lines and unreachable lines. `Grader::ARIA_ATTRS = %w[aria-expanded aria-controls aria-haspopup aria-describedby modal]` (`modal` added in review: computed `:modal` or `aria-modal="true"`; `aria-labelledby` is covered by the name in the snapshot; the driver fills `attrs` with exactly these keys; `aria-controls` is reported as `"resolves"`/`"dangling"`/nil, `aria-describedby` as the referenced text or nil). ARIA rules, per part, same count/skip logic as Look: -- `aria` (Playwright aria snapshot of the element) differs after `Normalize.value` → diff with key `snapshot`. +- `aria` (Playwright aria snapshot of the element) differs, compared exactly (hierarchy, case and numbers kept; `Normalize.value` is for Look only) → diff with key `snapshot`. - Each `ARIA_ATTRS` value differs (incl. present vs nil) → diff with key = attribute. - `divergences.match(dimension: "ARIA", part:, key:, …)` → `intentional`; theme/environment never apply to ARIA. - Output like Look but `diff` bucket → `warn`; `rubyui`/`shadcn` = `": "`, snapshots trimmed to 200 chars; grouped `pass` line `": accessibility tree matches"`, source `ARIA_SOURCE`. diff --git a/audit/lib/shadcn_audit/cli.rb b/audit/lib/shadcn_audit/cli.rb index c530ad8a..1d577087 100644 --- a/audit/lib/shadcn_audit/cli.rb +++ b/audit/lib/shadcn_audit/cli.rb @@ -113,7 +113,7 @@ def self.comment(args, out:, shell: Shell.new) end # Drives both fixtures in Chromium and grades the objective lines. - def self.collect(args, out:, driver_factory: Collect::Driver.method(:open), root: ShadcnAudit.root) + def self.collect(args, out:, driver_factory: Collect::Driver.method(:open), root: ShadcnAudit.root, theme: nil) run_dir = preview = nil urls = {} parser = OptionParser.new do |o| @@ -126,7 +126,7 @@ def self.collect(args, out:, driver_factory: Collect::Driver.method(:open), root raise ArgumentError, "usage: collect --run [--preview ] [--shadcn-url T] [--rubyui-url T]" unless folder && run_dir Collect::Runner.new(folder: folder, run_dir: run_dir, preview: preview, root: root, - driver_factory: driver_factory, **urls).call + theme: theme, driver_factory: driver_factory, **urls).call end def self.workspace(args, out:) diff --git a/audit/lib/shadcn_audit/collect/driver.rb b/audit/lib/shadcn_audit/collect/driver.rb index 36ba85c9..fdd96c36 100644 --- a/audit/lib/shadcn_audit/collect/driver.rb +++ b/audit/lib/shadcn_audit/collect/driver.rb @@ -8,6 +8,7 @@ class Driver STATES = %w[closed open focus after-esc].freeze VIEWPORT = {width: 1280, height: 800}.freeze TIMEOUT_MS = 15_000 + FIXTURE = "[data-audit-fixture]" MAX_TABS = 20 INSTALL_HINT = "run `pnpm install && pnpm exec playwright install chromium` in audit/" SIDES = %w[top right bottom left].freeze @@ -83,6 +84,13 @@ def with_page(url) page.goto(url, waitUntil: "load", timeout: TIMEOUT_MS) return page_error(errors) unless errors.empty? + # Client-rendered fixtures (the React sandbox) can mount after load. + begin + page.wait_for_selector(FIXTURE, timeout: TIMEOUT_MS) + rescue Playwright::TimeoutError + return unreached("fixture never rendered (no #{FIXTURE})") + end + yield page, errors ensure context&.close diff --git a/audit/lib/shadcn_audit/collect/grader.rb b/audit/lib/shadcn_audit/collect/grader.rb index 8ae4fadd..bae8a39c 100644 --- a/audit/lib/shadcn_audit/collect/grader.rb +++ b/audit/lib/shadcn_audit/collect/grader.rb @@ -165,13 +165,16 @@ def values_line(dimension, part, status, found, source:, basis: nil, note: nil) # Snapshot lines found on one side only, so the difference is visible # instead of buried in a truncated tree. def snapshot_delta(rv, sv) - r = rv.lines.map(&:strip).reject(&:empty?) - s = sv.lines.map(&:strip).reject(&:empty?) + r = rv.lines.map(&:rstrip).reject(&:empty?) + s = sv.lines.map(&:rstrip).reject(&:empty?) [only_here(r, s), only_here(s, r)] end + # Multiset difference: a node repeated more often on one side still shows, + # and indentation stays, so a nesting change isn't hidden. def only_here(lines, other) - extra = lines - other + remaining = other.tally + extra = lines.reject { |line| remaining[line].to_i.positive? && (remaining[line] -= 1) } return "only here: #{extra.join(" | ")}" unless extra.empty? lines.empty? ? "" : "same lines, different order" diff --git a/audit/lib/shadcn_audit/collect/normalize.rb b/audit/lib/shadcn_audit/collect/normalize.rb index e029c750..11fcd38b 100644 --- a/audit/lib/shadcn_audit/collect/normalize.rb +++ b/audit/lib/shadcn_audit/collect/normalize.rb @@ -6,7 +6,7 @@ module Normalize def self.value(value) value.to_s.strip.downcase.gsub(/\s+/, " ") - .gsub(DECIMAL) { |n| format("%.2f", n.to_f).sub(/\.?0+\z/, "") } + .gsub(DECIMAL) { |n| format("%.2f", n.to_f).sub(/\.?0+\z/, "").sub(/\A-0\z/, "0") } end end end diff --git a/audit/lib/shadcn_audit/collect/probes.js b/audit/lib/shadcn_audit/collect/probes.js index 04402180..476444a8 100644 --- a/audit/lib/shadcn_audit/collect/probes.js +++ b/audit/lib/shadcn_audit/collect/probes.js @@ -2,7 +2,7 @@ // entry is a function expression; the driver picks one by name. ({ // Same facts on both sides (interactions.md "Environment probe"). - environment: () => ({ + environment: async () => (await document.fonts.ready, { w: innerWidth, h: innerHeight, dpr: devicePixelRatio, font: getComputedStyle(document.body).fontFamily, fonts: [...new Set([...document.fonts].filter(f => f.status === "loaded").map(f => f.family))].sort(), @@ -23,6 +23,7 @@ // Waits for running animations (capped) and two frames. settle: async () => { + await document.fonts.ready const done = Promise.all(document.getAnimations().map(a => a.finished.catch(() => null))) await Promise.race([done, new Promise(r => setTimeout(r, 2000))]) await new Promise(r => requestAnimationFrame(() => requestAnimationFrame(r))) @@ -47,8 +48,8 @@ const v = e.getAttribute(name) if (name === "modal") a[name] = (e.matches(":modal") || v === "true" || e.getAttribute("aria-modal") === "true") ? "true" : null else if (v === null) a[name] = null - else if (name === "aria-controls") a[name] = v.split(/\s+/).every(id => document.getElementById(id)) ? "resolves" : "dangling" - else if (name === "aria-describedby") a[name] = v.split(/\s+/).map(text).join(" ") + else if (name === "aria-controls") a[name] = v.trim().split(/\s+/).every(id => document.getElementById(id)) ? "resolves" : "dangling" + else if (name === "aria-describedby") a[name] = v.trim().split(/\s+/).map(id => text(id) ?? "dangling").join(" ") else a[name] = v } return { index, styles: Object.fromEntries(props.map(p => [p, s.getPropertyValue(p)])), attrs: a } diff --git a/audit/lib/shadcn_audit/collect/runner.rb b/audit/lib/shadcn_audit/collect/runner.rb index 0ec8ccee..22f8a647 100644 --- a/audit/lib/shadcn_audit/collect/runner.rb +++ b/audit/lib/shadcn_audit/collect/runner.rb @@ -32,13 +32,14 @@ def call compared.select! { |part| selectors[part] } unselected = slots.keys - selectors.keys FileUtils.mkdir_p(@run_dir.join("screens")) + theme_diffs = theme["diffs"] # before the browser: a broken theme source fails fast observations, environment = @driver_factory.call(audit_dir: @root.join("audit")) do |driver| [observe_all(driver, STATES_BY_KIND[entry["kind"]], selectors.slice(*compared), slots), probe_environment(driver)] end environment_diffs, diff_keys = environment_diffs(environment) grader = Grader.new(folder: @folder, parts: compared, divergences: divergences, - theme_diffs: theme["diffs"], environment_diff_keys: diff_keys) + theme_diffs: theme_diffs, environment_diff_keys: diff_keys) lines = unselected.map { |part| missing_selector_line(part) } + grader.lines(observations) result = { diff --git a/audit/lib/shadcn_audit/component_map.rb b/audit/lib/shadcn_audit/component_map.rb index 3dd7329f..1ee7b003 100644 --- a/audit/lib/shadcn_audit/component_map.rb +++ b/audit/lib/shadcn_audit/component_map.rb @@ -81,7 +81,7 @@ def validate_selectors!(folder, entry) parts = entry["parts"] || {} selectors.each do |part, css| - raise ConfigError, "component_map.yml: #{folder}: selectors.#{part} is not a mapped part" unless parts.key?(part) + raise ConfigError, "component_map.yml: #{folder}: selectors.#{part} is not a mapped part" unless parts[part] next if css.nil? || (css.is_a?(String) && !css.strip.empty?) raise ConfigError, "component_map.yml: #{folder}: selectors.#{part} must be a non-empty CSS selector" diff --git a/audit/lib/shadcn_audit/divergences.rb b/audit/lib/shadcn_audit/divergences.rb index bdf9ea5e..2288ab14 100644 --- a/audit/lib/shadcn_audit/divergences.rb +++ b/audit/lib/shadcn_audit/divergences.rb @@ -65,7 +65,9 @@ def validate_match!(label, entry) raise ConfigError, "#{label}: match is only for Look or ARIA" unless MATCH_KEYS.key?(entry["dimension"]) key = MATCH_KEYS[entry["dimension"]] - raise ConfigError, "#{label}: #{entry["dimension"]} match needs #{key}" if m.is_a?(Hash) && !m.key?(key) && m.key?((MATCH_KEYS.values - [key]).first) + other = (MATCH_KEYS.values - [key]).first + raise ConfigError, "#{label}: #{entry["dimension"]} match needs #{key}" if m.is_a?(Hash) && !m.key?(key) && m.key?(other) + raise ConfigError, "#{label}: #{entry["dimension"]} match must use #{key} only" if m.is_a?(Hash) && m.key?(other) complete = m.is_a?(Hash) && %w[part rubyui shadcn].all? { |k| m.key?(k) && !m[k].nil? } && !m[key].to_s.strip.empty? raise ConfigError, "#{label}: match needs part, property|attribute, rubyui and shadcn" unless complete end diff --git a/audit/test/cli_collect_test.rb b/audit/test/cli_collect_test.rb index 671d151e..976aafa4 100644 --- a/audit/test/cli_collect_test.rb +++ b/audit/test/cli_collect_test.rb @@ -19,7 +19,8 @@ def test_missing_browser_is_one_error_object FileUtils.mkdir_p(File.join(dir, "audit")) File.write(File.join(dir, "audit/component_map.yml"), "dialog:\n kind: modal\n parts: {DialogContent: dialog-content}\n selectors: {DialogContent: div}\n") - status, result = run_cli(["collect", "dialog", "--run", File.join(dir, "run")], driver_factory: factory, root: Pathname.new(dir)) + status, result = run_cli(["collect", "dialog", "--run", File.join(dir, "run")], driver_factory: factory, root: Pathname.new(dir), + theme: {"diffs" => []}) assert_equal 1, status assert_includes result["error"], "pnpm exec playwright install chromium" end diff --git a/audit/test/collect/driver_smoke_test.rb b/audit/test/collect/driver_smoke_test.rb index b19dec5f..b8bea47f 100644 --- a/audit/test/collect/driver_smoke_test.rb +++ b/audit/test/collect/driver_smoke_test.rb @@ -83,4 +83,14 @@ def test_errors_visibility_modality_and_infinite_animations assert env["error"], env.inspect end end + + def test_waits_for_client_mounted_fixture_and_resolves_id_refs + with_driver do |driver| + open = driver.observe(url: url("e.html"), state: "open", selectors: {"Trigger" => "[data-audit-trigger]"}, + screenshot_path: File.join(@tmp, "e.png")) + assert open["reached"], open["reason"] + attrs = open["parts"]["Trigger"][0]["attrs"] + assert_equal({"aria-controls" => "resolves", "aria-describedby" => "dangling"}, attrs.slice("aria-controls", "aria-describedby")) + end + end end diff --git a/audit/test/collect/grader_test.rb b/audit/test/collect/grader_test.rb index 92c20a87..61421498 100644 --- a/audit/test/collect/grader_test.rb +++ b/audit/test/collect/grader_test.rb @@ -149,8 +149,8 @@ def test_snapshot_difference_shows_only_differing_lines r = side("DialogContent" => [el({}, "#{common}:\n - img\n - text: Close")]) s = side("DialogContent" => [el({}, common)]) line = aria(grader(parts: %w[DialogContent]).lines([obs("open", "light", r, s)])).first - assert_equal %(snapshot: only here: - button "Close": | - img | - text: Close), line["rubyui"] - assert_equal %(snapshot: only here: - button "Close"), line["shadcn"] + assert_equal %(snapshot: only here: - button "Close": | - img | - text: Close), line["rubyui"] + assert_equal %(snapshot: only here: - button "Close"), line["shadcn"] end def test_theme_explained_note_names_the_part @@ -197,4 +197,19 @@ def test_modal_fact_difference_is_warn line = aria(grader(parts: %w[Content]).lines([obs("open", "light", r, s)])).first assert_equal ["warn", "modal: true", "modal: (none)"], line.values_at("status", "rubyui", "shadcn") end + + def test_snapshot_delta_keeps_repeated_nodes + r = side("List" => [el({}, "- listitem: A\n- listitem: A\n- listitem: A")]) + s = side("List" => [el({}, "- listitem: A\n- listitem: A")]) + line = aria(grader(parts: %w[List]).lines([obs("open", "light", r, s)])).first + assert_equal "snapshot: only here: - listitem: A", line["rubyui"] + end + + def test_snapshot_delta_shows_nesting + r = side("List" => [el({}, %(- list:\n - button "Two"))]) + s = side("List" => [el({}, %(- list:\n- button "Two"))]) + line = aria(grader(parts: %w[List]).lines([obs("open", "light", r, s)])).first + assert_equal %(snapshot: only here: - button "Two"), line["rubyui"] + assert_equal %(snapshot: only here: - button "Two"), line["shadcn"] + end end diff --git a/audit/test/collect/normalize_test.rb b/audit/test/collect/normalize_test.rb index 104d42b2..8fe581d8 100644 --- a/audit/test/collect/normalize_test.rb +++ b/audit/test/collect/normalize_test.rb @@ -15,4 +15,8 @@ def test_collapses_whitespace_and_downcases def test_integers_unchanged assert_equal "10px", norm("10px") end + + def test_negative_zero_is_zero + assert_equal "0px", norm("-0.001px") + end end diff --git a/audit/test/collect/pixel_diff_test.rb b/audit/test/collect/pixel_diff_test.rb index 4a617ee6..8079601c 100644 --- a/audit/test/collect/pixel_diff_test.rb +++ b/audit/test/collect/pixel_diff_test.rb @@ -42,4 +42,15 @@ def test_size_mismatch_is_an_error assert_equal({"error" => "size 4x4 vs 4x3"}, compare(png("a.png"), png("b.png", 4, 3))) refute File.exist?(diff) end + + def test_tolerance_boundary + at16 = compare(png("a.png"), png("b.png", 4, 4, [0, 0] => ChunkyPNG::Color.rgba(116, 100, 100, 255))) + at17 = compare(png("c.png"), png("d.png", 4, 4, [0, 0] => ChunkyPNG::Color.rgba(117, 100, 100, 255))) + assert_equal [0, 1], [at16["pixels"], at17["pixels"]] + end + + def test_alpha_only_difference_counts + result = compare(png("a.png"), png("b.png", 4, 4, [0, 0] => ChunkyPNG::Color.rgba(100, 100, 100, 200))) + assert_equal 1, result["pixels"] + end end diff --git a/audit/test/collect/runner_test.rb b/audit/test/collect/runner_test.rb index ce0b82a5..dfc11db0 100644 --- a/audit/test/collect/runner_test.rb +++ b/audit/test/collect/runner_test.rb @@ -127,4 +127,11 @@ def driver.environment(url) = url.include?("3179") ? {"error" => "net::ERR_CONNE assert_includes result["environment_diffs"], "environment probe failed on rubyui (light): net::ERR_CONNECTION_REFUSED" assert File.exist?(File.join(@run, "collect.json")) end + + def test_theme_failure_happens_before_the_browser + factory = ->(audit_dir:, &block) { flunk "browser opened before theme was computed" } + runner = Runner.new(folder: "dialog", run_dir: @run, root: Pathname.new(@root), driver_factory: factory) + def runner.theme = raise(Errno::ENOENT, "themes.ts") + assert_raises(Errno::ENOENT) { runner.call } + end end diff --git a/audit/test/component_map_test.rb b/audit/test/component_map_test.rb index ff3747d3..7ab25eb7 100644 --- a/audit/test/component_map_test.rb +++ b/audit/test/component_map_test.rb @@ -167,4 +167,15 @@ def test_blank_selector_is_rejected error = assert_raises(ShadcnAudit::ConfigError) { selector_map("DialogContent" => " ") } assert_match(/selectors.DialogContent must be a non-empty CSS selector/, error.message) end + + def test_selectors_must_be_a_mapping + error = assert_raises(ShadcnAudit::ConfigError) { selector_map(["dialog > div"]) } + assert_match(/selectors must be a mapping/, error.message) + end + + def test_selector_on_part_without_shadcn_slot_is_rejected + map = {"dialog" => {"parts" => {"DialogMiddle" => nil}, "selectors" => {"DialogMiddle" => "div"}}} + error = assert_raises(ShadcnAudit::ConfigError) { ShadcnAudit::ComponentMap.new(map) } + assert_match(/selectors.DialogMiddle is not a mapped part/, error.message) + end end diff --git a/audit/test/divergences_test.rb b/audit/test/divergences_test.rb index 1809aa34..05ef79f8 100644 --- a/audit/test/divergences_test.rb +++ b/audit/test/divergences_test.rb @@ -147,4 +147,10 @@ def test_aria_match_is_exact refute_nil found.call(%(- button "Two")) assert_nil found.call(%(- button "two")) end + + def test_match_with_both_keys_is_rejected + match = look_match.merge("attribute" => "aria-expanded") + error = assert_raises(ShadcnAudit::ConfigError) { ShadcnAudit::Divergences.new(match_entry(match)) } + assert_match(/Look match must use property only/, error.message) + end end diff --git a/audit/test/fixtures/collect/a.html b/audit/test/fixtures/collect/a.html index 5e76d3a1..84094447 100644 --- a/audit/test/fixtures/collect/a.html +++ b/audit/test/fixtures/collect/a.html @@ -1,6 +1,6 @@ - + + From 195da52bb2e26858bef438ca0c7c03c909e56dab Mon Sep 17 00:00:00 2001 From: Cirdes Date: Sun, 1 Nov 2026 15:26:20 -0300 Subject: [PATCH 15/15] [Bug Fix] shadcn-audit: report a mount error instead of a missing fixture Co-Authored-By: Claude Opus 5.5 --- audit/docs/2026-11-01-collect-plan.md | 2 +- audit/lib/shadcn_audit/collect/driver.rb | 2 ++ audit/test/collect/driver_smoke_test.rb | 8 ++++++++ audit/test/fixtures/collect/f.html | 8 ++++++++ 4 files changed, 19 insertions(+), 1 deletion(-) create mode 100644 audit/test/fixtures/collect/f.html diff --git a/audit/docs/2026-11-01-collect-plan.md b/audit/docs/2026-11-01-collect-plan.md index 4fa9ddd5..c15f85a8 100644 --- a/audit/docs/2026-11-01-collect-plan.md +++ b/audit/docs/2026-11-01-collect-plan.md @@ -134,7 +134,7 @@ Output per part (in `parts` order), at most one line per non-empty bucket, in th - Test: `audit/test/collect/grader_test.rb` **Interfaces:** -- Produces: `Grader#lines` also emits ARIA lines and unreachable lines. `Grader::ARIA_ATTRS = %w[aria-expanded aria-controls aria-haspopup aria-describedby modal]` (`modal` added in review: computed `:modal` or `aria-modal="true"`; `aria-labelledby` is covered by the name in the snapshot; the driver fills `attrs` with exactly these keys; `aria-controls` is reported as `"resolves"`/`"dangling"`/nil, `aria-describedby` as the referenced text or nil). +- Produces: `Grader#lines` also emits ARIA lines and unreachable lines. `Grader::ARIA_ATTRS = %w[aria-expanded aria-controls aria-haspopup aria-describedby modal]` (`modal` added in review: computed `:modal` or `aria-modal="true"`; `aria-labelledby` is covered by the name in the snapshot; the driver fills `attrs` with exactly these keys; `aria-controls` is reported as `"resolves"`/`"dangling"`/nil, `aria-describedby` as the referenced text, `"dangling"` for a missing target, or nil). ARIA rules, per part, same count/skip logic as Look: - `aria` (Playwright aria snapshot of the element) differs, compared exactly (hierarchy, case and numbers kept; `Normalize.value` is for Look only) → diff with key `snapshot`. diff --git a/audit/lib/shadcn_audit/collect/driver.rb b/audit/lib/shadcn_audit/collect/driver.rb index fdd96c36..177ab4cf 100644 --- a/audit/lib/shadcn_audit/collect/driver.rb +++ b/audit/lib/shadcn_audit/collect/driver.rb @@ -88,6 +88,8 @@ def with_page(url) begin page.wait_for_selector(FIXTURE, timeout: TIMEOUT_MS) rescue Playwright::TimeoutError + return page_error(errors) unless errors.empty? + return unreached("fixture never rendered (no #{FIXTURE})") end diff --git a/audit/test/collect/driver_smoke_test.rb b/audit/test/collect/driver_smoke_test.rb index b8bea47f..295cc921 100644 --- a/audit/test/collect/driver_smoke_test.rb +++ b/audit/test/collect/driver_smoke_test.rb @@ -93,4 +93,12 @@ def test_waits_for_client_mounted_fixture_and_resolves_id_refs assert_equal({"aria-controls" => "resolves", "aria-describedby" => "dangling"}, attrs.slice("aria-controls", "aria-describedby")) end end + + def test_mount_error_is_reported_instead_of_missing_fixture + with_driver do |driver| + facts = driver.observe(url: url("f.html"), state: "closed", selectors: {}, screenshot_path: File.join(@tmp, "f.png")) + refute facts["reached"] + assert_includes facts["reason"], "mount failed" + end + end end diff --git a/audit/test/fixtures/collect/f.html b/audit/test/fixtures/collect/f.html new file mode 100644 index 00000000..cc3275c2 --- /dev/null +++ b/audit/test/fixtures/collect/f.html @@ -0,0 +1,8 @@ + + +
+ +