[zzylol]

Why

The cardinality demo compared the HLL sketch with exact counting at one fixed rate. Like the top-k demo before #40, it did not show the obvious alternative: store every raw point and count distinct values in the store when asked. It also could not show how each approach behaves as traffic grows or when an attribute's cardinality suddenly explodes, which is when a log store's distinct-count query gets expensive.

What

The cardinality dashboard (/d/asap-cardinality) now compares the same three approaches as the top-k demo:

• Store and query (method="raw"): the query job counts the distinct values of each attribute key in loki-raw with one LogQL query per key and reports them (asap_distinct{method="raw"}). A query Loki refuses (for example over its series limit) is reported as asap_query_failed.
• Answers side by side: per attribute key, for the latest window all three answered: exact distinct and class, sketch estimate, error and class, store and query and error.
• Class changes from both detectors (Loki events), relative error over time per key for the sketch and for store and query, throughput and distinct values per key over time, the freshness breakdown, and CPU, memory and bits per component of each approach, with the same layout as the top-k dashboard.
• Traffic: the rate per source doubles every 5 minutes (5k to 80k points/s), and at minute 10 http.route explodes from 20 to 20,000 values (as when an ID leaks into the route). user.id stays unique per point and k8s.pod.name keeps growing 5%/min.

How

• query_job.py: a raw answer per use case. Cardinality runs count(sum by (<key>) (count_over_time({service_name="checkout"}[60s]))) for each key exact counted, at the window's last millisecond; the backend query time is the sum over keys, including refused queries. A refused, timed-out or empty query is reported as asap_query_failed, never as 0. With each raw answer it republishes the sketch and exact distinct counts and classes of the same window (asap_compare_distinct, asap_compare_class, asap_compare_failed) and that window's end (asap_compare_window_end). These carry no timestamp, so the tables always show the latest compared window, however far store and query lags. The Prometheus answer query it times is now the use case's own answer metric.
• loki-raw: up to 20,000,000 series per query. Loki still refuses the user.id distinct count (HTTP 413) from about 2.4 million values per window.
• Dashboards: the top-k and cardinality dashboards are generated by grafana/gen_dashboards.py (now in the repository). The tables carry a panel naming the compared window and, for cardinality, a "refused" column; the error-over-time panels only compare samples of the same window; the throughput panels read each pipeline's last count within 2 minutes, so a stalled pipeline shows a gap, and say that raw counts are the points loki-raw acknowledged within 50 s of the window's end.
• Acceptance: a new check that store and query answers every key exact counted from loki-raw within 1% of exact, on the latest window in which it answered them all. The scaled copy drops a changes: key left empty after the ramp lines are removed (it would read as null).
• .gitignore ignores __pycache__/.

Before

• The cardinality scenario ran at a constant 20k points/s per source with fixed http.route cardinality.
• The dashboard compared sketch and exact only, and showed CPU, bytes and memory per pipeline.

After

Measured on the demo machine over the 30-minute ladder (2 sources, 1-minute windows; user.id is unique per point, so its distinct count equals the window's points):

Rate per source	user.id per window	Sketch: error, answer complete	Exact	Store and query
5k	600,000	±1.1%, 0.01 s	1.3 s after the window	all keys exact; 4 LogQL queries 15–21 s
10k	1.2M	±1.0%, 0.01 s	3.3 s	all keys exact; 38–43 s
20k (route explosion)	2.4M	±1.2%, 0.01 s	8.5 s	user.id query refused by Loki (HTTP 413); other keys exact; about 63 s
40k	4.8M	±1.1%, 0.01 s	19.8 s	loki-raw keeps about 50% of the points; user.id refused
80k	9.6M	±1.4%, 0.02 s	no answer: the exact window no longer fits one OTLP message (exporter encoding failures)	loki-raw keeps about 27%; no distinct counts

• Route explosion at minute 10: both detectors move http.route from stable to growing in the same window (20 → 19,989 values; the sketch estimates 19,749, −1.2%), and back to stable once the 10-window baseline has absorbed it.
• CPU at the top rate: sketch 1.2 cores (mostly the two branches), exact 1.38 while it still answered, store and query 3.5–6 (mostly loki-raw).
• Memory: each sketch stage stays at 36 MiB throughout. Exact grows to 621 MiB per branch and 475 MiB in the merge; loki-raw peaks at 1.7 GiB.

Verification

• The dashboard was checked in a browser on the running demo, and every table query was evaluated against it.
• python3 -m py_compile on the changed scripts; no Rust changes in this PR. CI runs the scaled acceptance, including the new check.
• Review follow-ups are in the second commit: the empty changes: in the scaled cardinality configs, every key required in the acceptance check, unstamped comparison gauges, same-window error panels, empty results as refused, the Loki comment, and the generator in the repository. The fixed query job and dashboards were checked against the running demo.