Skip to content

feat(product): record the loop from privacy-safe events and report it (#635) - #1130

Open
HugoXK wants to merge 3 commits into
mainfrom
dev4/635-loop-analytics
Open

HugoXK wants to merge 3 commits into
mainfrom
dev4/635-loop-analytics

Conversation

@HugoXK

@HugoXK HugoXK commented Nov 20, 2026

Copy link
Copy Markdown
Owner

Written-by: HugoXK
Author: Fledge Platform Dev-4

Fixes #635

What a user sees differently

Before: nothing on the instance answered "how far do the repositories people connect actually get, and how often does a human have to rescue a run?". The console counted jobs and cost per day; the time from connecting to a first merge, the stage where repositories drop out, the number of human rescues per failure and whether a finished job carried its evidence were not recorded at all, so every such answer was a one-off query someone wrote by hand.

After: a superadmin opens /admin/product (also listed as a module on /admin) and reads one page, for 7 or 30 days: the funnel (connected → first job → first PR → merged), the median time for each stage, human rescues per failure with the action mix, how many repositories were active in the window and were also active the week before, and how complete a finished job's evidence is. The same snapshot exports as CSV, and an operator can point one URL at the instance to receive the same aggregates once a day.

The record is an aggregate by construction

product_events is (id, name, at, repo_hash, duration_ms). There is no column for a user, an account, an e-mail, a prompt, code, a token or a URL, so the boundary is the schema rather than a filter applied later. name comes from a closed set in src/metrics/product.ts and recordProductEvent refuses anything else, which is why the one text column cannot hold a sentence or an address. repo_hash is sha256('repo:' + salt + ':' + repoId) truncated, with the salt in a single settings row for the installation: the same event from two different accounts produces rows that differ only in that digest, and deleteRepo recomputes it to remove the repository's rows. The row id is a digest of salt:name:subject, so a retried event is one row rather than a second count.

Three tests are the boundary rather than the happy path: the same event from two different users is indistinguishable in the row; an off switch writes no event row and mints no salt; and an event name that is not on the list is refused. The off switch is settings.product.analytics = 'off', and the CSV is not reachable while it is on off — the operator who switched the record off also said those counts do not exist.

The export and the optional receiver

  • GET /admin/product.csv renders loopCsv(store.productLoopMetrics(since)) — the page's own snapshot, not a second query — as section,metric,value. It answers 404 and is not linked while analytics is off, and the test asserts both.
  • PRODUCT_LOOP_WEBHOOK_URL is the only switch for the outbound push; unset means no request of any kind. Set, the scheduler pushes product.loop at most once per UTC day — product_push:last is written only after a 2xx, so a failed day retries, with a 15-minute memory backoff — and the payload is the funnel, durations, rescue counts and evidence rates with no repository fingerprint, account or free text. The test parses the payload and checks that neither the digest, nor the account id, nor the login, nor the repository id appears in it; a second call the same day makes no request.
  • product_events is a retention class of its own (RETENTION_PRODUCT_EVENTS_DAYS, default 730), and product_push: markers age out with the 30-day settings families, so a backup cannot carry the history past the operator's window.

Frames

Shot at the PR base 0079b96 (before) and this branch's tip 3e84850 (after), same seed — one approved superadmin, two repositories, three finished jobs — so the only variable is the tip. Frames live on evidence/635, not on the PR tree:

frame before 0079b96 after 3e84850
/admin, 1280, zh before 377764 B 1280x4719 after 391320 B 1280x4859
/admin, 1280, en before 386871 B 1280x4785 after 400918 B 1280x4926
/admin, 390, zh before 376303 B 390x6138 after 389627 B 390x6298
/admin/product, 1280, zh before 40554 B 1280x900 (404) after 192360 B 1280x2307
/admin/product, 1280, en before 44108 B 1280x900 (404) after 189765 B 1280x2397
/admin/product, 390, zh before 27442 B 390x900 (404) after 174586 B 390x2389

What the DOM said in the same run: /admin has no a[href="/admin/product"] before and exactly one after, every other module unchanged; /admin/product answers 404 at the base and 200 at this tip, where it renders [data-loop-terms], [data-loop-receiver], the CSV link and the funnel (connected 2, first job 2, first PR 1, merged 1 — a real 50% drop, not four equal bars). The screenshots are Chromium at 1280x900 and 390x900, locale zh-CN and en-US, fullPage: true, served from a built tree of each tip against a throwaway SQLite database.

Red first

At deb2c29 (this branch's previous tip, where the event store and the page already exist) the new file test/productLoop.test.ts is 15 tests, 12 pass, and the three cases this slice adds fail on their own assertions, not on a module or type error:

not ok 13 - the CSV export is the page snapshot and carries no digest or account
  error: '404 !== 200'                       # /admin/product.csv does not exist yet
not ok 14 - loop events age out on their own window and the preview counts them
  error: 'the loop table is a retention class, not an unmanaged table'
not ok 15 - the page names the receiver state it was handed
  error: 'the input did not match the regular expression /data-loop-receiver/'

test/productWebhook.test.ts fails to load at that tip (ERR_MODULE_NOT_FOUND for src/metrics/productWebhook.js), which is the push not existing yet. At the PR base 0079b96 the whole feature is absent: with the same tests copied in, all 15 store cases fail with store.recordProductEvent is not a function — there is no event table to read and no route to answer.

Gates

Ran:

  • npm run check at this tip (3e84850) — pass: lint: 11 warning(s) in 8 entries match the baseline, tsc --noEmit clean, # tests 3072 # pass 3072 # fail 0, tsc -p tsconfig.build.json OK
  • npx tsx --test test/productLoop.test.ts test/productWebhook.test.ts at this tip — # tests 17 # pass 17 # fail 0
  • the three cases above at deb2c29 with the new test files copied in — # tests 15 # pass 12 # fail 3, failing on the assertions quoted above; at 0079b96 the same file is # tests 15 # pass 0 # fail 15
  • npm run mock:check — mock-snapshot: ok (14 files)
  • node scripts/risk-gate.mjs --base 0079b964405c5c7fe0e44955b9ded1be4df79032 — risk-tier: T3; basis: scripts/lib/retention.d.mts (changed files: 20; src/db.ts, src/config.ts, src/jobs/service.ts and scripts/lib/retention.mjs are the strict paths)
  • npm run test:coverage then node scripts/coverage-diff.mjs --base 0079b964405c5c7fe0e44955b9ded1be4df79032 --head HEAD --report <the coverage run> --floor 80 — coverage-diff: ok — 420/437 changed lines covered (96.1%, floor 80%)
  • npm run visual:matrix at this tip (tree of 3e84850) — visual matrix: 6 pages x 9 widths x 2 locales + 1 narrow case(s) clean and visual gate: 6 pages x 3 widths x 2 schemes x 2 locales, touch/focus/name/leftover clean
  • node scripts/changelog.mjs check — changelog: 7 fragment(s) ok; node scripts/version.mjs check — version: 0.4.3 ok
  • checkBody(body, .github/agent/review-policy.json) locally — findings: none
    Not run: the Docker image; a real FlyBot dispatch (the event emission is exercised through the store, not through a live bridge); a real outbound push (the receiver tests use a fake fetch and an injected resolver, and the frame run pointed the instance at a non-resolving host so no request left the machine); pr-lint itself (it needs a GITHUB_TOKEN and a PR number). One note for the reviewer: the coverage run reported a single failure in test/nginxCheck.test.ts (a deploy-contract test this branch does not touch); it passes 6 of 6 when re-run alone and 4 of 4 on clean 0079b96, so that is a load-sensitive flake in the existing suite, not this diff.

Risk and rollback

T3 by path (src/db.ts, src/config.ts, src/jobs/service.ts, scripts/lib/retention.mjs), so this tip needs a PASS. The diff is additive: one table and its digest helpers, three read-model functions, one new page and one CSV route, one config field, one retention class and one settings family. It touches src/db.ts beside #655 and #369; the overlap is the new table, the columns-free helpers and a deleteRepo cascade line, and nothing in those cards' columns is renamed or repurposed.

  • The scheduler reads product_push:last from settings, so a restart does not repeat a day's push and a failed push is retried rather than lost.
  • A metric with no sample is null and prints n/a, never a zero; the rescue rate can pass 100% on purpose and the page says so.
  • Rollback: revert this branch. The table, the settings rows and the salts stay behind but nothing reads them; DELETE FROM product_events and the product.* settings rows are the cleanup, and retentionCounts still reports the table's age.

Fledge Platform Dev-4 added 3 commits November 19, 2026 17:16
Emit the funnel, first-PR, rescue and merge events behind a closed event-name
set, hash the repository with a per-installation salt, and render the read
model on /admin/product. The table has no user, text or URL column.
)

The loop page gains the two ways its numbers can leave the process, and both are
tested at the boundary rather than only for the happy path.

- GET /admin/product.csv renders loopCsv(store.productLoopMetrics(since)) — the
  page's own snapshot, not a second query — so the export cannot disagree with
  the page. While analytics is off it answers 404 and the page offers no link to
  it: an operator who switched the record off also said these counts do not
  exist.
- PRODUCT_LOOP_WEBHOOK_URL is the only switch for the outbound push. Unset means
  no request of any kind; set, the scheduler tick pushes the aggregate read model
  at most once per UTC day (product_push:last is written only after a 2xx, so a
  failed day retries) with a memory backoff so a refusing endpoint is not asked
  every minute. The payload has no repository fingerprint, account or free text.
- product_events becomes its own retention class (RETENTION_PRODUCT_EVENTS_DAYS,
  default 730) and the product_push: markers age out with the 30-day settings
  families, so a backup cannot carry the history past the operator's window.
- The admin page lists the loop page as a module, and the page states what the
  receiver is (or that none is configured) and names the last push.
- docs/observability.md and docs/architecture.md record the boundary, the off
  switch, the export and the push; .env.example and docs/deploy.md carry the new
  variable.

Written-by: Fledge Platform Dev-4

@HugoXK-bot HugoXK-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Written-by: HugoXK-bot
Verdict: CHANGES at 3e84850

Ran: npx tsx --test test/productLoop.test.ts test/productWebhook.test.ts at this tree → 17/17 pass; a probe on the same tree (analytics=off, then store.deleteRepo) left product.salt in settings (64 hex).
Not run: npm run check and mock:check at the tip (author reports 3072/3072 and 14 files; not repeated), the Docker image, a live dispatch.

  • src/jobs/service.ts:1917: a succeeded job with no PR — every review, and issue-output discover/iterate — is recorded as job_failed, so src/db.ts:5320-5321 counts it as a failed run and the rescue rate (src/metrics/product.ts:75-83) divides by failures that never happened; the tests seed job_failed by hand (test/productWebhook.test.ts:70), so none can catch it.
  • src/db.ts:6249: deleteRepo calls productRepoHash → productSalt() unconditionally, so deleting a repository with product.analytics=off mints the salt (measured on this tip), contradicting the invariant this PR states and tests (test/productLoop.test.ts:37-48).
  • src/views/product.ts:30: the drop cell divides by connected for a differently counted cohort and does not guard count > from, so from=1,count=3 renders −-200% (probe; the seeded frame only shows the 2→1 case).
  • pr-lint is red at this tip: no Large-PR: line (20 files, 1190 changed lines; scripts/pr-lint.mjs:164) — a body edit, not a commit.
  • Minor: src/metrics/productWebhook.ts:103 sends user-agent: 'fledge/0.2' while package.json:3 is 0.4.3.

@HugoXK

HugoXK commented Nov 20, 2026

Copy link
Copy Markdown
Owner Author

Security: PASS at 3e84850
Written-by: Fledge Platform Security Audit

  • Scope: 20 files, none under .github/, scripts/ci-ok*, review-gate* or deploy/ — no gate, permission, action pin or required-check change here (the only scripts/ file is scripts/lib/retention.mjs, a data-retention class).
  • Receiver: PRODUCT_LOOP_WEBHOOK_URL is operator-only and must be an absolute https: URL in production (src/config.ts:726); the push sends no credential and no signature (src/metrics/productWebhook.ts:101-104), and every hop is re-validated with no allowlist (src/lib/webhook.ts:118-159). Probed at this sha: 127.0.0.1, an http:// URL and a host resolving to 10.0.0.5 each return blocked with 0 fetches — stricter than NOTIFY_WEBHOOK_*, whose allowlist is not applied here; the flip side is that a private self-hosted receiver cannot be enabled at all.
  • CSV when off: src/app.ts:5689-5692 answers 404 with no CSV body and the page drops the link (src/views/product.ts:58); test/productLoop.test.ts + test/productWebhook.test.ts at this sha pass 17/17.
  • repo_hash: 128-bit truncation of sha256('repo:'+salt+':'+repoId) over a 32-byte per-install salt (src/db.ts:5259-5262); two installs give different digests for the same repo id (checked), so there is no cross-install join. It is not a defence against someone holding the DB — the salt and repos.id sit in the same SQLite file.
  • src/jobs/service.ts:1917: a succeeded job with no PR is recorded as job_failed, which inflates the failure count and the rescue rate on the page.
  • src/db.ts:6249: deleteRepo mints a salt via productRepoHash even with analytics off (so "off mints no salt" holds only until the first repo delete), and the account-purge path (src/db.ts:3861) removes repos by cascade but leaves their product_events rows until the 730-day window.

@HugoXK-bot HugoXK-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: CHANGES at 3e84850
Written-by: HugoXK-bot

Ran: npx tsx --test test/productLoop.test.ts test/productWebhook.test.ts at this tree → 17/17 pass; a probe on the same tree (analytics=off, then store.deleteRepo) left product.salt in settings (64 hex).
Not run: npm run check and mock:check at the tip (author reports 3072/3072 and 14 files; not repeated), the Docker image, a live dispatch.

  • src/jobs/service.ts:1917: a succeeded job with no PR — every review, and issue-output discover/iterate — is recorded as job_failed, so src/db.ts:5320-5321 counts it as a failed run and the rescue rate (src/metrics/product.ts:75-83) divides by failures that never happened; the tests seed job_failed by hand (test/productWebhook.test.ts:70), so none can catch it.
  • src/db.ts:6249: deleteRepo calls productRepoHash → productSalt() unconditionally, so deleting a repository with product.analytics=off mints the salt (measured on this tip), contradicting the invariant this PR states and tests (test/productLoop.test.ts:37-48).
  • src/views/product.ts:30: the drop cell divides by connected for a differently counted cohort and does not guard count > from, so from=1,count=3 renders −-200% (probe; the seeded frame only shows the 2→1 case).
  • pr-lint is red at this tip: no Large-PR: line (20 files, 1190 changed lines; scripts/pr-lint.mjs:164) — a body edit, not a commit.
  • Minor: src/metrics/productWebhook.ts:103 sends user-agent: 'fledge/0.2' while package.json:3 is 0.4.3.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: add privacy-respecting loop analytics for the one loop

2 participants