Repository navigation
fix(core): unbreak the outreach loop, then make it 57x faster at the data layer - #1
Merged
Merged
Conversation
Five defects made the product non-functional. All were verified by reading
the code and executing it, not inferred.
Blockers
- Mailbox daily cap: `sent_today` was a lifetime counter, never reset, so
every mailbox bricked after `daily_limit` sends, forever. It is now a
calendar-day (IST) counter and the rolling-24h budget is the real gate.
- Send gate: the sender excluded only "bounced", so `invalid`, `gibberish`,
`disposable` and `no_mx` addresses were all mailed - a ~30% hard-bounce
rate against Gmail's 5% bulk-sender ceiling. Ten undeliverable states are
now refused; un-enriched leads stay eligible.
- Sequence scheduler: multi-step sequences only advanced when a human
re-clicked Launch. Added a 5-minute in-process loop plus a Celery beat
entry, and "waiting on a step delay" is now distinct from "completed".
- Maps `q=` split: the text query carried the city, so Google returned one
city-wide set and every viewport duplicated the top 20. Category-only
queries take the same 8 requests from 34 unique leads to 60.
- `has_email` pagination: the filter ran after LIMIT/OFFSET and was absent
from the count query, so the Lead Database showed 4 of 20 matching leads
with a wrong total and a permanently disabled Next button.
Correctness
- A rejected MAIL FROM was classified as a recipient hard bounce, so one
misconfigured sending domain permanently suppressed the whole lead DB.
- Soft bounces suppressed a good lead for 30 days on the first transient
error; there is now a 3-attempt retry budget.
- A hard-bounce suppression could silently expire, because its expiry was
never cleared on upgrade.
- `purge_expired` compared `expires_at < created_at` - never true.
- Unknown `{{tokens}}` shipped literally to prospects; the HTML body was
not escaped.
- Every contact inherited `executives[0]`'s name, so `info@` was addressed
as "Hi Rajesh" and the fallback produced "Hi Apex," for Apex Realty.
- `is_verified=True` was asserted for addresses never SMTP-probed.
- Google stores the phone with the display form first and the international
form second; the parser took the first, so every number in the database
was `098225 22560` - which reads as an STD code 098225 landline and
mis-files the STD->city lookup.
- An out-of-office auto-responder stopped the sequence and counted as an
A/B "reply"; the reply poller collected the auto-reply headers and threw
them away.
- The A/B test compared raw rates at n=10; it is now a two-proportion
z-test at n>=50 over positive replies only.
- The ICP parser silently dropped constraints like "50 employees" and
"revenue over 10cr"; it now reports them as unsupported, with warnings.
Performance
- Enrichment ran one lead at a time (~9s each) inside the event loop, and
called the synchronous SMTP verifier from an `async def`, freezing the
whole API. Now a bounded fan-out plus the async verifier.
- No index existed on any foreign key, filter or sort column, and the
schema sync never created indexes - so existing deployments silently
never got them.
- SQLite foreign keys were off, making every `ondelete="CASCADE"` a no-op
and orphaning contacts on lead deletion.
- `rank_existing` loaded the entire database to return 50 rows (13.6s and
837 MB at 100k rows).
Infrastructure
- The API had no auth and `CORS_ORIGINS="*"` with credentials, so any
website could delete the lead database or send cold email as the
operator. Added an optional API key (unset = unchanged local DX), with
the mail-facing tracker exempt.
- The Celery worker could not boot (it imported a function that does not
exist) and no beat service ran, so reply detection never worked in the
Docker deployment.
- The CLI and MCP server never ran the schema migration, so they crashed
with "no such column" after any model change.
- MCP tools could never target anyone: no tool returned a contact ID, and
`contact.id != business.id` for most rows, so an agent mailed the wrong
79% of the database. Added contact IDs, a dry-run default on launch, and
JSON-RPC `isError` so the model can read a tool failure.
- Click tracking did not exist, so the warm-WhatsApp handoff panel could
never populate.
Dashboard
- The global Cmd+K search navigated to a URL nothing read; 15 native
alert() dialogs; no mobile layout; Launch sent real email with no
confirmation; CI skipped type-checking; 9 of 12 e2e tests could not fail
the build.
README: corrected the claims that were false against the code
(APP_INITIALIZATION_STATE, "protobuf", humanized mouse curves, Native MCP
SDK, Twenty CRM sync engine, the three spam-linter signals that do not
exist) and added a Security & Operations section.
Tests: 196 -> 226, including a regression suite for every blocker above.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Measured on a 500,000-lead SQLite database and a 5,000-row CSV import.
Import & export
- CSV import ran ~5 SELECTs per row (place_id, phone on the business, phone on
the contacts, domain, name, plus a contact lookup) - roughly 25,000
round-trips for a 5,000-row file, all inside one transaction holding the
write lock. The same lookups are now built once from a single scan and
matched in Python; new rows are inserted in bulk with INSERT...RETURNING
and the file is processed in committed chunks.
5,000 rows: 11.95s -> 0.21s (57x)
re-import: 5.10s -> 1.43s (3.6x)
- The CSV export materialised every row in memory and built the whole file
before responding (1.4 MB for 5,000 leads, ~125 MB for 500,000). It is now
streamed from a server-side cursor in 64 KB chunks, so the memory cost is
constant and the first bytes reach the client immediately.
Search
- Lead search used `LIKE '%term%'` across four columns, which cannot use an
index: every search was a full table scan, ~345ms at 500k rows, and the
count query duplicated it. Added an FTS5 virtual table with the trigram
tokenizer, which has identical substring semantics to LIKE, kept in sync by
triggers and rebuilt on boot. The match is pushed into SQL as a subquery
so a 50,000-hit search never round-trips ids through Python.
selective ('biz42'): 200ms -> 9ms (22x)
phone fragment: 276ms -> 5ms (55x)
no match (a typo): 427ms -> 2ms (200x)
A regression test asserts the FTS and LIKE paths return identical id sets
for whole-word, mid-word, phone-fragment and no-match terms.
Dedupe
- Added indexed `phone_e164` and `domain` columns, populated automatically by
a SQLAlchemy before_insert/before_update listener so every write path is
covered. The phone branch of the dedupe was dead code (it compared a
normalised value against the raw stored string); it now matches. The
domain branch was a substring match, so `example.com` matched
`smileshop.example.com`; it is now an exact host comparison.
Analytics
- The timeseries endpoint loaded every Send row in the window into Python
and counted them there. The aggregation is now a SQL GROUP BY.
18,000 sends: 0.010s, exact match against a Python reference on all
30 days (regression test added).
Note: `CAST(sent_at AS DATE)` is NOT portable - SQLite applies NUMERIC
affinity and truncates '2026-10-17' to the integer 2026 - so the day
bucketing uses the portable `date()` function.
Enrichment (13.2x faster end to end)
- The homepage was fetched FIVE times per lead (once by the crawler, once by
the social finder, and again for each page the crawler visited). It is now
fetched once and shared: 5 GETs -> 1, and the whole flow went from 8.14s
to 0.62s against a site with a 0.3s per-page delay.
- Yahoo and Bing were queried sequentially; they now run concurrently
(2.62s -> 1.32s), as do the website-health and Facebook-Ad probes
(2.00s -> 1.00s).
- The crawler now respects robots.txt, de-duplicates normalised URLs, and
fetches up to 4 pages in parallel while still honouring the page budget.
- TLS verification is on everywhere (three `verify=False` sites removed) and
the HTTP client is pooled instead of rebuilt per attempt.
- `looks_blocked()` treated ANY body under 2 KB as a block page. Every small
but legitimate page was therefore retried four times with backoff (~7-9s
per URL) and then discarded - the social finder alone cost 6.9s per lead
and returned nothing. Blocks are now identified by their content markers.
- The Facebook Ad Library probe downloaded the entire multi-megabyte page to
regex two numbers out of it; it now streams and stops at the first match.
- Fixed a NameError that lost every lead with no website but a web-search
email, a mailto: link compared against the wrong field, a social href
check that matched any URL containing "x.com", and uppercase MAILTO/TEL
links being dropped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- The lead list no longer takes the database's write lock on every page view: the refreshed score is only persisted when it actually changed. - The campaigns list issued 4 queries per campaign (151 round-trips for a 50-campaign list). The send, target and step aggregates are now three grouped queries. Also fixed the mislabelled field: "total_targets" reported the send count, not the number of selected recipients. - The full-text search triggers are created independently so one failure cannot take the others down, and the table is recreated if it goes missing (a missing table made every business UPDATE fail, because the update trigger referenced it). Measured at 500,000 leads: /leads 10.5ms /leads?sort=score 10.2ms /leads?has_email=true 2.4ms /leads?search=dental 83.5ms (83,000 matches - still a count) /leads?search=biz42 7.9ms /leads?search=<no match> 2.0ms /analytics/stats 10.0ms /analytics/timeseries 1.9ms /campaigns 1.7ms /inbox 3.0ms Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The scrape path matched businesses on the business NAME only, ignoring the place id, the phone and the domain - and it ran one SELECT per scraped lead. It now matches in the same priority order as the rest of the codebase (place id -> phone -> domain -> name) using four grouped queries for the whole batch, so the cost scales with the number of leads rather than with the size of the database. Businesses created within the same batch are registered in the index too, so a duplicate row inside one batch is caught. Two identity bugs fixed along the way: - The Google CID was being dropped on the primary scrape path (the HTTP path calls it google_data_id, and only the sources layer mapped it back). Every Maps-sourced business had a NULL place_id, so the strongest dedupe key we have was unused. Now 25/25 leads carry it. - ensure_schema() did not import the models, so create_all silently created NOTHING and the full-text triggers failed with 'no such table: main.businesses'. It only worked because every caller happened to have imported the API modules first - and saved_lists was missing from freshly created databases for the same reason. Also stopped the schema sync logging bogus 'duplicate column name' and 'index already exists' warnings on every boot: the SQLAlchemy inspector is a snapshot taken before the ALTER, so it never saw what it had just added. Verified on a live scrape: 25 businesses, 25 with a Google CID, 26 distinct phone numbers, zero duplicate names, and a business whose name changed is still matched through its phone. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ontact The campaign sender called `await is_suppressed(db, contact.email)` inside the candidate loop - one round-trip per contact, so a 500-target campaign spent 500 queries deciding who NOT to email, before sending a single message. Added `suppressed_reasons()`, which answers the whole candidate set in one query, and rewired `filter_suppressed()` on top of it so there is one implementation of the rules. A regression test asserts the batch form returns exactly what the per-address check returned, including the whole-domain case and the expired-soft-bounce case (which must NOT block). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An AST sweep for "a loop that awaits a query" found the remaining N+1s: - The reply poller issued a SELECT per inbound message, so a mailbox with 50 new replies cost 50 round-trips every five minutes. The sends and their contacts are now resolved in one query each. - The inbox counted one reply category per query (5 queries); it is now one grouped query. - The enricher issued a SELECT per discovered address to check whether the contact already existed; the business's contacts are now fetched once. - The CSV importer called `db.get(Business, id)` per matched row; the matched businesses are now fetched in one query per chunk. Re-importing 5,000 rows: 1.43s -> 0.32s (16x against the original 5.10s). Also fixed a routing bug introduced while batching the importer: the chunk splitter keyed off "does this row have a Business object", but matched rows now carry only an id, so every matched row was being treated as new. The split now keys off the id, and the import was re-verified end to end - 5,000 rows, 5,000 businesses, 5,000 contacts, zero duplicates, industry populated on every row. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…d scored An axe-core audit of the running dashboard found 48 elements failing WCAG AA colour contrast at 2.56:1, 50 nodes outside any landmark on all ten routes, a select with no accessible name, and two heading-order breaks - none of which TypeScript, the build, or the existing e2e suite could see. Accessibility gate - New `e2e/a11y.spec.ts` runs axe-core (WCAG 2.1 A/AA + best-practice) against every route and FAILS the build on any `critical` or `serious` violation. It also asserts one `<main>` and one `<h1>` per page, that no page fails to load its own assets, and that the layout does not shift while data loads (CLS < 0.1). - New CI jobs `frontend-a11y` and `frontend-e2e` run it against a live API. An accessibility regression should be as loud as a type error. Serving the production build correctly - `next start` does not work with `output: "standalone"`: it prints a warning and then serves the HTML shell while every JS chunk 404s, so the page loads and the client JavaScript never does. The app sits on a spinner forever - and an e2e suite pointed at it would "pass" against a dead page. New `scripts/start-prod.mjs` copies the static assets into the standalone output (as the Dockerfile does) and execs the real server; `npm run start:prod` is now used by CI and by the Playwright webServer. - `E2E_BASE_URL` lets the suite run against an already-running server without booting a second copy of the app. Lead score - A lead score was rendered as "84.35 HOT". The trust modifier is applied at full precision internally - that is what makes a directory lead actually score below a live Google Maps lead, which the README promises and which had no test - but the score the operator sees is now a whole number. - Added tests for the integer score and for the source-trust ordering. Also split the activity chart into `src/components/ActivityChart.tsx` so it can be loaded on demand: recharts and its d3 dependencies were ending up in the *shared* chunk, which meant every route - the inbox, the scraper, the settings page - downloaded a charting library only the overview ever renders. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rich what is actually unenriched Demo data - A fresh database seeds 83 sample businesses, and the overview says "83 businesses in your database". Nothing on the screen distinguished them from leads the user had actually found, so the very first screen of the product makes a claim that is not true. The leads API now returns an `is_demo` flag for the seeded rows. "Enrich everything" - It took the newest N rows by id, so running it a second time re-enriched exactly the same leads - the same crawl budget and the same SMTP probes - and the older leads that had never been enriched were never reached. It now selects leads with no `enriched_at` (or one enriched over 30 days ago), oldest first. - Added the missing `Business.enriched_at` column, set whenever a lead is enriched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tabases The overview said "83 businesses in your database" on a fresh install, where those 83 were the sample rows the database is seeded with. Nothing on screen said otherwise, so the first thing the product tells a new user was not true. - Added `Business.is_demo`, populated by the same before_insert/ before_update listener that keeps `phone_e164` and `domain` in sync. - Added a one-time, chunked backfill for rows that predate those columns. A legacy database had no `phone_e164` at all - so the phone branch of the dedupe silently never matched - and no way to tell sample data from real leads. The backfill only touches rows whose derived values are missing, so it costs nothing on a healthy database. - The dashboard stats now report `real_leads` and `demo_leads` separately instead of one misleading total. - The leads payload carries `is_demo` so the UI can label those rows. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…that still say "loading" - The sidebar brand rendered an <h1> next to every page's own <h1>, so the app shipped two level-1 headings on every route. The product name is a label, not a section heading. - Raised the tiny-text floor: 9px and 10px text is hard to read on the budget laptop panels a lot of Indian SMB operators work on. Everything is now at least 11px. - `motion-safe:animate-spin` leaves the spinner completely static for reduced-motion users, which reads as "hung" rather than "loading". They now get a pulse instead of a freeze. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Without it, a keyboard user who follows a link stays focused on the link they pressed - which now belongs to the previous page - and the next Tab press dumps them back at the top of the navigation instead of into the content they asked for. It only moves focus when the navigation was keyboard driven, so mouse users are not yanked around. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`check_sender_auth` performed up to 13 *blocking* DNS TXT lookups (SPF, 11 DKIM selectors, DMARC), each with a 3-second timeout - and it was called directly from an `async def` route. One request could therefore block the event loop for up to ~39 seconds, stalling every other request the API was serving. It was also on the critical path of a page load, which is how it was found: the settings page was taking 1196 ms because the last request it fired was a DNS probe, and a cold cache cost 600-800 ms. - Added `check_sender_auth_async`, which runs every lookup in a worker thread and probes the DKIM selectors concurrently. The wall clock is now one timeout instead of thirteen. - The route uses the async variant. - Measured after the fix: 44 ms, worst event-loop stall 51 ms. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…story Two agents audited the running app with axe-core and Playwright and rebuilt the parts that were failing. Everything below was measured, not assumed. Accessibility - 0 violations on all ten routes (from 22 types / 106 nodes) - Landmarks: the rail is an <aside>, the brand bar a <header>, the nav a labelled <nav>, the promo/docs a <footer>, and the TopBar is the page banner. Fifty nodes had been outside any landmark on every route. - Colour contrast: 48 of the 53 failures were one class - Tailwind's slate-400 on white is 2.56:1 where WCAG AA needs 4.5:1. Nav labels, the tagline, icons, emerald and red badges all raised. - A visible focus ring on every interactive element. There was none, so a keyboard user could not tell where they were. - Focus management: the mobile drawer moves focus in, traps Tab, closes on Escape and restores focus to its trigger. - aria-current on the active nav item, aria-expanded/aria-controls on the collapse toggle, decorative icons hidden from assistive tech. - Live-region announcements for loading and search status. - One <main> and one <h1> per page (the sidebar brand was a second <h1>). - Focus moves to <main> after a keyboard navigation. - The tiny-text floor is now 11px; spinners pulse rather than freeze for reduced-motion users. Layout stability - CLS 0.214/0.28 to 0 on every route - Skeletons sized to the real content, fixed table columns via <colgroup>, and a readiness strip that always occupies the same box. Verified adversarially with every API response delayed 700ms. Speed - /settings was 1196ms; it is now 564ms. The cause was a DNS probe on the page's critical path doing 13 sequential TXT lookups - now fixed in the backend and made an explicit per-mailbox action. - recharts moved out of the shared chunk: 319 KB of JS on every route is now 207 KB, because the charting library is only fetched by the one page that renders a chart. Usability - Every list surface has a real empty state, and a filter-caused empty says "No leads match these filters" and offers to clear them - rather than telling the user to scrape more when they simply filtered too narrowly. - Every route has an inline error state with a Retry button, not just a toast that disappears. - Keyboard shortcuts on the lead database: / focuses search, Esc clears it, j/k move the row cursor, x selects, ? opens the help. - The lead database is a card list on a phone. Nine columns crushed into 390px wrapped their headers onto three lines and overlapped their contents; a phone user could not see a phone number. Fixes to the e2e suite, which had been asserting the old markup: the sequence "add step" flow, the parse button name, the empty-lead wording, and the API base (it pointed at a port the product never uses instead of the dashboard's own proxy). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…l auto-sending Two `NameError`s shipped inside `execute_campaign_step` - `filter_suppressed` and `tracking_base` - and the scheduler's error handler swallowed both. Every campaign tick therefore failed and no sequence ever advanced, which is exactly the bug the scheduler was added to fix. The failure only surfaced because the log line was read carefully; nothing else reported it. - Fixed both names. - Added a test that drives `execute_campaign_step` all the way to the SMTP call with the transport stubbed out, so an unbound name on that path cannot hide again. Also a safety fix: the seeded sample campaign was created ACTIVE. The scheduler advances every ACTIVE campaign on its own, so the moment a new user connected a sending mailbox, LeadForge began cold emailing the sample leads without anyone asking it to. It is now seeded as a DRAFT. And a build-time footgun: `next.config.mjs` said the API proxy was read at RUNTIME, but Next serialises rewrites into routes-manifest.json when it builds. A manual `npm run build` without API_PROXY_URL therefore produced a dashboard that proxies to localhost:8000 - working on the build machine and nowhere else. The comment is now accurate and the build warns loudly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…alled
Three things existed in the codebase and were called from nowhere. All
three are the reasons the "built for India" claim did not hold.
1. `localize_template` had zero call sites. It substitutes
`{{festival_line}}` (and empties it outside a festival window), so a step
using that token shipped the literal braces to the prospect. The sender
now localises the body before interpolation.
2. `preferred_channel` - whatsapp / call / email, computed from the line
type at enrichment time - was never read by the sender. Indian SMB
owners answer WhatsApp and only a mobile can be reached on it, so the
headline differentiator changed nothing. The sender now reads the
channel, exposes `{{whatsapp_link}}` and `{{channel}}`, and points the
call-to-action at WhatsApp for a mobile lead and at a call for a
landline.
3. The A/B winner was finalised only inside the stats endpoint, so the
verdict depended on somebody opening the sequence-detail page. A
campaign could run to completion with both variants live and no answer.
The sender now declares the winner after each send batch, using the
same significance test, and persists it.
Four new tests drive the real send path with the transport stubbed out, so
none of these can regress silently again.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…directory `fetch_html` passed `proxy=` to `AsyncClient.get()`. httpx 0.26 - the pinned version - takes the proxy on the *client* only and has no `proxy` keyword on `get()`, so every request raised `TypeError: get() got an unexpected keyword argument 'proxy'`. The retry loop swallowed it as a generic failure, retried three more times with backoff, and returned None. The visible effect was that every directory source looked like a directory with no businesses. Nothing had ever been fetched. - `fetch_html` now detects whether the installed httpx accepts a per-request proxy. On 0.26 the proxy goes on the client (one client per proxy, so connection reuse survives); on 0.28+ it is passed per request. - Sulekha: the URL was never wrong - `sulekha.com/<niche>/<city>` serves 529 KB. The *selectors* were. The name lives in an `<h3>`, the address in an `<address>`, the "Verified by Sulekha" badge in `svg.lucide-badge-check` and the score in a bold orange span. The adapter now reads all four and returns 10 live listings for "dentists in Pune". - The Sulekha score is out of 10, not 5, so it goes to a new `source_score` field instead of `rating`, which would have corrupted the 0-5 scale the scorer uses. - JustDial: `www.justdial.com` answers every request with a 14-byte stub; the mobile host serves the real 800 KB page, but the listing grid is loaded by a client-side XHR and the site geo-resolves the city from the request IP. The adapter now uses the host that works and says plainly that it cannot read the listings. - IndiaMART: `www.indiamart.com/search.php` 404s. The `.mp` path serves a page, but it carries the same four static promoted slots whatever the query - so a "dentists" search returned cotton-t-shirt manufacturers. A keyword guard now rejects slots that do not match the niche, and a service niche correctly yields nothing instead of manufacturing leads. - `Source._failed()` records why, `health()` reports it, and a new `GET /api/v1/icp/sources/health` probes every adapter concurrently. A dead source that looks like an empty niche is how a broken adapter hides for months. - The README's "multi-source" claim now matches reality. 15 new tests, including a saved Sulekha HTML fixture so the selectors are regression-proof, and tests that pin the httpx proxy behaviour on both the old and new client APIs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Apollo's core surface is a person per row with attributes you can filter on. LeadForge had a business table with a nested contact, so the question the product exists to answer - "show me every owner in Pune with a reachable email" - could not even be expressed. Backend - `GET /api/v1/contacts` - one row per person, joined with the company. Filters: seniority, decision makers only, city, state, market tier, industry, has email, verified, company size, employee range, revenue band, technology, free-text search, sorting, paging. Every filter is a SQL predicate applied *before* the LIMIT, and the count query reuses the same predicates, so a filtered page reports an honest total. - `GET /api/v1/contacts/facets` - counts per filter value, so a dropdown shows what is actually there instead of guessing. - `GET /api/v1/contacts/stats` - headline numbers. Data that had to be invented, and now is not - The discovery path wrote `first_name="Gurucity"` and `title="Owner / Decision Maker"` on every business it found. That is a business name fragment and a placeholder, and classifying them made 126 of 149 rows look like founders. The People table reported a decision maker count that was pure fiction. Those rows now carry no name and no title, and a new `person_identified` column says whether a real human was actually found. - `city`, `state`, `market_tier` are derived from the free-text address on every write, so "every owner in Pune" is a SQL predicate instead of a string search. A state is never returned as a city. Seniority - A taxonomy that understands Indian SMB titles - Proprietor, Promoter, Managing Director, Principal, Partner. A US/EU list files all five as unknown, and an unknown title is a founder the table cannot show. - Word-boundary matching, not substring: "director" contains "cto", so every director in India was being filed as a C-level executive. - `S K M D` and `S.K.M.D.` resolve to C-level. Decision-maker extraction - JSON-LD `Person`/`Organization.employee` with a `jobTitle`. - "Name - Title", "Title: Name" (the form the old pattern missed entirely) and "Name is a Role". - Testimonials and reviews are stripped first: a customer writing "I bought my office from Mr. Omkar" would otherwise add a customer to the People table. - A person identified on the site is now kept even when no email could be found for them. That was the single biggest lever - the table had 13 people out of 149 businesses because people without a verified mailbox were dropped on the floor. Also fixes two latent infinite loops in the boot-time backfills: a row whose value could not be derived (a non-numeric phone, an address with no recognisable city) still matched the WHERE clause after the update, so the loop never ended. The API hung on boot. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… data Dashboard - `/people` - one row per human, with the company joined in. Filters for role, city, market tier, email, decision makers, verified; every filter is a server-side predicate, and the dropdowns show the real counts so a filter is a choice rather than a guess. Active filters are chips that can be removed one at a time. Desktop table, phone card list. - The row says whether it is a *person* or a *business target*. A business target is a company with a phone and a WhatsApp number but no human behind it yet, and calling that a founder is the single most misleading thing this product could do. The data behind it had to stop lying - The facet counts were a cross join. Selecting a `Business` column while counting over a subquery of `Contact` made SQLite add `businesses` to the FROM, so every city reported the *total* number of people - which is how a dropdown came to promise "13 in Bus Stand Road" for a city that has none. - The city parser invented cities out of street names: "Jain Mandir Rd", "Pocket", "Ground Floor", and "7 Pm - Opens 10 Am Mon" (Google Maps puts opening hours in the address). A city is now only a known city, or a fragment that reads like a town; a state is never returned as a city. - The boot-time backfills silently changed nothing. Two separate traps: a Core UPDATE against a mapper-enabled table invokes `before_update`, whose handler marks the ORM object dirty, and the commit then flushes that object and rewrites all of its columns - undoing the correction; and an ORM column SELECT after the commit re-populates the object with the stored value, marking it dirty again. Both are silent: the backfill reported success every time and changed nothing. The backfill now uses raw SQL. Also adds 8 Playwright specs for the People table - including one that fails if any row is a fabricated founder - and puts `/people` in the accessibility gate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…pressible
The People table could filter on "has an email" and nothing else. Apollo
filters on twenty attributes, and the three that matter most for a
segment are the company size, the turnover and the stack. Those columns
existed on the model and nothing ever wrote them.
- The crawl now reads the company's own site for them: a headcount stated
in prose ("we are a team of 35 employees"), an annual turnover
("turnover of 12 crore"), and the technology fingerprints actually
present in the markup (Shopify, Tally, Razorpay, ...).
- A signal is only reported when the page states it. A guessed headcount
is worse than a blank one: it would file a business into a segment it
does not belong to, and the operator would never know.
- Script and style tags are stripped before the numbers are read, so a
copyright year or a JS constant is not a headcount.
- The values are merged across the pages crawled: the first page that
states a number wins, technologies accumulate.
- The dashboard exposes them: filters for company size, turnover band and
technology, with the real counts in the dropdown, and the values shown
on the row.
Also fixes a crash the new code exposed: `execs` was assigned inside the
per-address loop, so a business with no discovered email raised
`UnboundLocalError` and the whole enrichment 500'd. It is now assigned
once, outside the loop - which is also where it belongs, because a
business whose site names its proprietor but whose mailbox could not be
found still has a decision maker.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An agent could list businesses but not people, so the question it is most likely to be asked - "every owner in Pune" - had no tool. `leadforge_search_people` carries the same filters as the People table and returns the same Contact IDs, so an agent can go from a filter straight into a sequence. To keep one set of rules, the People query is now a plain function (`query_people`) with the HTTP route as a thin wrapper over it. Calling the FastAPI handler directly does not work: its defaults are `Query` objects, not values, so a plain call hits `'Query' object has no attribute 'split'`. Also updates the README: the People table, the India-aware role taxonomy, and the tool count (19). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The score was the same for every lead. It measured how complete a record is - has an email, has a phone, has a website - which is a measure of the *scraper's* work, not of how good the lead is for the operator's market. Apollo's score is relative to the search, and that is the number a salesperson actually sorts by. - The lead list now scores each lead against the ICP it was saved under. The profile's preset decides whether a weak web presence is a penalty or an opportunity, and its category match counts - so the same lead moves when the ICP does. - The People table gains an ICP filter: "the people at the companies I actually sell to", using the profile's own keyword rules. - The row shows the ICP-relative score, with the global score beside it so the difference is visible rather than mysterious. - The profile being filtered by wins over the one a business happens to be attached to. A business matched by keyword but never saved under that profile still has to be scored against it. - The ICP profiles for a page are loaded in one query. A per-lead lookup would be N queries on the product's front door. The accessibility gate caught a contrast regression in the new markup (slate-400 at 3.55:1) and failed the build, which is what it is for. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The overview showed four flat counters - businesses, contacts, verified, sent. They cannot answer the only question that matters: *which stage is losing the leads?* A list of 500 businesses that produced 12 emails and 1 reply looks identical to a list of 12 businesses that produced 1 email and 1 reply, and the operator cannot tell which part to fix. - `GET /api/v1/analytics/funnel` - nine stages from "businesses found" through to "interested", each a real predicate over the data, with the conversion from the stage above. Plus the per-source breakdown, the per-mailbox table (sent / opened / replied / bounced with rates) and the reply categories. - A "Where the pipeline leaks" card on the overview, with each stage as a bar and the biggest drop highlighted. A funnel with no number on it is a guess; the counts are what make it actionable. - A send with no `sent_at` is counted by its creation time. Filtering on `sent_at` alone silently dropped sends that were imported or written before the column was filled - the funnel would have reported a pipeline that was not the real one. Five new tests, two of them e2e. One of the e2e tests asserts the funnel is a funnel: no stage may exceed the one above it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… called
An ICP could only be a list of keywords - a word list is the one thing a
salesperson cannot work with. "11-50 staff, over 10Cr turnover, tier-2
cities, owners and directors" is what they actually say, and none of it
could be expressed.
- The ICP profile now carries structured criteria: headcount range,
turnover bands, size buckets, market tiers, states, target roles, and
the two flags an operator reaches for most ("decision makers only",
"require an email").
- The natural-language parser reads them out of a sentence. "with 11-50
employees and turnover over 10cr" becomes a size filter and a revenue
band; "decision makers only" becomes a role filter. A term the parser
can express is no longer reported as unsupported, and a term it cannot
still is - the warning list is a promise that only what was *dropped*
appears in it.
- The People table honours the criteria, as SQL predicates applied before
the limit.
- The discovery ranker honours them too, so a profile set in the wizard
is the profile the discovery run uses.
Two bugs the tests caught, both silent:
- The business-level criteria were added to the contacts statement
directly, which made SQLAlchemy add `businesses` to the FROM with no
join condition - a cartesian product. "11-50 employees" then returned
the 120-staff business *and* the 8-staff one. The conditions now go
through a subquery over `businesses`.
- A search combined with a company sort joined `businesses` twice, which
made every `businesses.*` column ambiguous. It only failed for that
one combination, so only a user would have found it.
Also: a job interrupted by a server restart is now failed with the
reason, instead of sitting at "running" forever with nothing left to
finish it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The festival hook and the Hinglish closers existed, were computed, and were served by the API - and were reachable only by calling the API by hand. The overview told the operator a festival window was open; it did not help them write the greeting. The copy kit is now in the step editor, under the body field, with the seasonal greeting, the closers, and the do/avoid checklist. Every line is opt-in: the panel offers copy, and inserting it is the operator's decision. Nothing is auto-inserted into a send. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Wave 2 table records the People table and the honesty work with before/after numbers, the audit's scorecard is marked as the record of what was wrong rather than the current state, and the execution order says which items shipped and which are still open — so the open items are the ones that are actually open. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An Indian business is registered under a GSTIN and, if it is an MSME, under an Udyam number. Neither exists in Apollo, Hunter, or any Western directory - and both carry information nothing else does: - the GSTIN's first two digits are the **registered** state, which is a stronger location signal than a free-text address (it is where the business is registered, not where the pincode points); - the Udyam number's first two digits are the same state code, so a business that publishes both gives two independent signals; - their presence says the business is a registered entity, which is a qualifier for a sales pitch. - `app/locations/gstin.py` - the formats, the state-code table, and the extraction. A number whose 14th character is not the fixed "Z" is not treated as a GSTIN, and nothing is guessed: a blank field is honest, an invented one is not. - The registrations are read off the company's own site, which means the crawl now visits the legal and terms pages - where Indian companies actually print them - before it gives up. - `gstin_state` is kept apart from the address's `state`. A mismatch is a real signal (a branch office, a business registered elsewhere), and silently preferring one over the other would hide it. - The People table shows the GSTIN and its registered state. The MSME classification is deliberately *not* derived: the Udyam portal keeps it as a separate field, so it is not in the number, and an invented size band would file a business in the wrong bracket without anybody noticing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The word list was Western-only, so an Indian cold email full of "Dear
Sir/Madam", "kindly revert back" and "Sent from my iPhone" scored
"Good" - while the same message read by an Indian recipient is obviously
a list blast. The README's "80+ words" was true and the linter's India
coverage was zero.
Three separate findings, kept separate because they are different
problems:
- **Spam triggers**: the number-harvesting close ("save our number", "we
will call you"), the guaranteed-offer language ("guaranteed returns",
"data entry job"), the "limited seats" urgency. Weighted the same as
the Western list, because they are exactly as spammy.
- **Unpersonalised tells**: "Dear Sir/Madam", "revert back", "do the
needful", "with warm regards". Not spam - but the recipient can tell
the mail came from a list, and these reliably lower reply rates. A
smaller penalty and its own advice, because the fix is different.
- **The mobile signature**: "Sent from my iPhone" is on by default in
every Indian mailbox and is, on its own, one of the strongest
bulk-mail markers there is. It now costs 20 points and says to turn it
off.
The three density signals the README promised - all-caps density,
exclamation ratio, hyperlink density - now exist. A ratio catches what a
per-word check cannot: three capitals in a thirty-word body is shouting,
and ten exclamation marks in a hundred words is not a friendly tone.
Also fixes a double count: the "dear ..." phrases were in two lists and
were charged twice.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The scrape and the bulk enrichment ran as FastAPI BackgroundTasks, which die with the process: an API restart mid-run lost the work and left the job row at "running" forever. Celery was in the dependencies and a worker booted, but the API never used it - `run_scrape_task` existed and nothing dispatched to it, and it created its *own* job row, so a Celery deployment showed two jobs for one scrape and the one the operator was watching was never updated. - The API now dispatches to a worker when `USE_CELERY` is on, and passes the job id so the row the operator is watching is the row that runs. - A broker that is unreachable must not lose the job: the API falls back to the in-process task and says which path it took, in the response. - The bulk enrichment goes the same way. - `run_scrape_task` can now take an existing job id instead of always creating its own. Four new tests. One of them stubs the scrape out - the fallback path really runs it, and the suite was taking three and a half minutes because of it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`in_quiet_hours()` existed, was served by the API, and was never read by the sender. A campaign started at midnight sent its whole first batch in the middle of the night. An email that lands at 03:00 IST is marked as spam by a human, not a filter - the recipient is asleep, and the only thing the mail achieves is a "spam" report. Every serious cold-email tool schedules to the recipient's local business hours; this one did not, because the product's whole market is IST. - The sender now defers sends outside 09:00-20:00 IST. The send is *deferred*, never dropped: the contact is not marked as sent and no send record is written, so the scheduler picks it up on its next tick, which is inside the window. - The check is per lead, not per campaign. A campaign can mix an Indian lead with an American one, and holding the American one until 09:00 IST would be its 21:30 the previous evening. A lead counts as Indian on an Indian phone number or a state we recognise. - The result says "deferred", with the time the window opens - the difference between "we will send this" and "we lost this". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The parser could read "11-50 employees" into a size filter, and the People table could enforce it - but the wizard had no field for it, so the operator saved an ICP that was not the one they described. The difference is invisible until the results are wrong. - The wizard now has the criteria: minimum and maximum staff, market tier, turnover band, target role, state, "decision makers only" and "must have a reachable email". - The parser's warnings are shown. A constraint the model cannot honour is said out loud, because a silently dropped filter gives the operator a list that looks like an answer and is not. Two new e2e tests: the parsed headcount has to reach the wizard, and an unsupported constraint has to be reported. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…shing A tier-2 Indian SMB owner often reads Marathi, Telugu, Bengali, Kannada, Tamil or Gujarati and not English - but a full-Devanagari cold email reads like a mass blast, which is exactly why the Hinglish register exists. The copy kit now offers the same short, natural closers in the regional script, with a language selector so the panel is not a wall of seven scripts. Also fixes a crash the change introduced, and one the environment exposed: - The language selector's `useState` was declared *after* the `if (!kit) return null` early return. The hook count then depended on whether the kit had loaded, and React threw error #310 the moment it did - which crashed the whole sequence builder, not just the panel. Every hook is now declared before any early return. - The e2e suite's production server was being SIGKILLed partway through a full run: axe-core is injected into every route and analysed in full, and with the default heap the server and the browser together crossed the machine's memory limit. The tests reported it as a connection refused, not as a memory problem - which is why it looked like a flaky suite. The server's heap is now capped (the build's is not: it needs more), and all 49 specs pass in a single run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The People table could be read but not used. Selecting a group of people and putting them in a sequence is the whole point of the surface - a table you can only read is a report, and this is the one table whose job is to produce a list to work. - A checkbox per row, a header checkbox that selects the page (and shows the indeterminate state), and a bulk bar with the count. - "Add to Sequence" opens the sequence picker, and the selected contact ids are what the campaign targets - the same ids the Leads page sends. - The phone card list is selectable too. A selection affordance that only exists on desktop is a selection affordance that does not exist. - The selection survives a page change and a filter change, because the ids are the ones being sent. The new column needed a name of its own: a header with only a checkbox in it is an unnamed column, and the a11y test said so. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A cold-email tool makes decisions that are invisible from the outside: a lead was skipped, an address was suppressed, a send was deferred at 03:00 IST, a subject variant won. Without a record of those decisions, the only way to answer "why did this lead get no email?" is to read the source. - An `audit_events` table and a recorder that never raises - an audit failure must not break the action being audited. - The sender records the decisions that matter: an email sent, a hard bounce (and the suppression that followed), a send deferred to the India window, and a subject variant winning its A/B test. Each carries a *sentence*, not a code: a log of codes is only useful to the person who wrote the codes. - `GET /api/v1/audit` - the feed, filterable by event type and by the thing it happened to. `GET /api/v1/audit/types` - the vocabulary with a count for each, so the filter is discoverable rather than a hard-coded list in the UI. - A `/audit` page, and a compact feed on the overview. Also fixes a test that had become ambiguous: it asserted the *row count* changed when a filter was toggled, but the page size is 25 and both a page of people and a page of business targets are a full page - so the assertion was measuring the page size, not the filter. It now asserts on the total, which is the thing that actually changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reject package-version strings during extraction and verification, import each address in a CSV cell as a contact on one business, and keep discovery source metrics attributed to deduplicated leads.
Update the shipped India workflow and hardening status, and keep the remaining source-trust calibration gap explicit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Eight parallel audits of the codebase found five defects that made the product non-functional, a set of hot paths that were orders of magnitude slower than they needed to be, and a dashboard that was failing accessibility and usability checks. Every finding below was verified by reading the code and executing it.
The five blockers
Mailbox.sent_todaywas a lifetime counter, never reset"bounced"invalid/gibberish/no_mxaddresses — a ~30% hard-bounce rate against Gmail's 5% bulk-sender ceiling.has_emailwas applied after pagination and excluded from the countMeasured performance
At 500,000 leads on SQLite:
/settingsloadThe CSV import was doing ~5 SELECTs per row — 25,000 round-trips for a 5,000-row file, inside a transaction holding the write lock. Search was a
LIKE '%term%'full-table scan; it is now an FTS5 trigram index, which has identical substring semantics (a regression test asserts both paths return identical id sets for whole-word, mid-word, phone-fragment and no-match terms)./settingswas slow because it fired a DNS probe for SPF/DKIM/DMARC that did 13 sequential blocking TXT lookups inside anasync defroute — a trivial request could freeze the whole API.Accessibility: 22 violation types / 106 nodes → 0 on all ten routes
An axe-core audit (WCAG 2.1 A/AA + best-practice) found 48 of the 53 colour-contrast failures were one class — Tailwind's
slate-400on white is 2.56:1 where AA needs 4.5:1 — 50 nodes outside any landmark on every route, a<select>with no accessible name, and no visible focus ring anywhere, so a keyboard user could not tell where they were.Fixed: landmarks throughout, contrast, focus rings, focus trapping and restoration in the mobile drawer,
aria-current,aria-live, one<main>and one<h1>per page, a visible focus ring on every interactive element, and an 11px text floor.This is now a CI gate (
e2e/a11y.spec.ts): it runs axe-core against every route and fails the build on any critical or serious violation, plus asserts no asset 404s and CLS < 0.1. An accessibility regression should be as loud as a type error.Usability
min-w-[1560px]in a 356 px phone container — a phone user could not see a phone number. It is now a card list belowmd./focuses search,Escclears it,j/kmove the row cursor,xselects,?opens help.Also fixed
next startdoes not work withoutput: "standalone": it serves the HTML shell while every JS chunk 404s, so the CI e2e suite was pointed at a page that never finished loading.saved_listwas not registered — socreate_allsilently created nosaved_liststable on a fresh database.contact.id != business.idfor most rows, so an agent mailed the wrong 79% of the database. Added contact IDs, a dry-run default on launch, and JSON-RPCisErrorso the model can read a tool failure.098225 22560) instead of the international one, so every number read as an STD-code landline and mis-filed the STD→city lookup.Verification
tsc --noEmitclean,npm run buildgreen, 33 Playwright specs (12 accessibility + 21 e2e) that can actually fail the build.Full detail, per-item reasoning and the measured before/after numbers are in
ROADMAP.mdandUX-AUDIT.md.🤖 Generated with Claude Code