Skip to content

Replace single-key tracker storage with per-URL stores - #46

Merged
ryanmitchell merged 1 commit into
mainfrom
feature/per-url-tracker-storage
Nov 6, 2026
Merged

ryanmitchell merged 1 commit into
mainfrom
feature/per-url-tracker-storage

Conversation

@ryanmitchell

Copy link
Copy Markdown
Contributor

Why

Every tracked URL lived in one serialized array under tracker::urls. On a site with ~11k URLs that's a 13+ MB value, and:

  • every GET reaching the middleware read the whole thing for one has() check
  • every tracked render read it twice and rewrote all of it
  • every invalidation scanned every URL and rewrote all of it
  • concurrent writes overwrote each other, so URLs dropped out of tracking and went stale in the static cache
  • on Redis, each large GET or SET held up every other command on that Redis while it ran

What

Storage goes through a small Store interface with two implementations, picked automatically from the static cache store. There's no new config.

  • RedisStore (when the store is Redis): one hash of URL records plus one set per tag. Commands touch a single key, so it works on Redis Cluster, and writes are pipelined.
  • CacheStore (any other Laravel cache store): one key per URL, a URL index split across 256 keys, and one key per tag listing its URLs. Writes to those shared keys are locked. Leftover entries in a tag's list are checked and removed at lookup time.

Existing tracker::urls data is moved over automatically on first use, behind a lock so only one process does it.

The public Tracker API is unchanged. This also adds Tracker::findByTags(), which GetUrlsController now uses instead of scanning everything.

Why two drivers: the generic driver alone fixes reads and invalidation. Tracking a page, though, has to lock, read and rewrite the URL list of each tag used on nearly every page (globals, navs, partials). After a global or nav save invalidates the whole site, a re-warm queues every render on those locks. Redis adds a URL to a set (SADD) in a single call with no lock.

Numbers

11,218 URLs, 30 tags each, local Redis (no network time):

Old single key Redis driver Cache driver
has() 14.1 ms 0.02 ms 0.02 ms
add() 23.4 ms 0.07 ms 10.3 ms
invalidate one tag (42 URLs) 27.1 ms 0.14 ms 0.15 ms
wildcard invalidate (all URLs) 27 ms 30 ms 32 ms
one-off migration n/a 0.2 s 2.7 s

Notes

  • Wildcard tag invalidation still checks every URL, same as before. The addon never issues wildcard tags itself.
  • cache-tracker:list now sorts by URL, since all() order is no longer the order URLs were added.
  • Fixed it_doesnt_track_pages_already_in_the_manifest. It tracked / while the middleware tracks http://localhost, so it only passed because of array ordering.

Testing

  • New tests/Unit/StoreTest.php runs against both drivers. It covers stale tags, wildcards, import, flush and migration. The Redis cases are skipped when Redis isn't available.
  • CACHE_TRACKER_TEST_REDIS=1 vendor/bin/pest runs the whole suite against Redis (it uses db 15).
  • The full suite passes on both: 59 tests each.

🤖 Generated with Claude Code

All tracked URLs lived in one serialized array under tracker::urls, so
every GET, tracked render and invalidation read (and often rewrote) the
whole blob, and concurrent writes lost entries.

Storage now goes through a small Store interface with two implementations,
picked automatically from the static cache store:

- RedisStore: a hash of URL records plus one set per tag, using single-key
  commands only so it works on Redis Cluster.
- CacheStore: any Laravel cache store, with one key per URL, a sharded URL
  index and locked per-tag buckets whose stale entries are pruned on lookup.

Existing data is migrated from tracker::urls automatically on first use.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ryanmitchell
ryanmitchell merged commit 5123ee7 into main Nov 6, 2026
4 checks passed
@ryanmitchell
ryanmitchell deleted the feature/per-url-tracker-storage branch November 6, 2026 13:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant