Skip to content

Mirror provider EIDs to a cookie and run pull sync as an identity provider - #1489

Open
dhruv8sh wants to merge 12 commits into
feat/s2s-eid-collection-corefrom
feat/s2s-eid-collection-providers
Open

dhruv8sh wants to merge 12 commits into
feat/s2s-eid-collection-corefrom
feat/s2s-eid-collection-providers

Conversation

@dhruv8sh

@dhruv8sh dhruv8sh commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Turns pull sync into an identity provider that runs in the after-response phase from Collect server-side identity provider EIDs in a single page-load write #1487, so pull-synced partner IDs flow through the same coordinator and KV writes as other providers without adding page latency.
  • Mirrors provider-owned bidstream UIDs to a JS-readable __Host-ts-eids-mirror cookie so publisher-side code can read them, on a best-effort basis.

Stacked on #1487; review only the commits on top of it.

Changes

Commit Change
Run pull sync as an identity provider on navigations New trusted-server-identity-providers/src/pull_sync.rs; removes core ec/pull_sync.rs / ec/pull_sync_marker.rs and the old post-send pull sync path in the Fastly adapter. Pull-sync providers are built with collect = "after_response", so they only run for an EC whose consented row already exists
Mirror provider-owned bidstream UIDs to a JS-readable cookie New ec/mirror.rs; ec/finalize.rs, ec/cookies.rs, constants.rs write the __Host-ts-eids-mirror cookie
Make mirroring best effort and fix pull sync validation Mirror failures no longer fail the response; the mirror cookie is expired only when the browser sent one; pull sync config validation fixes
Stop mirroring ownerless UIDs and re-pulling held pull-sync sources Only owned UIDs are mirrored; ownerless batch records count as held for pull sync
Keep batch sync for pull-sync partners Batch sync rejects only sources owned by a configured identity provider, so a partner with both pull sync and an api_token keeps batch sync
Set the mirror cookie only when its value changes An unchanged mirror emits no Set-Cookie, so returning visitors' pages stay cacheable
Skip identity lookups when the rate limiter cannot count A rate-limiter error now skips the lookup (as pull sync did before) for every provider
Reject the pull-sync kind in identity provider config kind = "pull-sync" in [[ec.identity_providers]] is rejected; pull sync comes only from partner config with its token and rate limit
Refuse to key the template cache on the EID mirror cookie __Host-ts-eids-mirror joins the identity cookies the template cache key refuses
Rotate pull-sync partners hourly Partners beyond the per-navigation cap still get turns
Docs Rate-limit rounding, the per-navigation cap and the mirror exposure callout in docs/guide/edge-cookies.md and configuration.md
Tests Pull-sync results mirrored with their owner, non-navigations leaving the mirror untouched, withdrawn rows expiring it

Behaviour changes from main

  • Pull sync timing. Pull sync now runs in the after-response phase, within identity_after_response_budget_ms, and only for an EC whose consented row already exists.
  • pull_sync_concurrency is a cap per navigation. It used to be a batch width with every eligible partner pulled; partners now rotate hourly so each gets turns under the cap.
  • Rate limits. pull_sync_rate_limit is enforced per minute with a minimum of one on Fastly, so the default of 10 per hour behaves like 60.
  • Mirror exposure. Pull-sync UIDs now carry an owner, so for bidstream_enabled partners they reach the script-readable mirror cookie. Previously they lived only in KV.

Closes

Part of #1246 and #1245; the provider implementations that close them come in follow-up PRs.

Spec

Spec: reviewed separately; not included in this PR.

Test plan

  • cargo test-fastly && cargo test-fastly-reuse && cargo test-axum && cargo test-cloudflare && cargo test-spin
  • All eight clippy aliases
  • cargo fmt --all -- --check
  • Parity tests, native template build-digest test and clippy, scripts/lint-wasm-clock.sh
  • JS tests, format and lint
  • Docs format and Markdown prettier check
  • ./scripts/test-cli.sh, domain lint against the base branch, CI Node pin guard, docs-proposal scripts
  • HTML EOF checks (native harness, Axum and Fastly auction controls)
  • Fastly EC integration tests (left to CI)
  • Manual testing via fastly compute serve

Checklist

  • Feature PRs: spec linked above, ending in a review log of two or more agent review rounds
  • Changes follow AGENTS.md conventions
  • No unwrap() in production code — use expect("should ...")
  • Uses log macros (not println!)
  • New code has tests
  • No secrets or credentials committed

@dhruv8sh dhruv8sh self-assigned this Nov 2, 2026
@dhruv8sh
dhruv8sh force-pushed the feat/s2s-eid-collection-providers branch from db9e915 to bdd12a8 Compare November 3, 2026 14:40
@dhruv8sh
dhruv8sh marked this pull request as ready for review November 5, 2026 16:26
@aram356 aram356 added this to the 202611 milestone Nov 5, 2026
Pull-sync partners become provider-owned: browser ingest skips their sources and batch sync rejects them. The post-send dispatcher and the completeness marker are removed, and leftover marker cookies are expired.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
The mirror cookie now holds only UIDs with a provider owner, so browser, batch and legacy pull-sync records are never mirrored, and its size budget counts the pending EID suffix for later UIDs. A pull-sync source with any live UID counts as held, as legacy pull sync did, so ownerless records no longer send the EC ID to the partner on every navigation.

Config load warns when a pull-sync partner also sets a batch api_token, whose mappings batch sync rejects. Restore a test that an oversized pull-sync UID is dropped, and correct the guide on browser fallback ingest.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Batch sync rejects only sources owned by a configured identity provider. A source owned only by its partner's pull sync accepts batch mappings again, as it did before pull sync became a provider.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
The mirror was re-set on every consented navigation with a provider UID, and any Set-Cookie makes the response private and uncacheable, so returning users lost all caching of publisher HTML. The request's mirror value is now kept and the cookie is set only when the new value differs. An unchanged mirror is not re-extended; once it lapses the next navigation sets it again.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
…he cap

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
…rror exposure

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants