[novatechflow]

New LFS objects now go under their own prefix instead of inside the topic's segment folder.

Layout:

• Before: <ns>/<topic>/lfs/yyyy/mm/dd/obj-<uuid>, inside generation 0's topic folder.
• Now: <ns>/@lfs/<topic>/yyyy/mm/dd/obj-<uuid>, and <ns>/@lfs/<topic>@<N>/... after the Nth delete. This uses the same storage name as the segments (metadata.TopicStorageName).

Why @lfs: Kafka topic names only allow [a-zA-Z0-9._-], so @lfs can never be a topic prefix. @ already marks generations in storage names.

Generation: the proxy reads it from etcd when etcd is configured. Without etcd it uses 0. If the lookup fails, the upload fails instead of writing to the wrong generation.

Old objects: they are not moved. The envelope stores the full key, and the proxy download, the Go resolver and the SDKs read env.key as is. The download key check accepts both /lfs/ and /@lfs/.

Prune is unchanged. Its segment allowlist skips both old and new LFS keys. LFS pruning is not part of this PR.

Tests:

• TestBuildObjectKeyUsesOwnPrefix
• TestRewriteProduceWritesOwnPrefix
• TestDownloadOldPathStillReadable
• TestPruneSkipsLFSAndUnknownKeys, extended with @lfs keys

Metadata lookup timeout

The topic generation lookup runs with a timeout, 3000 ms by default, set with KAFSCALE_LFS_PROXY_METADATA_TIMEOUT_MS. When etcd is configured but not reachable, /lfs/produce and /lfs/uploads return 503 metadata_unavailable, and a Kafka produce with LFS records returns an error. Generations are looked up once per topic before any upload, so a failed lookup uploads nothing. The download key check accepts only <ns>/@lfs/... and the old <ns>/<topic>/lfs/... shape.

Generation recheck before produce

After the upload and right before the produce, the proxy looks up the topic generation again, with the same bounded lookup. This covers the Kafka rewrite path, HTTP /lfs/produce, and multipart complete. If the generation changed or the lookup fails, the proxy deletes the uploaded objects, reports objects it cannot delete as orphans, does not produce, and returns the metadata error: 503 metadata_unavailable over HTTP, and the retriable produce error on Kafka.

Remaining window: a delete and recreate that happens between this second lookup and the produce is not caught. The gap is one etcd read plus encoding the request, so it is small but not zero. Closing it fully would need the produce itself to be fenced on the generation.

[PaxMachinaOne]

🤖 PaxMachina automated review (worker-code)

The new layout works normally, but generation changes can place live references under the wrong incarnation.

• cmd/proxy/lfs_http.go:769 — Multipart uploads select and persist the generation-specific key only at upload initialization. A topic can be deleted and recreated before completion, after which handleHTTPUploadComplete publishes the envelope to the new incarnation while retaining the old generation’s key. This defeats incarnation isolation and could make a live object eligible for future old-generation pruning. Revalidate/guard the generation at completion and cleanly abort if it changed; the streaming and intercepted-produce paths have a smaller version of the same lookup-to-publish race.

[PaxMachinaOne]

🤖 PaxMachina automated review (worker-code)

Generation selection has a delete/recreate race that can place objects under the wrong incarnation.

• Correctness: cmd/proxy/lfs_rewrite.go:137 reads the generation once, then uploads later at line 180 without revalidating it or coupling it atomically to the Kafka produce. If the topic is deleted and recreated between those operations, the record can be produced to generation N+1 while its object is stored under @lfs/<topic>@N/. The HTTP and multipart paths have the same TOCTOU window at cmd/proxy/lfs_http.go:295 and line 769; multipart uploads can make it especially long. This defeats the promised per-incarnation separation and could make future generation-based LFS pruning unsafe.

[PaxMachinaOne]

🤖 PaxMachina automated review (worker-code)

The new prefix handling is sound, but multi-topic generation validation leaves a correctness race.

• cmd/proxy/lfs_rewrite.go:243 — Generation rechecks run sequentially over every topic, each with its own timeout. An early topic can therefore be deleted and recreated while later checks consume up to 3 seconds each, after which its record is produced with an object from the stale generation. For multi-topic requests, the documented “short gap” can grow to roughly topic count × metadata timeout. Recheck generations as one consistent etcd read/transaction, or at least concurrently immediately before dispatch.