Repository navigation
Conversation
RanVaknin
force-pushed
the
rvaknin/download-directory-temp-file-write-path
branch
from
November 19, 2026 06:47
229b916 to
cf93dcb
Compare
|
This pull request has been closed and the conversation has been locked. Comments on closed PRs are hard for our team to see. If you need more assistance, please open a new issue that references this one. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
When downloading 2 or more files in parallel, S3 TM can write both objects into the same local file causing mixed bytes. This is because S3 identifies these as distinct but an OS would treat as the same local destination. for example
photos/cat.jpgandphotos//cat.jpgare two distinct keys (in s3 they are just two different strings) but in OS land these would resolve to the same local destination.Changes
In this change we write each download as a temporary filename:
cat.jpg.s3tmp.<16 hex>in the same target directory, and then after the file download completes, we rename the file to its original namecat.jpg. The temporary rename + 16 hex avoids such collisions.In case the temp file name would exceed 255 bytes, we leave out the original name and keep it as
s3tmp.<16 hex>(255 byte size is an filesystem limitation)AI tools