Skip to content

S3 Trasnfer Manager - rename files to avoid potential collisions - #7567

Open
RanVaknin wants to merge 4 commits into
masterfrom
rvaknin/download-directory-temp-file-write-path
Open

RanVaknin wants to merge 4 commits into
masterfrom
rvaknin/download-directory-temp-file-write-path

Conversation

@RanVaknin

@RanVaknin RanVaknin commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Background

When downloading 2 or more files in parallel, S3 TM can write both objects into the same local file causing mixed bytes. This is because S3 identifies these as distinct but an OS would treat as the same local destination. for example photos/cat.jpg and photos//cat.jpg are two distinct keys (in s3 they are just two different strings) but in OS land these would resolve to the same local destination.

Changes

In this change we write each download as a temporary filename: cat.jpg.s3tmp.<16 hex> in the same target directory, and then after the file download completes, we rename the file to its original name cat.jpg. The temporary rename + 16 hex avoids such collisions.

In case the temp file name would exceed 255 bytes, we leave out the original name and keep it as s3tmp.<16 hex> (255 byte size is an filesystem limitation)

AI tools

  • I wrote this pull request description myself, not with an AI tool
  • I have reviewed all changes in this pull request myself, including any generated by AI tools

@RanVaknin
RanVaknin force-pushed the rvaknin/download-directory-temp-file-write-path branch from 229b916 to cf93dcb Compare November 19, 2026 06:47
@RanVaknin RanVaknin changed the title Rvaknin/download directory temp file write path S3 Trasnfer Manager - rename files to avoid potential collisions Nov 19, 2026
@RanVaknin
RanVaknin marked this pull request as ready for review November 19, 2026 22:21
@RanVaknin
RanVaknin requested a review from a team as a code owner November 19, 2026 22:21
@RanVaknin RanVaknin closed this Nov 19, 2026
@RanVaknin RanVaknin reopened this Nov 19, 2026
@github-actions

Copy link
Copy Markdown

This pull request has been closed and the conversation has been locked. Comments on closed PRs are hard for our team to see. If you need more assistance, please open a new issue that references this one.

@github-actions github-actions Bot locked as resolved and limited conversation to collaborators Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant