Skip to content

Serve pull-through blobs while their version commits - #2627

Open
scanner wants to merge 1 commit into
pulp:mainfrom
scanner:fix/pull-through-pending-blob-race
Open

scanner wants to merge 1 commit into
pulp:mainfrom
scanner:fix/pull-through-pending-blob-race

Conversation

@scanner

@scanner scanner commented Nov 1, 2026 •

Copy link
Copy Markdown

The content app could answer 404 for a pull-through blob in two windows while the task adding it to the repository committed:

  • finalize_new_version cleared the blob from pending_blobs, and that committed before pulpcore committed the version as complete.
  • get_by_digest read the repository version and looked the blob up in separate statements, so a commit in between hid the blob.

Clear pending content in on_new_version, after the version is complete, and look the blob up in pending_blobs before reading the repository version.

fixes #2626

Assisted-by: Claude Opus 5.5

📜 Checklist

  • Commits are cleanly separated with meaningful messages (simple features and bug fixes should be squashed to one commit)
  • A changelog entry or entries has been added for any significant changes
  • Follows the Pulp policy on AI Usage
  • (For new features) - User documentation and test coverage has been added

See: Pull Request Walkthrough

@gerrod3 gerrod3 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix! For the second issue can we try to just reverse the order of the lookup to try pending blobs first? I'm currently not liking the read only transaction.

@scanner

scanner commented Nov 12, 2026

Copy link
Copy Markdown
Author

Thanks for the fix! For the second issue can we try to just reverse the order of the lookup to try pending blobs first? I'm currently not liking the read only transaction.

Mmm... yeah, the more I think about it the less happy I am with it. It continues to feel a bit too hacky.

I will try to re-work it this weekend when I have time. I appreciate the feedback.

The content app could answer 404 for a pull-through blob in two windows
while the task adding it to the repository committed:

- finalize_new_version cleared the blob from pending_blobs, and that
  committed before pulpcore committed the version as complete.
- get_by_digest read the repository version and looked the blob up in
  separate statements, so a commit in between hid the blob.

Clear pending content in on_new_version, after the version is complete,
and look the blob up in pending_blobs before reading the repository
version.

fixes pulp#2626

Assisted-by: Claude Opus 5.5
@scanner
scanner force-pushed the fix/pull-through-pending-blob-race branch from 084bed1 to 6127c98 Compare November 15, 2026 20:06
@scanner

scanner commented Nov 15, 2026

Copy link
Copy Markdown
Author

Thanks for the fix! For the second issue can we try to just reverse the order of the lookup to try pending blobs first? I'm currently not liking the read only transaction.

Reworked as suggested. Dropping the transaction costs at most one query and makes the code easier to read. get_by_digest now:

  • gets the repository, and raises PathNotResolved if there is none;
  • looks in the repository's pending blobs, and serves the blob if it is there;
  • otherwise reads the latest repository version and looks there.

The version is read only after the pending lookup, so it can't miss a version that commits in between; test_blob_served_when_version_commits_between_lookups covers that. I also switched aget to afirst, since a blob has only one ContentArtifact for its digest.

@scanner
scanner requested a review from gerrod3 November 15, 2026 20:30

@gerrod3 gerrod3 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Realized this change exposed another bug I hadn't notice before. When using repository_version on a container distribution it shouldn't read from the repository's pending_blob/manifest content. Also, I believe it doesn't make sense to set repository_version and remote on the same distribution.

Let me know if you can work on this bug as well, otherwise we can scope this PR to just the race-condition bugfix.

Comment on lines +94 to +95
# A blob being added to the repository is always either pending or in the latest
# version, so look in the pending blobs first and in the latest version second.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code is easy to understand, the comment should reflect why the order has to be the way it is. Something like To avoid race condition content must always be read pending first, then latest_version, see ContainerRepository.on_new_version

)
ca = await blob_artifacts.filter(content__in=repository.pending_blobs.all()).afirst()

# Not pending: look in the latest version, read only now so that it is no older

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then you can remove the rest of these comments.

Comment on lines +88 to +89
repository = await sync_to_async(self._get_repository)(distribution)
if repository is None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This logic isn't what we want.

ContainerDistributions can have Repository or RepositoryVersion set. Only when setting Repository (i.e always use latest content) do we want to read the pending content. Else we should just read from the repo version's content. I know this isn't how it worked before, but I am now realizing that was a bug.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Heh.. so fixated on 'getting the version I just fetched returned' that should have been obvious.

Although the initial race was seen when I was doing a pull through fetch for something that was not in the cache at all, which means it would still fail (although the occurrence is probably rare), and a subsequent try would succeed.

@scanner

scanner commented Nov 18, 2026

Copy link
Copy Markdown
Author

Realized this change exposed another bug I hadn't notice before. When using repository_version on a container distribution it shouldn't read from the repository's pending_blob/manifest content. Also, I believe it doesn't make sense to set repository_version and remote on the same distribution.

Let me know if you can work on this bug as well, otherwise we can scope this PR to just the race-condition bugfix.

I am about to go on vacation for two weeks (back in the second week of December) if it is okay to wait until then, I can work on that bug.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Pull-through: content app returns 404 for a blob while the task that adds it commits

2 participants