fix(0152_alter_repositoryversion_content_ids): add a batch size to avoid timeout on large update - #8017
fix(0152_alter_repositoryversion_content_ids): add a batch size to avoid timeout on large update#8017bmx0r wants to merge 1 commit into
Conversation
…oid timeout on large update While trying to upgrade pulp this migration was killed after 10min, even after trying to tune db keepalives* parameters We are using an internal dbaas service so we do not have full access to the Database so this was the best solution maybe this could be integrated into guidelines for future migration script Assisted By: Claude Opus 4.8
|
For this specific migration we have a solution in place, where you can do the change beforehand to make the actual migration basically no-op. Classic RT*M: https://pulpproject.org/pulpcore/docs/admin/guides/data-repair/?h=content_id#repair-7465-populate-missing- Here it was introduced as a management command: The one thing I would think is missing here is a hint in the migration to maybe running this command in case the migration failed. |
|
@mdellweg we went from 3.76.1 → 3.116.0 and therefore we hadnt this endpoint available yet... Also I kinda expect that it run more or less the same kind of code resulting in the same kind of long transaction (I havent checked) so even if I totally agree there was something already this MR was more to give the idea to allways add batch mode to future migration of that kind :) about adding a hint, the migration was blocked so all changeset after 152 where not apply so we were probably not in a situation where we want the app to start... |
|
So the batching may make the operation faster, but i think it does not necessarily solve the issue completely. OTOH batching it up may further the chance to hit the 65536-parameter limit of a Postgres operation (not so sure about that one, but there are definitely other resource constraints involved.). OTOH, the whole "transform the rest of the repository-versions before changing the field definition" really needs to go into one transaction (as these migrations usually do). As long as we cannot guarantee the runtime below the timeout, we are kind of stuck. And the sadest part is these errors only turn up in the actual production environment. On the small test boxes we will never run into that timeout. In general, yes it's a good idea to keep migrations performant, and we have had similar issues before. The specifics with this one are (as you already noted with the small batchsizes) it is impossible to predict what the right batch is. |
|
Hi, Luckilly, I experience this on my "small" sandbox, so I will need to reproduce my workaroud in my production soon about the sizing of the table or the number of version per repo: could it be an idea to add it to https://analytics.pulpproject.org/ ? maybe that would give data to improve testing ar dev. guidelines in the future This situation also lead me to this: https://pulpproject.org/pulpcore/docs/user/guides/update-repo-retention/ (yes I read sometimes the docs ;) ) which i believe is the real root cause of my problem. again some data if it can help :) (The totals below don't match the raw core_repositoryversion counts because I filtered to a subset of repos via the label selector ) for i in $(pulp rpm repository list --label-select xxxx=xxxxxx --field name | jq -r .[].name); do echo $i;pulp rpm repository version list --repository "$i" --limit 10000| jq length; doneRepositoryVersion counts Sandbox
RepositoryVersion counts — prod
plotwist: applying retain_repo_versions synchronously deletes the excess versions inline, which blows the 5s immediate-task limit ❯ pulp rpm repository update --name MR-rhel-8-x86_64-baseos_os --retain-repo-versions 50
Started background task /pulp/api/v3/tasks/01a047e9-7f6d-76c3-bc85-e3f31594cfb4/
Error: Task /pulp/api/v3/tasks/01a047e9-7f6d-76c3-bc85-e3f31594cfb4/ failed: '[PLP0024] Immediate task 01a047e9-7f6d-76c3-bc85-e3f31594cfb4 (type: pulpcore.app.tasks.base.ageneral_update) timed out after 5 seconds.'I can do a loop and reduce it bit by bit, yet while i found this issue maybe this is a task that is a good candidate to be moved to deffered ( I saw TASK_PREFER_DEFER but it seems to be a server side task) That is a lot of thinks/thought, if some of it is usefull and you want me to open dedicated issues feel free to shoot, I'll be happy to do it if it can help Cheers |
|
Yeah, there is a workaround to disable the "immediate" running of tasks (https://pulpproject.org/pulpcore/docs/admin/reference/settings/#task_prefer_defer) but it would be a good idea to remove the behavior entirely for large repo version deletes. You're not the first to run into this. |
3 participants
While trying to upgrade pulp this migration was killed after 10min, even after trying to tune db keepalives* parameters
We are using an internal dbaas service so we do not have full access to the Database so this was the best solution maybe this could be integrated into guidelines for future migration script
Assisted By: Claude Opus 4.8
📜 Checklist