Batch resource_count UPDATEs across tags to reduce lock contention - #13915
Open
nvazquez wants to merge 3 commits into
Open
Batch resource_count UPDATEs across tags to reduce lock contention#13915nvazquez wants to merge 3 commits into
nvazquez wants to merge 3 commits into
Conversation
…pache#835) Concurrent restoreVirtualMachine on KVM clusters that have tagged storage limits configured (resource.limit.storage.tags) was failing at scale with MySQL "Lock wait timeout exceeded" (errcode 1205). On the worst-affected clusters the failure rate for reimage-vm reached ~43%, and >98% of failures during storm windows traced to RestoreVMCmdByAdmin bottoming out at ResourceCountDaoImpl.updateCountByDeltaForIds. Root cause: the volume and primary_storage resource-count entry points iterated the configured tag list (the untagged sentinel plus each storage tag) and issued one UPDATE cloud.resource_count per (type, tag) pair. Each UPDATE acquired X-locks on the account row and every parent domain row for that pair, all held until the outer restoreVirtualMachine transaction committed. With multiple sequential UPDATEs per restore and concurrent callers serializing on the shared rows, the in-transaction lock-acquire chain exceeded innodb_lock_wait_timeout (50s default). Fix: add removeResourceReservationIfNeededAndIncrementResourceCountForTags and decrementResourceCountForTags helpers that resolve the union of resource_count row IDs across the full tag list and issue a single batched UPDATE per ResourceType. Migrate all four affected entry points onto these helpers: - incrementVolumeResourceCount - decrementVolumeResourceCount - incrementVolumePrimaryStorageResourceCount - decrementVolumePrimaryStorageResourceCount For an N-tag configuration this collapses 2N sequential UPDATEs to 2 Signed-off-by: Ganesh Maharaj Mahalingam <g.mahalingam@apple.com> Co-authored-by: Ganesh Maharaj Mahalingam <g.mahalingam@apple.com> (cherry picked from commit a8bcb52cee166e362e4ec2744a322ae2588cfe5b)
Contributor
Author
|
@blueorangutan package |
|
@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress. |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## 4.22 #13915 +/- ##
============================================
- Coverage 17.77% 17.77% -0.01%
+ Complexity 15983 15981 -2
============================================
Files 5928 5928
Lines 534301 534347 +46
Branches 65382 65393 +11
============================================
- Hits 94987 94979 -8
- Misses 428568 428622 +54
Partials 10746 10746
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18907 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Concurrent restoreVirtualMachine on KVM clusters that have tagged storage limits configured (resource.limit.storage.tags) was failing at scale with MySQL "Lock wait timeout exceeded" (errcode 1205). On the worst-affected clusters the failure rate for reimage-vm reached ~43%, and >98% of failures during storm windows traced to RestoreVMCmdByAdmin bottoming out at ResourceCountDaoImpl.updateCountByDeltaForIds.
Root cause: the volume and primary_storage resource-count entry points iterated the configured tag list (the untagged sentinel plus each storage tag) and issued one UPDATE cloud.resource_count per (type, tag) pair. Each UPDATE acquired X-locks on the account row and every parent domain row for that pair, all held until the outer restoreVirtualMachine transaction committed. With multiple sequential UPDATEs per restore and concurrent callers serializing on the shared rows, the in-transaction lock-acquire chain exceeded innodb_lock_wait_timeout (50s default).
Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Bug Severity
Screenshots (if appropriate):
How Has This Been Tested?
How did you try to break this feature and the system with this change?