Repository navigation
vmware: reduce repeated requests for the same vCenter data - #634
Open
tyderian1978 wants to merge 1 commit into
Open
tyderian1978 wants to merge 1 commit into
tyderian1978 wants to merge 1 commit into
Conversation
Every read of a property of a pyVmomi managed object is a SOAP round trip which returns the whole top level property. The VMware source read e.g. a VM's 'config' about 9 times, its 'guest' once per NIC, the ~1 MB 'config' of a host 6 times per host, and walked the same folders and resource pools again for every VM. * read 'config', 'runtime', 'guest' of a VM and 'summary', 'config' of a host once per object * skip VMs already parsed in the first run over all VMs by their managed object id instead of requesting name and config again * cache names, parents and resource pools of objects for the run; the folder and resource pool walks for VM paths, tenants and resource pool filters use these caches * cache the tags attached to an object for the run (the tags of parent folders, clusters and datacenters are requested for every VM); tag and category definitions are already cached since ec1ba92 * cache custom attribute definitions per object type and only request them if the object has custom values * hand the already retrieved guest info to add_update_interface() through the new SourceBase.get_vmware_guest_info() hook get_parent_object_by_class() kept its depth in an instance attribute which was not reset when a walk ended at the root, so later lookups could give up early. The rewrite walks iteratively and has no such state. Measured against vcsim with the captured inventories, synced inventory unchanged: vchvr: 1469 -> 323 SOAP calls, 20.1 -> 3.6 MB vc001: 1707 -> 355 SOAP calls, 30.1 -> 6.0 MB
tyderian1978
force-pushed
the
perf/vmware-quick-wins
branch
from
November 18, 2026 23:55
3c9245f to
a2b1681
Compare
This was referenced Nov 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Disclaimer: AI generated, based off the work we're doing with Netbox.
This is one of three pull requests, aimed to significantly reduce both VMware inventory sync time and data usage.
What happens
pyVmomi caches nothing: every attribute read on a managed object is a SOAP
Fetchof the whole top-level property, sograb(vm, "config.uuid")andgrab(vm, "config.hardware.device")each transfer the VM's completeconfig. The VMware source read a VM'sconfigabout 9 times, itsguestonce per NIC, a host's ~1 MBconfig6 times, and walked the same folders and resource pools up the tree again for every VM. On thevchvrvcsim capture that's 1469 SOAP calls and 20.1 MB per run.Fix
config,runtimeandguestof a VM, andsummaryandconfigof a host, are read once per object.add_update_interface()gets the guest info that was already read through a newSourceBase.get_vmware_guest_info()hook.This also fixes a bug:
get_parent_object_by_class()kept its depth in an instance attribute that wasn't reset when a walk ended at the root, so later lookups could give up early. It now walks iteratively and keeps no state.Tests
tests/test_vmware_vcenter_requests.pyruns the sync against vcsim with the captured inventories and counts the SOAP requests per object and property:All of them fail on
developmentwithout the change. Full suite: 842 passed, 8 skipped.Numbers
developmentThe synced inventory is identical to
development's for both captures, with default settings and with the FC adapter, tenant and resource pool settings.A dry run against our production NetBox and five vCenters (about 1,500 VMs, over VPN, tag sync on) shows the same pattern. The VMware source went from 80,654 SOAP requests, 1.15 GB and 48 min on
developmentto 12,435 requests, 198 MB and 10.5 min. Its planned changes were identical to those of runs with all three PRs made directly before and after it. Those runs in turn matcheddevelopment's, apart from one VM that was removed from vCenter during the runs.