Repository navigation
vmware: retrieve object properties from vCenter in bulk - #635
Open
tyderian1978 wants to merge 2 commits into
Open
tyderian1978 wants to merge 2 commits into
tyderian1978 wants to merge 2 commits into
Conversation
Every read of a property of a pyVmomi managed object is a SOAP round trip which returns the whole top level property. The VMware source read e.g. a VM's 'config' about 9 times, its 'guest' once per NIC, the ~1 MB 'config' of a host 6 times per host, and walked the same folders and resource pools again for every VM. * read 'config', 'runtime', 'guest' of a VM and 'summary', 'config' of a host once per object * skip VMs already parsed in the first run over all VMs by their managed object id instead of requesting name and config again * cache names, parents and resource pools of objects for the run; the folder and resource pool walks for VM paths, tenants and resource pool filters use these caches * cache the tags attached to an object for the run (the tags of parent folders, clusters and datacenters are requested for every VM); tag and category definitions are already cached since ec1ba92 * cache custom attribute definitions per object type and only request them if the object has custom values * hand the already retrieved guest info to add_update_interface() through the new SourceBase.get_vmware_guest_info() hook get_parent_object_by_class() kept its depth in an instance attribute which was not reset when a walk ended at the root, so later lookups could give up early. The rewrite walks iteratively and has no such state. Measured against vcsim with the captured inventories, synced inventory unchanged: vchvr: 1469 -> 323 SOAP calls, 20.1 -> 3.6 MB vc001: 1707 -> 355 SOAP calls, 30.1 -> 6.0 MB
pyVmomi requests each property read from vCenter separately. Retrieve the properties the handlers need for many objects at once with the PropertyCollector (RetrievePropertiesEx over a ListView, paged with ContinueRetrievePropertiesEx) instead. The new PropertyCache wraps the session's SoapStubAdapter as its 'outerStub', so every managed object returned through it (prefetched objects, their parents, the host and resource pool of a VM, ...) reads prefetched properties from the cache. Properties which were not prefetched are still requested from vCenter: a path missing from the prefetch list costs time but never returns wrong data. The handlers are unchanged apart from reading the nested paths 'config.network', 'config.storageDevice.hostBusAdapter' and custom object attributes through get_prefetched_value(). * objects are prefetched and parsed in chunks of 250, the heavy properties of a chunk are dropped afterwards to bound memory * folders and resource pools are prefetched (name, parent) for the walks up the tree (datacenter, VM folder and resource pool paths) * only the parts of a host's ~1 MB 'config' which are used are requested * the second run over all VMs only requests the VMs not parsed yet * objects to reevaluate are prefetched again in one go * attached vCenter tags are requested per chunk with list_attached_tags_on_objects(), falling back to one request per object * custom object attribute paths vCenter refuses are dropped from the bulk request for the rest of the run and read object by object * if bulk retrieval fails, objects read their properties one by one as before Measured against vcsim with the captured inventories, synced inventory unchanged (relative to the previous commit / to development): vchvr: 323 / 1469 -> 68 SOAP calls, 3.6 / 20.1 -> 1.5 MB vc001: 355 / 1707 -> 66 SOAP calls, 6.0 / 30.1 -> 2.5 MB The number of requests no longer grows with the number of VMs (about 4 per 250 VMs plus one QueryNetworkHint per host).
tyderian1978
force-pushed
the
perf/vmware-bulk-fetch
branch
from
November 18, 2026 23:55
e47d33b to
6f2f5a7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Disclaimer: AI generated, based off the work we're doing with Netbox.
This is the second of three pull requests, aimed to significantly reduce both VMware inventory sync time and data usage.
Builds on #634: its commit is the first one here, and the change in this PR is the second commit (6f2f5a7).
What happens
Even with each property read only once (#634), every VM and host still costs several SOAP round trips: name, parent, config, runtime, guest and so on. The number of requests grows with the inventory, and over a WAN or VPN link the round trips dominate the run. In our production dry run the VMware source still spent 10.5 minutes, mostly waiting for vCenter.
Fix
The properties the handlers need are retrieved for many objects at once with the PropertyCollector:
RetrievePropertiesExover aListView, paged withContinueRetrievePropertiesEx. AListViewdrops objects deleted in the meantime instead of failing the request.The new
PropertyCache(module/sources/vmware/property_cache.py) wraps the session'sSoapStubAdapteras itsouterStub. As a result, every managed object returned through it (prefetched objects, their parents, a VM's host and resource pool, ...) reads prefetched properties from the cache. Properties that weren't prefetched are still requested from vCenter, so a path missing from the prefetch list costs time but never returns wrong data. The handlers are unchanged apart from readingconfig.network,config.storageDevice.hostBusAdapterand custom object attributes throughget_prefetched_value().configthat are used are requested.list_attached_tags_on_objects(), falling back to one request per object.InvalidProperty) are dropped from the bulk request for the rest of the run and read object by object.SoapStubAdapter, or bulk retrieval fails, objects read their properties one by one as before.For review
PropertyCacherelies on pyVmomi'sSoapStubAdapter.InvokeMethod(mo, info, args, outerStub)andStubAdapterAccessorMixin.InvokeAccessor. It was tested with pyvmomi 9.1.1.0, as pinned inrequirements.txt.vcf-sdk(create and list calls return nothing), and vcsim ignores invalid property paths. So the bulk tag call, its per-object fallback and theInvalidPropertyretry are covered by tests with fakes, not by vcsim. Against our production vCenters, tag sync ran without errors and gave the same tags asdevelopment.Tests
Added to
tests/test_vmware_vcenter_requests.py, against vcsim unless noted:InvalidProperty)All of the file's tests fail on
developmentexcept the unknown custom object attribute test, which guards behaviourdevelopmentalready has. Full suite: 851 passed, 8 skipped.Numbers
developmentThe number of requests no longer grows with the number of VMs: about 4 per 250 VMs, plus one
QueryNetworkHintper host. The synced inventory is identical todevelopment's for both captures, with default settings and with the FC adapter, tenant and resource pool settings.A dry run against our production NetBox and five vCenters (about 1,500 VMs, over VPN, tag sync on):
developmentThe planned changes are the same as
development's. The only differences come from one VM that was removed from vCenter between the runs.