Skip to content

inventory: index lookups instead of scanning every object - #636

Open
tyderian1978 wants to merge 1 commit into
bb-Ricardo:developmentfrom
tyderian1978:perf/inventory-indexes
Open

tyderian1978 wants to merge 1 commit into
bb-Ricardo:developmentfrom
tyderian1978:perf/inventory-indexes

Conversation

@tyderian1978

Copy link
Copy Markdown

Disclaimer: AI generated, based off the work we're doing with Netbox.

This is the third of three pull requests, aimed to significantly reduce both VMware inventory sync time and data usage.

Independent of the two VMware PRs (#634, #635); this one only touches the inventory and its callers.

What happens

Syncing N objects compares each one with every object of its type in several places, so a sync costs O(N²). Those places are:

  • finding a VM or device by name and cluster, and a VM interface or disk by name and VM
  • listing the interfaces, IPs, MACs and disks of an object
  • finding the longest matching prefix
  • matching VMs by MAC, serial or primary IP
  • resolving NetBox IDs

This doesn't show while vCenter reads dominate a run, but it does once they're fast. In a benchmark where the inventory already holds N VMs (1.5 NICs, MACs and IPs per VM, N/4 prefixes), running the VMware source's per-VM matching for every VM through the real code paths takes:

VMs development this PR
1000 8.6 s 0.36 s
2000 39 s 0.71 s
4000 164 s 1.43 s

The indexes cost memory: peak RSS at 4000 VMs goes from 102 MB to 130 MB.

Fix

The inventory keeps per-type indexes from an attribute value to the objects carrying it: NetBox ID, primary key, interface parent, IP address, assigned object, MAC address, prefix network, serial, asset tag and primary IPs. An index is built on the first lookup of its type.

  • An index only selects candidates. Every lookup still applies its original comparison to them in inventory order, so it returns exactly what the full scan returned (first match wins, case-insensitive names).
  • Keys depend only on the object's own data. The primary key index files an object under its lower-cased primary key value, which every display name starts with; a lookup tries the whole name and every part in front of a " (". Values without a stable string, i.e. objects, are filed as volatile and returned for every lookup.
  • NetBoxObject.data is an NBDataDict that reports every write, and assigning data or nb_id is reported too. So an object is re-keyed right away, whatever changes it: update(), resolve_relations() or a direct write.
  • An index is rebuilt if its object list was replaced or changed outside add_object(), as tests do.

SourceBase.ip_address_objects_for() from #622 keeps its interface and callers but now answers from the inventory's address index. That index follows address changes of existing objects and a replaced inventory list, which the append-only per-source index didn't, and the primary IP matching uses the same index, so there's one address index instead of two.

Unchanged:

  • With set_primary_ip = always, each existing primary IP still scans all devices and VMs to remove it from other objects.
  • A cable is still found by scanning all cables.
  • Nothing looks up cables through get_by_data(). An unlabelled cable's display name doesn't start with its primary key, so a future caller would need a look.

Tests

tests/test_inventory_indexes.py compares the indexed lookups with full scans. It covers:

  • case-insensitive names, renames and secondary key changes
  • duplicate names returning the first object in inventory order
  • names with parentheses and VLAN display names
  • ID changes
  • interfaces loaded from NetBox and then resolved
  • IP reassignment and readdressing
  • a list replaced behind the inventory
  • virtual disks following their VM
  • longest prefix match
  • MAC and primary IP candidates
  • a randomized sequence of mutations checked against full scans after every step

tests/test_ip_address_index.py gets a case for an address change after the first lookup.

test_inventory_indexes.py checks the index API itself, so it doesn't run on development. The new test_ip_address_index.py case fails there. Full suite: 850 passed, 8 skipped.

Numbers

In a dry run against our production NetBox (about 37,000 IPs, 1,500 VMs) and five vCenters, on top of the bulk fetch PR, the VMware source took 122 s without this PR and 52 s and 70 s in two runs with it. Without the SOAP requests, that's 91 s against 23 s and 28 s. The planned changes were byte-identical with and without this PR. Without bulk fetch, the saving is lost in the 48 min of vCenter requests.

Syncing N objects compared each one with every object of its type in
several places, so a sync cost O(N^2): finding a VM/device by name and
cluster or a VM interface/disk by name and VM, listing the interfaces,
IPs, MACs and disks of an object, the longest matching prefix, matching
VMs by MAC, serial or primary IP and resolving NetBox IDs.

The inventory now keeps per object type indexes from an attribute value to
the objects carrying it (NetBox ID, primary key, interface parent, IP
address, assigned object, MAC address, prefix network, serial, asset tag,
primary IPs). An index is built on the first lookup of its type.

Invariants:
* An index only selects candidates. Every lookup still applies its
  original comparison to them in inventory order, so it returns exactly
  what the full scan returned (first match wins, case-insensitive names).
* Keys depend only on the object's own data. The primary key index files
  an object under its lower-cased primary key value, which every display
  name starts with; a lookup tries the whole name and every part in front
  of a " (". Values without a stable string (i.e. objects) are filed as
  volatile and returned for every lookup.
* NetBoxObject.data is an NBDataDict which reports every write, and
  assigning data or nb_id is reported too, so an object is re-keyed right
  away whatever changes it (update(), resolve_relations(), direct writes).
* An index is rebuilt if its object list was replaced or changed outside
  add_object(), as tests do.

SourceBase.ip_address_objects_for() (4f5f686) keeps its interface and
callers but now answers from the inventory's address index. That index
follows address changes of existing objects and a replaced inventory list,
which the append-only per-source index did not, and it is the same index
the primary IP matching uses, so there is one address index instead of two.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant