CLVM: Added notes on CLVM Improvements and optimizations

source/adminguide/storage/clvm.rst

@@ -67,7 +67,7 @@ CLVM_NG (QCOW2 format):
 
 - Slight performance overhead from QCOW2 layer
 - More complex storage stack
-- Bitmaps lost during volume migration - hence the next snapshot will be a full snapshot (but subsequent snapshots will be incremental again)
+- After a VM migration with storage, the snapshot chain is ended: the next snapshot is a full snapshot and a new chain starts. Moving a volume to another volume group, even across clusters, keeps its bitmaps and the next snapshot stays incremental
 
 Key features
 ~~~~~~~~~~~~
@@ -87,7 +87,25 @@ CLVM/CLVM_NG use `lvmlockd` and `sanlock` to provide exclusive locking of logica
 - **Active Volumes**: Locked exclusively on the host running the VM
 - **Inactive Volumes**: Can be in shared mode or deactivated
 - **Lock Transfer**: Automatic during VM migration
-- **Lock Tracking**: CloudStack tracks which host owns each volume lock
+- **Lock Tracking**: CloudStack records which host owns each volume lock as a hint and verifies it against the real state, see *Finding the lock holder*
+
+Finding the lock holder
+^^^^^^^^^^^^^^^^^^^^^^^
+
+CloudStack records each volume's lock holder (the ``clvmLockHostId`` volume detail) only as a hint, and verifies it when an operation needs the holder:
+
+#. It asks the recorded host whether the volume is active there.
+#. If not, it reads the volume's sanlock lease from the hidden ``lvmlock`` volume of the volume group, which any host can read even when the holder is down. The lease names the owner's sanlock ``host_id``, which is mapped to a CloudStack host through the ``host.sanlock.id`` host detail and confirmed with that host.
+#. If the lease is free, or its owner has restarted since it took the lock, the volume is activated exclusively on a chosen host, preferring the destination of the operation.
+#. Only when the lease cannot be read, or the owner cannot be mapped to exactly one host, CloudStack asks every host in parallel (see ``clvm.lock.query.threads``).
+
+Right after a host failure or restart, an activation can be refused temporarily: ``locked by other host`` until the old holder's lease expires (140 seconds by default), and ``lock start in progress`` while the lockspace starts (about two minutes). These are retried until ``clvm.lock.recovery.retry.timeout``.
+
+Requirements:
+
+- Each host needs a unique numeric ``host_id`` in ``/etc/lvm/lvmlocal.conf`` (1 to 2000), the same for all volume groups of the host, and must have joined the lockspace of every volume group of its pools.
+- The KVM agent reports the ``host_id`` as ``host.sanlock.id`` each time it connects. After changing ``host_id``, make the host rejoin the lockspace (reboot it), then restart the agent.
+- A host without ``host.sanlock.id`` is logged with a warning, and a duplicate raises an alert. In both cases lookups ask every host.
 
 Lightweight Migration
 ^^^^^^^^^^^^^^^^^^^^^
@@ -201,6 +219,35 @@ clvm.secure.zero.fill
 - **Recommendation:** Enable for environments with strict security/compliance requirements (PCI-DSS, HIPAA). Disable for performance-critical environments where deletion speed matters more than data security
 - **Use Case:** Production environments handling sensitive data, multi-tenant environments with strict data isolation requirements
 
+clvm.lock.recovery.retry.timeout
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+- **Type:** Integer
+- **Default:** `180`
+- **Scope:** Global
+- **Category:** Advanced
+- **Description:** Maximum time in seconds, counted from the start of the lock holder lookup, to keep retrying the exclusive activation of a volume that is refused only temporarily: the previous holder's lease has not expired yet, or the lockspace of the volume group is still starting on the host. `0` disables the retry.
+- **Recommendation:** The lease expires 8 x `io_timeout` + 60 seconds after the holder stops renewing it (140 seconds with the default lvmlockd sanlock `io_timeout` of 10). If lvmlockd runs with a different `--sanlock-timeout`, set this to at least that value plus about 40 seconds.
+
+clvm.lock.recovery.retry.interval
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+- **Type:** Integer
+- **Default:** `10`
+- **Scope:** Global
+- **Category:** Advanced
+- **Description:** Time in seconds between retries of an activation that is refused only temporarily.
+
+clvm.lock.query.threads
+^^^^^^^^^^^^^^^^^^^^^^^
+
+- **Type:** Integer
+- **Default:** `5`
+- **Scope:** Global
+- **Category:** Advanced
+- **Description:** Number of threads a management server uses to send CLVM lock commands to several hosts in parallel: the fallback lookup that asks every host, and the deactivation of a template volume on all hosts before it is removed.
+- **Propagation:** Requires a management server restart.
+
 For incremental snapshots to work with CLVM_NG enabled the `kvm.incremental.snapshot` global setting must also be set to `true`.
 
 VM snapshots are currently not supported on CLVM/CLVM_NG volumes.
@@ -219,7 +266,7 @@ systemctl enable sanlock --now
 systemctl enable lvmlockd --now
 
 # Update /etc/lvm/lvm.conf to set use_lvmlockd = 1
-# Update /etc/lvm/lvmlocal.conf set host_id to a unique value for each host (e.g. hostname or IP or ID (1, 2, 3...))
+# Update /etc/lvm/lvmlocal.conf set host_id to a unique number (1 to 2000) for each host (e.g. 1, 2, 3...); it must be numeric and is used for every volume group of the host
 ```
 
 Just on one of the hosts, initialize the shared physical volume (PV) that will be used for CLVM storage pools and start the VG in shared mode:
@@ -238,6 +285,6 @@ vgchange lockstart sharedvg # Start the VG in shared mode
 
 .. note::
    It is reccommended to add CLVM / CLVM_NG storage pools as a Cluster scoped resource, so as to limited number of hosts that can access the pool for optimal performance and reliability.
-   Due to the lack of support from `lvm` to provide a centralized way to query which host has the lock for a given volume, CloudStack relies on tracking this information in the database. However, if there is a drift in this information, for example due to a host failure, CloudStack will attempt to query each host for the lock status of the volume. This can lead to performance issues if there are a large number of hosts in the cluster as each host will need to be queried for each volume.
+   `lvm` has no centralized way to query which host has the lock for a given volume, so CloudStack tracks it in the database and verifies it against the sanlock lease of the volume (see *Finding the lock holder*). Only when the lease cannot be read or mapped to a single host does CloudStack query each host of the pool, which is slower with a large number of hosts. This is another reason to add the pool as a cluster scoped resource.
 
-   VM snapshots and incremental volume snapshot are not supported for CLVM/CLVM_NG volumes as of CloudStack 4.23.0.0.
+   VM snapshots are not supported for CLVM/CLVM_NG volumes as of CloudStack 4.23.0.0. Incremental volume snapshots are supported on CLVM_NG only, and need the ``kvm.incremental.snapshot`` global setting set to ``true``.