Skip to content

About

Header-only C++20 lock-free shared pointers for Windows kernel drivers (x64 & ARM64, IRQL <= DISPATCH_LEVEL). Features DPC-based Quiescent State RCU, Epoch-based cache-aware Rundown with NUMA-sharded O(1) batch GC, and a 128-bit split-count atomic shared_ptr—eliminating per-object read-path RMWs and cache-line contention.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

3 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Windows Kernel Lock-Free Shared Pointers (KmRcuSharedPtr, KmRundownSharedPtr, KmAtomicSharedPtr)

Platform: Windows Kernel (x64 / ARM64) Language: C++20 (MSVC 2026) Environment: Kernel Mode

Project Highlights

  • Zero-RMW Reader Scaling: Eliminates per-object atomic read-modify-write (RMW) instructions for transient readers (0 atomic RMWs for RCU; CPU-sharded acquisition for Rundown), reducing cross-core cache invalidation on multi-core systems.
  • NUMA-Aware Retirement Sharding (RCU & Rundown): Routes displaced control blocks into node-local, cache-aligned lock-free SLIST_HEADER shards to reduce cross-socket interconnect traffic during pointer updates.
  • $O(1)$ Batch Synchronization: Replaces per-object OS synchronization waits with driver-wide Epoch-based and Quiescent-state collectors that drain and reclaim retired objects in batches.
  • Cache-Line Isolation: Single-allocation factories place payload data adjacent to control blocks at cache-aligned offsets, separating read-mostly payload fields from control-block metadata mutations.
  • User-Controlled Garbage Collection: Does not spawn internal background threads. Memory reclamation is scheduled and invoked explicitly by the client driver via high-watermark signaling or periodic maintenance routines.
  • Cross-Architecture Memory Ordering: Supports x64 Total Store Order (TSO) and weakly-ordered ARM64 pipelines using inner-shareable hardware memory fences and WDK-compliant IRQL transitions.

Table of Contents

  1. Overview
  2. Architecture & Algorithms
  3. Quick Start & API Overview
  4. Benchmarks & Scaling Performance
  5. Project Layout
  6. Building Instructions
  7. Choosing the Right Shared Pointer
  8. Conclusion
  9. License

Overview

Kernel-mode drivers frequently maintain global or per-device state—such as security policies, routing tables, filter configurations, and connection maps—that is read on every I/O fast path and updated periodically in the background. Protecting these pointers with traditional kernel primitives (EX_PUSH_LOCK wrapped in critical regions, spinlocks, or standard atomic reference counting) introduces scalability bottlenecks on multi-core and multi-socket NUMA systems. Even lock-free reference counting forces every concurrent reader to execute atomic read-modify-write (LOCK XADD / LDADD) instructions against a shared control-block cache line, causing cross-core cache invalidations and tail-latency spikes as core counts increase.

This repository provides three header-only smart pointer implementations with lock-free read and update paths designed for Windows kernel drivers (IRQL <= DISPATCH_LEVEL on non-paged memory). Primary emphasis is placed on two deferred-reclamation architectures—DPC-Based Quiescent State RCU (KmRcuSharedPtr) and Epoch-Based Cache-Aware Rundown (KmRundownSharedPtr)—which decouple transient read access from per-object reference counting and batch memory reclamation in $O(1)$ synchronization passes. For workloads requiring immediate deterministic reclamation without deferred garbage collection, a 128-bit Interlocked Split-Reference-Counted implementation (KmAtomicSharedPtr) based on Anthony Williams' algorithm is also provided.

Core Primitives Summary

Primitive & Header Reader Progress & Mechanism Reader RMWs Reclamation Model Primary Use Case
CRcuSharedPtr<T>
KmRcuSharedPtr.h
Wait-free
DISPATCH_LEVEL preemption guard (KeRaiseIrqlToDpcLevel)
0
(Acquire load)
Blocking GC pass: targeted HighImportance KDPC broadcast + batch free Low-latency non-blocking reads at <= DISPATCH_LEVEL
CRundownSharedPtr<T>
KmRundownSharedPtr.h
Lock-free
Per-CPU hazard slot + cache-aware OS rundown epoch shard
0 per-object
(CPU-local shard)
Blocking GC pass: 64-bit epoch swap + single OS rundown wait per batch Read-heavy PASSIVE_LEVEL / APC_LEVEL paths without raising IRQL
CAtomicSharedPtr<T>
KmAtomicSharedPtr.h
Lock-free
128-bit CAS loop + 64-bit atomic fetch-add reconciliation
2 on load()
1 on release
Immediate free when internal and external counts reach zero General shared ownership, CAS state machines, zero-GC paths

Key Architectural Highlights

  • Decoupled Transient & Persistent Access (KmRcuSharedPtr / KmRundownSharedPtr): Fast-path readers execute inside a scoped Read() lambda without mutating object reference counts. When a caller needs to retain an object across asynchronous I/O completions, work items, or blocking PASSIVE_LEVEL calls, LoadPersistent() promotes the active object to a reference-counted handle (CPersistentRcuPtr<T> or CPersistentRundownPtr<T>) while protected by the transient read section.
  • Explicit User-Controlled Garbage Collection: Neither deferred-reclamation pointer spawns internal background threads. Clients control when the garbage collection pipeline is scheduled and invoked.
  • Compile-Time Anti-Pattern Enforcement: Read() inspects the callback's return type via static_assert and blocks returning raw pointers or C++ references, preventing callers from leaking transient pointers outside the protected scope.
  • Cache-Line Isolation & Direct Payload Resolution: Single-allocation factories (MakeRcuShared, MakeRundownShared, MakeAtomicShared) place the control block and payload in adjacent, cache-line-aligned regions of a single non-paged pool block. RCU and Rundown readers resolve the payload address via compile-time constant offset arithmetic (GetPayloadOffset()), while CLocalSharedPtr<T> caches a direct payload pointer (m_pPtr), avoiding loads from the control block's mutable cache line during access.
  • NUMA-Affinitized Lock-Free Retirement Queues: Displaced RCU and Rundown blocks are pushed via lock-free InterlockedPushEntrySList operations into per-NUMA-node, cache-aligned SLIST_HEADER shards (ExAllocatePool3 node-affinitized allocations) with a statically embedded fallback SLIST_HEADER.
  • Cross-Architecture x64 & ARM64 Support: Implements explicit memory ordering across both x64 Total Store Order (TSO) and weakly-ordered ARM64 pipelines using ReadAcquire64 (LDAR), WriteRelease64 (STLR), inner-shareable hardware data memory barriers (__dmb(_ARM64_BARRIER_ISH)), and 128-bit atomic compare-and-swap intrinsics (_InterlockedCompareExchange128 / _acq / _rel).
  • IRQL & Driver Verifier Compliance: All anchor reads, persistent promotions, and pointer replacements (Store, load, store, exchange, compare_exchange) are safe at IRQL <= DISPATCH_LEVEL. All IRQL transitions use documented WDK kernel APIs (KeRaiseIrqlToDpcLevel / KeLowerIrql) to maintain compatibility with Driver Verifier IRQL and concurrency checks.

Architecture & Algorithms

High-Level Design: Transient Access vs. Persistent Ownership

In many kernel workloads, accesses to shared configuration or state structures are brief, read-only inspections: a dispatch routine or filter callback reads a few fields, makes a routing or policy decision, and exits. Standard smart pointers force these short-lived readers to pay the same atomic reference-counting cost as long-lived owners, generating cache invalidation traffic across CPUs.

KmRcuSharedPtr and KmRundownSharedPtr address this by splitting access into two distinct paths:

  • Transient Read Path (Read): Binds read access to a lexical scope via an execute-around lambda (anchor.Read([](const T* pObj) { ... })). Protection is established at the processor or epoch level rather than per object. The reader performs an acquire load of the anchor pointer, computes pObj by adding a compile-time constant byte offset, and executes the callback without modifying the object's control block.
  • Persistent Ownership Path (LoadPersistent): When an object must outlive the immediate scope (for example, attaching policy state to a pended IRP or reading fields across blocking PASSIVE_LEVEL calls), LoadPersistent() enters the transient read critical section, atomically increments m_lPersistentRefCount on the control block, exits the transient critical section, and returns a CPersistentRcuPtr<T> or CPersistentRundownPtr<T>.

User-Controlled Garbage Collection (RCU & Rundown)

While pointer retirement (Store) is lock-free and executes in a bounded number of instructions via InterlockedExchangePointer and InterlockedPushEntrySList, reclaiming retired memory requires waiting for in-flight readers across other processors to complete their read sections. Rather than spawning internal background worker threads that consume CPU cycles or queue DPCs unpredictably, the RCU and Rundown implementations rely on user-controlled garbage collection.

The underlying garbage collection engines (CQuiescentGarbageCollector for RCU, and CRundownPtrGarbageCollector for Rundown) operate as driver-wide global singletons. A single instance of each collector multiplexes and reclaims displaced blocks for all RCU or Rundown anchors instantiated within the driver module, avoiding per-anchor queue fragmentation.

The client driver is responsible for invoking ProcessGarbagePipeline() at PASSIVE_LEVEL through one of two mechanisms:

  • Event-Driven Wake Callback: The library provides a SetWakeCallback notification mechanism that triggers when the aggregate queue of retired objects reaches a configurable high-watermark threshold. Note: This callback must not execute the reclamation pass directly. Since the callback may be invoked at DISPATCH_LEVEL during a pointer swap, its role is to signal a synchronization event (KeSetEvent) or queue a system work item (IoQueueWorkItem) that wakes a client-managed PASSIVE_LEVEL thread to call ProcessGarbagePipeline().
  • Periodic Maintenance: Alternatively, drivers can omit the callback and invoke ProcessGarbagePipeline() periodically from an existing maintenance timer or housekeeping thread.

This separation keeps writer updates non-blocking while allowing driver authors to defer reclamation during latency-sensitive I/O bursts and consolidate cleanup into existing maintenance threads.


1. DPC-Based Quiescent State RCU (KmRcuSharedPtr.h)

KmRcuSharedPtr implements Quiescent State-Based Reclamation (QSBR) by mapping Read-Copy-Update (RCU) principles onto the Windows NT dispatcher's IRQL hierarchy.

Algorithmic Approach: QSBR via Scheduler Invariants

In operating systems with built-in RCU primitives, the scheduler tracks read-side critical sections across context switches. Since Windows kernel drivers cannot instrument the scheduler, KmRcuSharedPtr uses documented NT IRQL rules to enforce and observe quiescent states without per-object reference counting.

By binding transient read sections to DISPATCH_LEVEL, the implementation uses the NT DPC dispatcher's preemption masking to define a quiescent state constraint. To establish a system-wide grace period during garbage collection, the collector queues a targeted HighImportance DPC to every online processor and waits on a kernel event. Since a logical processor cannot service a newly queued DPC while an existing DISPATCH_LEVEL routine is executing on that core, waiting for all targeted DPCs to complete ensures that all pre-existing read sections have finished across the system.

Design Decisions

  • Zero-RMW Reader Scaling: Since grace-period synchronization is deferred to the collector and relies on DPC scheduling invariants, the Read() path is wait-free and executes zero atomic read-modify-write (RMW) instructions. This avoids the cross-socket LOCK contention and cache-line invalidation traffic associated with atomic reference counting on multi-core systems.
  • Degraded-Mode Reclamation Fallbacks: The collector handles edge cases where non-paged pool exhaustion prevents KDPC array allocation, or where an anchor is destroyed at an elevated IRQL where synchronous waiting is illegal. In these scenarios, the collector falls back to a sequential processor sweep (synchronous via an embedded KDPC at PASSIVE_LEVEL, or asynchronous via a state-machine KDPC when triggered at elevated IRQL) so reclamation can proceed without violating OS blocking rules.

2. Epoch-Based Cache-Aware Rundown (KmRundownSharedPtr.h)

While RCU avoids shared memory writes on the read path, elevating to DISPATCH_LEVEL is incompatible with read callbacks that must access pageable memory or invoke PASSIVE_LEVEL kernel APIs. KmRundownSharedPtr supports read scaling at IRQL <= DISPATCH_LEVEL without raising IRQL by combining Epoch-Based Reclamation (EBR) with Windows' native EX_RUNDOWN_REF_CACHE_AWARE facility.

Algorithmic Approach: Global Two-Shard Multiplexing

Allocating a dedicated EX_RUNDOWN_REF_CACHE_AWARE structure for every shared object scales well for readers but introduces reclamation overhead: it requires per-CPU memory allocations for each object and forces the garbage collector to wait sequentially on ExWaitForRundownProtectionReleaseCacheAware for every displaced pointer, creating Head-of-Line blocking.

KmRundownSharedPtr decouples rundown lifetime from individual objects by maintaining two global cache-aware rundown shards multiplexed across all shared pointer instances in the driver, representing Epoch 0 and Epoch 1:

  1. Readers sample the global 64-bit monotonic epoch counter and acquire rundown protection on the active epoch's CPU-sharded EX_RUNDOWN_REF_CACHE_AWARE instance (entering a critical region via KeEnterCriticalRegion() when called at PASSIVE_LEVEL so normal kernel APCs cannot suspend the thread while holding the epoch shard).
  2. During garbage collection, the collector flushes all retired objects into a pending batch and increments the global epoch, directing new readers to the alternate shard.
  3. The collector then blocks once on ExWaitForRundownProtectionReleaseCacheAware for the outgoing epoch shard to drain. Once active readers on that shard exit, the collector re-initializes the shard and frees the entire batch of retired objects across all NUMA nodes in a single pass.

Design Decisions: The Pre-Acquire Hazard Handshake

Multiplexing epoch rings with OS rundown protection introduces a lock-free linearization hazard: a reader thread at PASSIVE_LEVEL could load the anchor pointer and sample the epoch, experience an OS scheduling delay before calling ExAcquireRundownProtectionCacheAware, and resume after the collector has already advanced the epoch and re-initialized the shard.

To close this window without reader-side locks, the implementation combines a cache-line-isolated hazard slot array with a Dekker-style memory barrier handshake:

  • The Reader (Lock-Free Retry Loop): Before acquiring the OS rundown shard, the reader increments a CPU-indexed hazard slot (InterlockedIncrement), re-verifies that the anchor pointer has not changed, and acquires rundown protection on the sampled epoch shard before decrementing the hazard slot. If the anchor pointer changed or the epoch advanced prior to acquisition, the reader releases its claim and retries.
  • The Collector (Two-Phase Drain): The collector flushes displaced objects, waits for all hazard slots to reach zero (Barrier 1), increments the 64-bit epoch counter, and waits for all hazard slots to reach zero a second time (Barrier 2) before waiting on the outgoing OS rundown shard.

This handshake ensures that any preempted reader either observes the updated state and retries, or registered in its hazard slot prior to the epoch increment, causing the collector to wait until the reader finishes attaching to the outgoing OS rundown shard.


3. 128-Bit Split-Count Atomic Shared Pointer (KmAtomicSharedPtr.h)

For subsystems requiring standard value-semantics handles, lock-free atomic state-machine transitions (compare_exchange, exchange), and immediate deterministic destruction without a deferred garbage collection pipeline, KmAtomicSharedPtr.h adapts Anthony Williams' split-reference-counting algorithm for the Windows kernel.

Algorithmic Approach: Differential Split Reference Counting

In a lock-free reference-counted pointer, loading a control-block address from a shared anchor and incrementing the reference count inside that control block are separate memory operations; a concurrent writer can displace and free the control block between the load and the increment.

KmAtomicSharedPtr resolves this without hazard pointers or epochs by splitting the reference count into two differential components:

  1. The External Count (Anchor): The shared anchor is a 128-bit structure packing the control-block pointer alongside a transient "External" reader count and an ABA generation counter. A reader checking out a handle executes a 128-bit Compare-and-Swap (CAS) loop on the anchor to increment this External count, pinning the control block while the checkout is in flight.
  2. The Internal Count (Control Block): The control block maintains a packed 64-bit split counter containing the "Internal" reference count (tracking active handles and anchor ownership) and a mirrored "External" balance.
  3. Deferred Reconciliation: Once the 128-bit CAS succeeds on the anchor, the reader reconciles its checkout on the control block by atomically incrementing the Internal count and decrementing the mirrored External count. When an anchor is displaced via store() or exchange(), the displaced anchor's accumulated External count is added into the control block's External balance. The control block and payload are freed when both the Internal and External balances reach zero.

Design Decisions

  • Single-Instruction Carry-Compensated Reconciliation: Updating two 32-bit counters inside the control block via a 64-bit CAS loop would introduce retry contention under heavy reader traffic. Instead, apply_counts_atomic() packs both signed 32-bit deltas into a single bounded 64-bit atomic fetch-and-add (InterlockedExchangeAdd64), using two's-complement carry compensation to account for borrow/carry propagation across bit 32.
  • Bounded External-Count Flushing: If a thread displacing an anchor is preempted while concurrent readers continue checking out handles, the 32-bit External count on the anchor could approach overflow. To bound accumulation, any reader observing an External count at or above 65,536 flushes the accumulated claims into the control block's reconciliation counter via a 128-bit CAS and increments the anchor's generation counter.

4. Single-Allocation Cache-Isolated Memory Layout

All three primitives provide single-allocation factories (MakeRcuShared, MakeRundownShared, and MakeAtomicShared) to avoid secondary pool allocations and separate synchronization metadata from user payload data across cache lines (CACHE_LINE_SIZE: 64 bytes on x64, 128 bytes on ARM64).

Design Rationale

Placing read-only payload data on the same CPU cache line as frequently mutated synchronization fields (such as reference counts or intrusive list entries) causes false sharing, invalidating the payload cache line in the L1/L2 caches of reader cores whenever metadata is updated.

The factories allocate a single, cache-aligned non-paged pool block that places the payload behind the control block on a cache-line boundary. Since the deferred garbage collection mechanisms require different tracking metadata than the immediate-reclamation atomic model, the memory layouts are structured into two configurations:

Layout A: Deferred Reclamation (RCU & Rundown)

KmRcuSharedPtr and KmRundownSharedPtr embed the garbage collection tracking node (SLIST_ENTRY) within the control block. To avoid extra prefix padding for over-aligned pool allocations, the raw pool base pointer (m_pRawAllocation) is stored at byte offset 32 inside the control block's existing cache-line tail padding.

+-----------------------------------------------------------------------+
| Cache Line 0 (Offset 0..63 x64 / 0..127 ARM64): Control Block         |
|   - SLIST_ENTRY m_SListEntry          (16 bytes, 16-byte aligned)     |
|   - m_pfnDeleteCallback               (8 bytes)                       |
|   - volatile LONG m_lAnchorState      (4 bytes)                       |
|   - volatile LONG m_lPersistentRefCount (4 bytes)                     |
|   - void* m_pRawAllocation            (8 bytes, packed at offset 32)  |
|   - [Tail Padding to CACHE_LINE_SIZE Boundary]                        |
+-----------------------------------------------------------------------+
| Cache Line 1+ (Offset 64+ x64 / 128+ ARM64): Embedded Payload (T)     |
|   - Constructed in-place via placement new at GetPayloadOffset()      |
|   - Isolated on separate cache line(s) from control-block metadata    |
+-----------------------------------------------------------------------+

Layout B: Deterministic Split-Count (Atomic)

KmAtomicSharedPtr does not use deferred garbage collection queues. Its control block contains only the payload pointer, the 64-bit split counter, and an embedded-allocation flag, placing the payload on the next cache line boundary while CLocalSharedPtr caches a direct pointer to the payload.

+-----------------------------------------------------------------------+
| Cache Line 0 (Offset 0..63 x64 / 0..127 ARM64): TAtomicControlBlock   |
|   - T* m_pPtr                         (8 bytes)                       |
|   - volatile LONG64 m_llCounts        (8 bytes: Internal & External)  |
|   - bool m_bIsEmbedded                (1 byte)                        |
|   - [Tail Padding to CACHE_LINE_SIZE Boundary]                        |
+-----------------------------------------------------------------------+
| Cache Line 1+ (Offset 64+ x64 / 128+ ARM64): Embedded Payload (T)     |
|   - Constructed in-place via placement new at cbPayloadOffset         |
|   - Isolated on separate cache line(s) from control-block metadata    |
+-----------------------------------------------------------------------+

Quick Start & API Overview

CRcuSharedPtr<T> and CRundownSharedPtr<T> expose an identical high-level API surface (Read, LoadPersistent, Store, and singleton garbage-collector management), allowing drivers to switch between DISPATCH_LEVEL RCU protection and PASSIVE_LEVEL-compatible Rundown protection without restructuring call sites. CAtomicSharedPtr<T> follows the standard std::atomic<std::shared_ptr<T>> handle checkout model.

Detailed Usage Guides & API References

Comprehensive API specifications, edge-case contracts, and full driver integration patterns (including event-driven GC worker threads, IRP queue-node caching, and batched DISPATCH_LEVEL fast paths) are provided in dedicated documents:

Primitive Header Usage Guide & Design Patterns API Reference
DPC Quiescent RCU KmRcuSharedPtr.h RCU Usage Guide RCU API Reference
Epoch Cache-Aware Rundown KmRundownSharedPtr.h Rundown Usage Guide Rundown API Reference
128-Bit Split-Count Atomic KmAtomicSharedPtr.h Atomic Usage Guide Atomic API Reference

Core API Summary

Deferred-Reclamation Pointers (KmRcuSharedPtr.h / KmRundownSharedPtr.h)

Component / Operation RCU (KmRcuSharedPtr.h) Rundown (KmRundownSharedPtr.h) Max IRQL Description
Shared Anchor CRcuSharedPtr<T>
CCacheAlignedRcuSharedPtr<T>
CRundownSharedPtr<T>
CCacheAlignedRundownSharedPtr<T>
<= DISPATCH Thread-safe publication slot (standard 8-byte and cache-isolated variants).
Persistent Handle CPersistentRcuPtr<T> CPersistentRundownPtr<T> <= DISPATCH Reference-counted handle keeping payload T alive outside Read() scopes.
Single-Alloc Factory MakeRcuShared<T>(...)
MakeRcuSharedNuma<T>(...)
MakeRundownShared<T>(...)
MakeRundownSharedNuma<T>(...)
<= DISPATCH Allocates the control block and cache-isolated payload T in one non-paged block.
Transient Read anchor.Read(lambda) anchor.Read(lambda) <= DISPATCH Invokes lambda(T*) under RCU (DISPATCH_LEVEL) or Epoch Rundown with 0 per-object RMWs.
Persistent Checkout anchor.LoadPersistent() anchor.LoadPersistent() <= DISPATCH Promotes the active object to a persistent handle and immediately exits the transient guard.
Persistent Access handle->, *handle,
handle.Get()
handle->, *handle,
handle.Get()
<= DISPATCH Accesses payload fields at the caller's unchanged IRQL (supports PASSIVE_LEVEL blocking).
Pointer Replacement anchor.Store(desired) anchor.Store(desired) <= DISPATCH Atomically swaps the active pointer and stages the displaced block into the NUMA GC queue.
GC Singleton CQuiescentGarbageCollector CRundownPtrGarbageCollector PASSIVE
(Sweep/Teardown)
Driver-wide singleton: GetInstance(), SetWakeCallback(), ProcessGarbagePipeline(), Teardown().

Immediate-Reclamation Split-Count Pointer (KmAtomicSharedPtr.h)

Component / Operation Primitive / Method Max IRQL Description
Shared Anchor CAtomicSharedPtr<T> <= DISPATCH Cache-line-isolated anchor (64B on x64, 128B on ARM64) holding a 128-bit counted-pointer state managed via 128-bit CAS.
Local Handle CLocalSharedPtr<T> <= DISPATCH Thread-local reference-counted handle caching a direct T* payload pointer (->, *, get()).
Single-Alloc Factory MakeAtomicShared<T>(...) <= DISPATCH Allocates TAtomicControlBlock<T> and cache-aligned payload T in one non-paged block.
Checkout / Replace load() / store() /
exchange()
<= DISPATCH Lock-free handle checkout, unconditional replacement, and atomic swap-and-extract.
Conditional CAS compare_exchange_strong()
compare_exchange_weak()
<= DISPATCH Lock-free 128-bit CAS supporting lvalue and rvalue desired handles with spurious-failure filtering.

Quick Start: Deferred Reclamation (CRcuSharedPtr & CRundownSharedPtr)

The example below demonstrates the primary RCU workflow: wait-free transient reads via Read(), persistent checkout and PASSIVE_LEVEL read access via LoadPersistent() (CPersistentRcuPtr<T>), lock-free writer updates via Store(), and explicit garbage collection maintenance. Substituting CRundownSharedPtr, CPersistentRundownPtr, MakeRundownShared, and CRundownPtrGarbageCollector follows the exact same pattern when the Read() callback itself must remain at PASSIVE_LEVEL.

#include <ntddk.h>
#include "KmRcuSharedPtr.h" // Or "KmRundownSharedPtr.h" if Read() accesses pageable memory

struct SFilterPolicy
{
    ULONG   ulMaxConnections;
    ULONG   ulTimeoutMs;
    BOOLEAN bInspectionEnabled;

    SFilterPolicy(ULONG   maxConn,
                  ULONG   timeoutMs,
                  BOOLEAN enableInspection) noexcept : ulMaxConnections(maxConn),
                                                       ulTimeoutMs(timeoutMs),
                                                       bInspectionEnabled(enableInspection)
    {
    }
};

// Shared publication slot (can be global or embedded in a device/filter context)
CRcuSharedPtr<SFilterPolicy> g_PolicyAnchor;

// ----------------------------------------------------------------------------
// 1. Fast-Path Transient Reader (IRQL <= DISPATCH_LEVEL, 0 atomic RMWs)
// ----------------------------------------------------------------------------
BOOLEAN ShouldInspectPacket(ULONG ulCurrentConnections) noexcept
{
    return g_PolicyAnchor.Read([ulCurrentConnections](const SFilterPolicy* pPolicy) noexcept -> BOOLEAN
    {
        if (!pPolicy || !pPolicy->bInspectionEnabled)
        {
            return FALSE;
        }

        return (ulCurrentConnections < pPolicy->ulMaxConnections) ? TRUE : FALSE;
    });
}

// ----------------------------------------------------------------------------
// 2. Persistent Read via CPersistentRcuPtr (Safe for PASSIVE_LEVEL Blocking)
// ----------------------------------------------------------------------------
void ExecutePassiveInspectionWork() noexcept
{
    // LoadPersistent() briefly enters the RCU critical section, increments
    // m_lPersistentRefCount, and immediately restores the caller's IRQL.
    CPersistentRcuPtr<SFilterPolicy> pPolicy = g_PolicyAnchor.LoadPersistent();
    if (!pPolicy)
    {
        return;
    }

    // Read payload fields directly via pPolicy-> at PASSIVE_LEVEL outside RCU.
    // The caller can freely access pageable memory, acquire push locks, or invoke
    // blocking kernel APIs while pPolicy keeps the underlying object alive.
    if (pPolicy->bInspectionEnabled)
    {
        ULONG ulTimeout = pPolicy->ulTimeoutMs;
        UNREFERENCED_PARAMETER(ulTimeout);
    }
}

// ----------------------------------------------------------------------------
// 3. Writer Update (IRQL <= DISPATCH_LEVEL)
// ----------------------------------------------------------------------------
bool UpdateFilterPolicy(ULONG   ulNewMaxConn,
                        ULONG   ulNewTimeoutMs,
                        BOOLEAN bEnableInspection) noexcept
{
    CPersistentRcuPtr<SFilterPolicy> pNewPolicy = MakeRcuShared<SFilterPolicy>(ulNewMaxConn,
                                                                               ulNewTimeoutMs,
                                                                               bEnableInspection);
    if (!pNewPolicy)
    {
        return false;
    }

    // Atomically swaps the active pointer and pushes the displaced block to the NUMA SList
    return g_PolicyAnchor.Store(Ktl::move(pNewPolicy));
}

// ----------------------------------------------------------------------------
// 4. Garbage Collection & DriverUnload Teardown (PASSIVE_LEVEL)
// ----------------------------------------------------------------------------
void OnMaintenanceWorkerTick() noexcept
{
    // Invoked periodically or when woken by SetWakeCallback()
    CQuiescentGarbageCollector::GetInstance().ProcessGarbagePipeline(false);
}

void OnDriverUnload() noexcept
{
    g_PolicyAnchor.Store(nullptr);
    CQuiescentGarbageCollector::GetInstance().Teardown();
}

Quick Start: Immediate Reclamation (CAtomicSharedPtr)

When deferred garbage collection is not desired—or when threads must perform conditional Compare-And-Swap (compare_exchange_strong) state transitions—use CAtomicSharedPtr<T> and CLocalSharedPtr<T>:

#include <ntddk.h>
#include "KmAtomicSharedPtr.h"

// Uses struct SFilterPolicy defined in the example above
CAtomicSharedPtr<SFilterPolicy> g_AtomicPolicy;

// ----------------------------------------------------------------------------
// 1. Reader Checkout (IRQL <= DISPATCH_LEVEL)
// ----------------------------------------------------------------------------
BOOLEAN ShouldInspectPacketAtomic(ULONG ulCurrentConnections) noexcept
{
    // Atomically checks out a local reference-counted handle via 128-bit CAS.
    // The payload remains valid for the lifetime of pPolicy.
    CLocalSharedPtr<SFilterPolicy> pPolicy = g_AtomicPolicy.load();
    if (!pPolicy || !pPolicy->bInspectionEnabled)
    {
        return FALSE;
    }

    return (ulCurrentConnections < pPolicy->ulMaxConnections) ? TRUE : FALSE;
    // pPolicy destructor drops the reference; frees immediately if last owner
}

// ----------------------------------------------------------------------------
// 2. Unconditional Writer Update (IRQL <= DISPATCH_LEVEL)
// ----------------------------------------------------------------------------
bool UpdateFilterPolicyAtomic(ULONG   ulNewMaxConn,
                              ULONG   ulNewTimeoutMs,
                              BOOLEAN bEnableInspection) noexcept
{
    CLocalSharedPtr<SFilterPolicy> pNewPolicy = MakeAtomicShared<SFilterPolicy>(ulNewMaxConn,
                                                                                ulNewTimeoutMs,
                                                                                bEnableInspection);
    if (!pNewPolicy)
    {
        return false;
    }

    g_AtomicPolicy.store(Ktl::move(pNewPolicy));
    return true;
}

// ----------------------------------------------------------------------------
// 3. Conditional CAS State Transition (IRQL <= DISPATCH_LEVEL)
// ----------------------------------------------------------------------------
bool TryBumpConnectionLimit() noexcept
{
    CLocalSharedPtr<SFilterPolicy> pExpected = g_AtomicPolicy.load();
    if (!pExpected)
    {
        return false;
    }

    CLocalSharedPtr<SFilterPolicy> pDesired = MakeAtomicShared<SFilterPolicy>(pExpected->ulMaxConnections + 64,
                                                                              pExpected->ulTimeoutMs,
                                                                              pExpected->bInspectionEnabled);
    if (!pDesired)
    {
        return false;
    }

    // Atomically commits if g_AtomicPolicy still points to pExpected.
    // Displaced objects are freed immediately when the last handle goes out of scope.
    return g_AtomicPolicy.compare_exchange_strong(pExpected,
                                                  Ktl::move(pDesired));
}

Benchmarks & Scaling Performance

Kernel-mode synchronization primitives must be evaluated against hardware cache-coherency traffic, core migration, and scheduler preemption under multi-core contention. The included performance test suite measures memory reclamation correctness, throughput scaling, and read tail latency across bare-metal and virtualized SMP configurations.

All tests were performed on Windows 11, and the complete WDM test driver source code is included in the repository.


Test Methodology & Benchmark Architecture

The test harness is implemented as a native Windows kernel-mode driver (TestKmSharedPtrDrv.sys) executing across PASSIVE_LEVEL and DISPATCH_LEVEL. The suite evaluates four shared-pointer architectures under identical test conditions:

  • Baseline (CBaselineAtomicSharedPtr): A conventional Windows driver synchronization model protecting pointer swaps with an exclusive EX_PUSH_LOCK and managing object lifetimes with atomic reference counting.
  • Atomic (CAtomicSharedPtr): The 128-bit interlocked split-reference-counting implementation using Anthony Williams' algorithm.
  • Rundown (CRundownSharedPtr): The two-phase epoch-based reclamation implementation backed by OS cache-aware rundown protection (PEX_RUNDOWN_REF_CACHE_AWARE) and per-CPU hazard counters.
  • RCU (CRcuSharedPtr): The DPC-based quiescent state implementation using DISPATCH_LEVEL preemption suppression and targeted HighImportance DPC broadcasts.

The benchmark framework controls for allocator contention and measurement overhead:

  • Functional & Stress Validation: Covers handle lifecycle and CAS rollback correctness, sustained multi-producer multi-consumer contention, 6x CPU thread oversubscription, concurrent object replacement under active read dereferencing, 100% atomic exchange contention, voluntary thread yields, hardware-timed tail latency profiling, and multi-core thread scaling sweeps.
  • Kernel Memory Allocator Isolation: During thread scaling sweeps, worker threads operate on pre-allocated payload arrays. This keeps the working set cache-resident and prevents lock contention within the Windows kernel pool allocator (ExAllocatePool2) from skewing synchronization throughput measurements.
  • Amortized Batch Guards & Checkpointing: Readers in the scaling sweep use batch guards (CRundownBatchGuard and CRcuBatchGuard) that amortize IRQL and rundown setup across read bursts. To prevent targeted quiescent DPCs from being delayed during long read-heavy bursts (e.g., 99/1 and 99.9/0.1), CRcuBatchGuard::Checkpoint() periodically lowers IRQL every 32 reads if a quiescent barrier is active.
  • Tail-Latency Capture: Reader threads record individual operation durations into 1,048,576-entry power-of-two ring buffers using hardware timestamp counters (__rdtsc on x64, CNTVCT_EL0 on ARM64) while writer threads pace mutations every ~3 microseconds. Samples are sorted in-place using an iterative, non-recursive QuickSort with an explicit stack bounded to $\le 64$ entries.
  • Integer & Fixed-Point Timekeeping: Timer calibration and cooldown calculations use 64-bit integer cycle arithmetic and Q16 fixed-point math, avoiding kernel floating-point or vector register state saves.
  • Reclamation Verification: The harness tracks live allocations globally and asserts that all allocated objects are freed after every test run. Between scaling steps, synchronous garbage collection flushes drain pending queues to prevent cross-step interference.

Multi-Core Scaling Analysis: The Limits of PushLocks and Atomic Pointers

Performance benchmarks show that lock-based synchronization and per-checkout atomic CAS reference counting hit scalability ceilings as thread counts increase.

Multi-Core Throughput Scaling (80/20, 99/1, and 99.9/0.1 Read/Write Ratios) — Scaling comparison across Baseline (EX_PUSH_LOCK), KmAtomicSharedPtr (128-bit CAS), KmRundownSharedPtr (Epoch), and KmRcuSharedPtr (DPC Quiescent).

Intel Core i7-1165G7 Scaling Performance
Figure 1: Bare-Metal Client (Intel Core i7-1165G7 @ 2.80GHz — 4 Cores / 8 Threads)

Intel Core i7-1165G7: Across all three workload ratios (80/20, 99/1, and 99.9/0.1), KmRcuSharedPtr and KmRundownSharedPtr track closely and scale from ~7.5–8.1 Mops/s at 1 thread up to 50.2 Mops/s (80/20), 53.8 Mops/s (99/1), and 54.1 Mops/s (99.9/0.1) at 8 threads. In contrast, KmAtomicSharedPtr peaks near 4 threads (~12.4–14.5 Mops/s) and flattens at ~11.6–12.0 Mops/s due to cross-core 128-bit CAS cache-line invalidations, while EX_PUSH_LOCK peaks at 2–3 threads (~9.9–11.6 Mops/s) before regressing to ~8.2–8.5 Mops/s at 8 threads.


Intel Core i7-4800MQ Scaling Performance
Figure 2: Bare-Metal Legacy (Intel Core i7-4800MQ @ 2.70GHz — 4 Cores / 8 Threads)

Intel Core i7-4800MQ: On Haswell silicon, KmRcuSharedPtr and KmRundownSharedPtr scale from ~7.6–8.4 Mops/s at 1 thread up to 63.0 Mops/s and 60.5 Mops/s under 80/20 write load, and reach 68.8 Mops/s (99/1) and 69.3 Mops/s (99.9/0.1) at 8 threads (~8.3x scaling). KmAtomicSharedPtr peaks at 3 threads (~12.8–14.1 Mops/s) and declines to ~10.3–10.9 Mops/s at 8 threads as ring-bus coherency traffic saturates, while EX_PUSH_LOCK peaks at 2 threads (~9.0–10.5 Mops/s) and drops to ~8.1–8.5 Mops/s.


Intel Core i7-8086K Virtualized Guest Performance
Figure 3: Virtualized Guest (VMware Workstation on Intel Core i7-8086K @ 4.00GHz — 8 vCPUs)

Intel Core i7-8086K, VMware Guest: Under virtualized SMP scheduling, KmRcuSharedPtr and KmRundownSharedPtr scale from ~8.1–8.9 Mops/s on 1 vCPU to 63.0 Mops/s and 58.7 Mops/s at 8 vCPUs under 80/20 write contention, and reach 66.4–67.2 Mops/s across the 99/1 and 99.9/0.1 read-dominant sweeps (~7.5x scaling). KmAtomicSharedPtr plateaus at ~13.7–14.4 Mops/s beyond 3 vCPUs, and EX_PUSH_LOCK peaks at 2 vCPUs (~11.3–11.8 Mops/s) before declining to ~8.9–9.2 Mops/s at 8 vCPUs.


Why PushLocks Fail to Scale

The baseline EX_PUSH_LOCK experiences contention bottlenecks as thread counts increase:

  1. Lock Convoys & Negative Scaling: Across all tested platforms, PushLock throughput peaks at 2 threads (~10–11 Mops/sec) and regresses as core counts rise, dropping to ~8.2–9.2 Mops/sec at 8 threads. At an 80/20 read/write ratio, contending exclusive acquisitions force reader threads into kernel wait states.
  2. High Tail Latency Under Contention: Since acquiring an exclusive push lock deschedules contending threads, writer activity creates lock convoys. While median latency (P50) remains between 36 ns and 73 ns, tail latency rises to 39–55 microseconds at P99.9 and 368–520 microseconds at P99.99.

Why 128-Bit Atomic Split Counting Plateaus

KmAtomicSharedPtr avoids OS thread blocking and lock convoys, keeping bare-metal P99.9 latency under 510 ns and P99.99 latency near 3.2–3.5 µs:

  1. Cache-Line Bouncing: Every checkout via load() executes a 128-bit atomic Compare-and-Swap (LOCK CMPXCHG16B / CASP) against the shared anchor to increment m_lExternalCount.
  2. MESI Interconnect Saturation: The 128-bit atomic CAS transitions the cache line containing the anchor into the Exclusive/Modified state on the executing core, invalidating L1/L2 copies on other cores. Beyond 3 to 4 threads, coherency traffic limits throughput to ~10–14 Mops/sec across all read/write ratios.

The Rundown and RCU Scaling Advantage

KmRundownSharedPtr and KmRcuSharedPtr avoid shared control-block cache-line contention on the read path:

  1. Zero Per-Object RMWs on Read Paths: Neither primitive modifies the object's control block during Read(). RCU executes a wait-free read path using local IRQL elevation and an acquire load, while Rundown uses lock-free CPU-local hazard registration and CPU-sharded OS rundown acquisition.
  2. Multi-Core Throughput Scaling: Both primitives scale nearly linearly across the tested core counts. On an 11th Gen Core i7, throughput increases from ~8 Mops/sec at 1 thread to 53.8 Mops/sec at 8 threads (a 6.7x speedup on 8 logical cores). On an i7-4800MQ, throughput increases from ~8.3 Mops/sec to 69.3 Mops/sec (an 8.3x speedup).
  3. Bounded Read Tail Latency: In tail latency benchmarks, KmRcuSharedPtr achieves a median read latency of 19–62 ns and a P99.99 latency of 262–3,238 ns. KmRundownSharedPtr achieves a median latency of 35–80 ns and a P99.99 latency of 327–5,273 ns.

Tail Latency Under Continuous Paced Mutation

Reader latency was captured over 10-to-15-second intervals with 75% readers and 25% writers, where writers mutate the anchor every ~3 microseconds. All metrics are reported in nanoseconds.

Bare-Metal Client: Intel Core i7-1165G7 @ 2.80GHz

4 physical Performance cores / 8 logical threads, 12MB L3 Cache.

Architecture P50 (Median) P90 P99.9 P99.99
Baseline (EX_PUSH_LOCK) 65 ns 98 ns 46,062 ns 447,055 ns
KmAtomicSharedPtr 114 ns 139 ns 427 ns 3,483 ns
KmRundownSharedPtr 80 ns 105 ns 358 ns 3,330 ns
KmRcuSharedPtr 62 ns 77 ns 277 ns 3,238 ns

Bare-Metal Legacy: Intel Core i7-4800MQ @ 2.70GHz

4 physical cores / 8 logical threads, 6MB L3 Cache.

Architecture P50 (Median) P90 P99.9 P99.99
Baseline (EX_PUSH_LOCK) 73 ns 144 ns 54,649 ns 519,943 ns
KmAtomicSharedPtr 162 ns 258 ns 506 ns 3,177 ns
KmRundownSharedPtr 65 ns 83 ns 208 ns 327 ns
KmRcuSharedPtr 40 ns 42 ns 136 ns 283 ns

Virtualized Guest: VMware Workstation on Intel Core i7-8086K @ 4.00GHz

Windows 11 VM, 8 assigned virtual CPUs.

Architecture P50 (Median) P90 P99.9 P99.99
Baseline (EX_PUSH_LOCK) 36 ns 94 ns 39,389 ns 368,364 ns
KmAtomicSharedPtr 94 ns 130 ns 326 ns 6,195 ns
KmRundownSharedPtr 35 ns 51 ns 138 ns 5,273 ns
KmRcuSharedPtr 19 ns 26 ns 113 ns 262 ns

Tail Latency Results Analysis

  • Median & P90 Fast-Path Overhead: KmRcuSharedPtr records the lowest P50 (19–62 ns) and P90 (26–77 ns) latencies across all three environments because its read path performs a local IRQL elevation and a single acquire load with zero atomic RMW instructions. KmRundownSharedPtr follows closely (35–80 ns P50, 51–105 ns P90), as its CPU-indexed pre-acquire slot and cache-aware rundown shard typically remain resident in the executing core's L1/L2 cache. KmAtomicSharedPtr incurs 2x–4x higher baseline latency (94–162 ns P50, 130–258 ns P90) because every checkout and destruction executes a 128-bit CAS on the shared anchor followed by a 64-bit LOCK XADD reconciliation on the control block.
  • P99.9 & P99.99 Tail Behavior: Under 3 µs writer mutations, EX_PUSH_LOCK latency increases by three orders of magnitude—reaching 39.4–54.6 µs at P99.9 and 368.4–519.9 µs at P99.99—because readers colliding with an exclusive writer are descheduled by the kernel dispatcher. The three lock-free read paths avoid OS thread blocking, keeping P99.9 latency between 113 ns and 506 ns (more than 75x–100x lower than EX_PUSH_LOCK). At the P99.99 percentile, KmRcuSharedPtr holds to 262–283 ns on the i7-4800MQ and VMware guest because executing at DISPATCH_LEVEL prevents thread preemption during the measurement window, whereas KmRundownSharedPtr (327–5,273 ns) and KmAtomicSharedPtr (3,177–6,195 ns) reflect occasional retry backoff iterations or dispatcher interrupts when an anchor swap or epoch flip coincides with a reader probe at PASSIVE_LEVEL.

Sustained Contention & Oversubscription Soak Summary

  • MPMC Soak (75% Read / 25% Write at 1x Concurrency): Spawns one system thread per logical processor (6 dedicated readers continuously invoking Read() and 2 dedicated writers continuously allocating and publishing new objects via Store()). Unlike the isolated thread-scaling sweep, writers in this workload perform live non-paged pool allocations (ExAllocatePool3 / ExAllocatePool2) and trigger the test driver's client-managed garbage-collection pipeline and pool reclamation.
  • Heavy CPU Oversubscription (6x Concurrency, 48 Threads): Runs 48 concurrent kernel threads on an 8-logical-core system (36 readers and 12 writers, with writers continuously publishing new objects via Store() for RCU/Rundown and alternating store() / exchange() for CAtomicSharedPtr). This induces frequent OS scheduler preemption inside read and write windows, testing KmRundownSharedPtr's pre-acquire hazard slots, KmAtomicSharedPtr's 65,536 external-count flush threshold, and KmRcuSharedPtr's DISPATCH_LEVEL critical sections.
  • Concurrent Replacement (1.5x Concurrency, Extended Read Hold): Runs 12 concurrent threads (8 readers and 4 writers) where writers continuously displace the shared anchor while each reader executes a 25-iteration memory accumulation and invariant check on the payload inside Read(). This widens the active read window to verify grace-period enforcement against premature reclamation while measuring throughput across longer critical sections.

Across 10-to-15-second soak workloads under contention and oversubscription, the deferred-reclamation architectures sustain higher throughput than the lock-based and split-counting baselines (all values below are reported in Mops/s — millions of completed pointer operations per second):

Workload Scenario Platform Baseline (EX_PUSH_LOCK) Atomic (128-Bit CAS) Rundown (Epoch GC) RCU (DPC Quiescent)
MPMC Soak
(75% Read / 25% Write, 1x Cores)
i7-1165G7 13.53 17.11 66.84 56.85
i7-4800MQ 16.86 9.87 59.31 105.66
i7-8086K VM 31.10 18.05 89.06 118.73
Heavy Oversubscription
(6x Cores, 48 Threads)
i7-1165G7 12.31 14.84 42.20 48.55
i7-4800MQ 15.28 9.80 45.05 73.56
i7-8086K VM 29.44 18.49 64.10 80.98
Concurrent Replacement
(1.5x Cores, 25-Load Hold)
i7-1165G7 12.38 15.50 55.67 68.12
i7-4800MQ 14.52 9.88 53.41 109.52
i7-8086K VM 30.38 19.84 79.44 143.41

Soak Results Analysis

  • Live Allocation & Batch Reclamation Efficiency (MPMC Soak): With 25% of threads continuously allocating from the kernel non-paged pool and retiring displaced objects, KmRundownSharedPtr (59.31–89.06 Mops/s) and KmRcuSharedPtr (56.85–118.73 Mops/s) achieve 3.3x to 10.7x the throughput of EX_PUSH_LOCK and KmAtomicSharedPtr. Offloading reclamation into $O(1)$ batched background sweeps reduces writer publication to an InterlockedExchangePointer and InterlockedPushEntrySList, whereas KmAtomicSharedPtr drops to 9.87–18.05 Mops/s as writers and readers contend on the 128-bit CAS anchor.
  • Behavior Under 6x Thread Oversubscription: When 48 threads compete for 8 logical processors, KmRcuSharedPtr achieves the highest throughput across all tested platforms (48.55–80.98 Mops/s) because elevating to DISPATCH_LEVEL prevents the OS dispatcher from preempting a reader thread mid-access. KmRundownSharedPtr sustains 42.20–64.10 Mops/s because its pre-acquire hazard window spans only a few instructions and CScopedRundownGuard disables normal kernel APCs while holding rundown protection.
  • Impact of Critical Section Duration (Concurrent Replacement): Expanding the reader callback to 25 consecutive memory loads widens the gap between deferred reclamation and traditional primitives: KmRcuSharedPtr reaches 68.12–143.41 Mops/s and KmRundownSharedPtr reaches 53.41–79.44 Mops/s, compared to 12.38–30.38 Mops/s for EX_PUSH_LOCK and 9.88–19.84 Mops/s for KmAtomicSharedPtr. Since RCU and Rundown readers do not hold exclusive locks or dirty the control block's cache line, all 8 reader threads execute the 25-load loop concurrently from local L1 cache while writers continue replacing the anchor without blocking.

Project Layout

The repository separates the header-only kernel smart pointer implementations from the documentation, benchmark telemetry logs, and the WDM verification/benchmark driver.

├── Assets/                                 # Benchmark throughput and scaling charts (PNG)
├── Docs/                                   # Detailed usage guides and API reference documents
│   ├── AtomicSharedPtrApi.md               # API specification for KmAtomicSharedPtr.h
│   ├── AtomicSharedPtrUsage.md             # Usage patterns for CAtomicSharedPtr & CLocalSharedPtr
│   ├── RcuSharedPtrApi.md                  # API specification for KmRcuSharedPtr.h
│   ├── RcuSharedPtrUsage.md                # Usage patterns for CRcuSharedPtr & DPC Quiescent GC
│   ├── RundownSharedPtrApi.md              # API specification for KmRundownSharedPtr.h
│   └── RundownSharedPtrUsage.md            # Usage patterns for CRundownSharedPtr & Epoch Rundown GC
├── PerfData/                               # Raw verification & benchmark telemetry collected by the test driver
│   ├── 1165G7-KmSharedPtrPerf.txt          # Bare-metal client telemetry (Intel Core i7-1165G7)
│   ├── 4800MQ-KmSharedPtrPerf.txt          # Bare-metal legacy telemetry (Intel Core i7-4800MQ)
│   └── 8086K-VM_Guest_KmSharedPtrPerf.txt  # Virtualized guest telemetry (VMware on Intel Core i7-8086K)
├── SharedPtr/
│   ├── KmAtomicSharedPtr.h                 # 128-bit CAS split-reference-counted shared pointer
│   ├── KmRcuSharedPtr.h                    # DPC-based quiescent state RCU shared pointer
│   ├── KmRundownSharedPtr.h                # Epoch-based cache-aware rundown shared pointer
│   └── KmSharedPtrUtils.h                  # Shared allocators, memory barriers, NUMA & backoff utilities
└── TestDrv/
    ├── TestKmSharedPtr.h                   # Cross-architecture verification, soak, and benchmark suite
    └── TestKmSharedPtrDrv.cpp              # WDM kernel driver host, PAL, and telemetry logger

Building Instructions

Prerequisites

  • Visual Studio 2026: MSVC C++ compiler toolset supporting x64 (_M_AMD64) and ARM64 (_M_ARM64 / _M_ARM64EC).
  • C++20 Language Standard: Enable /std:c++20 in the project's C/C++ language flags.
  • Windows Driver Kit (WDK): Matching Windows 11 WDK installation integrated with Visual Studio.

Header-Only Integration

The smart pointer implementations in SharedPtr/ are header-only and require no static or dynamic library linking:

  1. Copy the SharedPtr/ directory into your driver project or add it to your project's Additional Include Directories.
  2. Include the desired header (KmRcuSharedPtr.h, KmRundownSharedPtr.h, or KmAtomicSharedPtr.h) after <ntddk.h> or <wdm.h>.
  3. Compile with /std:c++20 enabled.

Building the Test & Benchmark Driver (TestDrv)

Native MSVC 2026 solution and project files are provided in the repository to build the evaluation driver:

  • TestDrv (TestKmSharedPtrDrv.sys): Compiles the WDM kernel driver hosting the functional verification, stress-soak, tail-latency, and multi-core scaling suite (Release / x64 or ARM64 configurations recommended for benchmarking).

Execution, Configuration & Telemetry Output

  • Asynchronous Execution & Safe Teardown: DriverEntry spawns the benchmark coordinator and garbage-collection worker threads asynchronously so the Service Control Manager (SCM) start request returns STATUS_SUCCESS immediately. Stopping the service (DriverUnload) signals a global abort flag that slices sleeps into 50 ms ticks, allowing the driver to unwind active worker threads, flush both GC singletons, and unload cleanly at any point during the run.
  • Configurable Soak Duration (Optional): By default, each soak and latency phase runs for 10 seconds. Before starting the driver service, you can override this duration by creating a REG_DWORD value named TestPhaseDuration (in seconds) under the driver's parameters key: HKLM\SYSTEM\CurrentControlSet\Services\<ServiceName>\Parameters\TestPhaseDuration
  • Live Kernel Debugger Tracing: The driver emits real-time progress and throughput measurements via DbgPrintEx under the DPFLTR_IHVDRIVER_ID component filter at DPFLTR_ERROR_LEVEL, making output visible immediately in WinDbg or Sysinternals DebugView without modifying component filter masks.
  • Persistent Telemetry File: Upon completing the test suite, the driver flushes its 4 MB in-memory log buffer to \SystemRoot\Temp\KmSharedPtrPerf.txt (C:\Windows\Temp\KmSharedPtrPerf.txt).

Choosing the Right Shared Pointer

Selecting between CRcuSharedPtr, CRundownSharedPtr, and CAtomicSharedPtr depends on multi-core scaling requirements, tail-latency bounds under contention, read-path IRQL constraints, and whether the subsystem requires immediate reclamation or conditional Compare-And-Swap (CAS) primitives.

Decision Criterion CRcuSharedPtr<T> CRundownSharedPtr<T> CAtomicSharedPtr<T>
Multi-Core Read Scaling Linear
(0 atomic RMWs in Read())
Linear
(0 per-object RMWs in Read())
Poor (plateaus at 3–4 threads due to 128-bit CAS bouncing)
Sustained Contention & Oversubscription Highest (DISPATCH_LEVEL prevents mid-read preemption) High (15–40% below RCU under heavy oversubscription) Low (128-bit CAS + 64-bit XADD cache-line contention)
Read Latency (P50 / P99.99 Tail) Lowest (19–62 ns P50; tightest P99.99 bounds) Low (35–80 ns P50; occasional epoch-flip retry spikes) Highest (2x–4x higher P50; elevated P99.99 under writes)
IRQL Inside Read() DISPATCH_LEVEL
(elevated if < DISPATCH_LEVEL)
Caller's IRQL (<= DISPATCH; normal APCs disabled) N/A (handle checkout via load())
Pageable Memory / Blocking APIs Via LoadPersistent()
(restores IRQL immediately)
Directly inside Read()
(at PASSIVE_LEVEL) or LoadPersistent()
Directly via load()
(at PASSIVE_LEVEL)
Per-Object Atomic RMWs 0 in Read()
2 via LoadPersistent() (+1 / -1)
0 in Read()
2 via LoadPersistent() (+1 / -1)
2 per load() checkout
1 on handle release
Reclamation & Update Model Deferred batch GC;
unconditional Store()
Deferred batch GC;
unconditional Store()
Immediate free at refcount 0;
store(), exchange(), compare_exchange

1. Choose CRcuSharedPtr<T> (KmRcuSharedPtr.h) — Default for High Throughput & Lowest Tail Latency

  • Best overall contention and latency profile: Matches CRundownSharedPtr on linear multi-core scaling while delivering the lowest median and P99.99 tail latencies and the highest throughput under sustained write contention and 6x thread oversubscription (elevating to DISPATCH_LEVEL prevents the scheduler from preempting readers mid-access).
  • Non-blocking hot path with LoadPersistent() for blocking work: Use Read() for fast non-paged inspections (0 atomic RMWs), and call LoadPersistent() whenever a PASSIVE_LEVEL caller needs a CPersistentRcuPtr<T> handle to access pageable memory, acquire push locks, or invoke blocking APIs.
  • Native DISPATCH_LEVEL execution: Incurs zero IRQL transition overhead when called from NDIS receive paths, StorPort completion routines, or timer DPCs.

2. Choose CRundownSharedPtr<T> (KmRundownSharedPtr.h) — When Every Hot-Path Read Must Stay at PASSIVE_LEVEL

  • Linear scaling without raising IRQL: Provides the same multi-core scaling curve as RCU when the primary fast path itself must touch paged pool/code or invoke synchronous PASSIVE_LEVEL APIs on every read (where calling CRcuSharedPtr::LoadPersistent() on every access would reintroduce per-object refcount contention).
  • Trade-off vs. RCU: Because readers execute at PASSIVE_LEVEL (with normal kernel APCs disabled), they remain preemptible by DISPATCH_LEVEL/DIRQL activity and can experience retry backoff during concurrent epoch flips—resulting in slightly higher P99.99 tail latencies and lower throughput than RCU under heavy CPU oversubscription or rapid pointer replacement.
  • Extended read hold times: Holding an epoch rundown shard does not mask DPC delivery or thread scheduling on the local processor.

3. Choose CAtomicSharedPtr<T> (KmAtomicSharedPtr.h) — Only for Immediate Reclamation or CAS State Machines

  • Significant scaling and latency trade-off: Every load() and handle destruction executes a 128-bit CAS on the shared anchor plus a 64-bit atomic fetch-add on the control block. Cross-core cache-line invalidation causes throughput to plateau beyond 3–4 threads (~4x–6x slower than RCU/Rundown at 8 threads) and produces 2x–4x higher baseline and tail read latencies.
  • When to use: Restrict to low-to-moderate concurrency paths that strictly require immediate deterministic destruction upon releasing the final handle (no periodic/event-driven GC sweep) or lock-free compare_exchange / exchange state transitions.

Conclusion

This library addresses a common scalability bottleneck in Windows kernel driver design: protecting shared, read-mostly state across multi-core and multi-socket NUMA topologies without incurring the lock convoys of EX_PUSH_LOCK or the cross-core cache-line invalidation traffic of per-access atomic reference counting.

By separating transient read access (Read) from persistent reference-counted ownership (LoadPersistent), mapping Quiescent State-Based Reclamation (QSBR) and Epoch-Based Reclamation (EBR) onto documented Windows NT dispatcher and rundown facilities, and isolating payloads from control-block metadata within single-allocation pool blocks, CRcuSharedPtr and CRundownSharedPtr eliminate per-object atomic RMW operations from the read fast path. Paired with CAtomicSharedPtr for immediate-reclamation and CAS state-machine workloads, the suite provides x64 and ARM64 kernel drivers with Driver Verifier-compliant synchronization primitives that scale with core count while maintaining sub-microsecond read tail latency.


License

This project is licensed under the Apache License, Version 2.0.

You may not use this file except in compliance with the License. You may obtain a copy of the License at: http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations.

About

Header-only C++20 lock-free shared pointers for Windows kernel drivers (x64 & ARM64, IRQL <= DISPATCH_LEVEL). Features DPC-based Quiescent State RCU, Epoch-based cache-aware Rundown with NUMA-sharded O(1) batch GC, and a 128-bit split-count atomic shared_ptr—eliminating per-object read-path RMWs and cache-line contention.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages