kernel slab leak persists on the fixed kernel, ~25-48 MiB/h per compute member

[rldyourmnd]

What the record says

kernel-slab-recovery-rollout.json closed the 2026-08-27 OOM incident as a kernel defect:

• before: SUnreclaim 10.3–11.9 GiB, OOM kills 1 / 0 / 8 / 24, kernel 6.8.0-137
• upstream: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2098730, fixed floor 6.8.0-86
• remedy: pressure-close, natural drain, rolling reboot into 6.8.0-138
• after: members returned at 75–185 MiB

What is happening now

Same fleet, still on 6.8.0-138, 25 hours of uptime:

host	SUnreclaim	growth
gha-runner-1	2156 MiB	+48 MiB/h
gha-runner-2	1976 MiB	+31 MiB/h
gha-runner-3	2212 MiB	+28 MiB/h
gha-runner-4	2066 MiB	+25 MiB/h
gha-services	112 MiB	+0 MiB/h

Rates are from system_memory_usage{state="slab_unreclaimable"} over a 0.5–0.6 h window, not from spot samples.

The kernel upgrade slowed the leak; it did not remove it. On a fixed kernel a fixed defect should not grow at all.

The services host is the control

gha-services runs the same kernel and the same fleet software but no worker containers, and it is flat at 112 MiB. Every host that churns containers grows; the one that does not, does not. That points at container create/destroy rather than at the kernel in isolation.

Ruled out this round: AppArmor profile accumulation. 119 incus profiles are the nested set of the one live container, and only one distinct container name appears across all loaded profiles — profiles are being unloaded correctly.

Also unexplained: /proc/slabinfo accounts ~1020 MiB by pages while Slab: reports 2762 MiB, so most of the growth is not attributable to a named cache.

Urgency

Not an emergency. At the measured rates the fleet re-enters the 10 GiB region in roughly 7 days (runner-1 first). The kernel_slab_unreclaimable ticket fired at 2 GiB exactly as designed, with a week of runway.

The documented remedy still works and is cheap: pressure-close, natural drain, rolling reboot. What is missing is a reason it should be needed weekly on a kernel recorded as carrying the fix.

Suggested next step

Capture slab evidence before the next reboot resets it — slabtop -o, /proc/slabinfo, /sys/kernel/slab/*/objects — so the growing cache can be named. Every reboot so far has destroyed the evidence that would identify it.

https://claude.ai/code/session_01HMPUfMTrzqwrcgDCFvjnFG

[rldyourmnd]

Investigated further — two hypotheses ruled out, and the mechanism is narrower than the issue text implies

Dying cgroups are real but are not holding the memory

gha-runner-3 carried 142 dying cgroups against 93 live. drop_caches=3 cleared 90 of them (142 → 52), confirming page cache was pinning them — exactly what LWN describes.

But unreclaimable slab did not move: 2213 → 2204 MiB. So the dying cgroups were cheap; they are a symptom of churn, not the leak.

An earlier drop_caches=2 (dentries and inodes only) was equally ineffective: Slab 2668 → 2498 MiB, all of it reclaimable, SUnreclaim 2218 → 2213.

Neither form of cache dropping reclaims this memory. The mitigation the general literature suggests does not apply here.

AppArmor profile accumulation ruled out

119 incus profiles are the nested set belonging to the one live container, and only one distinct container name appears across all loaded profiles. Profiles unload correctly.

What is actually unexplained

On gha-runner-3, after drop_caches=3:

source	value
/proc/slabinfo, summed by pages across 387 caches	528 MiB
meminfo Slab:	2348 MiB
meminfo SUnreclaim:	2195 MiB
meminfo SReclaimable:	153 MiB

78% of slab is not attributable to any named cache. The sum is pagesperslab × num_slabs, so it already counts whole slab pages rather than live objects. Largest named caches are unremarkable: kmalloc-rnd-01-8k 119 MiB, ext4_inode_cache 56 MiB, kmalloc-2k 53 MiB.

Live cgroups account for almost none of it either: summing slab_unreclaimable across all 89 live cgroups gives 45 MiB against the host's 2214 MiB, and the root cgroup — hierarchical in v2, so it covers everything charged — reports 13 MiB.

So the memory is unreclaimable slab that belongs to no live cgroup, is named by no cache, and survives every cache drop.

What this changes

• Only a reboot reclaims it. The documented remedy stands and there is no cheaper substitute; drop_caches is not one.
• Identifying the cache needs kernel-level tooling — kmemleak, slub_debug, or a bpftrace probe on slab allocation — not /proc inspection.
• The suggestion in the issue body to capture slab evidence before the next reboot is now the only way forward, and it must include more than slabinfo, since slabinfo demonstrably cannot see 78% of it.

Runway is unchanged: ~7 days at 25–48 MiB/h.

https://claude.ai/code/session_01HMPUfMTrzqwrcgDCFvjnFG

[rldyourmnd]

Re-measured 2026-08-29 08:00 UTC. The rate is higher than this issue records and the runway is shorter than the estate plan assumed.

All four members are above the 2 GiB alert threshold

kernel_slab_unreclaimable has been firing continuously since 2026-08-28 12:25.

host	SUnreclaim	growth since last reset	reaches 8 GB (half of RAM)
gha-runner-1	3.24 GB	+74 MB/h	~3.0 days
gha-runner-2	3.02 GB	+53 MB/h	~4.4 days
gha-runner-3	3.29 GB	+75 MB/h	~2.9 days
gha-runner-4	3.09 GB	+53 MB/h	~4.3 days
gha-services	0.13 GB	+1 MB/h	~238 days

Recorded rate was 25–48 MiB/h. Measured is 53–75 MB/h — roughly double, and the plan's "about a week of runway" is three days on two members.

The services host is the useful control. Same kernel 6.8.0-138-generic, same fleet software, +1 MB/h — and it runs no containers. Everything that churns containers leaks; the one host that does not, does not. That narrows the search considerably and is measured rather than inferred.

For reference on how far this has drifted from the post-reboot baseline in the estate plan: all four members came back at 75–185 MiB and the accepted soak peaked near 215 MiB. They are now at 15× that.

A caution on capacity changes made while this is open

The fleet's CPU admission reservations were halved today after measurement showed per-instance CPU peak at p90 0.72 cores against a 2-unit reservation. That permits more concurrent workers, therefore more container churn per hour, on a leak that tracks container churn.

Fifty minutes of post-change data is inconclusive and points both ways — runner-1 74→137 MB/h and runner-4 69→115, but runner-2 78→56 and runner-3 81→74, and the window was dominated by one deploy wave. Nothing is attributed. It is recorded in the estate plan with an instruction to watch the aggregate over a full day at the same job mix and revert the reservation change if the rate holds materially higher.

The capture problem is now urgent rather than tidy

Only a reboot clears this and every previous reboot destroyed the evidence. At three days to half of RAM on two members, the choice between "reboot to stay up" and "capture before rebooting" arrives this week. kmemleak, slub_debug and a bpftrace of slab allocation sites all need to be in place before the next drain, not decided during it.

https://claude.ai/code/session_01BjpW1NtF1oik18aEMMGtMP

[rldyourmnd]

Narrowed it, with a same-kernel control and two hypotheses refuted. Also a measurement correction that matters for anyone repeating this.

The memory is in Slab and in no cache

host	slabinfo total	Slab	SUnreclaim	slabinfo accounts for
gha-runner-1	1303 MB	3954 MB	3180 MB	33%
gha-runner-3	1004 MB	3695 MB	3208 MB	27%
gha-services	1076 MB	1095 MB	121 MB	98%

gha-services is the control that makes this readable: same kernel 6.8.0-138-generic, same fleet software, and it runs no containers. There, /proc/slabinfo accounts for 98% of Slab. On the members, roughly 2.6 GB per host is counted in Slab and appears in no cache slabinfo reports.

This is not a runaway cache. The largest entries on a member are ordinary and smaller than the control's: ext4_inode_cache 443 MB on runner-1 against 641 MB on services.

Measurement correction

/proc/slabinfo must be summed as num_slabs × pagesperslab × PAGE_SIZE, not active_objs × objsize. The object form ignores partially-filled slabs and undercounts by about 30% — it gave me 918 MB where the real figure is 1303 MB. The "78% unattributed" in this issue and the 67–73% here are the same finding; the arithmetic just needs to be the slab form.

Two hypotheses refuted

Dying memory cgroups. Similar on all three hosts, including the one that does not leak: runner-1 267, runner-3 312, gha-services 215. Not the differentiator.

Destroyed-but-pinned kmem caches, which would be removed from slabinfo while their pages stay charged. There are none — /sys/kernel/slab minus symlink aliases equals the slabinfo line count exactly on every host: 387/387, 388/388, 383/383.

So it is neither a visible cache growing nor an invisible cache lingering.

The action this leaves

Everything reachable from userspace is now exhausted, which is why the capture has to be arranged before the next drain rather than during it. Concretely, the next reboot of a compute member should come up with slub_debug=U (or at minimum slub_debug=-,kmalloc-*) and kmemleak=on, so the following growth cycle is attributable instead of ending here again. Every previous reboot destroyed the evidence; at 53–75 MB/h and 3.0–3.3 GB already on all four members, the next one is days away, not weeks.

Worth noting the boot-parameter change is itself the whole ask — it costs one rolling reboot that is coming anyway.

https://claude.ai/code/session_01BjpW1NtF1oik18aEMMGtMP

[rldyourmnd]

Re-measured after the 2026-08-29 rolling reboot and the b18/b11/provider .89 wave. At ~29h uptime, SUnreclaim is 1.71–1.86 GiB on the four compute members versus 115 MiB on services; zero failed units, zero host OOM and low swap pressure. The leak therefore persists at roughly the previously observed order of growth and is not closed.

The running Ubuntu kernel has CONFIG_SLUB_DEBUG=y but no CONFIG_DEBUG_KMEMLEAK and no CONFIG_PAGE_OWNER. Kernel documentation advises enabling SLUB user tracking only for selected slabs to avoid system-wide performance impact; this incident still has no named slab (the unexplained bytes are absent from slabinfo), so enabling slub_debug=U fleet-wide would add production overhead without a target.

Operational mitigation remains the measured lossless one-member drain/reboot before the alert runway is exhausted. Warm capacity is now back on b18, but it remains one-job disposable and does not change the number of teardown cycles. Root cause stays open; it is not being mislabeled as completed.