Type of problem
CI timeout — ASan/MSan CAS-over-S3 stateless shards are killed by GitHub's 6h job limit when the test disk uses cas_gc_interval_sec 5.
Summary
The CAS-S3 test policy overrides the product default (60s) to 5s so reclamation happens during the run:
<cas_gc_enabled>1</cas_gc_enabled>
<cas_gc_interval_sec>5</cas_gc_interval_sec>
File: tests/config/config.d/cas_s3_storage_policy_for_merge_tree_by_default.xml
On MasterCI tag v26.6.2.20001.altinityantalya (SHA fc67ca28aab) those shards were cancelled at 6h, not hung on one test:
| Job |
Progress at kill |
Median test time |
Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 1/2) |
4333/5581 (78%) |
~15s |
Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 2/2) |
4380/5604 (78%) |
~14s |
Stateless tests (amd_msan, cas s3 storage, parallel, 2/3) |
2073/3768 (55%) |
~30s |
Stateless tests (amd_msan, cas s3 storage, parallel, 3/3) |
1996/3668 (54%) |
~27s |
Throughput dropped over the run (ASan ~458 → ~183 tests / 30 min). The same-run non-CAS ASan shards finished in ~1.5h; TSan plain S3 in ~2h.
Control
Throwaway PRs against antalya-26.6 (do not merge) with only that XML changed:
cas_gc_interval_sec 60 (product default): PR #2296 — ASan 1/2 3h14m, 2/2 2h59m (finished; OOMs tracked separately)cas_gc_enabled 0: PR #2297 — ASan 1/2 2h07m, 2/2 2h51m (finished; no OOM storm)
GC at 5s is what makes this lane miss 6h. GC is not runtime-reloadable; the interval is baked into the test disk config.
Likely fix direction
Stop using a 5s GC interval in the CAS-S3 (and possibly local CAS) test policy. Product default is already 60s. Keep a shorter interval only in a dedicated GC-focused test, not the full sanitizer stateless suite.
Links
Related: https://github.com/Altinity/ClickHouse/actions/runs/32969478793
Related: https://github.com/Altinity/ClickHouse/actions/runs/32969478793/job/98559503678
Related: #2296
Related: #2297
Project Antalya Build Version: 26.6.2.20001.altinityantalya
Related: #2299
Type of problem
CI timeout — ASan/MSan CAS-over-S3 stateless shards are killed by GitHub's 6h job limit when the test disk uses
cas_gc_interval_sec5.Summary
The CAS-S3 test policy overrides the product default (60s) to 5s so reclamation happens during the run:
File:
tests/config/config.d/cas_s3_storage_policy_for_merge_tree_by_default.xmlOn MasterCI tag
v26.6.2.20001.altinityantalya(SHAfc67ca28aab) those shards were cancelled at 6h, not hung on one test:Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 1/2)Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 2/2)Stateless tests (amd_msan, cas s3 storage, parallel, 2/3)Stateless tests (amd_msan, cas s3 storage, parallel, 3/3)Throughput dropped over the run (ASan ~458 → ~183 tests / 30 min). The same-run non-CAS ASan shards finished in ~1.5h; TSan plain S3 in ~2h.
Control
Throwaway PRs against
antalya-26.6(do not merge) with only that XML changed:cas_gc_interval_sec60 (product default): PR #2296 — ASan 1/2 3h14m, 2/2 2h59m (finished; OOMs tracked separately)cas_gc_enabled0: PR #2297 — ASan 1/2 2h07m, 2/2 2h51m (finished; no OOM storm)GC at 5s is what makes this lane miss 6h. GC is not runtime-reloadable; the interval is baked into the test disk config.
Likely fix direction
Stop using a 5s GC interval in the CAS-S3 (and possibly local CAS) test policy. Product default is already 60s. Keep a shorter interval only in a dedicated GC-focused test, not the full sanitizer stateless suite.
Links
Related: https://github.com/Altinity/ClickHouse/actions/runs/32969478793
Related: https://github.com/Altinity/ClickHouse/actions/runs/32969478793/job/98559503678
Related: #2296
Related: #2297
Project Antalya Build Version: 26.6.2.20001.altinityantalya
Related: #2299