diff --git a/POD-STORAGE-QUOTA.md b/POD-STORAGE-QUOTA.md new file mode 100644 index 0000000..4498fc7 --- /dev/null +++ b/POD-STORAGE-QUOTA.md @@ -0,0 +1,538 @@ +# Pod Storage Quota — Performance Analysis & Design (2026-08-11) + +Pivot runs on the Community Solid Server (CSS) file backend with a per-pod quota +(`css:config/storage/backend/pod-quota-file.json`, enabled from `config/prod.json`). + +**Problem:** the quota is enforced by recursively walking the whole pod tree on +every write. With tens of thousands of files (e.g. a large inbox) this becomes +extremely slow and, under concurrent writes, fills memory and takes the server +down. Disabling the quota (config-only) removes the cost but loses the feature. + +--- + +## 1. How quota works today (CSS v7.x) + +Three moving parts: + +### 1.1 `QuotaValidator` (`dist/storage/validators/QuotaValidator.js`) +Runs on every write/PATCH pipeline and: +1. `getAvailableSpace()` **before** the write → full pod walk. +2. `createQuotaGuard()` — a streaming guard wrapped around the write body. +3. `getAvailableSpace()` **again** after the write (`afterWrite` flush). + +### 1.2 `QuotaStrategy.createQuotaGuard()` — the real killer +```js +async transform(chunk, enc, done) { + total += await reporter.calculateChunkSize(chunk); + const availableSpace = await that.getAvailableSpace(identifier); // ← FULL POD WALK, PER CHUNK + ... +} +``` +`getAvailableSpace()` → `getTotalSpaceUsed()` → `reporter.getSize(podRoot)` → +full recursive `FileSizeReporter.getTotalSize()` walk. So a single multi-chunk +upload walks the **entire pod once per stream chunk**. This is `chunks × O(N)`. + +### 1.3 `FileSizeReporter.getTotalSize()` (`dist/storage/size-reporter/FileSizeReporter.js`) +```js +if (stat.isFile()) return stat.size; +const childFiles = await fs.readdir(fileLocation); // one big dirent array +let totalSize = stat.size; +for (const current of childFiles) { + // skip ignoreFolders (e.g. /.internal/) + totalSize += await this.getTotalSize(childFileLocation); // recursion +} +``` +Sequential `stat` + `readdir` per entry — O(N) syscalls per walk, one large +dirent array per directory in the Node heap. + +### Cost profile +| | Per write | +|---|---| +| Pre-check | 1 full walk | +| Streaming guard | 1 full walk **per chunk** | +| Post-check | 1 full walk | + +Concurrent writes multiply the walks → memory exhaustion (dirent arrays + stat +results + in-flight async walks pile up in the Node heap). + +--- + +## 2. Backend impact (memory / database) + +CSS has **only one size reporter: `FileSizeReporter`** (filesystem-specific). +There is no memory or database reporter. + +| Backend | Quota wired? | Notes | +|---|---|---| +| **File** (`file.json`, `pod-quota-file.json`) | ✅ | The O(N)-per-write problem above. | +| **Memory** (`memory.json`, `MemoryDataAccessor`) | ❌ none | No quota at all. 10k-file inbox lives in RAM (inherent to backend). If quota wanted, size calc is trivial (sum contentLength in-memory) and an incremental counter needs no persistence. | +| **Database** (`sparql.json`) | ❌ none | No quota. "Size" is ambiguous (bytes vs quads); counter would be a table + delta queries. | + +The recursive-walk problem is specific to the **file backend** (what pivot uses). +Design C targets the file backend first, keeping the reporter/counter pluggable. + +--- + +## 3. Fix directions + +### A. Stop the per-chunk full walk (mandatory, lowest risk) +Compute `availableSpace` **once** before the stream; during streaming only track +the current write's byte delta and compare: + +```js +availableSpace = await strategy.getAvailableSpace(identifier); // once +transform(chunk) { + total += chunk.length; + if (availableSpace.amount < total) throw Quota exceeded; + push(chunk); +} +``` +Correct: `getAvailableSpace` already subtracts the overwritten resource's size, +and nothing else about the pod changes mid-write. Atomic writes go to +`/.internal/` (ignored by the walk) anyway. +Cost: `chunks × O(N)` → **~2 × O(N) per write** (pre + post). Kills the memory +blowup from concurrent chunk-streams. + +### B. Cache / memoize the pod size +Wrap `getSize(pod)` with a per-pod cache: +- **TTL variant:** cache `{ size, expiresAt }` (e.g. 1–5 s); one walk per expiry. +- **Invalidation variant:** drop the pod's cache entry when a write completes. +Combined with A, roughly **1 walk per pod per TTL window / per invalidation**. +Caveat: TTL window staleness (concurrent writes may exceed the limit undetected +within the window; CSS already documents tolerance of races). + +### C. Incremental per-pod counter (durable end-state) +Per-write O(1); one full walk only for bootstrap/recovery. See §4. + +--- + +## 4. Design C — incremental per-pod byte counter + +### 4.1 Counter store +- **In-memory:** `Map` — O(1) reads. +- **Persistence:** per-pod sidecar, e.g. `/.internal/pivot-quota.json` + (`.internal/` is already ignored by size accounting). Written **atomically** + (write temp + rename) on each write → crash-safe. + +### 4.2 Recount (bootstrap / recovery) +- First access to a pod with no valid counter → **one** full walk to seed it. +- Startup: load sidecar if present (no walk); else lazy recount on first access. +- Crash between write and persist: mark dirty / recount on next access. + +### 4.3 Delta hooks +A pivot store wrapper (like the existing `RdfPatchingStore`) around the backend: +- **Before:** `oldSize = accessor.getSize(identifier)` (O(1) single `stat`). +- Perform write/delete. +- **After:** `newSize = accessor.getSize(identifier)`. +- `Δ = newSize − oldSize` → `counter[pod] += Δ`. + +Cases: +- Create: `Δ = file size`. Overwrite: `Δ = new − old` (can be negative). +- Delete: `Δ = −oldSize`. Auxiliary writes (`.acl`/`.meta`): included (matches today). +- Atomic temp files: no special handling (rename exposes only the final file). + +### 4.4 Read path +`getSize(pod)` → `counter[pod].total` if valid, else recount. Plugs into the +**existing** `PodQuotaStrategy`/`QuotaValidator` unchanged — only +`urn:solid-server:default:SizeReporter` is replaced (plus the delta store +wrapper). No validator/strategy changes. + +### 4.5 Concurrency +- Per-pod mutex serializes `counter += Δ`. +- Sidecar atomic rename per write (cheap). Optional mode: persist periodically + + always recount on startup (simpler, costs a startup walk per pod). + +### 4.6 Edge cases / staleness +- Out-of-band file changes → stale counter. Mitigations: on-demand recount + endpoint, or periodic background recount. Documented limitation. +- Pod deletion → remove counter entry. + +### 4.7 Config wiring (pivot) +- New pivot components: `IncrementalSizeReporter` (counter + recount) and a + quota-delta store wrapper. +- Override `urn:solid-server:default:SizeReporter` and insert the wrapper in the + store chain via `pivot:config/storage/backend/...`, keeping + `pod-quota-file.json`'s validator/strategy. No CSS fork. + +### 4.8 Staleness detection & recovery (added 2026-08-15) + +The counter is **only an optimization — the filesystem is the source of +truth**, and `du` is a cheap way to re-derive truth. So a de-synchronized +counter is never permanent or catastrophic: it self-heals on next access. + +**Causes of de-sync** + +| Cause | How it happens | +|---|---| +| Out-of-band changes | Files added/removed directly on disk (admin, scripts, restore, sync tool) — bypasses the delta hook | +| Crash between write & persist | Data write lands, but the process dies before the sidecar rename — counter short by that one delta | +| Migration/bootstrap | Pods created before the feature → no sidecar (handled lazily at first access) | +| Auxiliary writes bypassing the hook | A code path (`.acl`/`.meta`, temp files, future store change) that forgets to report its delta | +| Manual tampering | Sidecar edited/deleted/restored from a backup without the pod | + +**Detection — cheap validity checks on read** (no walk required): + +1. `valid` flag / generation marker — sidecar records `{ total, valid, + version }`. Any path that can't guarantee a delta flips `valid: false`. +2. **Pod-root mtime comparison** — sidecar stores the pod root's + `lastRecordedMtime`; on read, one `stat` of the pod root — if newer than + `lastRecordedMtime`, the counter may be behind → invalidate → recount. + Catches out-of-band writes for the cost of a single `stat`. +3. **Sanity bound** — if `total` is wildly inconsistent with expectation + (e.g. 0 but the pod has files), treat as dirty. + +**Recovery — reuse the lazy path**: invalidation just means "delete/flag the +sidecar"; the existing lazy-bootstrap path does the rest — next access to +that pod runs one `du` and re-seeds. + +- On suspicion (mtime/flag/`valid:false`) → mark dirty → recount on next + access. One `du`, non-blocking. +- Crash window — atomic rename keeps the sidecar internally consistent (old + or new, never torn); add `fsync` before the rename completes the response, + shrinking the window. If a crash still lands in it, the pod-root mtime + check catches it on next access. +- Admin/on-demand recount — small CLI/HTTP/admin hook to force recount of one + pod or all pods (delete sidecars → next accesses recount, or an explicit + sweep). +- Optional background reconciliation — low-priority idle job walks pods with + `du` every few hours to bound drift over time. Never at startup, never + blocking writes. + +**Bottom line:** de-sync is detected cheaply (mtime/valid flag, one `stat`) +and fixed cheaply (`du` recount on next access). Worst persistent state is +"counter slightly behind until next access of that pod", which then +self-heals. + +--- + +## 5. Using `du` as the fast walk / recount primitive + +Instead of the Node recursive walk, shell out to GNU `du` — C-level `fts` +traversal: + +- **Speed:** 10–100× faster than the per-entry Node `stat`/`readdir` loop. +- **Memory:** the walk runs in a child process — dirent arrays / stat buffers + never touch the Node heap (directly fixes the memory blowup). +- **Unit:** `du -sb` = apparent bytes (sum of `st_size`) — identical semantics + to today's `FileSizeReporter` and the **chosen unit (2026-08-15)**: portable + across servers/filesystems and user-manageable. `du -s --block-size=1` = + disk usage — **rejected**: the result is server/filesystem-dependent + (cluster size, compression, COW), not user-manageable, and Solid pods are + small-file-heavy so cluster rounding would dominate. + +### Critical caveat +**Never call `du` per stream chunk** — spawning a process per chunk is a spawn +storm. `du` must be combined with fix **A** (walk only at pre/post check) and +ideally with **B**/**C** (walk only on recount). In design C, `du` is the +recount/seed engine only; steady-state writes are O(1) with zero spawns. + +### Practical concerns +| Concern | Handling | +|---|---| +| `ignoreFolders` (`.internal/`) | GNU `du --exclude=PATTERN` (or `--exclude-from`) — must match current behavior. | +| Path safety | `execFile('du', ['-sb', '--exclude', ..., podPath])` — never shell interpolation; CSS already containment-checks mapped paths. | +| Portability | `du` is coreutils/BSD — fine on Linux/WSL test servers, **not native on Windows**. Fall back to the Node walk when `du` is unavailable. | +| Symlinks/hardlinks | `du` doesn't follow symlinks by default (FileSizeReporter's `stat` does). Minor edge case — document. | +| Concurrency | With A+B/C, `du` spawns are rare → no process storm. | + +### Recommendation +- **Short/medium term:** `du` as the walk engine behind `getTotalSize`, **plus** + fix A (no per-chunk walk), **plus** a small TTL cache (B) → ~1 `du` spawn per + write. Small change, keeps the architecture, kills time + memory problems. +- **Long term:** same `du` as the recount engine inside **C** — incremental + counter, deltas per write, `du -sb` only to seed/repair. O(1) writes with + native-speed recovery. + +### Platform / flags + +**Performance: apparent-size vs disk-blocks — no meaningful difference.** +Both variants traverse the identical tree (same `readdir`/`stat` syscalls); +apparent size sums `st_size`, disk usage sums `st_blocks` (already in the +`stat` result). No extra syscalls either way → pick the unit purely on quota +semantics, not performance. + +`du` availability: + +| Platform | `du` | Flags | +|---|---|---| +| **Linux** (incl. WSL) | ✅ GNU coreutils | `-sb` (apparent bytes), `--block-size=1` / `-B 1` (disk bytes), `--exclude=PATTERN` | +| **macOS** | ✅ BSD `du` (always present) | **No GNU long options.** Apparent size: `-A`; bytes: `-B 1`; exclude: `-I pattern`; default block unit 512 B. So `du -s -A -B 1` ≈ GNU `du -sb`, `du -s -B 1` ≈ GNU `du -s --block-size=1` | +| **Windows** | ❌ **no native `du`** | None in cmd/PowerShell. Sysinternals `du.exe` (different CLI, not GNU-compatible), or GNU `du` via Git Bash / MSYS2 / Cygwin / WSL. PowerShell `Get-ChildItem -Recurse \| Measure-Object Length -Sum` is the slow JS-style walk | + +**Implementation note:** GNU-first, with probe & fallback: +1. Detect GNU vs BSD (`du --version` succeeds on GNU, fails on BSD → use + `-A`/`-B 1`/`-I`). +2. Fall back to the existing Node recursive walk when no compatible `du` is + found (e.g. bare Windows) — correct, just slower. + +### Unit consistency: apparent bytes everywhere (decided 2026-08-15) + +The chosen unit is **apparent bytes** (sum of `st_size` — same as today's +`FileSizeReporter`). This makes the unit fully portable: the same content +measures the same on any server/filesystem, and users can reason about it +("I used 500 MB of 1 GB"). Disk blocks (`st_blocks`) were considered and +rejected because the result is server-dependent (cluster size, compression, +COW) and small-file-heavy Solid pods would be dominated by cluster rounding. + +With apparent bytes there is **no platform inconsistency**: + +| Path | Unit | How | +|---|---|---| +| `du` (Linux/macOS) | apparent bytes | `du -sb` / BSD `du -s -A -B 1` | +| Node walk (Windows fallback) | apparent bytes | sum `stat.size` | + +Both paths sum `st_size` — identical quantity everywhere. (Verified: Node +`fs.stat` also exposes `blocks` on Windows, e.g. a 1000-byte file → +`blocks*512: 4096`, but we deliberately do NOT use it — that would introduce +server-dependent cluster rounding.) + +Edge cases: +- Sparse files: apparent bytes counts them at their logical size (matches + today; a disk-space quota would count less). +- Symlinks: `stat` follows symlinks (matches today's `FileSizeReporter`); + `du` doesn't by default — already noted as a minor edge case. + +--- + +## 6. Decision points before implementing + +### Decisions (2026-08-15) +1. **Unit: apparent bytes** (`du -sb` / sum of `st_size`) — portable across + servers/filesystems and **user-manageable**; identical semantics to + today's `FileSizeReporter`, so the existing 70 MB limit in + `customise-me.json` keeps its meaning. Disk blocks considered and + rejected (server/filesystem-dependent result; small-file-heavy pods). +2. **Persistence mode: persist-per-write** (atomic sidecar + `/.internal/pivot-quota.json`, exact across restarts). **Restart does + NOT trigger a bulk recount of all pods** — counters are loaded from disk; + the only full walks are lazy per-pod (first access of a pre-existing pod / + crash-dirty pod / de-synced pod — i.e. effectively *at login* for that + pod, never a startup sweep). +3. **Scope: A+B first** (quick win: no per-chunk walk + du-based walk with + TTL cache), then build C on top later. +4. **Staleness/recovery:** mtime/valid-flag check on read + lazy recount + + optional background reconciliation (see §4.8). + +### Remaining to verify before implementing +- Where before/after sizes are cleanest in the `AtomicFileDataAccessor` / + store stack (determines how much of C is new plumbing vs reusing existing + hooks). +- Exact `du` platform handling (GNU `--block-size=1` / BSD `-B 1 -s`, probe + & fallback to the Node walk; never per-chunk spawns). +- TTL window semantics with concurrent writes (CSS already tolerates race; + the `QuotaValidator` after-write flush check still catches breaches). + +--- + +## 7. Verification: benchmark + equivalence proof (2026-08-15) + +Two scripts in `scripts/` demonstrate the improvement and prove the size +calculation is unchanged. + +### 7.1 Benchmark — `scripts/benchmark-quota.js` + +Measures exactly what A+B optimizes: the pod walk and the per-write quota +guard. Run `node scripts/benchmark-quota.js [fileCount] [fileBytes]`. + +Results on a pod with **5 000 files × 1 KB** (body write = 4 MB in 64 KB +chunks): + +| Measure | Old (FileSizeReporter + QuotaStrategy) | New (DuSizeReporter + FastQuotaStrategy) | +|---|---|---| +| Full pod walk (`getSize`) | 669.6 ms | 657.8 ms first call; **0.13 ms cached** | +| Per-write quota guard | **36 279 ms** (≈36 s; full walk **per chunk**) | **3.3 ms** (walk once + cache) | +| Guard speedup | — | **~11 000×** | + +Notes: +- On bare Windows both paths use the Node walk (no `du`), so the walk line + shows ~1×; on Linux/WSL the `du` walk is 10–100× faster than the Node walk. +- The 36 s guard is the exact production bug: a single 4 MB upload into a + 5 000-file pod triggered 64 full pod walks. + +### 7.2 Equivalence — `scripts/verify-size-equivalence.js` + +Proves the new reporter returns **byte-for-byte identical** apparent-byte +totals to CSS's `FileSizeReporter`. Generates random pod trees (nested dirs, +random-size files, empty dirs, root-level `.internal` temp files) and asserts +equality for **both** the `du` path and the Node-walk fallback. + +Run `node scripts/verify-size-equivalence.js [iterations] [maxFiles]` +(with GNU `du` on PATH to exercise the real `du` path — e.g. Git Bash's +`C:\Program Files\Git\usr\bin` on Windows). + +Result: **12/12 random trees match exactly** on both paths. + +Findings / documented differences: +- **Node-walk fallback is mathematically identical** to `FileSizeReporter` + (same recursive `stat.size` sum) — always equivalent. +- **`du` path** sums the same `st_size` values (directory sizes included by + both) — equivalent. +- **Exclusion semantics caveat** (found by the proof): `du --exclude=.internal` + matches a `.internal` component at *any* depth, while CSS's anchored regex + `^/\.internal$` only excludes it at the *pod root*. Identical for real pods + (`.internal` is only ever created at the root — CSS temp files). A nested + `.internal` would be excluded by `du` but counted by the Node walk — a + documented, non-issue-in-practice difference. +- **Symlinks** (`du` doesn't follow by default; Node `stat` does) and + **hardlinks** (`du` counts once; Node per directory entry) differ — neither + occurs in normal Solid pod storage. + +--- + +## 8. Design C — incremental per-pod counter (implemented 2026-08-15) + +**Files:** `src/storage/quota/QuotaCounter.ts`, `src/storage/quota/IncrementalSizeReporter.ts`, +`src/storage/quota/QuotaDeltaDataAccessor.ts`, `config/storage/backend/quota-counter-file.json`, +`src/index.ts`, `config/customise-me.json`, `test/unit/storage/*`, `scripts/benchmark-quota-c.js`, +`scripts/smoke-design-c.js` + +Steady-state writes become **O(1)**: a per-pod byte counter is updated by a +delta hook and read by the quota strategy; a full `du`/Node walk happens only +once per pod (bootstrap / recovery). + +### 8.1 Components + +| Component | Role | +|---|---| +| **`QuotaCounter`** | In-memory `Map` + per-pod mutex; sidecar `/.internal/pivot-quota.json` written atomically (temp + rename) on every delta; lazy recount via an uncached `DuSizeReporter`; §4.8 mtime staleness check on read; `register / isPodRoot / getSize / add / remove / sizeOfResource / walk`. | +| **`IncrementalSizeReporter`** | `SizeReporter` replacing `urn:solid-server:default:SizeReporter`: pod root → O(1) counter read; any other resource → single `stat`. | +| **`QuotaDeltaDataAccessor`** | `PassthroughDataAccessor` wrapping the top of the accessor chain; before/after apparent size (data + `.meta` stat; containers walked) on `writeDocument` / `writeContainer` / `writeMetadata` / `deleteResource` → `counter.add(pod, Δ)`. Pod discovery mirrors `PodQuotaStrategy.searchPimStorage` (metadata walk for `pim:Storage`, cached per path); deleting the pod root drops the counter. | + +### 8.2 Config wiring (no CSS fork) + +`config/storage/backend/quota-counter-file.json` (imported by `customise-me.json`): +- `QuotaCounter` instance (fileIdentifierMapper, rootFilePath, `ignoreFolders: ["^/\\.internal$"]`) +- `Override` `urn:solid-server:default:SizeReporter` → `IncrementalSizeReporter` +- `Override` `urn:solid-server:default:FileDataAccessor` → `QuotaDeltaDataAccessor` + (wrapping the original FilterMetadata → Validating → Atomic chain, preserving + the content-length filter) +- `Override` `urn:solid-server:default:QuotaStrategy` → `FastQuotaStrategy` (70 MB) + +`QuotaValidator`, `ValidatingDataAccessor`, `AtomicFileDataAccessor` unchanged. + +### 8.3 Verification — `scripts/benchmark-quota-c.js` + +Old vs A+B vs C under identical conditions, with COLD (first write) and WARM +(steady state) separated. Same host, 5 000 files × 1 KB, 4 MB write in 64 KB +chunks: + +| Measure | old | A+B | C | +|---|---|---|---| +| walk `getSize(podRoot)` | 664.8 ms | 745.7 ms cold / 0.19 ms cached | 0.48 ms | +| write guard **cold** | 36 660 ms (64 walks) | 673.7 ms | 630.4 ms (bootstrap) | +| write guard **warm** | — (no cache) | 32.8 ms | **3.38 ms** | + +At 10 000 files (WSL1): old = **108 670 ms**, A+B warm = 49.1 ms, +**C warm = 10.6 ms** — C is the only write that stays O(1)/flat. + +Notes: +- The early A+B benchmark's "3.3 ms" was warm-cache with the Node-walk fallback + (no `du`), which is why it differed from later runs — the corrected script + separates cold/warm and uses the same pod layout for all three. +- C warm's remaining per-write cost is a few `fs.stat`s (pod-root mtime check + + overwritten-resource stat) plus identifier mapping — no walks. On WSL1 + (drvfs) each stat crosses the WSL→Windows boundary (~1 ms); on native Linux + (ext4) it is microseconds, so C warm is sub-millisecond and flat. +- The mtime staleness check (one stat per quota read, §4.8) is a deliberate + robustness tradeoff; it can be made periodic/configurable if stat latency + matters. + +### 8.4 Implementation notes / fixes found by the test run + +- `discoverPod` must return the pod root **identifier** (URL), not a filesystem + path (counter methods map identifiers). +- Tests/config must pass `ignoreFolders: ["^/\\.internal$"]` or the sidecar + inflates recounts. +- `QuotaCounter.add` records the pod-root mtime **after** persisting + (`persistWithMtime`): creating `.internal/` bumps the pod-root mtime, so + recording before would cause a spurious recount on every read. +- The delta hook registers the pod even when a write's delta is 0 (e.g. an + empty container on filesystems reporting directory size 0), so the reporter + routes pod-root reads to the counter. +- `scripts/smoke-design-c.js` verifies the compiled dist without jest: + accumulate, sidecar reload, staleness recount, remove/re-bootstrap, reporter + O(1)+stat, delta accessor create/overwrite/meta/delete == real walk, pod-root + delete drops the counter — all pass. + +--- + +## 9. Production incident: IDP lock expiry on `/.internal` (2026-08-16) + +**Symptom** (pivot-test, quota-counter config, during real logins): +``` +[WrappedExpiringReadWriteLocker] error: Lock expired after 6000ms on https://.../.internal/idp/adapter/AuthorizationCode/ +[WrappedExpiringStorage] error: Error during interval callback: Failed to remove expired entries - Lock expired after 6000ms ... +``` + +**Why `/.internal` was affected.** The IDP's `KeyValueStorage` (AuthorizationCode +store, `/.internal/idp/adapter/`) is backed by `ResourceStore` → +`ResourceStore_Backend` → `FileDataAccessor` (see CSS +`config/storage/key-value/resource-store.json`). Since `QuotaDeltaDataAccessor` +overrides `FileDataAccessor`, **every** internal write ran through the quota +chain: + +1. `QuotaDeltaDataAccessor.track()` — stat + pod-discovery walk + counter sidecar. +2. **`QuotaValidator`** (inside `ValidatingFileDataAccessor`) — calls + `getAvailableSpace` **before and after** every write + `createQuotaGuard` + mid-stream. For `/.internal/*`, `getAvailableSpace` did pod discovery + (`searchPimStorage`) + `reporter.getSize(pod)` — a **full pod walk** whenever + the pod wasn't counter-registered → pushed the IDP write past the 6 s + `WrappedExpiringReadWriteLocker` expiry. + +`ignoreFolders: ["^/\\.internal$"]` on the reporter only excludes `.internal` +from `du`/walk sizes — it does **not** stop the QuotaValidator from running. + +**Fix (three commits on `pod-quota-counter`):** +- `d8c1135` — `QuotaDeltaDataAccessor`: skip delta tracking for `/.internal` + paths (still performs the write). +- `6692db8` — `FastQuotaStrategy.getAvailableSpace` returns unlimited for + `/.internal` paths, short-circuiting the QuotaValidator's before/after checks + and the guard's available-space computation — no pod discovery/walk on + internal writes. +- `5e54d02` — **root cause of the guard never matching:** `ResourceIdentifier.path` + is the **full canonical URL** (e.g. `https://pivot-test.solidproject.org:3000/.internal/...`), + not a bare path — identifier strategies test it against URL regexes + (`createSubdomainRegexp`). The initial `startsWith('/.internal/')` check in + both `d8c1135`/`6692db8` never matched, so the quota chain kept running on + internal writes (hence the subdomain/suffix difference observed on + pivot-test: only prod-sized pods blew the 6 s lock). New shared helper + `src/storage/quota/InternalPath.ts` extracts `new URL(path).pathname` before + comparing, covering suffix *and* subdomain modes (and bare-path fallback). + +**Verification:** `benchmark-quota-c.js` C-warm unchanged (~3 ms flat); +`smoke-design-c.js` ALL CHECKS PASSED (calculation intact); unit check of +`isInternalPath` matches subdomain/suffix internal URLs and rejects pod URLs; +lock errors gone on pivot-test after deploying all three commits. + +**Lesson:** quota hooks must treat CSS-internal paths (`/.internal/`) as exempt +at **every** layer (delta accessor *and* quota validator/strategy), not just in +the size-reporter's walk excludes — and remember that `ResourceIdentifier.path` +is a URL, so prefix checks must go through `new URL(...).pathname`. + +--- + +## 10. Subdomain-mode pod discovery bug (2026-08-17) + +**Symptom:** in subdomain mode no `pivot-quota.json` sidecar is ever created +(e.g. `data-subdomain/alice` after a real write), while suffix mode works +(`data-suffix/bourgeoa` has one). + +**Root cause:** `QuotaDeltaDataAccessor.discoverPod` (copied from CSS's +`PodQuotaStrategy.searchPimStorage`) tests `identifierStrategy.isRootContainer()` +**before** reading metadata. In subdomain mode every pod root IS a root +container (`SubdomainIdentifierStrategy.isRootContainer(alice.localhost/)` -> +true), so discovery bails with "no pod" without ever checking `pim:Storage` — +no counter, no delta updates, and `getAvailableSpace` returns unlimited. This +is an **upstream CSS limitation** too: standard `PodQuotaStrategy` never finds +pods in subdomain mode, so pod quota is silently unlimited there. + +**Fix:** new shared `src/storage/quota/PodDiscovery.ts` — reads metadata +**first**, returns the pod if it has `pim:Storage`, and only then falls back to +the root-container stop. Used by both `QuotaDeltaDataAccessor` and +`FastQuotaStrategy.getAvailableSpace` (which no longer delegates to +`super.getAvailableSpace` — it replicates `QuotaStrategy.getAvailableSpace` +semantics: pod total minus the overwritten resource's size). + +**Verification:** subdomain smoke (`pod registered: true`, sidecar created, +`getAvailableSpace` returns `limit - used`); suffix `smoke-design-c.js` ALL +CHECKS PASSED in WSL; tsc clean. diff --git a/config/customise-me.json b/config/customise-me.json index 9c5f4e6..ddbe66f 100644 --- a/config/customise-me.json +++ b/config/customise-me.json @@ -4,6 +4,10 @@ "https://linkedsoftwaredependencies.org/bundles/npm/@solid/community-server/^7.0.0/components/context.jsonld", "https://linkedsoftwaredependencies.org/bundles/npm/@solid/pivot/^1.0.0/components/context.jsonld" ], + "import": [ + "pivot:config/storage/backend/quota-counter-file.json", + "pivot:config/storage/resource-locker/long-expiry.json" + ], "@graph": [ { "comment": "The settings of your email server.", @@ -37,7 +41,7 @@ "overrideInstance": { "@id": "urn:solid-server:default:QuotaStrategy" }, - "overrideParameters": { + "overrideParameters": { "@type": "PodQuotaStrategy", "limit_amount": 70000000, "limit_unit": "bytes" diff --git a/config/dev-http-subdomain.json b/config/dev-http-subdomain.json index e735f3d..2ffc205 100644 --- a/config/dev-http-subdomain.json +++ b/config/dev-http-subdomain.json @@ -29,6 +29,7 @@ "css:config/storage/key-value/resource-store.json", "css:config/storage/location/pod.json", "pivot:config/storage/middleware/default.json", + "pivot:config/storage/profile-card-guard.json", "css:config/util/auxiliary/acl.json", "css:config/util/identifiers/subdomain.json", @@ -37,7 +38,8 @@ "css:config/util/representation-conversion/default.json", "css:config/util/resource-locker/file.json", "css:config/util/variables/default.json", - "pivot:config/pivot-overrides.json" + "pivot:config/pivot-overrides.json", + "pivot:config/pivot-file-locker-overrides.json" ], "@graph": [ { diff --git a/config/dev-http-suffix.json b/config/dev-http-suffix.json index 3b60c04..06f5842 100644 --- a/config/dev-http-suffix.json +++ b/config/dev-http-suffix.json @@ -29,6 +29,7 @@ "css:config/storage/key-value/resource-store.json", "css:config/storage/location/pod.json", "pivot:config/storage/middleware/default.json", + "pivot:config/storage/profile-card-guard.json", "css:config/util/auxiliary/acl.json", "css:config/util/identifiers/suffix.json", @@ -37,7 +38,8 @@ "css:config/util/representation-conversion/default.json", "css:config/util/resource-locker/file.json", "css:config/util/variables/default.json", - "pivot:config/pivot-overrides.json" + "pivot:config/pivot-overrides.json", + "pivot:config/pivot-file-locker-overrides.json" ], "@graph": [ { diff --git a/config/pivot-file-locker-overrides.json b/config/pivot-file-locker-overrides.json new file mode 100644 index 0000000..5dcb487 --- /dev/null +++ b/config/pivot-file-locker-overrides.json @@ -0,0 +1,20 @@ +{ + "comment": "Overrides specific to configs that use the file-system resource locker. Imported only by prod/dev configs, NOT by test.json (which uses the memory locker).", + "@context": [ + "https://linkedsoftwaredependencies.org/bundles/npm/@solid/community-server/^7.0.0/components/context.jsonld", + "https://linkedsoftwaredependencies.org/bundles/npm/@solid/pivot/^1.0.0/components/context.jsonld" + ], + "@graph": [ + { + "comment": "Bound the lock acquisition retries so a contended lock fails fast instead of spinning the event loop and filesystem threadpool forever (~30s max wait).", + "@type": "Override", + "overrideInstance": { "@id": "urn:solid-server:default:FileSystemResourceLocker" }, + "overrideParameters": { + "@type": "FileSystemResourceLocker", + "attemptSettings_retryCount": 600, + "attemptSettings_retryDelay": 50, + "attemptSettings_retryJitter": 30 + } + } + ] +} diff --git a/config/pivot-overrides.json b/config/pivot-overrides.json index 0abcd00..99a07e8 100644 --- a/config/pivot-overrides.json +++ b/config/pivot-overrides.json @@ -61,6 +61,82 @@ "templateEngine": { "@id": "urn:solid-server:default:TemplateEngine" }, "template": "./templates/main.html.ejs" } + }, + { + "@type": "Override", + "overrideInstance": { "@id": "urn:solid-server:default:AccountStorage" }, + "overrideParameters": { + "@type": "SafeBaseLoginAccountStorage", + "storage": { "@id": "urn:solid-server:default:IndexedStorage" } + } + }, + { + "comment": "Scope the cookie sweep to its own container so entries()/sweeps do not walk the entire /.internal/ tree (sustained high CPU with many accounts).", + "@type": "Override", + "overrideInstance": { "@id": "urn:solid-server:default:CookieStorage" }, + "overrideParameters": { + "@type": "WrappedExpiringStorage", + "source": { + "@type": "MaxKeyLengthStorage", + "source": { + "@type": "JsonResourceStorage", + "source": { "@id": "urn:solid-server:default:ResourceStore_Backend" }, + "baseUrl": { "@id": "urn:solid-server:default:variable:baseUrl" }, + "container": "/.internal/accounts/cookies/" + } + } + } + }, + { + "comment": "Scope the forgot-password sweep to its own container.", + "@type": "Override", + "overrideInstance": { "@id": "urn:solid-server:default:ForgotPasswordStorage" }, + "overrideParameters": { + "@type": "WrappedExpiringStorage", + "source": { + "@type": "MaxKeyLengthStorage", + "source": { + "@type": "JsonResourceStorage", + "source": { "@id": "urn:solid-server:default:ResourceStore_Backend" }, + "baseUrl": { "@id": "urn:solid-server:default:variable:baseUrl" }, + "container": "/.internal/accounts/forgot-password/" + } + } + } + }, + { + "comment": "Scope the token sweep to its own container.", + "@type": "Override", + "overrideInstance": { "@id": "urn:solid-server:default:ExpiringTokenStorage" }, + "overrideParameters": { + "@type": "WrappedExpiringStorage", + "source": { + "@type": "MaxKeyLengthStorage", + "source": { + "@type": "JsonResourceStorage", + "source": { "@id": "urn:solid-server:default:ResourceStore_Backend" }, + "baseUrl": { "@id": "urn:solid-server:default:variable:baseUrl" }, + "container": "/.internal/idp/tokens/" + } + } + } + }, + { + "comment": "Scope the adapter sweep to its own container. Requires the @id added to the installed CSS adapter-factory.json by the patch-package override.", + "@type": "Override", + "overrideInstance": { "@id": "urn:solid-server:default:ExpiringAdapterStorage" }, + "overrideParameters": { + "@type": "WrappedExpiringStorage", + "source": { + "@type": "MaxKeyLengthStorage", + "source": { + "@type": "JsonResourceStorage", + "source": { "@id": "urn:solid-server:default:ResourceStore_Backend" }, + "baseUrl": { "@id": "urn:solid-server:default:variable:baseUrl" }, + "container": "/.internal/idp/adapter/" + } + } + } } ] } diff --git a/config/prod.json b/config/prod.json index db33fe3..ef539f8 100644 --- a/config/prod.json +++ b/config/prod.json @@ -29,6 +29,7 @@ "css:config/storage/key-value/resource-store.json", "css:config/storage/location/pod.json", "pivot:config/storage/middleware/default.json", + "pivot:config/storage/profile-card-guard.json", "css:config/util/auxiliary/acl.json", "css:config/util/identifiers/subdomain.json", "css:config/util/logging/winston.json", @@ -36,7 +37,8 @@ "css:config/util/representation-conversion/default.json", "css:config/util/resource-locker/file.json", "css:config/util/variables/default.json", - "pivot:config/pivot-overrides.json" + "pivot:config/pivot-overrides.json", + "pivot:config/pivot-file-locker-overrides.json" ], "@graph": [ { @@ -70,6 +72,11 @@ "@id": "urn:solid-server:default:ExpiringTokenStorage", "@type": "WrappedExpiringStorage", "timeout": 1 + }, + { + "@id": "urn:solid-server:default:ExpiringAdapterStorage", + "@type": "WrappedExpiringStorage", + "timeout": 1 } ] } diff --git a/config/storage/backend/quota-counter-file.json b/config/storage/backend/quota-counter-file.json new file mode 100644 index 0000000..c60fd4a --- /dev/null +++ b/config/storage/backend/quota-counter-file.json @@ -0,0 +1,90 @@ +{ + "comment": "Design C: incremental per-pod byte counter. QuotaCounter + IncrementalSizeReporter + QuotaDeltaDataAccessor (delta hook) + FastQuotaStrategy. Writes are O(1) after bootstrap; a full du/Node walk only seeds/repairs a pod's counter.", + "@context": [ + "https://linkedsoftwaredependencies.org/bundles/npm/@solid/community-server/^7.0.0/components/context.jsonld", + "https://linkedsoftwaredependencies.org/bundles/npm/@solid/pivot/^1.0.0/components/context.jsonld" + ], + "@graph": [ + { + "comment": "The shared per-pod counter (in-memory + sidecar + recount engine).", + "@id": "urn:solid-server:default:QuotaCounter", + "@type": "QuotaCounter", + "fileIdentifierMapper": { + "@id": "urn:solid-server:default:FileIdentifierMapper" + }, + "rootFilePath": { + "@id": "urn:solid-server:default:variable:rootFilePath" + }, + "ignoreFolders": [ + "(^|/)\\.internal$" + ] + }, + { + "comment": "SizeReporter backed by the counter: O(1) pod reads, single stat for resources.", + "@type": "Override", + "overrideInstance": { + "@id": "urn:solid-server:default:SizeReporter" + }, + "overrideParameters": { + "@type": "IncrementalSizeReporter", + "counter": { + "@id": "urn:solid-server:default:QuotaCounter" + } + } + }, + { + "comment": "Delta hook: wraps the accessor chain, feeds per-write deltas to the counter. Preserves the content-length filter.", + "@type": "Override", + "overrideInstance": { + "@id": "urn:solid-server:default:FileDataAccessor" + }, + "overrideParameters": { + "@type": "QuotaDeltaDataAccessor", + "fileIdentifierMapper": { + "@id": "urn:solid-server:default:FileIdentifierMapper" + }, + "identifierStrategy": { + "@id": "urn:solid-server:default:IdentifierStrategy" + }, + "counter": { + "@id": "urn:solid-server:default:QuotaCounter" + }, + "accessor": { + "@type": "FilterMetadataDataAccessor", + "accessor": { + "@id": "urn:solid-server:default:ValidatingFileDataAccessor" + }, + "filters": [ + { + "@type": "FilterPattern", + "predicate": "http://www.w3.org/2011/http-headers#content-length" + } + ] + } + } + }, + { + "comment": "Pod quota strategy that computes the available space once per write (no per-chunk walk).", + "@type": "Override", + "overrideInstance": { + "@id": "urn:solid-server:default:QuotaStrategy" + }, + "overrideParameters": { + "@type": "FastQuotaStrategy", + "limit_amount": 70000000, + "limit_unit": "bytes", + "reporter": { + "@id": "urn:solid-server:default:SizeReporter" + }, + "identifierStrategy": { + "@id": "urn:solid-server:default:IdentifierStrategy" + }, + "accessor": { + "@id": "urn:solid-server:default:AtomicFileDataAccessor" + }, + "baseUrl": { "@id": "urn:solid-server:default:variable:baseUrl" }, + "internalFolder": "/.internal/" + } + } + ] +} diff --git a/config/storage/backend/quota-fast-file.json b/config/storage/backend/quota-fast-file.json new file mode 100644 index 0000000..f020ef5 --- /dev/null +++ b/config/storage/backend/quota-fast-file.json @@ -0,0 +1,52 @@ +{ + "comment": "Fast per-pod quota: DuSizeReporter (du + TTL cache) and FastQuotaStrategy (no per-chunk pod walk).", + "@context": [ + "https://linkedsoftwaredependencies.org/bundles/npm/@solid/community-server/^7.0.0/components/context.jsonld", + "https://linkedsoftwaredependencies.org/bundles/npm/@solid/pivot/^1.0.0/components/context.jsonld" + ], + "@graph": [ + { + "comment": "SizeReporter backed by du with a per-path TTL cache, measuring apparent bytes (portable, user-manageable). Falls back to the Node walk when du is unavailable.", + "@type": "Override", + "overrideInstance": { + "@id": "urn:solid-server:default:SizeReporter" + }, + "overrideParameters": { + "@type": "DuSizeReporter", + "fileIdentifierMapper": { + "@id": "urn:solid-server:default:FileIdentifierMapper" + }, + "rootFilePath": { + "@id": "urn:solid-server:default:variable:rootFilePath" + }, + "ignoreFolders": [ + "^/\\.internal$" + ], + "ttl": 5000 + } + }, + { + "comment": "Pod quota strategy that computes the available space once per write instead of per stream chunk.", + "@type": "Override", + "overrideInstance": { + "@id": "urn:solid-server:default:QuotaStrategy" + }, + "overrideParameters": { + "@type": "FastQuotaStrategy", + "limit_amount": 70000000, + "limit_unit": "bytes", + "reporter": { + "@id": "urn:solid-server:default:SizeReporter" + }, + "identifierStrategy": { + "@id": "urn:solid-server:default:IdentifierStrategy" + }, + "accessor": { + "@id": "urn:solid-server:default:AtomicFileDataAccessor" + }, + "baseUrl": { "@id": "urn:solid-server:default:variable:baseUrl" }, + "internalFolder": "/.internal/" + } + } + ] +} diff --git a/config/storage/profile-card-guard.json b/config/storage/profile-card-guard.json new file mode 100644 index 0000000..9a28e84 --- /dev/null +++ b/config/storage/profile-card-guard.json @@ -0,0 +1,48 @@ +{ + "@context": [ + "https://linkedsoftwaredependencies.org/bundles/npm/@solid/community-server/^7.0.0/components/context.jsonld", + "https://linkedsoftwaredependencies.org/bundles/npm/@solid/pivot/^1.0.0/components/context.jsonld" + ], + "comment": "Guards the profile card of WebIDs registered on this server, keeping the `solid:oidcIssuer` triple.", + "@graph": [ + { + "comment": "Guards registered WebID cards from being deleted or losing their `solid:oidcIssuer` triple.", + "@id": "urn:solid-server:default:ResourceStore_CardGuard", + "@type": "ProfileCardGuard", + "source": { "@id": "urn:solid-server:default:ResourceStore_Converting" }, + "webIdStore": { "@id": "urn:solid-server:default:WebIdStore" }, + "baseUrl": { "@id": "urn:solid-server:default:variable:baseUrl" }, + "converter": { "@id": "urn:solid-server:default:RepresentationConverter" }, + "relativeWebIdPaths": [ + "/profile/card#me" + ] + }, + { + "comment": "WebID store that also exposes whether a WebID is registered (hasWebId), as required by the card guard.", + "@type": "Override", + "overrideInstance": { "@id": "urn:solid-server:default:WebIdStore" }, + "overrideParameters": { + "@type": "GuardedWebIdStore", + "storage": { "@id": "urn:solid-server:default:AccountStorage" } + } + }, + { + "comment": "Insert the card guard between pivot's RDF patching store and the converting store, so PUT/POST as well as PATCH results are all validated.", + "@type": "Override", + "overrideInstance": { "@id": "urn:solid-server:default:ResourceStore_RdfPatching" }, + "overrideParameters": { + "@type": "RdfPatchingStore", + "source": { "@id": "urn:solid-server:default:ResourceStore_CardGuard" } + } + }, + { + "comment": "Route the internal key-value storage (accounts, WebID links, ...) through the converting store directly, bypassing the guard.", + "@type": "Override", + "overrideInstance": { "@id": "urn:solid-server:default:JsonResourceStorage" }, + "overrideParameters": { + "@type": "JsonResourceStorage", + "source": { "@id": "urn:solid-server:default:ResourceStore_Converting" } + } + } + ] +} diff --git a/config/storage/resource-locker/long-expiry.json b/config/storage/resource-locker/long-expiry.json new file mode 100644 index 0000000..18c29da --- /dev/null +++ b/config/storage/resource-locker/long-expiry.json @@ -0,0 +1,26 @@ +{ + "comment": "Safety margin for the expiring read/write lock. The default WrappedExpiringReadWriteLocker (file.json) expires after 6000ms; on production disks a large internal container listing (e.g. the IDP AuthorizationCode store) can exceed 6s, aborting the WrappedExpiringStorage cleanup and letting expired entries accumulate (2026-08-16 incident: ~3900 stale auth codes). This override keeps the same file-based locker but allows 30s per operation. Import AFTER css:config/util/resource-locker/file.json (e.g. from customise-me.json).", + "@context": [ + "https://linkedsoftwaredependencies.org/bundles/npm/@solid/community-server/^7.0.0/components/context.jsonld", + "https://linkedsoftwaredependencies.org/bundles/npm/@solid/pivot/^1.0.0/components/context.jsonld" + ], + "@graph": [ + { + "comment": "Same as the default file-based locker, but with a 30s expiring lock instead of 6s.", + "@type": "Override", + "overrideInstance": { + "@id": "urn:solid-server:default:ResourceLocker" + }, + "overrideParameters": { + "@type": "WrappedExpiringReadWriteLocker", + "locker": { + "@type": "PartialReadWriteLocker", + "locker": { + "@id": "urn:solid-server:default:FileSystemResourceLocker" + } + }, + "expiration": 30000 + } + } + ] +} diff --git a/config/suffix.json b/config/suffix.json index 81045bb..15196d5 100644 --- a/config/suffix.json +++ b/config/suffix.json @@ -29,6 +29,7 @@ "css:config/storage/key-value/resource-store.json", "css:config/storage/location/pod.json", "pivot:config/storage/middleware/default.json", + "pivot:config/storage/profile-card-guard.json", "css:config/util/auxiliary/acl.json", "css:config/util/identifiers/suffix.json", "css:config/util/logging/winston.json", @@ -36,7 +37,8 @@ "css:config/util/representation-conversion/default.json", "css:config/util/resource-locker/file.json", "css:config/util/variables/default.json", - "pivot:config/pivot-overrides.json" + "pivot:config/pivot-overrides.json", + "pivot:config/pivot-file-locker-overrides.json" ], "@graph": [ { @@ -70,6 +72,11 @@ "@id": "urn:solid-server:default:ExpiringTokenStorage", "@type": "WrappedExpiringStorage", "timeout": 1 + }, + { + "@id": "urn:solid-server:default:ExpiringAdapterStorage", + "@type": "WrappedExpiringStorage", + "timeout": 1 } ] } diff --git a/jest.config.js b/jest.config.js index 6b29e80..5527dfc 100644 --- a/jest.config.js +++ b/jest.config.js @@ -2,6 +2,10 @@ module.exports = { transform: { '^.+\\.ts$': [ 'ts-jest', { tsconfig: 'tsconfig.json', + // Transpile-only: don't hard-fail on missing ambient test types (e.g. + // @types/jest not installed on servers) and silence the TS151002 hybrid + // module-kind warning. + isolatedModules: true, }], }, // Only run tests in the unit and integration folders. diff --git a/package-lock.json b/package-lock.json index 6e1a48c..f34ac29 100644 --- a/package-lock.json +++ b/package-lock.json @@ -7,10 +7,13 @@ "": { "name": "@solid/pivot", "version": "1.9.1", + "hasInstallScript": true, "license": "MIT", "dependencies": { "@inrupt/solid-client-authn-core": "^3.1.1", + "@rdfjs/types": "^1.1.2", "@solid/community-server": "^7.3.0", + "arrayify-stream": "^2.0.1", "mashlib": "^3.0.0", "patch-package": "^8.0.1", "rdflib": "^2.4.1" @@ -20,8 +23,10 @@ "@types/jest": "^30.0.0", "@types/node-fetch": "^2.6.13", "componentsjs-generator": "^3.1.2", + "cross-fetch": "^4.1.0", "jest": "^30.5.2", "jest-rdf": "^2.0.0", + "n3": "^1.26.0", "node-fetch": "^3.3.2", "ts-jest": "^29.4.15", "typescript": "^5.9.3" @@ -80,16 +85,6 @@ } } }, - "node_modules/@awesome.me/webawesome/node_modules/@types/node": { - "version": "24.19.2", - "resolved": "https://registry.npmjs.org/@types/node/-/node-24.19.2.tgz", - "integrity": "sha512-93d49wCWJkCyN87MxLpNN6KXPCwFw7bVUg79mGUV1OS40JGWwVCr8n61DXCSqz7CV6/VfPkb7xec7uP2M+QHWQ==", - "extraneous": true, - "license": "MIT", - "dependencies": { - "undici-types": ">=7.24.0 <7.24.7" - } - }, "node_modules/@awesome.me/webawesome/node_modules/marked": { "version": "11.2.0", "resolved": "https://registry.npmjs.org/marked/-/marked-11.2.0.tgz", @@ -102,13 +97,6 @@ "node": ">= 18" } }, - "node_modules/@awesome.me/webawesome/node_modules/undici-types": { - "version": "7.24.6", - "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-7.24.6.tgz", - "integrity": "sha512-WRNW+sJgj5OBN4/0JpHFqtqzhpbnV0GuB+OozA9gCL7a993SmU+1JBZCzLNxYsbMfIeDL+lTsphD5jN5N+n0zg==", - "extraneous": true, - "license": "MIT" - }, "node_modules/@babel/code-frame": { "version": "7.29.7", "resolved": "https://registry.npmjs.org/@babel/code-frame/-/code-frame-7.29.7.tgz", @@ -7677,16 +7665,6 @@ "vitest-axe": "^0.1.0" } }, - "node_modules/chat-pane/node_modules/@types/node": { - "version": "26.6.6", - "resolved": "https://registry.npmjs.org/@types/node/-/node-26.6.6.tgz", - "integrity": "sha512-MzQj+8Kx3XI40OmUAE6Qoh58ZMNstNNMwn2nfd3qWsfe7nDNSqJNzes4njhz6AHwrSHOffSLwXU6yb474Q4aog==", - "extraneous": true, - "license": "MIT", - "dependencies": { - "undici-types": "~8.9.0" - } - }, "node_modules/chat-pane/node_modules/@vitest/mocker": { "version": "4.1.12", "resolved": "https://registry.npmjs.org/@vitest/mocker/-/mocker-4.1.12.tgz", @@ -7725,13 +7703,6 @@ "url": "https://github.com/chalk/chalk?sponsor=1" } }, - "node_modules/chat-pane/node_modules/undici-types": { - "version": "8.9.0", - "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-8.9.0.tgz", - "integrity": "sha512-KTDyRTYX8sWmKXAikPHHSyc63CRPETMctyjKFupcC6OBLXT3xsN0e9aF7m+mIXutFWpUXuedtowG7iLOzp0kQg==", - "extraneous": true, - "license": "MIT" - }, "node_modules/chat-pane/node_modules/vite": { "version": "8.3.5", "resolved": "https://registry.npmjs.org/vite/-/vite-8.3.5.tgz", @@ -9387,16 +9358,6 @@ "vitest-axe": "^0.1.0" } }, - "node_modules/folder-pane/node_modules/@types/node": { - "version": "26.6.6", - "resolved": "https://registry.npmjs.org/@types/node/-/node-26.6.6.tgz", - "integrity": "sha512-MzQj+8Kx3XI40OmUAE6Qoh58ZMNstNNMwn2nfd3qWsfe7nDNSqJNzes4njhz6AHwrSHOffSLwXU6yb474Q4aog==", - "extraneous": true, - "license": "MIT", - "dependencies": { - "undici-types": "~8.9.0" - } - }, "node_modules/folder-pane/node_modules/@vitest/mocker": { "version": "4.1.12", "resolved": "https://registry.npmjs.org/@vitest/mocker/-/mocker-4.1.12.tgz", @@ -9435,13 +9396,6 @@ "url": "https://github.com/chalk/chalk?sponsor=1" } }, - "node_modules/folder-pane/node_modules/undici-types": { - "version": "8.9.0", - "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-8.9.0.tgz", - "integrity": "sha512-KTDyRTYX8sWmKXAikPHHSyc63CRPETMctyjKFupcC6OBLXT3xsN0e9aF7m+mIXutFWpUXuedtowG7iLOzp0kQg==", - "extraneous": true, - "license": "MIT" - }, "node_modules/folder-pane/node_modules/vite": { "version": "8.3.5", "resolved": "https://registry.npmjs.org/vite/-/vite-8.3.5.tgz", @@ -13372,16 +13326,6 @@ "vitest-axe": "^0.1.0" } }, - "node_modules/profile-pane/node_modules/@types/node": { - "version": "26.6.6", - "resolved": "https://registry.npmjs.org/@types/node/-/node-26.6.6.tgz", - "integrity": "sha512-MzQj+8Kx3XI40OmUAE6Qoh58ZMNstNNMwn2nfd3qWsfe7nDNSqJNzes4njhz6AHwrSHOffSLwXU6yb474Q4aog==", - "extraneous": true, - "license": "MIT", - "dependencies": { - "undici-types": "~8.9.0" - } - }, "node_modules/profile-pane/node_modules/@vitest/mocker": { "version": "4.1.12", "resolved": "https://registry.npmjs.org/@vitest/mocker/-/mocker-4.1.12.tgz", @@ -13420,13 +13364,6 @@ "url": "https://github.com/chalk/chalk?sponsor=1" } }, - "node_modules/profile-pane/node_modules/undici-types": { - "version": "8.9.0", - "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-8.9.0.tgz", - "integrity": "sha512-KTDyRTYX8sWmKXAikPHHSyc63CRPETMctyjKFupcC6OBLXT3xsN0e9aF7m+mIXutFWpUXuedtowG7iLOzp0kQg==", - "extraneous": true, - "license": "MIT" - }, "node_modules/profile-pane/node_modules/vite": { "version": "8.3.5", "resolved": "https://registry.npmjs.org/vite/-/vite-8.3.5.tgz", @@ -14815,16 +14752,6 @@ "node": ">=18" } }, - "node_modules/solid-logic/node_modules/@types/node": { - "version": "26.6.6", - "resolved": "https://registry.npmjs.org/@types/node/-/node-26.6.6.tgz", - "integrity": "sha512-MzQj+8Kx3XI40OmUAE6Qoh58ZMNstNNMwn2nfd3qWsfe7nDNSqJNzes4njhz6AHwrSHOffSLwXU6yb474Q4aog==", - "extraneous": true, - "license": "MIT", - "dependencies": { - "undici-types": "~8.9.0" - } - }, "node_modules/solid-logic/node_modules/@vitest/mocker": { "version": "4.1.12", "resolved": "https://registry.npmjs.org/@vitest/mocker/-/mocker-4.1.12.tgz", @@ -14851,13 +14778,6 @@ } } }, - "node_modules/solid-logic/node_modules/undici-types": { - "version": "8.9.0", - "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-8.9.0.tgz", - "integrity": "sha512-KTDyRTYX8sWmKXAikPHHSyc63CRPETMctyjKFupcC6OBLXT3xsN0e9aF7m+mIXutFWpUXuedtowG7iLOzp0kQg==", - "extraneous": true, - "license": "MIT" - }, "node_modules/solid-logic/node_modules/vite": { "version": "8.3.5", "resolved": "https://registry.npmjs.org/vite/-/vite-8.3.5.tgz", @@ -15203,16 +15123,6 @@ "vitest-axe": "^0.1.0" } }, - "node_modules/source-pane/node_modules/@types/node": { - "version": "26.6.6", - "resolved": "https://registry.npmjs.org/@types/node/-/node-26.6.6.tgz", - "integrity": "sha512-MzQj+8Kx3XI40OmUAE6Qoh58ZMNstNNMwn2nfd3qWsfe7nDNSqJNzes4njhz6AHwrSHOffSLwXU6yb474Q4aog==", - "extraneous": true, - "license": "MIT", - "dependencies": { - "undici-types": "~8.9.0" - } - }, "node_modules/source-pane/node_modules/@vitest/mocker": { "version": "4.1.12", "resolved": "https://registry.npmjs.org/@vitest/mocker/-/mocker-4.1.12.tgz", @@ -15251,13 +15161,6 @@ "url": "https://github.com/chalk/chalk?sponsor=1" } }, - "node_modules/source-pane/node_modules/undici-types": { - "version": "8.9.0", - "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-8.9.0.tgz", - "integrity": "sha512-KTDyRTYX8sWmKXAikPHHSyc63CRPETMctyjKFupcC6OBLXT3xsN0e9aF7m+mIXutFWpUXuedtowG7iLOzp0kQg==", - "extraneous": true, - "license": "MIT" - }, "node_modules/source-pane/node_modules/vite": { "version": "8.3.5", "resolved": "https://registry.npmjs.org/vite/-/vite-8.3.5.tgz", diff --git a/package.json b/package.json index 55f0cf8..b0b370e 100644 --- a/package.json +++ b/package.json @@ -27,6 +27,7 @@ "build:components": "componentsjs-generator -s src -c dist/components -i .componentsignore -r pivot", "build:ts": "tsc", "prepare": "npm run build", + "postinstall": "patch-package", "test": "jest" }, "files": [ @@ -37,6 +38,8 @@ ], "dependencies": { "@inrupt/solid-client-authn-core": "^3.1.1", + "@rdfjs/types": "^1.1.2", + "arrayify-stream": "^2.0.1", "@solid/community-server": "^7.3.0", "mashlib": "^3.0.0", "patch-package": "^8.0.1", @@ -47,8 +50,10 @@ "@types/jest": "^30.0.0", "@types/node-fetch": "^2.6.13", "componentsjs-generator": "^3.1.2", + "cross-fetch": "^4.1.0", "jest": "^30.5.2", "jest-rdf": "^2.0.0", + "n3": "^1.26.0", "node-fetch": "^3.3.2", "ts-jest": "^29.4.15", "typescript": "^5.9.3" diff --git a/patches/@solid+community-server+7.2.0.patch b/patches/@solid+community-server+7.2.0.patch new file mode 100644 index 0000000..0307c59 --- /dev/null +++ b/patches/@solid+community-server+7.2.0.patch @@ -0,0 +1,12 @@ +diff --git a/node_modules/@solid/community-server/config/identity/handler/base/adapter-factory.json b/node_modules/@solid/community-server/config/identity/handler/base/adapter-factory.json +index c2b9890..c406ba6 +--- a/node_modules/@solid/community-server/config/identity/handler/base/adapter-factory.json ++++ b/node_modules/@solid/community-server/config/identity/handler/base/adapter-factory.json +@@ -13,6 +13,7 @@ + "source": { + "@type": "ExpiringAdapterFactory", + "storage": { ++ "@id": "urn:solid-server:default:ExpiringAdapterStorage", + "@type": "WrappedExpiringStorage", + "source": { + "@type": "ContainerPathStorage", diff --git a/patches/proper-lockfile+4.1.2.patch b/patches/proper-lockfile+4.1.2.patch new file mode 100644 index 0000000..e2a8b2d --- /dev/null +++ b/patches/proper-lockfile+4.1.2.patch @@ -0,0 +1,18 @@ +diff --git a/node_modules/proper-lockfile/lib/lockfile.js b/node_modules/proper-lockfile/lib/lockfile.js +index 97b6637..25bd0a2 100644 +--- a/node_modules/proper-lockfile/lib/lockfile.js ++++ b/node_modules/proper-lockfile/lib/lockfile.js +@@ -99,6 +99,13 @@ function removeLock(file, options, callback) { + function updateLock(file, options) { + const lock = locks[file]; + ++ // Guard against a renewal timer racing with lock release: ++ // `locks[file]` may already be deleted when an in-flight fs callback recurses. ++ /* istanbul ignore if */ ++ if (!lock) { ++ return; ++ } ++ + // Just for safety, should never happen + /* istanbul ignore if */ + if (lock.updateTimeout) { diff --git a/scripts/benchmark-quota-c.js b/scripts/benchmark-quota-c.js new file mode 100644 index 0000000..38fa088 --- /dev/null +++ b/scripts/benchmark-quota-c.js @@ -0,0 +1,148 @@ +/** + * Rigorous benchmark: original CSS quota chain vs A+B vs design C. + * + * Everything is compared under identical conditions, with COLD (first write, + * no cache / bootstrap) and WARM (steady state) numbers separated: + * + * 1. WALK — a single getSize(podRoot): old Node walk / du walk / cached / + * counter (O(1)). + * 2. WRITE — quota guard over a 4 MB body in 64 KB chunks: + * old : per-chunk full walks (no cache — always cold) + * A+B : du walk once per write; COLD = first write, WARM = cached + * C : counter; COLD = bootstrap recount, WARM = O(1) + * + * Run: node scripts/benchmark-quota-c.js [fileCount] [fileBytes] + * (Put Git Bash du on PATH to use the real du path: add + * "C:\Program Files\Git\usr\bin" to PATH on Windows.) + */ +const { performance } = require('node:perf_hooks'); +const fsSync = require('node:fs'); +const { promises: fs } = require('node:fs'); +const os = require('node:os'); +const path = require('node:path'); +const { QuotaStrategy, FileSizeReporter } = require('@solid/community-server'); +const { DuSizeReporter } = require('../dist/storage/size-reporter/DuSizeReporter.js'); +const { FastQuotaStrategy } = require('../dist/storage/quota/FastQuotaStrategy.js'); +const { QuotaCounter } = require('../dist/storage/quota/QuotaCounter.js'); +const { IncrementalSizeReporter } = require('../dist/storage/quota/IncrementalSizeReporter.js'); + +const FILE_COUNT = Number(process.argv[2]) || 5000; +const FILE_BYTES = Number(process.argv[3]) || 1024; +const WRITE_BYTES = 4 * 1024 * 1024; +const CHUNK_BYTES = 64 * 1024; +const IGNORE = [ '^/\\.internal$' ]; + +function makeMapper(root) { + return { + async mapUrlToFilePath(identifier) { + const url = new URL(identifier.path); + return { identifier, filePath: path.join(root, url.pathname), contentType: undefined, isMetadata: false }; + }, + async mapFilePathToUrl() { throw new Error('n/a'); }, + }; +} + +class OldStrategy extends QuotaStrategy { + constructor(reporter, limit, pod) { super(reporter, limit); this.pod = pod; } + async getTotalSpaceUsed() { return this.reporter.getSize(this.pod); } +} +class FastStrategy extends FastQuotaStrategy { + constructor(reporter, limit, pod) { super(limit, reporter, {}, {}); this.pod = pod; } + async getTotalSpaceUsed() { return this.reporter.getSize(this.pod); } +} + +function seedPod(podRootPath, count, bytes) { + const inbox = path.join(podRootPath, 'inbox'); + fsSync.mkdirSync(inbox, { recursive: true }); + const buf = Buffer.alloc(bytes, 7); + for (let i = 0; i < count; i++) { + fsSync.writeFileSync(path.join(inbox, `f-${i}.bin`), buf); + } +} + +function writeThroughGuard(guard, totalBytes, chunkBytes) { + return new Promise((resolve, reject) => { + guard.on('data', () => {}); + guard.on('end', resolve); + guard.on('error', reject); + let remaining = totalBytes; + while (remaining > 0) { + const size = Math.min(chunkBytes, remaining); + guard.write(Buffer.alloc(size, 1)); + remaining -= size; + } + guard.end(); + }); +} + +async function timed(fn) { + const start = performance.now(); + await fn(); + return performance.now() - start; +} + +function pad(s, w) { return String(s).padStart(w); } + +async function main() { + const limit = { unit: 'bytes', amount: 10 * 1024 * 1024 * 1024 }; + const pod = { path: 'http://example.com/alice/' }; + const resource = { path: 'http://example.com/alice/new-file' }; + + const root = fsSync.mkdtempSync(path.join(os.tmpdir(), 'quota-c-')); + console.log(`Pod: ${FILE_COUNT} files × ${FILE_BYTES} B (write body ${WRITE_BYTES / 1024 / 1024} MB in ${CHUNK_BYTES / 1024} KB chunks)`); + seedPod(path.join(root, 'alice'), FILE_COUNT, FILE_BYTES); + const mapper = makeMapper(root); + + // ---- 1. WALK / read path ---- + console.log('\n1. WALK — getSize(podRoot):'); + const oldReporter = new FileSizeReporter(mapper, root); + const duReporter = new DuSizeReporter(mapper, root, IGNORE); + const counter = new QuotaCounter(mapper, root, IGNORE); + + const tOldWalk = await timed(() => oldReporter.getSize(pod)); + const tDuCold = await timed(() => duReporter.getSize(pod)); + const tDuWarm = await timed(() => duReporter.getSize(pod)); + await counter.register(pod); + await counter.add(pod, (await new DuSizeReporter(mapper, root, IGNORE).getSize(pod)).amount); + const incReporter = new IncrementalSizeReporter(counter); + const tCounter = await timed(() => incReporter.getSize(pod)); + + console.log(` old (Node walk) : ${pad(tOldWalk.toFixed(1), 8)} ms`); + console.log(` A+B du (cold walk) : ${pad(tDuCold.toFixed(1), 8)} ms`); + console.log(` A+B du (cached) : ${pad(tDuWarm.toFixed(3), 8)} ms`); + console.log(` C counter (O(1)) : ${pad(tCounter.toFixed(3), 8)} ms`); + + // ---- 2. WRITE / guard ---- + console.log('\n2. WRITE — quota guard:'); + const chunks = Math.ceil(WRITE_BYTES / CHUNK_BYTES); + + // old — always cold (no cache), per-chunk walks. + const oldStrategy = new OldStrategy(oldReporter, limit, pod); + const tOld = await timed(async () => writeThroughGuard(await oldStrategy.createQuotaGuard(resource), WRITE_BYTES, CHUNK_BYTES)); + + // A+B — cold (fresh reporter) then warm (pre-warmed cache). + const duFastCold = new FastStrategy(new DuSizeReporter(mapper, root, IGNORE), limit, pod); + const tDuColdWrite = await timed(async () => writeThroughGuard(await duFastCold.createQuotaGuard(resource), WRITE_BYTES, CHUNK_BYTES)); + // Warm: ensure the cache is warm, then measure. + await duReporter.getSize(pod); + const duFastWarm = new FastStrategy(duReporter, limit, pod); + const tDuWarmWrite = await timed(async () => writeThroughGuard(await duFastWarm.createQuotaGuard(resource), WRITE_BYTES, CHUNK_BYTES)); + + // C — cold (fresh counter, bootstrap recount) then warm (counter ready). + const coldCounter = new QuotaCounter(mapper, root, IGNORE); + const cCold = new FastStrategy(new IncrementalSizeReporter(coldCounter), limit, pod); + const tCCold = await timed(async () => writeThroughGuard(await cCold.createQuotaGuard(resource), WRITE_BYTES, CHUNK_BYTES)); + const cWarm = new FastStrategy(incReporter, limit, pod); + const tCWarm = await timed(async () => writeThroughGuard(await cWarm.createQuotaGuard(resource), WRITE_BYTES, CHUNK_BYTES)); + + console.log(` old (per-chunk walks) : ${pad(tOld.toFixed(1), 8)} ms (${chunks} walks)`); + console.log(` A+B cold (1 du walk) : ${pad(tDuColdWrite.toFixed(1), 8)} ms`); + console.log(` A+B warm (cached) : ${pad(tDuWarmWrite.toFixed(3), 8)} ms`); + console.log(` C cold (bootstrap) : ${pad(tCCold.toFixed(1), 8)} ms`); + console.log(` C warm (O(1)) : ${pad(tCWarm.toFixed(3), 8)} ms`); + + fsSync.rmSync(root, { recursive: true, force: true }); + console.log('\nNote: "old" has no cache (always cold). On Linux/WSL, du walks are 10-100x faster than on Windows+Git Bash.'); +} + +main().catch((e) => { console.error(e); process.exit(1); }); diff --git a/scripts/benchmark-quota.js b/scripts/benchmark-quota.js new file mode 100644 index 0000000..aebd87d --- /dev/null +++ b/scripts/benchmark-quota.js @@ -0,0 +1,145 @@ +/** + * Benchmark: CSS pod-quota chain — old (FileSizeReporter + QuotaStrategy) + * vs pivot (DuSizeReporter + FastQuotaStrategy). + * + * Measures exactly what was optimized: + * 1. Full pod walk cost (old recursive Node walk vs du walk) + * 2. The per-write quota guard (old: full walk PER CHUNK; new: walk once) + * 3. The TTL cache effect on repeated size queries + * + * Run: node scripts/benchmark-quota.js [fileCount] [fileBytes] + * e.g. node scripts/benchmark-quota.js 10000 1024 + * + * NOTE: on bare Windows there is no `du`, so the walk times will be similar + * between the two (the guard + cache wins still show). On Linux/WSL the du + * walk is 10-100x faster. + */ +const { performance } = require('node:perf_hooks'); +const fs = require('node:fs'); +const os = require('node:os'); +const path = require('node:path'); +const { PassThrough } = require('node:stream'); +const { + FileSizeReporter, + QuotaStrategy, +} = require('@solid/community-server'); +const { DuSizeReporter } = require('../dist/storage/size-reporter/DuSizeReporter.js'); +const { FastQuotaStrategy } = require('../dist/storage/quota/FastQuotaStrategy.js'); + +const FILE_COUNT = Number(process.argv[2]) || 5000; +const FILE_BYTES = Number(process.argv[3]) || 1024; +const WRITE_BYTES = 4 * 1024 * 1024; // simulated write body +const CHUNK_BYTES = 64 * 1024; + +function makeMapper(root) { + return { + async mapUrlToFilePath(identifier) { + const url = new URL(identifier.path); + return { identifier, filePath: path.join(root, url.pathname), contentType: undefined, isMetadata: false }; + }, + async mapFilePathToUrl() { throw new Error('n/a'); }, + }; +} + +// Old strategy: base QuotaStrategy whose pod size = reporter.getSize(podRoot) +class OldBenchStrategy extends QuotaStrategy { + constructor(reporter, limit, podRoot) { + super(reporter, limit); + this.podRoot = podRoot; + } + async getTotalSpaceUsed() { + return this.reporter.getSize(this.podRoot); + } +} + +// New strategy: FastQuotaStrategy with the same pod-size hook +class NewBenchStrategy extends FastQuotaStrategy { + constructor(reporter, limit, podRoot) { + super(limit, reporter, {}, {}); + this.podRoot = podRoot; + } + async getTotalSpaceUsed() { + return this.reporter.getSize(this.podRoot); + } +} + +function seedPod(root, count, bytes) { + const inbox = path.join(root, 'inbox'); + fs.mkdirSync(inbox, { recursive: true }); + const buf = Buffer.alloc(bytes, 7); + for (let i = 0; i < count; i++) { + fs.writeFileSync(path.join(inbox, `file-${i}.bin`), buf); + } +} + +function writeThroughGuard(guard, totalBytes, chunkBytes) { + return new Promise((resolve, reject) => { + const chunks = []; + let remaining = totalBytes; + guard.on('data', () => {}); + guard.on('end', () => resolve(chunks)); + guard.on('error', reject); + while (remaining > 0) { + const size = Math.min(chunkBytes, remaining); + guard.write(Buffer.alloc(size, 1)); + remaining -= size; + } + guard.end(); + }); +} + +async function timed(fn) { + const start = performance.now(); + await fn(); + return performance.now() - start; +} + +async function main() { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'quota-bench-')); + const podRoot = { path: 'http://example.com/' }; + const limit = { unit: 'bytes', amount: 10 * 1024 * 1024 * 1024 }; + + console.log(`Seeding ${FILE_COUNT} files × ${FILE_BYTES} B in ${root} …`); + const seedStart = performance.now(); + seedPod(root, FILE_COUNT, FILE_BYTES); + console.log(` seeded in ${(performance.now() - seedStart).toFixed(0)} ms\n`); + + const mapper = makeMapper(root); + const reporterOld = new FileSizeReporter(mapper, root); + const reporterNew = new DuSizeReporter(mapper, root, [ '^/\\.internal$' ]); + + // --- 1. Full pod walk (single getSize) --- + const walkOld = await timed(() => reporterOld.getSize(podRoot)); + const walkNew = await timed(() => reporterNew.getSize(podRoot)); + const walkCached = await timed(() => reporterNew.getSize(podRoot)); + + console.log('1. FULL POD WALK (getSize of pod root)'); + console.log(` old (Node recursive walk): ${walkOld.toFixed(1)} ms`); + console.log(` new (du, first call): ${walkNew.toFixed(1)} ms`); + console.log(` new (du, cached, TTL hit): ${walkCached.toFixed(3)} ms`); + if (walkNew > 0) { + console.log(` walk speedup (first call): ${(walkOld / walkNew).toFixed(1)}×`); + } + + // --- 2. Per-write guard (write body through the quota guard) --- + const strategyOld = new OldBenchStrategy(reporterOld, limit, podRoot); + const strategyNew = new NewBenchStrategy(reporterNew, limit, podRoot); + + const guardOld = await strategyOld.createQuotaGuard({ path: 'http://example.com/inbox/file-new.bin' }); + const guardNew = await strategyNew.createQuotaGuard({ path: 'http://example.com/inbox/file-new.bin' }); + + const guardOldMs = await timed(() => writeThroughGuard(guardOld, WRITE_BYTES, CHUNK_BYTES)); + const guardNewMs = await timed(() => writeThroughGuard(guardNew, WRITE_BYTES, CHUNK_BYTES)); + + console.log(`\n2. PER-WRITE QUOTA GUARD (${(WRITE_BYTES / 1024 / 1024).toFixed(0)} MB body, ${CHUNK_BYTES / 1024} KB chunks)`); + console.log(` old (walk per chunk, ${Math.ceil(WRITE_BYTES / CHUNK_BYTES)} chunks): ${guardOldMs.toFixed(1)} ms`); + console.log(` new (walk once + cache): ${guardNewMs.toFixed(1)} ms`); + if (guardNewMs > 0) { + console.log(` guard speedup: ${(guardOldMs / guardNewMs).toFixed(1)}×`); + } + + fs.rmSync(root, { recursive: true, force: true }); + console.log('\nDone.'); +} + +main().catch((err) => { console.error(err); process.exit(1); }); diff --git a/scripts/smoke-design-c.js b/scripts/smoke-design-c.js new file mode 100644 index 0000000..2d773be --- /dev/null +++ b/scripts/smoke-design-c.js @@ -0,0 +1,146 @@ +// Smoke test for design C against the compiled dist — replicates the jest +// scenarios that failed (mtime/sidecar handling + pod discovery). +// Run: node scripts/smoke-design-c.js +const { promises: fs } = require('node:fs'); +const os = require('node:os'); +const path = require('node:path'); +const { SingleRootIdentifierStrategy } = require('@solid/community-server'); +const { QuotaCounter } = require('../dist/storage/quota/QuotaCounter.js'); +const { IncrementalSizeReporter } = require('../dist/storage/quota/IncrementalSizeReporter.js'); +const { QuotaDeltaDataAccessor } = require('../dist/storage/quota/QuotaDeltaDataAccessor.js'); +const { DuSizeReporter } = require('../dist/storage/size-reporter/DuSizeReporter.js'); + +const IGNORE = [ '^/\\.internal$' ]; +const PIM_STORAGE = 'http://www.w3.org/ns/pim/space#Storage'; + +function createMapper(root) { + return { + async mapUrlToFilePath(identifier, isMetadata) { + const url = new URL(identifier.path); + const base = path.join(root, url.pathname); + return { identifier, filePath: isMetadata ? `${base}.meta` : base, contentType: undefined, isMetadata }; + }, + async mapFilePathToUrl() { throw new Error('n/a'); }, + }; +} + +async function walkExpected(root, mapper, pod) { + return (await new DuSizeReporter(mapper, root, IGNORE).getSize(pod)).amount; +} + +function createAccessor(root) { + const mapper = createMapper(root); + const meta = (isStorage) => ({ getAll: () => (isStorage ? [ { value: PIM_STORAGE } ] : []) }); + return { + async canHandle() {}, + async getData(id) { const { filePath } = await mapper.mapUrlToFilePath(id, false); return fs.createReadStream(filePath); }, + async getMetadata(id) { return meta(id.path.endsWith('/') && id.path !== 'http://example.com/'); }, + getChildren() { return (async function*() {})(); }, + async writeDocument(id, data) { + const { filePath } = await mapper.mapUrlToFilePath(id, false); + const chunks = []; + for await (const c of data) chunks.push(Buffer.from(c)); + await fs.mkdir(path.dirname(filePath), { recursive: true }); + await fs.writeFile(filePath, Buffer.concat(chunks)); + }, + async writeContainer(id) { + const { filePath } = await mapper.mapUrlToFilePath(id, false); + await fs.mkdir(filePath, { recursive: true }); + }, + async writeMetadata(id) { + const { filePath } = await mapper.mapUrlToFilePath(id, true); + await fs.mkdir(path.dirname(filePath), { recursive: true }); + await fs.writeFile(filePath, '{}'); + }, + async deleteResource(id) { + const data = await mapper.mapUrlToFilePath(id, false); + const meta = await mapper.mapUrlToFilePath(id, true); + await fs.rm(data.filePath, { recursive: true, force: true }); + await fs.rm(meta.filePath, { force: true }); + }, + }; +} + +let failures = 0; +function check(label, actual, expected) { + const ok = actual === expected; + console.log(` ${ok ? 'OK ' : 'FAIL'} ${label}: expected ${expected}, got ${actual}`); + if (!ok) failures++; +} + +async function main() { + const root = await fs.mkdtemp(path.join(os.tmpdir(), 'smoke-c-')); + const mapper = createMapper(root); + await fs.mkdir(path.join(root, 'alice')); + + // --- QuotaCounter --- + console.log('QuotaCounter:'); + const counter = new QuotaCounter(mapper, root, IGNORE); + const POD = { path: 'http://example.com/alice/' }; + await counter.register(POD); + await counter.add(POD, 100); + await counter.add(POD, 50); + check('accumulate 100+50', (await counter.getSize(POD)).amount, 150); + + // persistence across instances + const first = new QuotaCounter(mapper, root, IGNORE); + await first.register(POD); + await first.add(POD, 200); + const second = new QuotaCounter(mapper, root, IGNORE); + check('reload from sidecar (no re-walk)', (await second.getSize(POD)).amount, 200); + + // staleness + await fs.writeFile(path.join(root, 'alice', 'extra.bin'), Buffer.alloc(400)); + const stale = await second.getSize(POD); + check('staleness recount', stale.amount, await walkExpected(root, mapper, POD)); + + // remove → re-bootstrap + await second.remove(POD); + check('isPodRoot false after remove', await second.isPodRoot(POD), false); + check('re-bootstrap after remove', (await second.getSize(POD)).amount, await walkExpected(root, mapper, POD)); + + // --- IncrementalSizeReporter --- + console.log('IncrementalSizeReporter:'); + const counter2 = new QuotaCounter(mapper, root, IGNORE); + await counter2.register(POD); + await counter2.add(POD, 123); + const reporter = new IncrementalSizeReporter(counter2); + check('pod root → counter total', (await reporter.getSize(POD)).amount, 123); + await fs.writeFile(path.join(root, 'alice', 'foo'), Buffer.alloc(64)); + check('resource → stat', (await reporter.getSize({ path: 'http://example.com/alice/foo' })).amount, 64); + + // --- QuotaDeltaDataAccessor --- + console.log('QuotaDeltaDataAccessor (pod discovery + delta tracking):'); + const dRoot = await fs.mkdtemp(path.join(os.tmpdir(), 'smoke-delta-')); + const dMapper = createMapper(dRoot); + const dCounter = new QuotaCounter(dMapper, dRoot, IGNORE); + const dAccessor = new QuotaDeltaDataAccessor( + createAccessor(dRoot), + new SingleRootIdentifierStrategy('http://example.com/'), + dCounter, + dMapper, + ); + const dPOD = { path: 'http://example.com/alice/' }; + const dRes = { path: 'http://example.com/alice/foo' }; + await dAccessor.writeContainer(dPOD, {}); + check('pod registered after createContainer', await dCounter.isPodRoot(dPOD), true); + await dAccessor.writeDocument(dRes, (async function*() { yield Buffer.alloc(100); })(), {}); + check('after create doc(100) == walk', (await dCounter.getSize(dPOD)).amount, await walkExpected(dRoot, dMapper, dPOD)); + await dAccessor.writeDocument(dRes, (async function*() { yield Buffer.alloc(150); })(), {}); + check('after overwrite(150) == walk', (await dCounter.getSize(dPOD)).amount, await walkExpected(dRoot, dMapper, dPOD)); + await dAccessor.writeMetadata(dRes, {}); + check('after writeMetadata == walk', (await dCounter.getSize(dPOD)).amount, await walkExpected(dRoot, dMapper, dPOD)); + await dAccessor.deleteResource(dRes); + check('after delete == walk', (await dCounter.getSize(dPOD)).amount, await walkExpected(dRoot, dMapper, dPOD)); + // pod root delete → counter dropped + await dAccessor.writeContainer(dPOD, {}); + await dAccessor.deleteResource(dPOD); + check('pod root delete drops counter', await dCounter.isPodRoot(dPOD), false); + + await fs.rm(root, { recursive: true, force: true }); + await fs.rm(dRoot, { recursive: true, force: true }); + console.log(failures === 0 ? '\nALL CHECKS PASSED' : `\n${failures} CHECK(S) FAILED`); + process.exit(failures === 0 ? 0 : 1); +} + +main().catch((e) => { console.error(e); process.exit(1); }); diff --git a/scripts/verify-size-equivalence.js b/scripts/verify-size-equivalence.js new file mode 100644 index 0000000..04407e5 --- /dev/null +++ b/scripts/verify-size-equivalence.js @@ -0,0 +1,120 @@ +/** + * Equivalence proof: DuSizeReporter vs CSS FileSizeReporter. + * + * Generates random pod trees and asserts both reporters return the EXACT same + * apparent-byte total (same ignoreFolders), for the du path and the Node-walk + * fallback. Exits non-zero on any mismatch. + * + * Run with GNU du on PATH to exercise the real du path, e.g. on Windows: + * $env:PATH = "C:\Program Files\Git\usr\bin;$env:PATH" + * node scripts/verify-size-equivalence.js [iterations] [maxFiles] + */ +const fs = require('node:fs'); +const os = require('node:os'); +const path = require('node:path'); +const { FileSizeReporter } = require('@solid/community-server'); +const { DuSizeReporter } = require('../dist/storage/size-reporter/DuSizeReporter.js'); + +const ITERATIONS = Number(process.argv[2]) || 10; +const MAX_FILES = Number(process.argv[3]) || 60; +const IGNORE = [ '^/\\.internal$' ]; + +class ForcedDu extends DuSizeReporter { + async detectDu() { return 'gnu'; } +} +class ForcedNode extends DuSizeReporter { + async detectDu() { return 'none'; } +} + +function makeMapper(root) { + return { + async mapUrlToFilePath(identifier) { + const url = new URL(identifier.path); + return { identifier, filePath: path.join(root, url.pathname), contentType: undefined, isMetadata: false }; + }, + async mapFilePathToUrl() { throw new Error('n/a'); }, + }; +} + +// Random names for nested content. .internal is handled separately below: +// CSS only ever places .internal at the pod ROOT (temp files), where the +// anchored regex ^/\.internal$ and du's basename --exclude=.internal agree. +const SEGMENTS = [ 'a', 'b', 'c', 'd', 'e', 'inbox', 'public', 'private', 'settings' ]; + +function randomTree(root, maxFiles) { + const files = []; + const count = 1 + Math.floor(Math.random() * maxFiles); + for (let i = 0; i < count; i++) { + // 1-4 nested segments, each a subdirectory (mkdirp on write). + const depth = 1 + Math.floor(Math.random() * 3); + const segs = []; + for (let d = 0; d < depth; d++) { + segs.push(SEGMENTS[Math.floor(Math.random() * SEGMENTS.length)]); + } + const dir = path.join(root, ...segs); + const file = path.join(dir, `f${i}.bin`); + const size = Math.floor(Math.random() * 50_000); + files.push({ dir, file, size }); + } + return files; +} + +function writeTree(root, files) { + for (const { dir, file, size } of files) { + fs.mkdirSync(dir, { recursive: true }); + fs.writeFileSync(file, Buffer.alloc(size, 42)); + } + // Root-level .internal (excluded by both paths identically). + fs.mkdirSync(path.join(root, '.internal', 'tempFiles'), { recursive: true }); + fs.writeFileSync(path.join(root, '.internal', 'tempFiles', 'tmp.bin'), Buffer.alloc(777, 9)); + // Some empty dirs too. + fs.mkdirSync(path.join(root, 'empty-dir-1'), { recursive: true }); + fs.mkdirSync(path.join(root, 'a', 'empty-dir-2'), { recursive: true }); +} + +function assertEqual(label, a, b) { + const same = a.amount === b.amount; + console.log(` ${same ? 'OK ' : 'FAIL'} ${label}: FileSizeReporter=${a.amount} DuSizeReporter=${b.amount}${same ? '' : ' <-- MISMATCH'}`); + return same; +} + +let allOk = true; + +async function main() { + for (let iter = 1; iter <= ITERATIONS; iter++) { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'equiv-')); + const files = randomTree(root, MAX_FILES); + writeTree(root, files); + + const mapper = makeMapper(root); + const oldReporter = new FileSizeReporter(mapper, root, IGNORE); + const duReporter = new ForcedDu(mapper, root, IGNORE); + const nodeReporter = new ForcedNode(mapper, root, IGNORE); + const podId = { path: 'http://example.com/' }; + + const oldSize = await oldReporter.getSize(podId); + const duSize = await duReporter.getSize(podId); + const nodeSize = await nodeReporter.getSize(podId); + + console.log(`Iteration ${iter} (${files.length} files):`); + const okDu = assertEqual('du path ', oldSize, duSize); + const okNode = assertEqual('node fallback', oldSize, nodeSize); + if (!okDu || !okNode) allOk = false; + console.log(''); + + fs.rmSync(root, { recursive: true, force: true }); + } +} + +main().then(() => { + if (allOk) { + console.log(`EQUIVALENT: DuSizeReporter matches FileSizeReporter across ${ITERATIONS} random trees.`); + process.exit(0); + } else { + console.error('MISMATCH FOUND — sizes are NOT equivalent.'); + process.exit(1); + } +}).catch((err) => { + console.error(err); + process.exit(1); +}); diff --git a/src/identity/interaction/account/util/SafeBaseLoginAccountStorage.ts b/src/identity/interaction/account/util/SafeBaseLoginAccountStorage.ts new file mode 100644 index 0000000..05ba222 --- /dev/null +++ b/src/identity/interaction/account/util/SafeBaseLoginAccountStorage.ts @@ -0,0 +1,47 @@ +import { + ACCOUNT_TYPE, + BaseLoginAccountStorage, + createErrorMessage, + getLoggerFor, +} from '@solid/community-server'; + +const LOGIN_COUNT = 'linkedLoginsCount'; + +/** + * A {@link BaseLoginAccountStorage} that prevents the periodic cleanup of accounts + * without login methods from crashing the process when a lock timeout occurs. + * + * The storage is typed as `any` (and the class is not generic) because the + * componentsjs-generator cannot resolve the `IndexedStorage`/`IndexTypeCollection` + * generic types from the installed Community Solid Server `.d.ts` (TS2415-safe + * `logger` is renamed to `safeLogger` for the same reason). + */ +export class SafeBaseLoginAccountStorage extends BaseLoginAccountStorage { + // Renamed (not `logger`) because the base class already declares a private `logger`, + // which cannot be re-declared in a subclass (TS2415). + private readonly safeLogger = getLoggerFor(this); + + private readonly timeout: number; + + public constructor(storage: any, expiration = 30 * 60) { + super(storage, expiration); + this.timeout = expiration * 1000; + } + + protected createAccountTimeout(id: string): void { + // eslint-disable-next-line @typescript-eslint/no-misused-promises + const timer = setTimeout(async(): Promise => { + try { + const account = await this.storage.get(ACCOUNT_TYPE, id); + if (account && account[LOGIN_COUNT] === 0) { + this.safeLogger.debug(`Removing account with no login methods ${id}`); + await this.storage.delete(ACCOUNT_TYPE, id); + } + } catch (error: unknown) { + // Prevent an unhandled rejection (e.g. a lock timeout) from crashing the process. + this.safeLogger.error(`Error during account cleanup of ${id}: ${createErrorMessage(error)}`); + } + }, this.timeout); + timer.unref(); + } +} diff --git a/src/identity/interaction/webid/util/GuardedWebIdStore.ts b/src/identity/interaction/webid/util/GuardedWebIdStore.ts new file mode 100644 index 0000000..172bda4 --- /dev/null +++ b/src/identity/interaction/webid/util/GuardedWebIdStore.ts @@ -0,0 +1,36 @@ +import { + AccountLoginStorage, + BaseWebIdStore, + WEBID_STORAGE_DESCRIPTION, + WEBID_STORAGE_TYPE, +} from '@solid/community-server'; + +type WebIdStorage = AccountLoginStorage<{ [WEBID_STORAGE_TYPE]: typeof WEBID_STORAGE_DESCRIPTION }>; + +/** + * A {@link BaseWebIdStore} that also determines whether a WebID is registered + * to any account on this server (`hasWebId`). + * + * The card guard needs to know whether a WebID is registered, which the + * upstream WebIdStore does not expose (yet). This subclass keeps the full + * upstream behavior and only adds the exact indexed lookup on the webId index. + */ +export class GuardedWebIdStore extends BaseWebIdStore { + private readonly webIdStorage: WebIdStorage; + + // Loosely typed so the Components.js generator does not need to resolve the + // external generic; Components.js does not type-check constructor arguments. + public constructor(storage: any) { + super(storage); + this.webIdStorage = storage as unknown as WebIdStorage; + } + + /** + * Determines if the given WebID is registered to an account on this server. + * + * @param webId - WebID to check. + */ + public async hasWebId(webId: string): Promise { + return (await this.webIdStorage.find(WEBID_STORAGE_TYPE, { webId })).length > 0; + } +} diff --git a/src/index.ts b/src/index.ts index 2d9b1a8..247d4b3 100644 --- a/src/index.ts +++ b/src/index.ts @@ -1,6 +1,14 @@ export * from "./storage/RdfPatchingStore"; +export * from './storage/ProfileCardGuard'; export * from "./storage/patch/ThrowingN3Patcher"; +export * from "./storage/quota/FastQuotaStrategy"; +export * from "./storage/quota/IncrementalSizeReporter"; +export * from "./storage/quota/QuotaCounter"; +export * from "./storage/quota/QuotaDeltaDataAccessor"; +export * from "./storage/size-reporter/DuSizeReporter"; export * from './FedcmHttpHandler'; export * from './http/output/PivotResponseWriter'; +export * from './identity/interaction/account/util/SafeBaseLoginAccountStorage'; export * from './identity/interaction/password/MigratedPasswordLoginHandler'; +export * from './identity/interaction/webid/util/GuardedWebIdStore'; export * from './identity/PivotOidcHttpHandler'; diff --git a/src/storage/ProfileCardGuard.ts b/src/storage/ProfileCardGuard.ts new file mode 100644 index 0000000..f931998 --- /dev/null +++ b/src/storage/ProfileCardGuard.ts @@ -0,0 +1,171 @@ +import arrayifyStream from 'arrayify-stream'; +import type { Quad } from '@rdfjs/types'; +import { + BadRequestHttpError, + ForbiddenHttpError, + INTERNAL_QUADS, + PassthroughStore, + RepresentationConverter, + SOLID, + cloneRepresentation, + createErrorMessage, + trimTrailingSlashes, +} from '@solid/community-server'; +import type { + ChangeMap, + Conditions, + Representation, + ResourceIdentifier, + ResourceStore, +} from '@solid/community-server'; +import type { GuardedWebIdStore } from '../identity/interaction/webid/util/GuardedWebIdStore'; + +/** + * Guards the profile card of WebIDs registered on this server. + * + * The location of these cards is given by the `relativeWebIdPaths` parameter, + * which lists the relative paths (including fragment) that are used for the + * WebIDs generated by this server, e.g. `/profile/card#me`. Multiple paths can + * be provided to also cover WebIDs generated with a previous configuration. + * Whether a document is actually protected is always verified against the + * {@link GuardedWebIdStore}, so only WebIDs that are registered to an account + * are guarded, independent of the storage backend used. + */ +export class ProfileCardGuard extends PassthroughStore { + private readonly webIdStore: GuardedWebIdStore; + private readonly baseUrl: string; + private readonly converter: RepresentationConverter; + private readonly relativeWebIdPaths: string[]; + + /** + * @param source - Store wrapped by this guard. + * @param webIdStore - Store tracking which WebIDs are registered to accounts. + * @param baseUrl - Base URL of the server, used to verify the `solid:oidcIssuer` triple. + * @param converter - Converts incoming representations to quads for validation. + * @param relativeWebIdPaths - Relative paths (including fragment) of generated WebIDs, + * e.g. `/profile/card#me`. + */ + public constructor( + source: ResourceStore, + webIdStore: GuardedWebIdStore, + baseUrl: string, + converter: RepresentationConverter, + relativeWebIdPaths: string[], + ) { + super(source); + this.webIdStore = webIdStore; + this.baseUrl = trimTrailingSlashes(baseUrl); + this.converter = converter; + this.relativeWebIdPaths = relativeWebIdPaths; + } + + public async setRepresentation( + identifier: ResourceIdentifier, + representation: Representation, + conditions?: Conditions, + ): Promise { + await this.validateCard(identifier, representation); + return this.source.setRepresentation(identifier, representation, conditions); + } + + public async addResource( + container: ResourceIdentifier, + representation: Representation, + conditions?: Conditions, + ): Promise { + // The final URL of a new resource is determined by the backend, + // so only validate if it is already known. A newly created resource + // cannot host a registered WebID yet anyway. + const identifier = representation.metadata.identifier?.value; + if (identifier) { + await this.validateCard({ path: identifier }, representation); + } + return this.source.addResource(container, representation, conditions); + } + + public async deleteResource(identifier: ResourceIdentifier, conditions?: Conditions): Promise { + if ((await this.getRegisteredWebIds(identifier)).length > 0) { + throw new ForbiddenHttpError( + `The profile card ${identifier.path} cannot be deleted while its WebID is registered to an account.`, + ); + } + return this.source.deleteResource(identifier, conditions); + } + + /** + * Returns the WebIDs registered on this server whose card is the given document. + * A WebID is considered if its relative path matches one of the configured + * `relativeWebIdPaths` and if it is still registered in the {@link GuardedWebIdStore}. + */ + protected async getRegisteredWebIds(identifier: ResourceIdentifier): Promise { + const webIds = new Set(); + for (const relativeWebIdPath of this.relativeWebIdPaths) { + const hashIndex = relativeWebIdPath.lastIndexOf('#'); + const relativePath = hashIndex >= 0 ? relativeWebIdPath.slice(0, hashIndex) : relativeWebIdPath; + if (!identifier.path.endsWith(relativePath)) { + continue; + } + const webId = hashIndex >= 0 ? `${identifier.path}#${relativeWebIdPath.slice(hashIndex + 1)}` : identifier.path; + if (await this.webIdStore.hasWebId(webId)) { + webIds.add(webId); + } + } + return [ ...webIds ]; + } + + /** + * Converts the representation of a document to quads, without consuming the + * original representation, which remains usable by the wrapped store. + */ + protected async parseToQuads(identifier: ResourceIdentifier, representation: Representation): Promise { + const copy = await cloneRepresentation(representation); + if (copy.metadata.contentType === INTERNAL_QUADS) { + return arrayifyStream(copy.data); + } + const converted = await this.converter.handleSafe({ + identifier, + representation: copy, + preferences: { type: { [INTERNAL_QUADS]: 1 }}, + }); + if (converted.metadata.contentType !== INTERNAL_QUADS) { + representation.data.destroy(); + throw new BadRequestHttpError(`Invalid profile card ${identifier.path}: could not be parsed as RDF.`); + } + return arrayifyStream(converted.data); + } + + /** + * Ensures that a write to the card of a registered WebID results in valid RDF + * that still contains the `solid:oidcIssuer` triple for that WebID. + */ + protected async validateCard(identifier: ResourceIdentifier, representation: Representation): Promise { + const webIds = await this.getRegisteredWebIds(identifier); + if (webIds.length === 0) { + return; + } + + let quads: Quad[]; + try { + quads = await this.parseToQuads(identifier, representation); + } catch (cause: unknown) { + representation.data.destroy(); + const message = `Invalid profile card ${identifier.path}: not valid RDF. ${createErrorMessage(cause)}`; + throw new BadRequestHttpError(message, { cause }); + } + + const valid = webIds.every((webId): boolean => + quads.some(({ subject, predicate, object }): boolean => + subject.value === webId && + predicate.value === SOLID.terms.oidcIssuer.value && + object.termType === 'NamedNode' && + object.value === `${this.baseUrl}/`)); + + if (!valid) { + representation.data.destroy(); + const expected = [ ...webIds ] + .map((webId): string => `<${webId}> solid:oidcIssuer <${this.baseUrl}/>`) + .join(' and '); + throw new BadRequestHttpError(`Invalid profile card ${identifier.path}: missing the ${expected} triple.`); + } + } +} diff --git a/src/storage/quota/FastQuotaStrategy.ts b/src/storage/quota/FastQuotaStrategy.ts new file mode 100644 index 0000000..fcecd8d --- /dev/null +++ b/src/storage/quota/FastQuotaStrategy.ts @@ -0,0 +1,107 @@ +import { PassThrough } from 'node:stream'; +import { + PodQuotaStrategy, + guardStream, +} from '@solid/community-server'; +// PayloadHttpError is not exported from the package root; deep import is +// needed to surface a 413 (Payload Too Large) on quota breaches. +import { PayloadHttpError } from '@solid/community-server/dist/util/errors/PayloadHttpError'; +import type { Guarded } from '@solid/community-server'; +import type { DataAccessor } from '@solid/community-server'; +import type { IdentifierStrategy } from '@solid/community-server'; +import type { ResourceIdentifier } from '@solid/community-server'; +import type { Size } from '@solid/community-server'; +import type { SizeReporter } from '@solid/community-server'; +import type { DuSizeReporter } from '../size-reporter/DuSizeReporter'; +import { isInternalPath } from './InternalPath'; +import { discoverPod } from './PodDiscovery'; + +/** + * Pod quota strategy that avoids the per-chunk full pod walk. + * + * CSS's default `QuotaStrategy.createQuotaGuard` calls `getAvailableSpace` + * (which performs a full pod walk) for EVERY stream chunk — `chunks × O(N)`. + * This override computes the available space ONCE before streaming, then only + * tracks the current write's own byte delta per chunk. When the write + * completes, the reporter's size cache is invalidated so the QuotaValidator's + * post-write check (and the next write's pre-check) re-walk fresh. + */ +export class FastQuotaStrategy extends PodQuotaStrategy { + private readonly discoveryStrategy: IdentifierStrategy; + private readonly discoveryAccessor: DataAccessor; + + public constructor( + limit: Size, + reporter: SizeReporter, + identifierStrategy: IdentifierStrategy, + accessor: DataAccessor, + baseUrl: string, + internalFolder?: string, + ) { + super(limit, reporter, identifierStrategy, accessor, baseUrl, internalFolder); + this.discoveryStrategy = identifierStrategy; + this.discoveryAccessor = accessor; + } + + /** + * Exempt CSS internal paths from quota checks. The QuotaValidator calls + * `getAvailableSpace` before AND after every write (and `createQuotaGuard` + * mid-stream); for `/.internal/*` writes (e.g. the IDP AuthorizationCode + * store) the inherited pod-discovery + size walk could take longer than the + * `WrappedExpiringReadWriteLocker` 6s expiry. Returning unlimited here + * short-circuits the validator without any pod walk. + */ + public override async getAvailableSpace(identifier: ResourceIdentifier): Promise { + if (isInternalPath(identifier)) { + return { amount: Number.MAX_SAFE_INTEGER, unit: 'bytes' }; + } + // Corrected pod discovery (metadata before root-container test) so + // subdomain-mode pods are found — CSS's searchPimStorage returns unlimited + // for every subdomain pod root. + const pod = await discoverPod(identifier, this.discoveryAccessor, this.discoveryStrategy); + if (pod === null) { + return { amount: Number.MAX_SAFE_INTEGER, unit: this.limit.unit }; + } + // Pod total, minus the resource's own size (it will be overwritten, so its + // space counts as available) — mirrors QuotaStrategy.getAvailableSpace. + const totalUsed = (await this.reporter.getSize(pod)).amount + - (await this.reporter.getSize(identifier)).amount; + return { amount: this.limit.amount - totalUsed, unit: this.limit.unit }; + } + + public async createQuotaGuard(identifier: ResourceIdentifier): Promise> { + // Compute the available space ONCE. getAvailableSpace already subtracts + // the overwritten resource's own size, and nothing else about the pod + // changes mid-write (atomic writes go to /.internal/, excluded by the + // reporter), so this single value is safe for the whole stream. + const availableSpace = await this.getAvailableSpace(identifier); + const reporter = this.reporter as DuSizeReporter & SizeReporter; + let total = 0; + + return guardStream(new PassThrough({ + async transform(chunk: any, _encoding: string, done: () => void): Promise { + total += await reporter.calculateChunkSize(chunk); + if (availableSpace.amount < total) { + this.destroy(new PayloadHttpError( + `Quota exceeded by ${total - availableSpace.amount} ${availableSpace.unit} during write`, + )); + } + this.push(chunk); + done(); + }, + async flush(done: (error?: Error) => void): Promise { + // Drop the cached sizes (resource + its ancestors incl. the pod root) + // so the QuotaValidator's after-write check re-walks and sees the new + // state. Best-effort: a failure to invalidate must not fail the write. + if (typeof reporter.invalidate === 'function') { + try { + await reporter.invalidate(identifier); + } catch { + // Ignore cache invalidation errors. + } + } + done(); + }, + })); + } +} diff --git a/src/storage/quota/IncrementalSizeReporter.ts b/src/storage/quota/IncrementalSizeReporter.ts new file mode 100644 index 0000000..5cc23a8 --- /dev/null +++ b/src/storage/quota/IncrementalSizeReporter.ts @@ -0,0 +1,50 @@ +import { + UNIT_BYTES, +} from '@solid/community-server'; +import type { + RepresentationMetadata, + ResourceIdentifier, + Size, + SizeReporter, +} from '@solid/community-server'; +import type { QuotaCounter } from './QuotaCounter'; + +/** + * {@link SizeReporter} backed by the incremental {@link QuotaCounter}. + * + * - `getSize(podRoot)` → **O(1)** counter read (recount only on bootstrap / + * staleness). + * - `getSize(any other resource)` → single stat (used by + * `QuotaStrategy.getAvailableSpace` to subtract the overwritten resource). + * + * Replaces `urn:solid-server:default:SizeReporter` in design C. The apparent + * byte unit is unchanged, so the 70 MB limit keeps its meaning. + */ +export class IncrementalSizeReporter implements SizeReporter { + private readonly counter: QuotaCounter; + + public constructor(counter: QuotaCounter) { + this.counter = counter; + } + + public getUnit(): string { + return UNIT_BYTES; + } + + public async getSize(identifier: ResourceIdentifier): Promise { + if (await this.counter.isPodRoot(identifier)) { + return this.counter.getSize(identifier); + } + return { unit: UNIT_BYTES, amount: await this.counter.sizeOfResource(identifier) }; + } + + /** The size of a chunk is simply its length in bytes. */ + public async calculateChunkSize(chunk: unknown): Promise { + return Buffer.isBuffer(chunk) ? chunk.length : Number((chunk as any)?.length) || 0; + } + + /** The estimated size of a resource is simply the content-length header. */ + public async estimateSize(metadata: RepresentationMetadata): Promise { + return metadata.contentLength; + } +} diff --git a/src/storage/quota/InternalPath.ts b/src/storage/quota/InternalPath.ts new file mode 100644 index 0000000..dd355ba --- /dev/null +++ b/src/storage/quota/InternalPath.ts @@ -0,0 +1,20 @@ +import type { ResourceIdentifier } from '@solid/community-server'; + +/** + * CSS internal storage (locks, IDP adapter, ...) lives under `/.internal/`. + * + * IMPORTANT: `ResourceIdentifier.path` is the full canonical URL (e.g. + * `https://pod.example.org/.internal/...`), not a bare path — identifier + * strategies (e.g. `SubdomainIdentifierStrategy`) test it against URL regexes. + * We must extract the URL pathname before comparing, otherwise the check never + * matches. This works in both suffix and subdomain deployment modes. + */ +export function isInternalPath(identifier: ResourceIdentifier): boolean { + let path = identifier.path; + try { + path = new URL(identifier.path).pathname; + } catch { + // Not a parseable URL — use the raw path as-is. + } + return path === '/.internal' || path.startsWith('/.internal/'); +} diff --git a/src/storage/quota/PodDiscovery.ts b/src/storage/quota/PodDiscovery.ts new file mode 100644 index 0000000..b8d1526 --- /dev/null +++ b/src/storage/quota/PodDiscovery.ts @@ -0,0 +1,62 @@ +import { + NotFoundHttpError, +} from '@solid/community-server'; +import type { + DataAccessor, + IdentifierStrategy, + RepresentationMetadata, + ResourceIdentifier, +} from '@solid/community-server'; + +const RDF_TYPE = 'http://www.w3.org/1999/02/22-rdf-syntax-ns#type'; +const PIM_STORAGE = 'http://www.w3.org/ns/pim/space#Storage'; +const TYPE_TERM = { termType: 'NamedNode', value: RDF_TYPE }; + +/** + * Finds the closest parent container that has `pim:Storage` as metadata. + * + * NOTE: unlike CSS's own `PodQuotaStrategy.searchPimStorage`, this reads the + * metadata BEFORE testing for a root container. CSS checks `isRootContainer` + * first and bails without reading metadata — in SUBDOMAIN mode that returns + * "no pod" for every pod root (each subdomain root IS a root container), so + * pod quota is silently unlimited and no pod is ever discovered there (and no + * counter sidecar is ever created). Design C fixes the order so pod discovery + * works in both suffix and subdomain modes. + */ +export async function discoverPod( + identifier: ResourceIdentifier, + accessor: DataAccessor, + identifierStrategy: IdentifierStrategy, +): Promise { + let metadata: RepresentationMetadata; + try { + metadata = await accessor.getMetadata(identifier); + } catch (error: unknown) { + if (NotFoundHttpError.isInstance(error)) { + // Resource and/or its metadata do not exist — walk up, but stop at a + // root container to avoid unbounded recursion. + if (identifierStrategy.isRootContainer(identifier)) { + return null; + } + return discoverPod( + identifierStrategy.getParentContainer(identifier), + accessor, + identifierStrategy, + ); + } + throw error; + } + const hasPimStorage = metadata.getAll(TYPE_TERM as any) + .some((term): boolean => term.value === PIM_STORAGE); + if (hasPimStorage) { + return identifier; + } + if (identifierStrategy.isRootContainer(identifier)) { + return null; + } + return discoverPod( + identifierStrategy.getParentContainer(identifier), + accessor, + identifierStrategy, + ); +} diff --git a/src/storage/quota/QuotaCounter.ts b/src/storage/quota/QuotaCounter.ts new file mode 100644 index 0000000..f2936a9 --- /dev/null +++ b/src/storage/quota/QuotaCounter.ts @@ -0,0 +1,259 @@ +import { promises as fs } from 'node:fs'; +import { join } from 'node:path'; +import { + UNIT_BYTES, + joinFilePath, + normalizeFilePath, +} from '@solid/community-server'; +import type { + FileIdentifierMapper, + ResourceIdentifier, + Size, +} from '@solid/community-server'; +import { DuSizeReporter } from '../size-reporter/DuSizeReporter'; + +// In-memory counter entry for one pod. +interface CounterEntry { + total: number; + valid: boolean; + podMtimeMs: number; +} + +/** + * Incremental per-pod byte counter (design C). + * + * Keeps the apparent-byte total of every pod in memory, updated O(1) per + * write by the {@link QuotaDeltaDataAccessor} delta hook, and persisted to a + * per-pod sidecar (`/.internal/pivot-quota.json`, atomic rename) so + * counters survive restarts. A full `du`/Node walk (via DuSizeReporter) is + * only used to bootstrap a pod (first access, no sidecar) or recover a + * de-synchronized counter. + * + * The counter is a cache; the filesystem is the source of truth. Staleness + * (out-of-band changes, crash window) is detected cheaply by comparing the + * pod root directory's mtime against the recorded one, then re-walking once. + */ +export class QuotaCounter { + private readonly fileIdentifierMapper: FileIdentifierMapper; + private readonly rootFilePath: string; + private readonly sidecarRelativePath: string; + private readonly walker: DuSizeReporter; + private readonly entries: Map = new Map(); + private readonly locks: Map> = new Map(); + + public constructor( + fileIdentifierMapper: FileIdentifierMapper, + rootFilePath: string, + ignoreFolders: string[] = [], + sidecarRelativePath = '/.internal/pivot-quota.json', + ) { + this.fileIdentifierMapper = fileIdentifierMapper; + this.rootFilePath = normalizeFilePath(rootFilePath); + this.sidecarRelativePath = sidecarRelativePath; + // Dedicated walker with no cache — every call is a fresh recount. + this.walker = new DuSizeReporter(fileIdentifierMapper, rootFilePath, ignoreFolders, 0); + } + + /** The QuotaCounter always reports in bytes. */ + public getUnit(): string { + return UNIT_BYTES; + } + + /** + * Returns the pod's current total, performing a recount (walk) only when + * the pod has no valid counter (first access, no sidecar, or staleness). + */ + public async getSize(podIdentifier: ResourceIdentifier): Promise { + const path = await this.mapDataPath(podIdentifier); + const entry = await this.ensureEntry(path, podIdentifier); + return { unit: UNIT_BYTES, amount: entry.total }; + } + + /** + * Marks a path as a pod root so {@link IncrementalSizeReporter} routes + * pod-root identifiers to the counter. Called by the delta hook when it + * first discovers a pod. + */ + public async register(podIdentifier: ResourceIdentifier): Promise { + const path = await this.mapDataPath(podIdentifier); + if (!this.entries.has(path)) { + this.entries.set(path, { total: 0, valid: false, podMtimeMs: 0 }); + } + } + + /** Whether the given identifier maps to a known pod root. */ + public async isPodRoot(identifier: ResourceIdentifier): Promise { + return this.entries.has(await this.mapDataPath(identifier)); + } + + /** + * Applies a size delta to the pod. Updates the in-memory counter and + * persists the sidecar atomically. Per-pod mutex serializes concurrent + * writes. + */ + public async add(podIdentifier: ResourceIdentifier, delta: number): Promise { + const path = await this.mapDataPath(podIdentifier); + await this.withLock(path, async (): Promise => { + const entry = this.entries.get(path) ?? { total: 0, valid: false, podMtimeMs: 0 }; + entry.total += delta; + entry.valid = true; + this.entries.set(path, entry); + await this.persistWithMtime(path, entry); + }); + } + + /** Drops the counter for a pod and removes its sidecar (pod deletion). */ + public async remove(podIdentifier: ResourceIdentifier): Promise { + const path = await this.mapDataPath(podIdentifier); + await this.withLock(path, async (): Promise => { + this.entries.delete(path); + await fs.rm(this.sidecarPath(path), { force: true }).catch(() => undefined); + }); + } + + /** + * Apparent size of a single resource (not a pod root) — used by the + * reporter for the overwritten-resource subtraction in + * `QuotaStrategy.getAvailableSpace`. Single stat for a document; walk for a + * container. + */ + public async sizeOfResource(identifier: ResourceIdentifier): Promise { + const filePath = await this.mapDataPath(identifier); + try { + const stat = await fs.stat(filePath); + if (stat.isFile()) { + return stat.size; + } + // Container — walk it (rare: only the overwritten resource is a file). + return (await this.walker.getSize(identifier)).amount; + } catch { + return 0; + } + } + + /** Maps an identifier to its data file path (normalized). */ + public async mapDataPath(identifier: ResourceIdentifier): Promise { + const { filePath } = await this.fileIdentifierMapper.mapUrlToFilePath(identifier, false); + return normalizeFilePath(filePath); + } + + /** + * Full apparent-byte walk of a resource/container (used by the delta hook + * for container before/after sizing — rare, e.g. create/delete container). + */ + public async walk(identifier: ResourceIdentifier): Promise { + return (await this.walker.getSize(identifier)).amount; + } + + // --- Internals --- + + private async ensureEntry(path: string, podIdentifier: ResourceIdentifier): Promise { + let entry = this.entries.get(path); + if (entry && entry.valid) { + const mtime = await this.podRootMtime(path); + if (entry.podMtimeMs === mtime) { + return entry; + } + // Pod root mtime moved — the counter may be stale, recount below. + } + // Try the sidecar first (persisted counter), then a full walk. + return this.withLock(path, async (): Promise => { + entry = this.entries.get(path); + if (entry && entry.valid) { + const mtime = await this.podRootMtime(path); + if (entry.podMtimeMs === mtime) { + return entry; + } + } + const loaded = await this.loadSidecar(path); + if (loaded && loaded.valid) { + const mtime = await this.podRootMtime(path); + if (loaded.podMtimeMs === mtime) { + this.entries.set(path, loaded); + return loaded; + } + } + // No valid counter — full walk (bootstrap / recovery). + const total = (await this.walker.getSize(podIdentifier)).amount; + const fresh: CounterEntry = { total, valid: true, podMtimeMs: 0 }; + this.entries.set(path, fresh); + await this.persistWithMtime(path, fresh); + return fresh; + }); + } + + private async podRootMtime(path: string): Promise { + try { + const stat = await fs.stat(path); + return stat.isDirectory() ? stat.mtimeMs : 0; + } catch { + return 0; + } + } + + private sidecarPath(podRootPath: string): string { + return normalizeFilePath(joinFilePath(podRootPath, this.sidecarRelativePath)); + } + + private async loadSidecar(path: string): Promise { + try { + const raw = await fs.readFile(this.sidecarPath(path), 'utf8'); + const parsed = JSON.parse(raw); + if (typeof parsed?.total === 'number' && typeof parsed?.podMtimeMs === 'number') { + return { total: parsed.total, valid: true, podMtimeMs: parsed.podMtimeMs }; + } + } catch { + // Missing or malformed sidecar → recount. + } + return undefined; + } + + private async persist(path: string, entry: CounterEntry): Promise { + const sidecar = this.sidecarPath(path); + const tmp = `${sidecar}.tmp`; + try { + await fs.mkdir(join(path, '.internal'), { recursive: true }); + await fs.writeFile(tmp, JSON.stringify({ + version: 1, + total: entry.total, + podMtimeMs: entry.podMtimeMs, + updatedAt: new Date().toISOString(), + })); + await fs.rename(tmp, sidecar); + } catch { + // Persistence is best-effort: keep the in-memory counter authoritative. + } + } + + /** + * Persist, then record the pod root mtime AFTER the persist. Persisting the + * sidecar may create the `.internal/` directory (a new pod-root child), which + * bumps the pod root's mtime — recording the mtime before would leave every + * subsequent read thinking the counter is stale. + */ + private async persistWithMtime(path: string, entry: CounterEntry): Promise { + await this.persist(path, entry); + entry.podMtimeMs = await this.podRootMtime(path); + await this.persist(path, entry); + } + + private async withLock(key: string, fn: () => Promise): Promise { + const previous = this.locks.get(key) ?? Promise.resolve(); + let release: () => void = () => undefined; + const gate = new Promise((resolve): void => { + release = resolve; + }); + // Waiters chain on `previous`, then wait for this call's `gate` to open. + const next = previous.then(() => gate); + this.locks.set(key, next); + await previous; + try { + return await fn(); + } finally { + release(); + if (this.locks.get(key) === next) { + this.locks.delete(key); + } + } + } +} diff --git a/src/storage/quota/QuotaDeltaDataAccessor.ts b/src/storage/quota/QuotaDeltaDataAccessor.ts new file mode 100644 index 0000000..d436e33 --- /dev/null +++ b/src/storage/quota/QuotaDeltaDataAccessor.ts @@ -0,0 +1,156 @@ +import { promises as fs } from 'node:fs'; +import type { Readable } from 'node:stream'; +import { + PassthroughDataAccessor, +} from '@solid/community-server'; +import type { + DataAccessor, + FileIdentifierMapper, + IdentifierStrategy, + RepresentationMetadata, + ResourceIdentifier, +} from '@solid/community-server'; +import type { Guarded } from '@solid/community-server'; +import type { QuotaCounter } from './QuotaCounter'; +import { isInternalPath } from './InternalPath'; +import { discoverPod } from './PodDiscovery'; + +/** + * Delta hook for design C. Wraps the top of the file accessor chain and, for + * every mutation, computes the resource's apparent-byte delta (before/after) + * and feeds it to the {@link QuotaCounter}. + * + * Plugs into the existing chain untouched: this wrapper's `accessor` is the + * original `FileDataAccessor` (FilterMetadata → Validating → Atomic), so the + * quota validation and content-length filtering are preserved. + * + * Cases handled: + * - create/overwrite document: Δ = new − old (data + metadata file) + * - delete resource: Δ = −(data + metadata size) + * - create/delete container: Δ from a walk (captures directory sizes) + */ +export class QuotaDeltaDataAccessor extends PassthroughDataAccessor { + private readonly identifierStrategy: IdentifierStrategy; + private readonly counter: QuotaCounter; + private readonly fileIdentifierMapper: FileIdentifierMapper; + private readonly podCache: Map = new Map(); + + public constructor( + accessor: DataAccessor, + identifierStrategy: IdentifierStrategy, + counter: QuotaCounter, + fileIdentifierMapper: FileIdentifierMapper, + ) { + super(accessor); + this.identifierStrategy = identifierStrategy; + this.counter = counter; + this.fileIdentifierMapper = fileIdentifierMapper; + } + + public async writeDocument( + identifier: ResourceIdentifier, + data: Guarded, + metadata: RepresentationMetadata, + ): Promise { + await this.track(identifier, (): Promise => this.accessor.writeDocument(identifier, data, metadata)); + } + + public async writeContainer(identifier: ResourceIdentifier, metadata: RepresentationMetadata): Promise { + await this.track(identifier, (): Promise => this.accessor.writeContainer(identifier, metadata)); + } + + public async writeMetadata(identifier: ResourceIdentifier, metadata: RepresentationMetadata): Promise { + await this.track(identifier, (): Promise => this.accessor.writeMetadata(identifier, metadata)); + } + + public async deleteResource(identifier: ResourceIdentifier): Promise { + if (isInternalPath(identifier)) { + await this.accessor.deleteResource(identifier); + return; + } + const before = await this.sizeOf(identifier); + await this.accessor.deleteResource(identifier); + const after = await this.sizeOf(identifier); + const delta = after - before; + const pod = await this.findPod(identifier); + if (pod === null) { + return; + } + // Deleting the pod root itself → drop the counter entirely. + if (pod.path === identifier.path) { + await this.counter.remove(identifier); + return; + } + if (delta !== 0) { + await this.counter.register(pod); + await this.counter.add(pod, delta); + } + } + + // --- Delta tracking --- + + private async track(identifier: ResourceIdentifier, op: () => Promise): Promise { + // Skip the delta bookkeeping on CSS internal paths: the stat + pod-discovery + // walk + counter sidecar work can otherwise push internal writes (e.g. IDP + // authorization codes) past the WrappedExpiringReadWriteLocker's lock expiry. + if (isInternalPath(identifier)) { + await op(); + return; + } + const before = await this.sizeOf(identifier); + await op(); + const after = await this.sizeOf(identifier); + const pod = await this.findPod(identifier); + if (pod === null) { + return; + } + // Always register the pod so the reporter routes its reads to the counter + // (even when this particular write has a zero delta, e.g. an empty + // container on a filesystem that reports directory size 0). + await this.counter.register(pod); + const delta = after - before; + if (delta !== 0) { + await this.counter.add(pod, delta); + } + } + + /** Apparent size of the resource: data file + metadata file (+ walk for containers). */ + private async sizeOf(identifier: ResourceIdentifier): Promise { + const data = await this.stat(identifier, false); + const meta = await this.stat(identifier, true); + return data + meta; + } + + private async stat(identifier: ResourceIdentifier, isMetadata: boolean): Promise { + try { + const { filePath } = await this.fileIdentifierMapper.mapUrlToFilePath(identifier, isMetadata); + const stat = await fs.stat(filePath); + if (stat.isDirectory()) { + return this.counter.walk(identifier); + } + return stat.size; + } catch { + return 0; + } + } + + // --- Pod discovery (mirrors PodQuotaStrategy.searchPimStorage) --- + + private async findPod(identifier: ResourceIdentifier): Promise { + const path = await this.counter.mapDataPath(identifier); + const cached = this.podCache.get(path); + if (cached !== undefined) { + return cached; + } + const pod = await this.discoverPod(identifier); + this.podCache.set(path, pod); + return pod; + } + + private async discoverPod(identifier: ResourceIdentifier): Promise { + // Uses the corrected discovery (metadata check before the root-container + // test) so subdomain-mode pod roots are found — CSS's own searchPimStorage + // bails at root containers and never finds subdomain pods. + return discoverPod(identifier, this.accessor, this.identifierStrategy); + } +} diff --git a/src/storage/size-reporter/DuSizeReporter.ts b/src/storage/size-reporter/DuSizeReporter.ts new file mode 100644 index 0000000..eb1f80f --- /dev/null +++ b/src/storage/size-reporter/DuSizeReporter.ts @@ -0,0 +1,206 @@ +import { execFile } from 'node:child_process'; +import { promises as fsPromises } from 'node:fs'; +import { promisify } from 'node:util'; +import { + UNIT_BYTES, + joinFilePath, + normalizeFilePath, + trimTrailingSlashes, +} from '@solid/community-server'; +import type { + FileIdentifierMapper, + RepresentationMetadata, + ResourceIdentifier, + Size, + SizeReporter, +} from '@solid/community-server'; + +const execFileAsync = promisify(execFile); + +// Cache entry: computed size + expiry timestamp +interface CacheEntry { + size: number; + expiresAt: number; +} + +/** + * A {@link SizeReporter} that measures a resource (and its children) in + * apparent bytes, using GNU/BSD `du` as a fast C-level walk with a per-path + * TTL cache, falling back to a plain Node walk when no compatible `du` + * exists (e.g. bare Windows). + * + * The unit is apparent bytes (sum of `st_size`) — identical to CSS's + * `FileSizeReporter`, portable across servers/filesystems and + * user-manageable. `stat.blocks` (disk usage) is deliberately NOT used: + * the result would depend on the server's filesystem cluster size. + */ +export class DuSizeReporter implements SizeReporter { + private readonly fileIdentifierMapper: FileIdentifierMapper; + private readonly rootFilePath: string; + private readonly ignoreFolders: RegExp[]; + private readonly ttlMs: number; + private readonly cache: Map = new Map(); + private duFlavor: 'gnu' | 'bsd' | 'none' | null = null; + + public constructor( + fileIdentifierMapper: FileIdentifierMapper, + rootFilePath: string, + ignoreFolders: string[] = [], + ttl: number = 5000, + ) { + this.fileIdentifierMapper = fileIdentifierMapper; + this.rootFilePath = normalizeFilePath(rootFilePath); + this.ignoreFolders = ignoreFolders.map((folder): RegExp => new RegExp(folder, 'u')); + this.ttlMs = ttl; + } + + /** The DuSizeReporter always returns data in the form of bytes. */ + public getUnit(): string { + return UNIT_BYTES; + } + + /** + * Returns the size of the given resource (and its children) in apparent + * bytes, using the per-path TTL cache when possible. + */ + public async getSize(identifier: ResourceIdentifier): Promise { + const { filePath } = await this.fileIdentifierMapper.mapUrlToFilePath(identifier, false); + const normalized = normalizeFilePath(filePath); + const cached = this.cache.get(normalized); + if (cached && cached.expiresAt > Date.now()) { + return { unit: UNIT_BYTES, amount: cached.size }; + } + const amount = await this.computeTotalSize(normalized); + this.cache.set(normalized, { size: amount, expiresAt: Date.now() + this.ttlMs }); + return { unit: UNIT_BYTES, amount }; + } + + /** + * Drop the cached size for the given resource and all of its ancestors + * (e.g. the pod root). Called when a write to the resource completes, so + * the next size query re-walks and reflects the new content. + */ + public async invalidate(identifier: ResourceIdentifier): Promise { + try { + const { filePath } = await this.fileIdentifierMapper.mapUrlToFilePath(identifier, false); + const normalized = normalizeFilePath(filePath); + for (const key of this.cache.keys()) { + if (key === normalized || normalized.startsWith(key)) { + this.cache.delete(key); + } + } + } catch { + // Best-effort: if the resource cannot be mapped, leave the cache as-is. + } + } + + /** The size of a chunk is simply its length in bytes. */ + public async calculateChunkSize(chunk: unknown): Promise { + return Buffer.isBuffer(chunk) ? chunk.length : Number((chunk as any)?.length) || 0; + } + + /** The estimated size of a resource is simply the content-length header. */ + public async estimateSize(metadata: RepresentationMetadata): Promise { + return metadata.contentLength; + } + + // --- Walk implementations --- + + private async computeTotalSize(fileLocation: string): Promise { + const flavor = await this.detectDu(); + if (flavor !== 'none') { + try { + return await this.computeTotalSizeWithDu(fileLocation, flavor); + } catch { + // du failed (e.g. no du after all, permission error) — fall back to the Node walk. + } + } + return this.computeTotalSizeWithNode(fileLocation); + } + + private async computeTotalSizeWithDu(fileLocation: string, flavor: 'gnu' | 'bsd'): Promise { + const args: string[] = flavor === 'gnu' ? [ '-sb' ] : [ '-s', '-A', '-B', '1' ]; + for (const pattern of this.duExcludePatterns()) { + args.push(flavor === 'gnu' ? '--exclude' : '-I', pattern); + } + args.push(fileLocation); + const { stdout } = await execFileAsync('du', args, { maxBuffer: 64 * 1024 * 1024 }); + const amount = Number(stdout.trim().split(/\s+/)[0]); + if (!Number.isFinite(amount)) { + throw new Error(`Could not parse du output: ${stdout.trim()}`); + } + return amount; + } + + /** + * Plain Node recursive walk — the same semantics as CSS's + * `FileSizeReporter.getTotalSize`. Used when no compatible `du` exists. + */ + private async computeTotalSizeWithNode(fileLocation: string): Promise { + let stat; + try { + stat = await fsPromises.stat(fileLocation); + } catch { + return 0; + } + // If the file's location points to a file, simply return the file's size. + if (stat.isFile()) { + return stat.size; + } + // Recursively add all sizes of children to the total. + const childFiles = await fsPromises.readdir(fileLocation); + const rootFilePathLength = trimTrailingSlashes(this.rootFilePath).length; + let totalSize = stat.size; + for (const current of childFiles) { + const childFileLocation = normalizeFilePath(joinFilePath(fileLocation, current)); + // Exclude internal files, matching FileSizeReporter's behavior. + if (!this.ignoreFolders.some((folder): boolean => folder.test(childFileLocation.slice(rootFilePathLength)))) { + totalSize += await this.computeTotalSizeWithNode(childFileLocation); + } + } + return totalSize; + } + + protected async detectDu(): Promise<'gnu' | 'bsd' | 'none'> { + if (this.duFlavor) { + return this.duFlavor; + } + try { + await execFileAsync('du', [ '--version' ], { timeout: 1000 }); + this.duFlavor = 'gnu'; + } catch (error: any) { + // ENOENT: no `du` at all (e.g. bare Windows). Anything else means the + // GNU long option was rejected → assume BSD; if BSD flags fail at use + // time, the caller falls back to the Node walk. + this.duFlavor = error?.code === 'ENOENT' ? 'none' : 'bsd'; + } + return this.duFlavor; + } + + /** + * Convert the configured ignore-folder regexes into `du` exclude patterns. + * GNU/BSD `du` matches exclude patterns against path components/basenames + * (not against the full leading-slash path), so a regex like `^/\.internal$` + * becomes the exclude `.internal`. + * + * Only simple anchored folder patterns are convertible + * (`^/name$`); complex regexes that cannot be expressed as a du exclude are + * skipped here — the Node-walk fallback still applies them verbatim. + */ + private duExcludePatterns(): string[] { + const patterns: string[] = []; + for (const regex of this.ignoreFolders) { + // RegExp.source always escapes '/' as '\/' (e.g. `^/\.internal$` → + // `^\/\.internal$`). Strip the leading '^' + '/' (possibly '\/'), drop + // the trailing '$', then unescape the remaining escapes. + let src = regex.source.replace(/^\^?\\?\//, ''); + src = src.replace(/\$?$/, ''); + src = src.replace(/\\./g, '.'); + // Keep only patterns with no remaining regex metacharacters. + if (/^[A-Za-z0-9._-]+$/.test(src)) { + patterns.push(src); + } + } + return patterns; + } +} diff --git a/test/integration/ProfileCardGuard.test.ts b/test/integration/ProfileCardGuard.test.ts new file mode 100644 index 0000000..3475946 --- /dev/null +++ b/test/integration/ProfileCardGuard.test.ts @@ -0,0 +1,148 @@ +import fetch from 'cross-fetch'; +import type { App } from '@solid/community-server'; +import { joinUrl } from '@solid/community-server'; +import { getPort } from '../util/Util'; +import { getDefaultVariables, getTestConfigPath, instantiateFromConfig } from './Config'; + +const port = getPort('ProfileCardGuard'); +const baseUrl = `http://localhost:${port}/`; + +/** + * The full lifecycle of a WebID profile card on a server with the profile card guard: + * 1. a pod is created with a WebID, registered to the account, and the server is its IDP; + * 2. the card is protected as long as the WebID is registered; + * 3. after unlinking the WebID from the account, the card is no longer protected. + */ +describe('A server with the profile card guard', (): void => { + let app: App; + let cookie: string; + let controls: any; + let cardUrl: string; + let webId: string; + const email = 'test@example.com'; + const password = 'secret!'; + + beforeAll(async(): Promise => { + const instances = await instantiateFromConfig( + 'urn:solid-server:test:Instances', + getTestConfigPath('server-memory-guard.json'), + getDefaultVariables(port, baseUrl), + ) as Record; + ({ app } = instances); + await app.start(); + + // Fetch the account controls + let res = await fetch(joinUrl(baseUrl, '.account/')); + if (res.status !== 200) { + throw new Error(`Fetching the controls failed: ${await res.text()}`); + } + ({ controls } = await res.json()); + + // Create an account, which also logs us in + res = await fetch(controls.account.create, { method: 'POST' }); + if (res.status !== 200) { + throw new Error(`Creating the account failed: ${await res.text()}`); + } + const setCookie = res.headers.get('set-cookie')!; + cookie = setCookie.slice(0, setCookie.indexOf(';')); + + // Get the account-specific controls + res = await fetch(joinUrl(baseUrl, '.account/'), { headers: { cookie }}); + if (res.status !== 200) { + throw new Error(`Fetching the account controls failed: ${await res.text()}`); + } + ({ controls } = await res.json()); + + // Add a password login method + res = await fetch(controls.password.create, { + method: 'POST', + headers: { cookie, 'content-type': 'application/json' }, + body: JSON.stringify({ email, password }), + }); + if (res.status !== 200) { + throw new Error(`Adding the login method failed: ${await res.text()}`); + } + + // Create a pod, which registers its WebID to the account + res = await fetch(controls.account.pod, { + method: 'POST', + headers: { cookie, 'content-type': 'application/json' }, + body: JSON.stringify({ name: 'test' }), + }); + if (res.status !== 200) { + throw new Error(`Creating the pod failed: ${await res.text()}`); + } + const { webId: createdWebId } = await res.json(); + ({ webId } = { webId: createdWebId }); + cardUrl = webId.slice(0, webId.indexOf('#')); + }); + + afterAll(async(): Promise => { + await app.stop(); + }); + + it('creates the card with the server as its issuer.', async(): Promise => { + const res = await fetch(cardUrl, { headers: { accept: 'text/turtle' }}); + expect(res.status).toBe(200); + await expect(res.text()).resolves.toContain('solid:oidcIssuer'); + expect(webId).toBe(`${cardUrl}#me`); + }); + + it('forbids deleting the card while its WebID is registered.', async(): Promise => { + const res = await fetch(cardUrl, { method: 'DELETE' }); + expect(res.status).toBe(403); + }); + + it('rejects a card update that drops the issuer triple.', async(): Promise => { + const turtle = `<${webId}> "Alice".`; + const res = await fetch(cardUrl, { + method: 'PUT', + headers: { 'content-type': 'text/turtle' }, + body: turtle, + }); + expect(res.status).toBe(400); + }); + + it('accepts a card update that keeps the issuer triple.', async(): Promise => { + const turtle = `@prefix solid: . +<${webId}> solid:oidcIssuer <${baseUrl}>; solid:name "Alice".`; + const res = await fetch(cardUrl, { + method: 'PUT', + headers: { 'content-type': 'text/turtle' }, + body: turtle, + }); + expect(res.status).toBe(205); + }); + + it('does not protect documents that host no registered WebID.', async(): Promise => { + const note = joinUrl(baseUrl, 'test/notes'); + let res = await fetch(note, { + method: 'PUT', + headers: { 'content-type': 'text/turtle' }, + body: ' "note".', + }); + expect(res.status).toBe(201); + res = await fetch(note, { method: 'DELETE' }); + expect(res.status).toBe(205); + }); + + it('no longer protects the card after the WebID is unlinked.', async(): Promise => { + // Unlink the WebID from the account + let res = await fetch(controls.account.webId, { headers: { cookie }}); + expect(res.status).toBe(200); + const { webIdLinks } = await res.json(); + expect(webIdLinks[webId]).toBeDefined(); + + res = await fetch(webIdLinks[webId], { method: 'DELETE', headers: { cookie }}); + expect(res.status).toBe(200); + + // Verify the WebID is no longer registered + res = await fetch(controls.account.webId, { headers: { cookie }}); + expect(res.status).toBe(200); + expect((await res.json()).webIdLinks[webId]).toBeUndefined(); + + // The card can now be deleted + res = await fetch(cardUrl, { method: 'DELETE' }); + expect(res.status).toBe(205); + }); +}); diff --git a/test/integration/config/server-memory-guard.json b/test/integration/config/server-memory-guard.json new file mode 100644 index 0000000..7da93c4 --- /dev/null +++ b/test/integration/config/server-memory-guard.json @@ -0,0 +1,59 @@ +{ + "@context": [ + "https://linkedsoftwaredependencies.org/bundles/npm/@solid/community-server/^7.0.0/components/context.jsonld", + "https://linkedsoftwaredependencies.org/bundles/npm/@solid/pivot/^1.0.0/components/context.jsonld" + ], + "import": [ + "css:config/app/init/initialize-root.json", + "css:config/app/main/default.json", + "css:config/http/handler/default.json", + "css:config/http/middleware/default.json", + "css:config/http/notifications/all.json", + "css:config/http/server-factory/http.json", + "css:config/http/static/default.json", + "css:config/identity/access/public.json", + "css:config/identity/handler/default.json", + "css:config/identity/oidc/default.json", + "css:config/identity/ownership/token.json", + "css:config/identity/pod/static.json", + "css:config/ldp/authentication/dpop-bearer.json", + "css:config/ldp/authorization/allow-all.json", + "css:config/ldp/handler/default.json", + "css:config/ldp/metadata-parser/default.json", + "css:config/ldp/metadata-writer/default.json", + "css:config/ldp/modes/default.json", + "css:config/storage/backend/memory.json", + "css:config/storage/key-value/resource-store.json", + "css:config/storage/location/root.json", + "pivot:config/storage/middleware/default.json", + "pivot:config/storage/profile-card-guard.json", + "css:config/util/auxiliary/acl.json", + "css:config/util/identifiers/suffix.json", + "css:config/util/index/default.json", + "css:config/util/logging/winston.json", + "css:config/util/representation-conversion/default.json", + "css:config/util/resource-locker/memory.json", + "css:config/util/variables/default.json" + ], + "@graph": [ + { + "@id": "urn:solid-server:test:Instances", + "@type": "RecordObject", + "record": [ + { + "RecordObject:_record_key": "app", + "RecordObject:_record_value": { "@id": "urn:solid-server:default:App" } + } + ] + }, + { + "@id": "urn:solid-server:default:EmailSender", + "@type": "BaseEmailSender", + "args_senderName": "Solid Server", + "args_emailConfig_host": "smtp.example.email", + "args_emailConfig_port": 587, + "args_emailConfig_auth_user": "alice@example.email", + "args_emailConfig_auth_pass": "NYEaCsqV7aVStRCbmC" + } + ] +} diff --git a/test/unit/storage/DuSizeReporter.test.ts b/test/unit/storage/DuSizeReporter.test.ts new file mode 100644 index 0000000..ae013e5 --- /dev/null +++ b/test/unit/storage/DuSizeReporter.test.ts @@ -0,0 +1,135 @@ +import { promises as fs } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import type { FileIdentifierMapper, ResourceIdentifier, Size } from '@solid/community-server'; +import { DuSizeReporter } from '../../../src/storage/size-reporter/DuSizeReporter'; + +// Force the du-based path (works even where du is absent — the Node walk +// produces the same apparent-byte sum for a simple tree). +class ForceDuReporter extends DuSizeReporter { + protected override async detectDu(): Promise<'gnu' | 'bsd' | 'none'> { + return 'gnu'; + } +} + +// Force the Node-walk fallback path. +class ForceNodeReporter extends DuSizeReporter { + protected override async detectDu(): Promise<'gnu' | 'bsd' | 'none'> { + return 'none'; + } +} + +function createMapper(root: string): FileIdentifierMapper { + return { + async mapUrlToFilePath(identifier: ResourceIdentifier): Promise { + const url = new URL(identifier.path); + return { identifier, filePath: join(root, url.pathname), contentType: undefined, isMetadata: false }; + }, + async mapFilePathToUrl(): Promise { + throw new Error('Not implemented'); + }, + }; +} + +describe('A DuSizeReporter', (): void => { + let root: string; + let mapper: FileIdentifierMapper; + + beforeEach(async(): Promise => { + root = await fs.mkdtemp(join(tmpdir(), 'du-size-reporter-')); + mapper = createMapper(root); + }); + + afterEach(async(): Promise => { + await fs.rm(root, { recursive: true, force: true }); + }); + + it('returns the apparent size of a file.', async(): Promise => { + const reporter = new DuSizeReporter(mapper, root); + await fs.writeFile(join(root, 'a.txt'), Buffer.alloc(100)); + const size = await reporter.getSize({ path: 'http://example.com/a.txt' }); + expect(size).toEqual({ unit: 'bytes', amount: 100 }); + }); + + it('reports the same size whether du or the Node fallback is used.', async(): Promise => { + await fs.mkdir(join(root, 'dir')); + await fs.writeFile(join(root, 'dir', 'a.txt'), Buffer.alloc(100)); + await fs.writeFile(join(root, 'dir', 'b.txt'), Buffer.alloc(50)); + const viaDu = await new ForceDuReporter(mapper, root).getSize({ path: 'http://example.com/dir/a.txt' }); + const viaNode = await new ForceNodeReporter(mapper, root).getSize({ path: 'http://example.com/dir/a.txt' }); + expect(viaDu.amount).toBe(100); + expect(viaNode.amount).toBe(100); + }); + + it('serves a cached result within the TTL window without re-walking.', async(): Promise => { + const reporter = new ForceDuReporter(mapper, root, [], 60_000); + await fs.writeFile(join(root, 'a.txt'), Buffer.alloc(100)); + const first = await reporter.getSize({ path: 'http://example.com/a.txt' }); + // Change the file without invalidating — the cache must still serve the old size. + await fs.writeFile(join(root, 'a.txt'), Buffer.alloc(200)); + const cached = await reporter.getSize({ path: 'http://example.com/a.txt' }); + expect(first.amount).toBe(100); + expect(cached.amount).toBe(100); + }); + + it('recomputes after invalidation.', async(): Promise => { + const reporter = new DuSizeReporter(mapper, root, [], 60_000); + await fs.writeFile(join(root, 'a.txt'), Buffer.alloc(100)); + await reporter.getSize({ path: 'http://example.com/a.txt' }); + await fs.writeFile(join(root, 'a.txt'), Buffer.alloc(200)); + await reporter.invalidate({ path: 'http://example.com/a.txt' }); + const after = await reporter.getSize({ path: 'http://example.com/a.txt' }); + expect(after.amount).toBe(200); + }); + + it('invalidates ancestor entries (e.g. the pod root) as well.', async(): Promise => { + const reporter = new DuSizeReporter(mapper, root, [], 60_000); + await fs.mkdir(join(root, 'dir')); + await fs.writeFile(join(root, 'dir', 'a.txt'), Buffer.alloc(100)); + const rootSize = await reporter.getSize({ path: 'http://example.com/' }); + expect(rootSize.amount).toBeGreaterThanOrEqual(100); + await fs.writeFile(join(root, 'dir', 'a.txt'), Buffer.alloc(300)); + await reporter.invalidate({ path: 'http://example.com/dir/a.txt' }); + const newRootSize = await reporter.getSize({ path: 'http://example.com/' }); + expect(newRootSize.amount).toBe(rootSize.amount + 200); + }); + + it('excludes the ignoreFolders from the total.', async(): Promise => { + const reporter = new DuSizeReporter(mapper, root, [ '^/\\.internal$' ]); + await fs.mkdir(join(root, '.internal')); + await fs.writeFile(join(root, '.internal', 'x.txt'), Buffer.alloc(1000)); + const without = await reporter.getSize({ path: 'http://example.com/' }); + // Adding a file inside .internal must not change the reported size. + await fs.writeFile(join(root, '.internal', 'y.txt'), Buffer.alloc(1000)); + await reporter.invalidate({ path: 'http://example.com/' }); + const still = await reporter.getSize({ path: 'http://example.com/' }); + expect(still.amount).toBe(without.amount); + // Adding a normal file must increase it. + await fs.writeFile(join(root, 'a.txt'), Buffer.alloc(50)); + await reporter.invalidate({ path: 'http://example.com/' }); + const increased = await reporter.getSize({ path: 'http://example.com/' }); + expect(increased.amount).toBe(without.amount + 50); + }); + + it('returns the content-length as the estimated size.', async(): Promise => { + const reporter = new DuSizeReporter(mapper, root); + await expect(reporter.estimateSize({ contentLength: 42 } as any)).resolves.toBe(42); + await expect(reporter.estimateSize({} as any)).resolves.toBeUndefined(); + }); + + it('calculates the chunk size as the buffer length.', async(): Promise => { + const reporter = new DuSizeReporter(mapper, root); + await expect(reporter.calculateChunkSize(Buffer.alloc(17))).resolves.toBe(17); + }); + + it('returns the byte unit.', async(): Promise => { + const reporter = new DuSizeReporter(mapper, root); + expect(reporter.getUnit()).toBe('bytes'); + }); + + it('reports a size of 0 for a missing resource.', async(): Promise => { + const reporter = new DuSizeReporter(mapper, root); + const size: Size = await reporter.getSize({ path: 'http://example.com/nope' }); + expect(size.amount).toBe(0); + }); +}); diff --git a/test/unit/storage/FastQuotaStrategy.test.ts b/test/unit/storage/FastQuotaStrategy.test.ts new file mode 100644 index 0000000..3700793 --- /dev/null +++ b/test/unit/storage/FastQuotaStrategy.test.ts @@ -0,0 +1,130 @@ +import { PassThrough } from 'node:stream'; +import type { + DataAccessor, + IdentifierStrategy, + RepresentationMetadata, + ResourceIdentifier, + Size, + SizeReporter, +} from '@solid/community-server'; +import { FastQuotaStrategy } from '../../../src/storage/quota/FastQuotaStrategy'; + +const PIM_STORAGE = 'http://www.w3.org/ns/pim/space#Storage'; +const POD = { path: 'http://example.com/' }; + +// A strategy with a fixed pod total so createQuotaGuard can be exercised +// without a real pod/accessor setup. Discovery is mocked so `getAvailableSpace` +// finds POD and the reporter reports `podTotal` for it and `resourceSize` for +// the resource being written. With `noPod` discovery reports no pod (quota +// does not apply → unlimited). +function createStrategy( + podTotal: number, + resourceSize: number, + limit: Size, + reporter: jest.Mocked> & { invalidate: jest.Mock }, + noPod = false, +): FastQuotaStrategy { + const accessor: any = { + getMetadata: jest.fn(async (id: ResourceIdentifier): Promise => ({ + getAll: (): any[] => (id.path === POD.path ? [{ value: PIM_STORAGE }] : []), + })), + }; + const identifierStrategy: any = { + isRootContainer: jest.fn((): boolean => noPod), + getParentContainer: jest.fn((): ResourceIdentifier => POD), + }; + reporter.getSize.mockImplementation(async (id: ResourceIdentifier): Promise => + id.path === POD.path ? { unit: 'bytes', amount: podTotal } : { unit: 'bytes', amount: resourceSize }); + // The PodQuotaStrategy constructor requires the server base URL to build its + // internal-folder path; these tests never exercise it. + return new FastQuotaStrategy(limit, reporter, identifierStrategy, accessor, 'http://example.com/'); +} + +function mockReporter(oldResourceSize: number): jest.Mocked> & { invalidate: jest.Mock } { + const reporter: any = { + getUnit: jest.fn((): string => 'bytes'), + getSize: jest.fn(async(): Promise => ({ unit: 'bytes', amount: oldResourceSize })), + calculateChunkSize: jest.fn(async(chunk: Buffer): Promise => chunk.length), + estimateSize: jest.fn(async(): Promise => undefined), + invalidate: jest.fn(async(): Promise => undefined), + }; + return reporter; +} + +// Writes all chunks into the guard and waits for it to end (or error). +function writeChunks(guard: PassThrough, chunks: Buffer[]): Promise { + return new Promise((resolve, reject): void => { + guard.on('data', (): void => { + // consume + }); + guard.on('end', resolve); + guard.on('error', reject); + for (const chunk of chunks) { + guard.write(chunk); + } + guard.end(); + }); +} + +describe('A FastQuotaStrategy', (): void => { + const identifier: ResourceIdentifier = { path: 'http://example.com/foo' }; + // available space = limit - pod total + overwritten resource size + // = 100 - 90 + 10 = 20 bytes + const limit: Size = { unit: 'bytes', amount: 100 }; + + it('calls getAvailableSpace (the pod walk) only once per write, not per chunk.', async(): Promise => { + const reporter = mockReporter(10); + const strategy = createStrategy(90, 10, limit, reporter); + const guard = await strategy.createQuotaGuard(identifier); + // Two 5-byte chunks → 10 bytes total, under the 20-byte budget. + await writeChunks(guard, [ Buffer.alloc(5), Buffer.alloc(5) ]); + // getSize is called exactly twice by getAvailableSpace (once for the pod, + // once for the overwritten resource) and must NOT be called again per chunk. + expect(reporter.getSize).toHaveBeenCalledTimes(2); + }); + + it('passes chunks through while the write stays under the available space.', async(): Promise => { + const reporter = mockReporter(10); + const strategy = createStrategy(90, 10, limit, reporter); + const guard = await strategy.createQuotaGuard(identifier); + const received: Buffer[] = []; + guard.on('data', (chunk: Buffer): void => { + received.push(chunk); + }); + await writeChunks(guard, [ Buffer.alloc(5), Buffer.alloc(10) ]); + expect(received.reduce((sum, chunk): number => sum + chunk.length, 0)).toBe(15); + }); + + it('errors when the write exceeds the available space.', async(): Promise => { + const reporter = mockReporter(10); + const strategy = createStrategy(90, 10, limit, reporter); + const guard = await strategy.createQuotaGuard(identifier); + // 25 bytes > 20 available. + await expect(writeChunks(guard, [ Buffer.alloc(25) ])).rejects.toThrow(/Quota exceeded/); + }); + + it('invalidates the reporter cache when the write completes.', async(): Promise => { + const reporter = mockReporter(10); + const strategy = createStrategy(90, 10, limit, reporter); + const guard = await strategy.createQuotaGuard(identifier); + await writeChunks(guard, [ Buffer.alloc(5) ]); + expect(reporter.invalidate).toHaveBeenCalledTimes(1); + expect(reporter.invalidate).toHaveBeenLastCalledWith(identifier); + }); + + it('does not fail the write when cache invalidation fails.', async(): Promise => { + const reporter = mockReporter(10); + reporter.invalidate.mockRejectedValue(new Error('mapping failed')); + const strategy = createStrategy(90, 10, limit, reporter); + const guard = await strategy.createQuotaGuard(identifier); + await expect(writeChunks(guard, [ Buffer.alloc(5) ])).resolves.toBeUndefined(); + }); + + it('never errors when the quota does not apply (infinite space).', async(): Promise => { + const reporter = mockReporter(0); + // noPod → discovery finds no storage → unlimited space. + const strategy = createStrategy(0, 0, limit, reporter, true); + const guard = await strategy.createQuotaGuard(identifier); + await expect(writeChunks(guard, [ Buffer.alloc(10_000) ])).resolves.toBeUndefined(); + }); +}); diff --git a/test/unit/storage/IncrementalSizeReporter.test.ts b/test/unit/storage/IncrementalSizeReporter.test.ts new file mode 100644 index 0000000..a27d7cf --- /dev/null +++ b/test/unit/storage/IncrementalSizeReporter.test.ts @@ -0,0 +1,63 @@ +import { promises as fs } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import type { FileIdentifierMapper, ResourceIdentifier } from '@solid/community-server'; +import { IncrementalSizeReporter } from '../../../src/storage/quota/IncrementalSizeReporter'; +import { QuotaCounter } from '../../../src/storage/quota/QuotaCounter'; + +const IGNORE = [ '^/\\.internal$' ]; + +function createMapper(root: string): FileIdentifierMapper { + return { + async mapUrlToFilePath(identifier: ResourceIdentifier, isMetadata: boolean): Promise { + const url = new URL(identifier.path); + const base = join(root, url.pathname); + return { + identifier, + filePath: isMetadata ? `${base}.meta` : base, + contentType: undefined, + isMetadata, + }; + }, + async mapFilePathToUrl(): Promise { + throw new Error('Not implemented'); + }, + }; +} + +const POD = { path: 'http://example.com/alice/' }; +const RESOURCE = { path: 'http://example.com/alice/foo' }; + +describe('An IncrementalSizeReporter', (): void => { + let root: string; + let counter: QuotaCounter; + let reporter: IncrementalSizeReporter; + + beforeEach(async(): Promise => { + root = await fs.mkdtemp(join(tmpdir(), 'inc-reporter-')); + await fs.mkdir(join(root, 'alice')); + counter = new QuotaCounter(createMapper(root), root, IGNORE); + reporter = new IncrementalSizeReporter(counter); + }); + + afterEach(async(): Promise => { + await fs.rm(root, { recursive: true, force: true }); + }); + + it('returns the counter total for a registered pod root.', async(): Promise => { + await counter.register(POD); + await counter.add(POD, 123); + await expect(reporter.getSize(POD)).resolves.toEqual({ unit: 'bytes', amount: 123 }); + }); + + it('stats a regular resource (not a pod root).', async(): Promise => { + await fs.writeFile(join(root, 'alice', 'foo'), Buffer.alloc(64)); + await expect(reporter.getSize(RESOURCE)).resolves.toEqual({ unit: 'bytes', amount: 64 }); + }); + + it('returns the byte unit and chunk/content-length helpers.', async(): Promise => { + expect(reporter.getUnit()).toBe('bytes'); + await expect(reporter.calculateChunkSize(Buffer.alloc(9))).resolves.toBe(9); + await expect(reporter.estimateSize({ contentLength: 42 } as any)).resolves.toBe(42); + }); +}); diff --git a/test/unit/storage/ProfileCardGuard.test.ts b/test/unit/storage/ProfileCardGuard.test.ts new file mode 100644 index 0000000..1f992f7 --- /dev/null +++ b/test/unit/storage/ProfileCardGuard.test.ts @@ -0,0 +1,249 @@ +import type { Quad } from '@rdfjs/types'; +import { DataFactory } from 'n3'; +import { + BasicRepresentation, + BadRequestHttpError, + ForbiddenHttpError, + INTERNAL_QUADS, + RepresentationMetadata, + SOLID, + guardedStreamFrom, + readableToString, +} from '@solid/community-server'; +import type { Representation, ResourceIdentifier, ResourceStore } from '@solid/community-server'; +import type { GuardedWebIdStore } from '../../../src/identity/interaction/webid/util/GuardedWebIdStore'; +import { ProfileCardGuard } from '../../../src/storage/ProfileCardGuard'; + +const { namedNode, quad } = DataFactory; + +const CARD: ResourceIdentifier = { path: 'http://example.com/alice/profile/card' }; +const OTHER: ResourceIdentifier = { path: 'http://example.com/alice/notes' }; +const WEBID = 'http://example.com/alice/profile/card#me'; +const ISSUER = 'http://example.com/'; +const WEB_ID_PATHS = [ '/profile/card#me' ]; + +function representation(data: any, contentType: string, identifier?: string): Representation { + const metadata = identifier ? new RepresentationMetadata({ path: identifier }) : new RepresentationMetadata(); + metadata.contentType = contentType; + return new BasicRepresentation(data, metadata); +} + +function quadRepresentation(quads: Quad[]): Representation { + return representation(guardedStreamFrom(quads), INTERNAL_QUADS); +} + +function issuerQuad(webId = WEBID, issuer = ISSUER): Quad { + return quad(namedNode(webId), namedNode(SOLID.terms.oidcIssuer.value), namedNode(issuer)); +} + +describe('A ProfileCardGuard', (): void => { + const source: jest.Mocked = { + getRepresentation: jest.fn(async(): Promise => 'get'), + addResource: jest.fn(async(): Promise => 'add'), + setRepresentation: jest.fn(async(): Promise => 'set'), + deleteResource: jest.fn(async(): Promise => 'delete'), + modifyResource: jest.fn(), + } as any; + const converter: { handleSafe: jest.Mock } = { handleSafe: jest.fn() }; + let registered: string[]; + let webIdStore: { hasWebId: jest.Mock }; + let guard: ProfileCardGuard; + + beforeEach(async(): Promise => { + jest.clearAllMocks(); + registered = [ WEBID ]; + webIdStore = { + hasWebId: jest.fn(async(webId: string): Promise => registered.includes(webId)), + }; + converter.handleSafe.mockResolvedValue(quadRepresentation([ issuerQuad() ])); + guard = new ProfileCardGuard( + source, + webIdStore as unknown as GuardedWebIdStore, + ISSUER, + converter as any, + WEB_ID_PATHS, + ); + }); + + it('passes through writes to documents outside the configured card locations.', async(): Promise => { + const rep = representation('data', 'text/plain'); + await guard.setRepresentation(OTHER, rep); + expect(source.setRepresentation).toHaveBeenCalledTimes(1); + expect(source.setRepresentation).toHaveBeenLastCalledWith(OTHER, rep, undefined); + expect(webIdStore.hasWebId).toHaveBeenCalledTimes(0); + expect(converter.handleSafe).toHaveBeenCalledTimes(0); + }); + + it('passes through writes to a card location that hosts no registered WebID.', async(): Promise => { + registered = []; + const rep = representation('anything', 'text/plain'); + await guard.setRepresentation(CARD, rep); + expect(webIdStore.hasWebId).toHaveBeenCalledTimes(1); + expect(webIdStore.hasWebId).toHaveBeenLastCalledWith(WEBID); + expect(converter.handleSafe).toHaveBeenCalledTimes(0); + expect(source.setRepresentation).toHaveBeenCalledTimes(1); + }); + + it('passes through a valid card write that keeps the issuer triple.', async(): Promise => { + const turtle = ' solid:oidcIssuer .'; + const rep = representation(turtle, 'text/turtle'); + await guard.setRepresentation(CARD, rep); + expect(converter.handleSafe).toHaveBeenCalledTimes(1); + expect(source.setRepresentation).toHaveBeenCalledTimes(1); + expect(source.setRepresentation).toHaveBeenLastCalledWith(CARD, rep, undefined); + // The original data is still streamable + await expect(readableToString(rep.data)).resolves.toContain('solid:oidcIssuer'); + }); + + it('does not require the card to be sent as Turtle.', async(): Promise => { + const rep = representation('{ "@id": "http://example.com/alice/profile/card#me" }', 'application/ld+json'); + await guard.setRepresentation(CARD, rep); + expect(converter.handleSafe).toHaveBeenCalledTimes(1); + expect(source.setRepresentation).toHaveBeenCalledTimes(1); + }); + + it('accepts a card write provided as internal quads, as produced by a PATCH.', async(): Promise => { + const rep = representation(guardedStreamFrom([ issuerQuad() ]), INTERNAL_QUADS); + await guard.setRepresentation(CARD, rep); + expect(converter.handleSafe).toHaveBeenCalledTimes(0); + expect(source.setRepresentation).toHaveBeenCalledTimes(1); + expect(source.setRepresentation).toHaveBeenLastCalledWith(CARD, rep, undefined); + }); + + it('rejects a card write without the issuer triple.', async(): Promise => { + converter.handleSafe.mockResolvedValueOnce(quadRepresentation([ + quad(namedNode(WEBID), namedNode('http://purl.org/dc/terms/name'), namedNode('Alice')), + ])); + const rep = representation('data', 'text/turtle'); + await expect(guard.setRepresentation(CARD, rep)).rejects.toThrow(BadRequestHttpError); + expect(source.setRepresentation).toHaveBeenCalledTimes(0); + expect(rep.data.destroyed).toBe(true); + }); + + it('rejects a card write pointing to another issuer.', async(): Promise => { + converter.handleSafe.mockResolvedValueOnce(quadRepresentation([ issuerQuad(WEBID, 'http://other.example/') ])); + const rep = representation('data', 'text/turtle'); + await expect(guard.setRepresentation(CARD, rep)).rejects.toThrow(BadRequestHttpError); + expect(source.setRepresentation).toHaveBeenCalledTimes(0); + expect(rep.data.destroyed).toBe(true); + }); + + it('rejects a card write that is not valid RDF.', async(): Promise => { + converter.handleSafe.mockRejectedValueOnce(new Error('could not parse')); + const rep = representation('not valid', 'text/turtle'); + await expect(guard.setRepresentation(CARD, rep)).rejects.toThrow(BadRequestHttpError); + expect(source.setRepresentation).toHaveBeenCalledTimes(0); + expect(rep.data.destroyed).toBe(true); + }); + + it('rejects a card write when the conversion result is not quads.', async(): Promise => { + converter.handleSafe.mockResolvedValueOnce(representation('not quads', 'text/turtle')); + const rep = representation('data', 'text/turtle'); + await expect(guard.setRepresentation(CARD, rep)).rejects.toThrow(BadRequestHttpError); + expect(source.setRepresentation).toHaveBeenCalledTimes(0); + expect(rep.data.destroyed).toBe(true); + }); + + it('protects cards at any pod root, as the path only has to match the suffix.', async(): Promise => { + const rootCard: ResourceIdentifier = { path: 'http://example.com/profile/card' }; + const rootWebId = 'http://example.com/profile/card#me'; + registered = [ rootWebId ]; + converter.handleSafe.mockResolvedValue(quadRepresentation([ issuerQuad(rootWebId) ])); + const rep = representation('data', 'text/turtle'); + await guard.setRepresentation(rootCard, rep); + expect(webIdStore.hasWebId).toHaveBeenLastCalledWith(rootWebId); + expect(source.setRepresentation).toHaveBeenCalledTimes(1); + }); + + it('supports multiple configured WebID locations, e.g. after a config change.', async(): Promise => { + const oldCard: ResourceIdentifier = { path: 'http://example.com/alice/old/location' }; + const oldWebId = 'http://example.com/alice/old/location#this'; + registered = [ WEBID, oldWebId ]; + guard = new ProfileCardGuard( + source, + webIdStore as unknown as GuardedWebIdStore, + ISSUER, + converter as any, + [ '/profile/card#me', '/old/location#this' ], + ); + + converter.handleSafe.mockResolvedValue(quadRepresentation([ issuerQuad(oldWebId) ])); + await guard.setRepresentation(oldCard, representation('data', 'text/turtle')); + expect(webIdStore.hasWebId).toHaveBeenLastCalledWith(oldWebId); + expect(source.setRepresentation).toHaveBeenCalledTimes(1); + }); + + it('requires the issuer triple for every registered WebID on the document.', async(): Promise => { + const secondWebId = 'http://example.com/alice/profile/card#this'; + registered = [ WEBID, secondWebId ]; + guard = new ProfileCardGuard( + source, + webIdStore as unknown as GuardedWebIdStore, + ISSUER, + converter as any, + [ '/profile/card#me', '/profile/card#this' ], + ); + + // Only one of the two WebIDs keeps its issuer triple + converter.handleSafe.mockResolvedValueOnce(quadRepresentation([ issuerQuad() ])); + await expect(guard.setRepresentation(CARD, representation('data', 'text/turtle'))) + .rejects.toThrow(BadRequestHttpError); + expect(source.setRepresentation).toHaveBeenCalledTimes(0); + + // Both WebIDs keep their issuer triple + converter.handleSafe.mockResolvedValueOnce(quadRepresentation([ + issuerQuad(), + issuerQuad(secondWebId), + ])); + await guard.setRepresentation(CARD, representation('data', 'text/turtle')); + expect(source.setRepresentation).toHaveBeenCalledTimes(1); + }); + + it('validates a card created through addResource when its URL is known.', async(): Promise => { + const rep = representation('data', 'text/turtle', CARD.path); + await guard.addResource({ path: 'http://example.com/alice/' }, rep); + expect(webIdStore.hasWebId).toHaveBeenLastCalledWith(WEBID); + expect(converter.handleSafe).toHaveBeenCalledTimes(1); + expect(source.addResource).toHaveBeenCalledTimes(1); + }); + + it('does not validate addResource calls without a known URL.', async(): Promise => { + const rep = representation('data', 'text/plain'); + await guard.addResource({ path: 'http://example.com/alice/' }, rep); + // The metadata only contains a generated blank-node identifier, so nothing is validated + expect(converter.handleSafe).toHaveBeenCalledTimes(0); + expect(source.addResource).toHaveBeenCalledTimes(1); + }); + + it('supports configured WebID paths without a fragment.', async(): Promise => { + const plainCard: ResourceIdentifier = { path: 'http://example.com/alice/plain' }; + const plainWebId = 'http://example.com/alice/plain'; + registered = [ plainWebId ]; + guard = new ProfileCardGuard( + source, + webIdStore as unknown as GuardedWebIdStore, + ISSUER, + converter as any, + [ '/plain' ], + ); + converter.handleSafe.mockResolvedValue(quadRepresentation([ issuerQuad(plainWebId) ])); + await guard.setRepresentation(plainCard, representation('data', 'text/turtle')); + expect(webIdStore.hasWebId).toHaveBeenLastCalledWith(plainWebId); + expect(source.setRepresentation).toHaveBeenCalledTimes(1); + }); + + it('forbids deleting the card of a registered WebID.', async(): Promise => { + await expect(guard.deleteResource(CARD)).rejects.toThrow(ForbiddenHttpError); + expect(webIdStore.hasWebId).toHaveBeenLastCalledWith(WEBID); + expect(source.deleteResource).toHaveBeenCalledTimes(0); + }); + + it('allows deleting documents that host no registered WebID.', async(): Promise => { + registered = []; + await guard.deleteResource(CARD); + expect(source.deleteResource).toHaveBeenCalledTimes(1); + + await guard.deleteResource(OTHER); + expect(source.deleteResource).toHaveBeenCalledTimes(2); + }); +}); diff --git a/test/unit/storage/QuotaCounter.test.ts b/test/unit/storage/QuotaCounter.test.ts new file mode 100644 index 0000000..22eca40 --- /dev/null +++ b/test/unit/storage/QuotaCounter.test.ts @@ -0,0 +1,120 @@ +import { promises as fs } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import type { FileIdentifierMapper, ResourceIdentifier } from '@solid/community-server'; +import { QuotaCounter } from '../../../src/storage/quota/QuotaCounter'; +import { DuSizeReporter } from '../../../src/storage/size-reporter/DuSizeReporter'; + +const IGNORE = [ '^/\\.internal$' ]; + +function createMapper(root: string): FileIdentifierMapper { + return { + async mapUrlToFilePath(identifier: ResourceIdentifier, isMetadata: boolean): Promise { + const url = new URL(identifier.path); + const base = join(root, url.pathname); + return { + identifier, + filePath: isMetadata ? `${base}.meta` : base, + contentType: undefined, + isMetadata, + }; + }, + async mapFilePathToUrl(): Promise { + throw new Error('Not implemented'); + }, + }; +} + +// The same walk engine the counter uses for recounts. +async function expectedWalk(root: string, mapper: FileIdentifierMapper, pod: ResourceIdentifier): Promise { + return (await new DuSizeReporter(mapper, root, IGNORE).getSize(pod)).amount; +} + +const POD = { path: 'http://example.com/alice/' }; +const RESOURCE = { path: 'http://example.com/alice/foo' }; + +describe('A QuotaCounter', (): void => { + let root: string; + let mapper: FileIdentifierMapper; + + beforeEach(async(): Promise => { + root = await fs.mkdtemp(join(tmpdir(), 'quota-counter-')); + await fs.mkdir(join(root, 'alice')); + mapper = createMapper(root); + }); + + afterEach(async(): Promise => { + await fs.rm(root, { recursive: true, force: true }); + }); + + it('accumulates deltas and returns the total (O(1)).', async(): Promise => { + const counter = new QuotaCounter(mapper, root, IGNORE); + await counter.register(POD); + await counter.add(POD, 100); + await counter.add(POD, 50); + await expect(counter.getSize(POD)).resolves.toEqual({ unit: 'bytes', amount: 150 }); + }); + + it('bootstraps by walking when no counter or sidecar exists.', async(): Promise => { + await fs.writeFile(join(root, 'alice', 'a.txt'), Buffer.alloc(120)); + const counter = new QuotaCounter(mapper, root, IGNORE); + const size = await counter.getSize(POD); + expect(size.amount).toBe(await expectedWalk(root, mapper, POD)); + expect(size.amount).toBeGreaterThanOrEqual(120); + }); + + it('persists the sidecar and reloads it on a fresh instance (no re-walk).', async(): Promise => { + const first = new QuotaCounter(mapper, root, IGNORE); + await first.register(POD); + await first.add(POD, 200); + // Fresh counter — same pod, sidecar matches mtime → loaded, no walk. + const second = new QuotaCounter(mapper, root, IGNORE); + await expect(second.getSize(POD)).resolves.toEqual({ unit: 'bytes', amount: 200 }); + }); + + it('detects staleness (out-of-band change) and recounts.', async(): Promise => { + const counter = new QuotaCounter(mapper, root, IGNORE); + await counter.register(POD); + await counter.add(POD, 100); + // Out-of-band change: a direct child appears in the pod root. + await fs.writeFile(join(root, 'alice', 'extra.bin'), Buffer.alloc(400)); + // Directory mtime has microsecond granularity: if this write lands in the + // same tick as the counter's sidecar persist, the staleness check sees an + // identical mtime and skips the recount (observed on WSL2 ext4). Nudge the + // pod root's mtime so the out-of-band change is unambiguous. + const bumped = new Date(Date.now() + 2000); + await fs.utimes(join(root, 'alice'), bumped, bumped); + const size = await counter.getSize(POD); + expect(size.amount).toBe(await expectedWalk(root, mapper, POD)); + expect(size.amount).toBeGreaterThanOrEqual(400); + }); + + it('returns the size of a single resource via stat.', async(): Promise => { + await fs.writeFile(join(root, 'alice', 'foo'), Buffer.alloc(77)); + const counter = new QuotaCounter(mapper, root, IGNORE); + await expect(counter.sizeOfResource(RESOURCE)).resolves.toBe(77); + }); + + it('returns 0 for a missing resource.', async(): Promise => { + const counter = new QuotaCounter(mapper, root, IGNORE); + await expect(counter.sizeOfResource(RESOURCE)).resolves.toBe(0); + }); + + it('drops the entry and sidecar on remove, then bootstraps again.', async(): Promise => { + const counter = new QuotaCounter(mapper, root, IGNORE); + await counter.register(POD); + await counter.add(POD, 100); + await counter.remove(POD); + expect(await counter.isPodRoot(POD)).toBe(false); + // The sidecar is gone and the counter is dropped → next read re-walks. + const size = await counter.getSize(POD); + expect(size.amount).toBe(await expectedWalk(root, mapper, POD)); + }); + + it('serializes concurrent adds with a per-pod lock.', async(): Promise => { + const counter = new QuotaCounter(mapper, root, IGNORE); + await counter.register(POD); + await Promise.all([ counter.add(POD, 10), counter.add(POD, 20), counter.add(POD, 30) ]); + await expect(counter.getSize(POD)).resolves.toEqual({ unit: 'bytes', amount: 60 }); + }); +}); diff --git a/test/unit/storage/QuotaDeltaDataAccessor.test.ts b/test/unit/storage/QuotaDeltaDataAccessor.test.ts new file mode 100644 index 0000000..f634d24 --- /dev/null +++ b/test/unit/storage/QuotaDeltaDataAccessor.test.ts @@ -0,0 +1,166 @@ +import { createReadStream } from 'node:fs'; +import { promises as fs } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { + SingleRootIdentifierStrategy, +} from '@solid/community-server'; +import type { + DataAccessor, + FileIdentifierMapper, + RepresentationMetadata, + ResourceIdentifier, +} from '@solid/community-server'; +import { QuotaCounter } from '../../../src/storage/quota/QuotaCounter'; +import { QuotaDeltaDataAccessor } from '../../../src/storage/quota/QuotaDeltaDataAccessor'; +import { DuSizeReporter } from '../../../src/storage/size-reporter/DuSizeReporter'; + +const IGNORE = [ '^/\\.internal$' ]; +const PIM_STORAGE = 'http://www.w3.org/ns/pim/space#Storage'; + +function createMapper(root: string): FileIdentifierMapper { + return { + async mapUrlToFilePath(identifier: ResourceIdentifier, isMetadata: boolean): Promise { + const url = new URL(identifier.path); + const base = join(root, url.pathname); + return { identifier, filePath: isMetadata ? `${base}.meta` : base, contentType: undefined, isMetadata }; + }, + async mapFilePathToUrl(): Promise { + throw new Error('Not implemented'); + }, + }; +} + +async function readStream(stream: NodeJS.ReadableStream): Promise { + const chunks: Buffer[] = []; + for await (const chunk of stream as any) { + chunks.push(Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk)); + } + return Buffer.concat(chunks); +} + +// Recursive walk — same engine as the counter (du/Node fallback). +async function expectedWalk(root: string, mapper: FileIdentifierMapper): Promise { + return (await new DuSizeReporter(mapper, root, IGNORE).getSize({ path: 'http://example.com/alice/' })).amount; +} + +// A minimal accessor that stores to the real filesystem and reports +// pim:Storage on the pod root (so pod discovery works). +function createAccessor(root: string): DataAccessor { + const mapper = createMapper(root); + const meta = (isStorage: boolean): RepresentationMetadata => ({ + getAll: (): any[] => (isStorage ? [{ value: PIM_STORAGE }] : []), + } as any); + + return { + async canHandle(): Promise { /* no-op */ }, + async getData(identifier: ResourceIdentifier): Promise { + const { filePath } = await mapper.mapUrlToFilePath(identifier, false); + return createReadStream(filePath) as any; + }, + async getMetadata(identifier: ResourceIdentifier): Promise { + return meta(identifier.path.endsWith('/') && identifier.path !== 'http://example.com/'); + }, + getChildren(): AsyncIterableIterator { + return (async function*(): AsyncIterableIterator { })(); + }, + async writeDocument(identifier: ResourceIdentifier, data: any): Promise { + const { filePath } = await mapper.mapUrlToFilePath(identifier, false); + const buffer = await readStream(data); + await fs.mkdir(join(filePath, '..'), { recursive: true }); + await fs.writeFile(filePath, buffer); + }, + async writeContainer(identifier: ResourceIdentifier): Promise { + const { filePath } = await mapper.mapUrlToFilePath(identifier, false); + await fs.mkdir(filePath, { recursive: true }); + }, + async writeMetadata(identifier: ResourceIdentifier, metadata: RepresentationMetadata): Promise { + const { filePath } = await mapper.mapUrlToFilePath(identifier, true); + await fs.mkdir(join(filePath, '..'), { recursive: true }); + await fs.writeFile(filePath, '{}'); + }, + async deleteResource(identifier: ResourceIdentifier): Promise { + const data = await mapper.mapUrlToFilePath(identifier, false); + const meta = await mapper.mapUrlToFilePath(identifier, true); + await fs.rm(data.filePath, { recursive: true, force: true }); + await fs.rm(meta.filePath, { force: true }); + }, + }; +} + +const POD = { path: 'http://example.com/alice/' }; +const RESOURCE = { path: 'http://example.com/alice/foo' }; +const SUB = { path: 'http://example.com/alice/sub/' }; +const SUB_RESOURCE = { path: 'http://example.com/alice/sub/bar' }; + +async function writeDoc(accessor: QuotaDeltaDataAccessor, identifier: ResourceIdentifier, size: number): Promise { + const stream = (async function*(): AsyncIterableIterator { + yield Buffer.alloc(size, 1); + })(); + await accessor.writeDocument(identifier, stream as any, {} as RepresentationMetadata); +} + +describe('A QuotaDeltaDataAccessor', (): void => { + let root: string; + let counter: QuotaCounter; + let accessor: QuotaDeltaDataAccessor; + + beforeEach(async(): Promise => { + root = await fs.mkdtemp(join(tmpdir(), 'delta-')); + const mapper = createMapper(root); + const source = createAccessor(root); + counter = new QuotaCounter(mapper, root, IGNORE); + accessor = new QuotaDeltaDataAccessor( + source, + new SingleRootIdentifierStrategy('http://example.com/'), + counter, + mapper, + ); + }); + + afterEach(async(): Promise => { + await fs.rm(root, { recursive: true, force: true }); + }); + + it('tracks container creation, writes, overwrites and deletes so the counter equals a real walk.', async(): Promise => { + // Create the pod root container. + await accessor.writeContainer(POD, {} as RepresentationMetadata); + // Create a document (100 bytes). + await writeDoc(accessor, RESOURCE, 100); + expect((await counter.getSize(POD)).amount).toBe(await expectedWalk(root, createMapper(root))); + + // Overwrite it with a bigger body (150 bytes) → +50. + await writeDoc(accessor, RESOURCE, 150); + expect((await counter.getSize(POD)).amount).toBe(await expectedWalk(root, createMapper(root))); + + // Nested container + resource. + await accessor.writeContainer(SUB, {} as RepresentationMetadata); + await writeDoc(accessor, SUB_RESOURCE, 40); + expect((await counter.getSize(POD)).amount).toBe(await expectedWalk(root, createMapper(root))); + + // Write metadata for a resource (the .meta file). + await accessor.writeMetadata(RESOURCE, {} as RepresentationMetadata); + expect((await counter.getSize(POD)).amount).toBe(await expectedWalk(root, createMapper(root))); + + // Delete the document → counter drops back to the walk. + await accessor.deleteResource(RESOURCE); + expect((await counter.getSize(POD)).amount).toBe(await expectedWalk(root, createMapper(root))); + }); + + it('drops the counter entirely when the pod root itself is deleted.', async(): Promise => { + await accessor.writeContainer(POD, {} as RepresentationMetadata); + await writeDoc(accessor, RESOURCE, 50); + expect(await counter.isPodRoot(POD)).toBe(true); + await accessor.deleteResource(POD); + expect(await counter.isPodRoot(POD)).toBe(false); + }); + + it('does not track resources outside any pod (no pim:Storage).', async(): Promise => { + const outside = { path: 'http://example.com/root-file' }; + await writeDoc(accessor, outside, 999); + // No pod was registered → the file is not counted anywhere. + expect(await counter.isPodRoot({ path: 'http://example.com/' })).toBe(false); + // And it doesn't crash. + await accessor.deleteResource(outside); + }); +}); diff --git a/test/unit/storage/QuotaDeltaDataAccessorSubdomain.test.ts b/test/unit/storage/QuotaDeltaDataAccessorSubdomain.test.ts new file mode 100644 index 0000000..cd94c85 --- /dev/null +++ b/test/unit/storage/QuotaDeltaDataAccessorSubdomain.test.ts @@ -0,0 +1,156 @@ +import { createReadStream } from 'node:fs'; +import { promises as fs } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { + SubdomainIdentifierStrategy, +} from '@solid/community-server'; +import type { + DataAccessor, + FileIdentifierMapper, + RepresentationMetadata, + ResourceIdentifier, +} from '@solid/community-server'; +import { QuotaCounter } from '../../../src/storage/quota/QuotaCounter'; +import { QuotaDeltaDataAccessor } from '../../../src/storage/quota/QuotaDeltaDataAccessor'; +import { DuSizeReporter } from '../../../src/storage/size-reporter/DuSizeReporter'; + +const IGNORE = [ '^/\\.internal$' ]; +const PIM_STORAGE = 'http://www.w3.org/ns/pim/space#Storage'; +const BASE = 'http://example.com/'; +const BASE_HOST = 'example.com'; + +// Subdomain-aware mapper: http://alice.example.com/foo -> /alice/foo +function createMapper(root: string): FileIdentifierMapper { + return { + async mapUrlToFilePath(identifier: ResourceIdentifier, isMetadata: boolean): Promise { + const url = new URL(identifier.path); + const host = url.hostname; + const pod = host === BASE_HOST ? '' : host.slice(0, -(BASE_HOST.length + 1)); + const base = join(root, pod, url.pathname); + return { identifier, filePath: isMetadata ? `${base}.meta` : base, contentType: undefined, isMetadata }; + }, + async mapFilePathToUrl(): Promise { + throw new Error('Not implemented'); + }, + }; +} + +async function readStream(stream: NodeJS.ReadableStream): Promise { + const chunks: Buffer[] = []; + for await (const chunk of stream as any) { + chunks.push(Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk)); + } + return Buffer.concat(chunks); +} + +// Recursive walk — same engine as the counter (du/Node fallback). +async function expectedWalk(root: string, mapper: FileIdentifierMapper, pod: ResourceIdentifier): Promise { + return (await new DuSizeReporter(mapper, root, IGNORE).getSize(pod)).amount; +} + +// A minimal accessor that stores to the real filesystem and reports +// pim:Storage on the subdomain pod root (so pod discovery works). +function createAccessor(root: string): DataAccessor { + const mapper = createMapper(root); + const meta = (isStorage: boolean): RepresentationMetadata => ({ + getAll: (): any[] => (isStorage ? [{ value: PIM_STORAGE }] : []), + } as any); + + return { + async canHandle(): Promise { /* no-op */ }, + async getData(identifier: ResourceIdentifier): Promise { + const { filePath } = await mapper.mapUrlToFilePath(identifier, false); + return createReadStream(filePath) as any; + }, + async getMetadata(identifier: ResourceIdentifier): Promise { + // A subdomain pod root (ends with '/', is not the base root) is a storage. + return meta(identifier.path.endsWith('/') && identifier.path !== BASE); + }, + getChildren(): AsyncIterableIterator { + return (async function*(): AsyncIterableIterator { })(); + }, + async writeDocument(identifier: ResourceIdentifier, data: any): Promise { + const { filePath } = await mapper.mapUrlToFilePath(identifier, false); + const buffer = await readStream(data); + await fs.mkdir(join(filePath, '..'), { recursive: true }); + await fs.writeFile(filePath, buffer); + }, + async writeContainer(identifier: ResourceIdentifier): Promise { + const { filePath } = await mapper.mapUrlToFilePath(identifier, false); + await fs.mkdir(filePath, { recursive: true }); + }, + async writeMetadata(identifier: ResourceIdentifier, metadata: RepresentationMetadata): Promise { + const { filePath } = await mapper.mapUrlToFilePath(identifier, true); + await fs.mkdir(join(filePath, '..'), { recursive: true }); + await fs.writeFile(filePath, '{}'); + }, + async deleteResource(identifier: ResourceIdentifier): Promise { + const data = await mapper.mapUrlToFilePath(identifier, false); + const meta = await mapper.mapUrlToFilePath(identifier, true); + await fs.rm(data.filePath, { recursive: true, force: true }); + await fs.rm(meta.filePath, { force: true }); + }, + }; +} + +const POD = { path: 'http://alice.example.com/' }; +const RESOURCE = { path: 'http://alice.example.com/foo' }; + +async function writeDoc(accessor: QuotaDeltaDataAccessor, identifier: ResourceIdentifier, size: number): Promise { + const stream = (async function*(): AsyncIterableIterator { + yield Buffer.alloc(size, 1); + })(); + await accessor.writeDocument(identifier, stream as any, {} as RepresentationMetadata); +} + +describe('A QuotaDeltaDataAccessor in subdomain mode', (): void => { + let root: string; + let counter: QuotaCounter; + let accessor: QuotaDeltaDataAccessor; + + beforeEach(async(): Promise => { + root = await fs.mkdtemp(join(tmpdir(), 'delta-sub-')); + const mapper = createMapper(root); + const source = createAccessor(root); + counter = new QuotaCounter(mapper, root, IGNORE); + accessor = new QuotaDeltaDataAccessor( + source, + new SubdomainIdentifierStrategy(BASE), + counter, + mapper, + ); + }); + + afterEach(async(): Promise => { + await fs.rm(root, { recursive: true, force: true }); + }); + + it('discovers subdomain pod roots and tracks deltas (regression: CSS searchPimStorage bails at root containers).', async(): Promise => { + // Create the pod root container. In subdomain mode the pod root IS a root + // container — discovery must read its metadata (pim:Storage) before the + // root-container stop, otherwise no pod is ever found (and no counter is + // created). This used to fail: `pod registered` stayed false. + await accessor.writeContainer(POD, {} as RepresentationMetadata); + expect(await counter.isPodRoot(POD)).toBe(true); + + // Create a document (100 bytes) inside the subdomain pod. + await writeDoc(accessor, RESOURCE, 100); + expect((await counter.getSize(POD)).amount).toBe(await expectedWalk(root, createMapper(root), POD)); + + // Overwrite it with a bigger body (150 bytes) → +50. + await writeDoc(accessor, RESOURCE, 150); + expect((await counter.getSize(POD)).amount).toBe(await expectedWalk(root, createMapper(root), POD)); + + // The sidecar is persisted per pod: /alice/.internal/pivot-quota.json + const sidecar = join(root, 'alice', '.internal', 'pivot-quota.json'); + await expect(fs.stat(sidecar)).resolves.toBeTruthy(); + }); + + it('does not treat the base root as a pod (writes outside any pod are untracked).', async(): Promise => { + const baseFile = { path: 'http://example.com/root-file' }; + await writeDoc(accessor, baseFile, 999); + expect(await counter.isPodRoot({ path: BASE })).toBe(false); + await accessor.deleteResource(baseFile); + }); +}); diff --git a/test/util/Util.ts b/test/util/Util.ts index b22ea00..555763e 100644 --- a/test/util/Util.ts +++ b/test/util/Util.ts @@ -20,6 +20,7 @@ const portNames = [ 'Middleware', 'N3Patch', 'PermissionTable', + 'ProfileCardGuard', 'PodCreation', 'PodQuota', 'RedisLocker',