From e2c9ab3aded8cac07aafc908b37be1b6c29aed83 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Wed, 18 Nov 2026 11:11:16 +0100 Subject: [PATCH 1/5] feat(zarr-storage)!: share thread pools and operation limits across backends Add two resources that backends share, after TensorStore's context resources: ThreadPool runs blocking work, and Limiter caps operations in flight for threads and tasks on any event loop in one first-come, first-served queue. Pass one to several backends to bound them together; left out, a backend uses a process-wide default (ThreadPool.default(), max(4, cpu_count) threads; one Limiter(32) per obstore store class). LocalBackend takes `threads` and runs async operations and batches on it; closing the backend no longer shuts a pool down, since whoever creates a pool owns it. ObjectStoreBackend takes `concurrency`, a Limiter, and holds a slot for every operation, sync and async, single and batched, instead of bounding each async batch on its own. A slot is re-entrant within one operation, so an operation that calls another can't wait on itself. A batch started on one of its pool's own threads runs on that thread rather than wait on the pool. Pickling keeps resources shared: pickle stores each object once, and defaults unpickle as the receiving process's defaults. BREAKING CHANGE: LocalBackend's `concurrency: int` is now `threads: ThreadPool`, and ObjectStoreBackend's `concurrency` is a Limiter instead of an int. LocalBackend.aclose is gone. Assisted-by: ClaudeCode:claude-opus-5-5 --- packages/zarr-storage/DESIGN.md | 49 ++- .../zarr-storage/src/zarr_storage/__init__.py | 3 + .../zarr-storage/src/zarr_storage/_batch.py | 20 +- .../src/zarr_storage/_resources.py | 364 ++++++++++++++++++ .../src/zarr_storage/backends/_local.py | 95 +++-- .../src/zarr_storage/backends/_obstore.py | 295 ++++++++------ packages/zarr-storage/tests/test_batch.py | 9 +- packages/zarr-storage/tests/test_local.py | 5 - packages/zarr-storage/tests/test_resources.py | 299 ++++++++++++++ 9 files changed, 940 insertions(+), 199 deletions(-) create mode 100644 packages/zarr-storage/src/zarr_storage/_resources.py create mode 100644 packages/zarr-storage/tests/test_resources.py diff --git a/packages/zarr-storage/DESIGN.md b/packages/zarr-storage/DESIGN.md index 46edd3d635..62ace954e5 100644 --- a/packages/zarr-storage/DESIGN.md +++ b/packages/zarr-storage/DESIGN.md @@ -35,8 +35,10 @@ AsyncStore the same names as coroutines; `aclose()`, `async with`; `as_sync()` A **backend** is an object that implements some capability protocols (`Readable`, `Writable`, `Listable`, `WritableIfMatch`, `DeletableIfMatch`, and their async twins) on *backend paths*: -`/`-separated strings, already joined with the root. Backends never see relative keys and never join or strip paths. They hold everything that must exist once: the dict, -the filesystem root, a client, locks, concurrency limits. +`/`-separated strings, already joined with the root. Backends never see relative keys and never +join or strip paths. They hold everything that must exist once per backend: the dict, the +filesystem root, a client, locks. Thread pools and limits on operations in flight are shared +resources the backend is given (see below). A **`Store`** or **`AsyncStore`** is the only object users hold. Each is a view: a backend plus a root. All key validation, root joining, listing relativization and read-only enforcement happen in @@ -166,10 +168,12 @@ key (`a/b` when `a` is a key): that reads as absent. | `LocalBackend` | `(dev, inode, mtime_ns, size)`, with each write stamping a unique ns mtime | yes / yes, on filesystems that keep ns times | no (`KeyConflictError`) | the root | | `ObjectStoreBackend` | ETag and version (obstore's `LocalStore` ETag can repeat on ext4) | if the store supports it / not implemented | as the store does | as the store does | -Every backend has `put_if_absent` and `get_if_changed`. `LocalBackend` runs batches, sync and -async, `concurrency` operations at once on its threads; `ObjectStoreBackend` keeps `concurrency` -requests of an async batch in flight and runs sync batches one request at a time; `MemoryBackend` -uses `Store`'s serial batches, which are as fast for it. +Every backend has `put_if_absent` and `get_if_changed`. `LocalBackend` runs its async operations +and its batches, sync and async, on its `threads`, as many batch operations at once as the pool has +threads. `ObjectStoreBackend` holds a slot of its `concurrency` limiter for every operation, sync +and async, single and batched; async batches start at most `concurrency.limit` operations at once, +and sync batches run one at a time. `MemoryBackend` uses `Store`'s serial batches, which are as fast +for it. `LocalBackend` stages each write in a reserved directory under the root and renames it into place, holding the lock of the key's directory; the lock lives in that directory, so overlapping roots and @@ -210,10 +214,39 @@ step, each stored key reads back as the model says, through `get` and `get_many` `get_if_changed` with its generation is `Unchanged`. The obstore tests also fail any async run that makes a blocking obstore call. +## Shared resources + +Threads and limits on operations in flight are resources that backends share, after TensorStore's +context resources (`file_io_concurrency`, `s3_request_concurrency`). zarrs has no I/O limits of its +own (its `Resources` carry only a compute target, from Rayon's global pool); it bounds object-store +requests by wrapping the store in object_store's `LimitStore` and sharing that store. + +- `ThreadPool(max_workers)` runs blocking work: `LocalBackend(root, threads=...)`. +- `Limiter(limit)` caps operations in flight, for threads and tasks on any event loop in one + first-come, first-served queue: `ObjectStoreBackend(store, concurrency=...)`. `limiter.slot()` is + a sync or async context manager. A slot is held for a whole operation, and an operation that calls + another through a limiter it already holds takes no second slot (a context variable records the + slots an operation holds), so `get_into`, which reads through `get`, can't wait on itself. +- Pass one resource to several backends to bound them together. Left out, a backend uses a + process-wide default, as TensorStore does: `ThreadPool.default()` of `max(4, cpu_count)` threads, + and one `Limiter(32)` per obstore store class, so every S3 backend in a process shares one limit + and an HTTP backend doesn't share it. A default is shared state, but it changes only speed, never + what an operation does; pass your own resource to isolate a backend. +- Whoever creates a resource owns it: a backend never shuts down a pool it was given, and closing a + `LocalBackend` refuses new operations without stopping the pool. The default pool can't be shut + down. +- Pickling a resource pickles its settings, and pickle stores each object once, so backends that + shared a resource share one new resource after a round trip; a default unpickles as the receiving + process's default. +- A batch started on one of its pool's own threads runs on that thread, one operation at a time: + waiting there for work queued on the same pool could wait forever once every thread does it. +- Thread pools for I/O and for compute should stay separate (TensorStore's `file_io_concurrency` + and `data_copy_concurrency`): compute waiting on I/O in one pool starves it. + ## Configuration -No global configuration. Every knob (local I/O concurrency, how many requests an obstore batch -keeps in flight) is a constructor parameter with its default in the signature. +No global configuration. Every knob is a constructor parameter with its default in the signature; +the shared resources above are passed the same way. ## Open diff --git a/packages/zarr-storage/src/zarr_storage/__init__.py b/packages/zarr-storage/src/zarr_storage/__init__.py index 510c1e6ef0..d18fb9ee4a 100644 --- a/packages/zarr-storage/src/zarr_storage/__init__.py +++ b/packages/zarr-storage/src/zarr_storage/__init__.py @@ -39,6 +39,7 @@ WritableMany, WritableManyAsync, ) +from zarr_storage._resources import Limiter, ThreadPool from zarr_storage._store import AsyncStore, Store from zarr_storage._types import ( ByteRange, @@ -79,6 +80,7 @@ "InvalidKeyError", "Key", "KeyConflictError", + "Limiter", "ListItem", "Listable", "ListableAsync", @@ -101,6 +103,7 @@ "Store", "StoreClosedError", "SuffixRequest", + "ThreadPool", "Unchanged", "UnsupportedOperationError", "Writable", diff --git a/packages/zarr-storage/src/zarr_storage/_batch.py b/packages/zarr-storage/src/zarr_storage/_batch.py index e850519e57..c3a9145599 100644 --- a/packages/zarr-storage/src/zarr_storage/_batch.py +++ b/packages/zarr-storage/src/zarr_storage/_batch.py @@ -14,7 +14,9 @@ if TYPE_CHECKING: from collections.abc import AsyncIterator, Callable, Coroutine, Iterator, Sequence - from concurrent.futures import Executor, Future + from concurrent.futures import Future + + from zarr_storage._resources import ThreadPool __all__ = ["run_many", "run_many_async"] @@ -27,13 +29,17 @@ def run_many( fn: Callable[[Unpack[Ts]], R], requests: Sequence[tuple[*Ts]], - executor: Executor | None = None, + pool: ThreadPool | None = None, window: int = 1, ) -> Iterator[tuple[int, R]]: - """Call `fn(*request)` for each request: one at a time, or on `executor`, `window` at once.""" - if executor is None: + """Call `fn(*request)` for each request: one at a time, or on `pool`, `window` at once. + + Called from one of `pool`'s own threads, it runs one at a time on that thread: waiting there + for work queued behind it on the same pool could wait forever. + """ + if pool is None or pool.in_worker(): return _run_serially(fn, requests) - return _run_on(executor, fn, requests, window) + return _run_on(pool, fn, requests, window) def _run_serially( @@ -48,7 +54,7 @@ def _run_serially( def _run_on( - executor: Executor, + pool: ThreadPool, fn: Callable[[Unpack[Ts]], R], requests: Sequence[tuple[*Ts]], window: int, @@ -59,7 +65,7 @@ def _run_on( try: while pending or (not errors and started < len(requests)): while not errors and started < len(requests) and len(pending) < window: - pending[executor.submit(fn, *requests[started])] = started + pending[pool.submit(fn, *requests[started])] = started started += 1 done, _ = wait(pending, return_when=FIRST_COMPLETED) for future in done: diff --git a/packages/zarr-storage/src/zarr_storage/_resources.py b/packages/zarr-storage/src/zarr_storage/_resources.py new file mode 100644 index 0000000000..5f4671e01b --- /dev/null +++ b/packages/zarr-storage/src/zarr_storage/_resources.py @@ -0,0 +1,364 @@ +"""Resources that backends share: thread pools and limits on operations in flight. + +Modeled on TensorStore's context resources (`file_io_concurrency`, `s3_request_concurrency`): a +resource is an object you create and pass to as many backends as should share it. A backend never +closes a resource it was given. Left out, a backend uses a process-wide default, as TensorStore +does, because threads and request slots are process-wide in the end; pass your own to isolate a +backend from the rest of the process. A resource changes how fast operations run, never what they +do. + +Pickling a resource pickles its settings. Pickle stores each object once, so backends that shared +a resource before pickling share one new resource after; a default resource unpickles as the +default of the receiving process. +""" + +from __future__ import annotations + +import asyncio +import os +import threading +from collections import deque +from concurrent.futures import ThreadPoolExecutor +from contextvars import ContextVar +from functools import partial +from typing import TYPE_CHECKING, Final, TypeVar, TypeVarTuple + +if TYPE_CHECKING: + from collections.abc import Callable + from concurrent.futures import Future + from contextvars import Token + from types import TracebackType + +__all__ = ["Limiter", "Slot", "ThreadPool", "default_limiter", "default_thread_pool"] + +R = TypeVar("R") +Ts = TypeVarTuple("Ts") + +DEFAULT_THREADS: Final = max(4, os.cpu_count() or 1) +"""Threads in the default pool, as TensorStore's `file_io_concurrency` default.""" + +DEFAULT_LIMIT: Final = 32 +"""Operations in flight under a default limiter, as TensorStore's `s3_request_concurrency`.""" + + +def _check_size(name: str, value: int) -> None: + if type(value) is not int: + raise TypeError(f"{name} must be an int; got {type(value).__name__}.") + if value < 1: + raise ValueError(f"{name} must be >= 1; got {value}.") + + +# ---------------------------------------------------------------------------- thread pools + +_worker = threading.local() +"""Per thread: the `ThreadPool` the thread works for, if any.""" + + +def _mark_worker(pool: ThreadPool) -> None: + _worker.pool = pool + + +class ThreadPool: + """Threads that run blocking operations, such as file I/O, for the backends given it. + + The threads start when first needed. Whoever creates a pool owns it and may `shutdown` it once + no backend needs it; the default pool lives as long as the process. + + Parameters + ---------- + max_workers + The most threads the pool runs at once. + """ + + def __init__(self, max_workers: int) -> None: + _check_size("max_workers", max_workers) + self._max_workers = max_workers + self._lock = threading.Lock() + self._executor: ThreadPoolExecutor | None = None + + @classmethod + def default(cls) -> ThreadPool: + """The process-wide pool that backends use when given none.""" + return default_thread_pool() + + @property + def max_workers(self) -> int: + return self._max_workers + + def __repr__(self) -> str: + return f"{type(self).__name__}(max_workers={self._max_workers})" + + def __reduce__(self) -> tuple[object, ...]: + return (ThreadPool, (self._max_workers,)) + + def _pool(self) -> ThreadPoolExecutor: + with self._lock: + if self._executor is None: + self._executor = ThreadPoolExecutor( + self._max_workers, + thread_name_prefix="zarr-storage", + initializer=_mark_worker, + initargs=(self,), + ) + return self._executor + + def in_worker(self) -> bool: + """Whether the calling thread is one of this pool's. + + Work started from one of the pool's own threads must not wait for other work queued on the + pool, which may never start; the batch runners run such work inline. + """ + return getattr(_worker, "pool", None) is self + + def submit(self, fn: Callable[[*Ts], R], /, *args: *Ts) -> Future[R]: + """Run `fn(*args)` on one of the pool's threads.""" + return self._pool().submit(fn, *args) + + async def run(self, fn: Callable[[*Ts], R], /, *args: *Ts) -> R: + """Await `fn(*args)`, run on one of the pool's threads.""" + loop = asyncio.get_running_loop() + return await loop.run_in_executor(self._pool(), partial(fn, *args)) + + def shutdown(self, *, wait: bool = True) -> None: + """Stop the pool's threads; with `wait`, after the work already given them. + + The pool starts new threads if it is used again. Raises `ValueError` for the default pool, + which other code may be using. + """ + with self._lock: + executor, self._executor = self._executor, None + if executor is not None: + executor.shutdown(wait=wait) + + +class _DefaultThreadPool(ThreadPool): + """The process-wide pool: never shut down, and unpickled as the receiver's own default.""" + + def __reduce__(self) -> tuple[object, ...]: + return (default_thread_pool, ()) + + def shutdown(self, *, wait: bool = True) -> None: + raise ValueError("The default thread pool is shared by the whole process.") + + +_default_pool_lock = threading.Lock() +_default_pool: list[ThreadPool] = [] + + +def default_thread_pool() -> ThreadPool: + """The process-wide `ThreadPool`, with `DEFAULT_THREADS` threads.""" + with _default_pool_lock: + if not _default_pool: + _default_pool.append(_DefaultThreadPool(DEFAULT_THREADS)) + return _default_pool[0] + + +# ---------------------------------------------------------------------------- limiters + +_held: ContextVar[frozenset[Limiter]] = ContextVar( + "zarr_storage_held_limiters", default=frozenset() +) +"""The limiters whose slots the current operation holds, so nested operations don't wait again.""" + + +class _ThreadWaiter: + """A thread blocked on a limiter.""" + + def __init__(self) -> None: + self._event = threading.Event() + + def grant(self) -> bool: + self._event.set() + return True + + def wait(self) -> None: + self._event.wait() + + +def _resolve(future: asyncio.Future[None]) -> None: + if not future.done(): + future.set_result(None) + + +class _TaskWaiter: + """A task blocked on a limiter, on its own event loop. Fields change under the limiter's lock.""" + + def __init__(self, loop: asyncio.AbstractEventLoop) -> None: + self.loop = loop + self.future: asyncio.Future[None] = loop.create_future() + self.granted = False + self.cancelled = False + + def grant(self) -> bool: + if self.cancelled: + return False + try: + self.loop.call_soon_threadsafe(_resolve, self.future) + except RuntimeError: # the loop is closed; nothing waits any more + return False + self.granted = True + return True + + +class Limiter: + """A limit on how many operations run at once, across every backend given it. + + Threads and tasks on any event loop wait in one first-come, first-served queue. + + Parameters + ---------- + limit + The most operations that run at once. + """ + + def __init__(self, limit: int) -> None: + _check_size("limit", limit) + self._limit = limit + self._lock = threading.Lock() + self._in_use = 0 + self._waiters: deque[_ThreadWaiter | _TaskWaiter] = deque() + + @property + def limit(self) -> int: + return self._limit + + @property + def in_use(self) -> int: + """How many slots are held now.""" + return self._in_use + + def __repr__(self) -> str: + return f"{type(self).__name__}(limit={self._limit}, in_use={self._in_use})" + + def __reduce__(self) -> tuple[object, ...]: + return (Limiter, (self._limit,)) + + def slot(self) -> Slot: + """A context manager, sync (`with`) or async (`async with`), that holds one slot. + + Prefer it to `acquire` and `release`: it releases on error, and inside an operation that + already holds a slot of this limiter it takes no second one, so an operation that calls + another can't wait on itself. + """ + return Slot(self) + + def _take(self) -> bool: + """Take a free slot if no one is waiting; called under the lock.""" + if self._in_use < self._limit and not self._waiters: + self._in_use += 1 + return True + return False + + def acquire(self) -> None: + """Take a slot, blocking the calling thread until one is free.""" + with self._lock: + if self._take(): + return + waiter = _ThreadWaiter() + self._waiters.append(waiter) + waiter.wait() + + async def acquire_async(self) -> None: + """Take a slot, waiting without blocking the event loop until one is free.""" + with self._lock: + if self._take(): + return + waiter = _TaskWaiter(asyncio.get_running_loop()) + self._waiters.append(waiter) + try: + await waiter.future + except asyncio.CancelledError: + with self._lock: + granted = waiter.granted + if not granted: + waiter.cancelled = True + self._waiters.remove(waiter) + if granted: + self.release() + raise + + def release(self) -> None: + """Give back a slot, to the longest waiter if there is one. + + Raises `RuntimeError` if no slot is held. + """ + with self._lock: + while self._waiters: + if self._waiters.popleft().grant(): + return # the slot passes to the waiter + if self._in_use == 0: + raise RuntimeError("Released a limiter slot that was not held.") + self._in_use -= 1 + + +class Slot: + """One use of a `Limiter`: holds a slot from entry to exit, unless the operation holds one.""" + + __slots__ = ("_limiter", "_token") + + def __init__(self, limiter: Limiter) -> None: + self._limiter = limiter + self._token: Token[frozenset[Limiter]] | None = None + + def _needed(self) -> bool: + return self._limiter not in _held.get() + + def _mark(self) -> None: + self._token = _held.set(_held.get() | {self._limiter}) + + def _unmark(self) -> None: + if self._token is None: + return + _held.reset(self._token) + self._token = None + self._limiter.release() + + def __enter__(self) -> None: + if self._needed(): + self._limiter.acquire() + self._mark() + + def __exit__( + self, + exc_type: type[BaseException] | None, + exc: BaseException | None, + tb: TracebackType | None, + ) -> None: + self._unmark() + + async def __aenter__(self) -> None: + if self._needed(): + await self._limiter.acquire_async() + self._mark() + + async def __aexit__( + self, + exc_type: type[BaseException] | None, + exc: BaseException | None, + tb: TracebackType | None, + ) -> None: + self._unmark() + + +class _DefaultLimiter(Limiter): + """A process-wide limiter for one key; unpickled as the receiver's limiter for that key.""" + + def __init__(self, key: str) -> None: + super().__init__(DEFAULT_LIMIT) + self._key = key + + def __reduce__(self) -> tuple[object, ...]: + return (default_limiter, (self._key,)) + + +_default_limiters_lock = threading.Lock() +_default_limiters: dict[str, Limiter] = {} + + +def default_limiter(key: str) -> Limiter: + """The process-wide `Limiter` for `key` (such as an obstore store class), of `DEFAULT_LIMIT`.""" + with _default_limiters_lock: + limiter = _default_limiters.get(key) + if limiter is None: + limiter = _default_limiters[key] = _DefaultLimiter(key) + return limiter diff --git a/packages/zarr-storage/src/zarr_storage/backends/_local.py b/packages/zarr-storage/src/zarr_storage/backends/_local.py index e6a22f68eb..5545ce5f02 100644 --- a/packages/zarr-storage/src/zarr_storage/backends/_local.py +++ b/packages/zarr-storage/src/zarr_storage/backends/_local.py @@ -2,7 +2,6 @@ from __future__ import annotations -import asyncio import contextlib import errno import os @@ -12,10 +11,8 @@ import threading import time import uuid -from concurrent.futures import ThreadPoolExecutor -from functools import partial from pathlib import Path -from typing import TYPE_CHECKING, ParamSpec, TypedDict, TypeVar +from typing import TYPE_CHECKING, TypedDict, TypeVar, TypeVarTuple from zarr_storage._batch import run_many, run_many_async from zarr_storage._buffers import byte_view @@ -26,6 +23,7 @@ StoreClosedError, UnsupportedOperationError, ) +from zarr_storage._resources import ThreadPool, default_thread_pool from zarr_storage._types import ( Copied, Deleted, @@ -73,14 +71,14 @@ ) _WINDOWS_FORBIDDEN_CHARACTERS = frozenset('<>:"\\|?*') | frozenset(chr(i) for i in range(32)) -P = ParamSpec("P") +Ts = TypeVarTuple("Ts") R = TypeVar("R") T = TypeVar("T") class _State(TypedDict): root: str - concurrency: int + threads: ThreadPool closed: bool @@ -396,24 +394,26 @@ class LocalBackend: create Create `root` if it doesn't exist. With `create=False`, a missing root raises `FileNotFoundError`. - concurrency - The number of threads that run the async methods' file operations, and the number of - operations a batch (`get_many`, `put_many`, ...) runs at once. + threads + The threads that run the async methods' file operations and the operations of batches + (`get_many`, `put_many`, ...), which run as many at once as the pool has threads. Share one + pool between backends to bound their threads together; the default is the process-wide + `ThreadPool.default()`. The backend never shuts the pool down. """ def __init__( - self, root: str | os.PathLike[str], *, create: bool = True, concurrency: int = 16 + self, + root: str | os.PathLike[str], + *, + create: bool = True, + threads: ThreadPool | None = None, ) -> None: - if concurrency < 1: - raise ValueError(f"concurrency must be >= 1; got {concurrency}.") self._root = Path(root).resolve() if create: self._root.mkdir(parents=True, exist_ok=True) elif not self._root.is_dir(): raise FileNotFoundError(f"The root directory {str(self._root)!r} does not exist.") - self._concurrency = concurrency - self._executor: ThreadPoolExecutor | None = None - self._executor_lock = threading.Lock() + self._threads = default_thread_pool() if threads is None else threads self._probe_lock = threading.Lock() self._exact_times: bool | None = None self._closed = False @@ -423,17 +423,20 @@ def root(self) -> Path: """The directory that holds the keys.""" return self._root + @property + def threads(self) -> ThreadPool: + """The threads that run the async methods and batches.""" + return self._threads + def __repr__(self) -> str: return f"LocalBackend({str(self._root)!r})" def __getstate__(self) -> _State: - return {"root": str(self._root), "concurrency": self._concurrency, "closed": self._closed} + return {"root": str(self._root), "threads": self._threads, "closed": self._closed} def __setstate__(self, state: _State) -> None: self._root = Path(state["root"]) - self._concurrency = state["concurrency"] - self._executor = None - self._executor_lock = threading.Lock() + self._threads = state["threads"] self._probe_lock = threading.Lock() self._exact_times = None self._closed = state["closed"] @@ -449,15 +452,8 @@ def writable(self) -> bool: return True def close(self) -> None: - with self._executor_lock: - self._closed = True - executor, self._executor = self._executor, None - if executor is not None: - executor.shutdown(wait=True) - - async def aclose(self) -> None: - """`close` on a worker thread: it waits for running file operations to finish.""" - await asyncio.to_thread(self.close) + """Refuse further operations. Operations already running finish; the pool stays up.""" + self._closed = True # --- paths @@ -710,39 +706,38 @@ def list_dir(self, directory: str) -> Iterator[ListItem]: return iter(()) return _classify(entries) - # --- batches: the operations above, `concurrency` at once on the backend's threads + # --- batches: the operations above, as many at once as the pool has threads def get_many( self, requests: Sequence[tuple[str, ByteRange | None]] ) -> Iterator[tuple[int, Found | Missing]]: - return run_many(self.get, requests, self._pool(), self._concurrency) + return run_many(self.get, requests, self._open_threads(), self._threads.max_workers) def get_into_many( self, requests: Sequence[tuple[str, Buffer, ByteRange | None]] ) -> Iterator[tuple[int, Copied | Missing]]: - return run_many(self.get_into, requests, self._pool(), self._concurrency) + return run_many(self.get_into, requests, self._open_threads(), self._threads.max_workers) def put_many(self, requests: Sequence[tuple[str, memoryview]]) -> Iterator[tuple[int, Written]]: - return run_many(self.put, requests, self._pool(), self._concurrency) + return run_many(self.put, requests, self._open_threads(), self._threads.max_workers) def delete_many(self, paths: Sequence[str]) -> Iterator[tuple[int, Deleted | Missing]]: - return run_many(self.delete, [(path,) for path in paths], self._pool(), self._concurrency) + return run_many( + self.delete, + [(path,) for path in paths], + self._open_threads(), + self._threads.max_workers, + ) # --- async twins: run the sync methods on the backend's threads - def _pool(self) -> ThreadPoolExecutor: - with self._executor_lock: - if self._closed: - raise StoreClosedError(f"{self!r} is closed.") - if self._executor is None: - self._executor = ThreadPoolExecutor( - max_workers=self._concurrency, thread_name_prefix="zarr-storage-local" - ) - return self._executor + def _open_threads(self) -> ThreadPool: + if self._closed: + raise StoreClosedError(f"{self!r} is closed.") + return self._threads - async def _run(self, fn: Callable[P, T], *args: P.args, **kwargs: P.kwargs) -> T: - loop = asyncio.get_running_loop() - return await loop.run_in_executor(self._pool(), partial(fn, *args, **kwargs)) + async def _run(self, fn: Callable[[*Ts], T], *args: *Ts) -> T: + return await self._open_threads().run(fn, *args) async def get_async(self, path: str, byte_range: ByteRange | None) -> Found | Missing: return await self._run(self.get, path, byte_range) @@ -790,22 +785,24 @@ async def delete_dir_async(self, directory: str) -> None: def get_many_async( self, requests: Sequence[tuple[str, ByteRange | None]] ) -> AsyncIterator[tuple[int, Found | Missing]]: - return run_many_async(self.get_async, requests, self._concurrency) + return run_many_async(self.get_async, requests, self._threads.max_workers) def get_into_many_async( self, requests: Sequence[tuple[str, Buffer, ByteRange | None]] ) -> AsyncIterator[tuple[int, Copied | Missing]]: - return run_many_async(self.get_into_async, requests, self._concurrency) + return run_many_async(self.get_into_async, requests, self._threads.max_workers) def put_many_async( self, requests: Sequence[tuple[str, memoryview]] ) -> AsyncIterator[tuple[int, Written]]: - return run_many_async(self.put_async, requests, self._concurrency) + return run_many_async(self.put_async, requests, self._threads.max_workers) def delete_many_async( self, paths: Sequence[str] ) -> AsyncIterator[tuple[int, Deleted | Missing]]: - return run_many_async(self.delete_async, [(path,) for path in paths], self._concurrency) + return run_many_async( + self.delete_async, [(path,) for path in paths], self._threads.max_workers + ) async def list_async(self, directory: str) -> AsyncIterator[str]: for path in await self._run(_materialize, self.list(directory)): diff --git a/packages/zarr-storage/src/zarr_storage/backends/_obstore.py b/packages/zarr-storage/src/zarr_storage/backends/_obstore.py index 0bdb8fbae4..27e7ed771f 100644 --- a/packages/zarr-storage/src/zarr_storage/backends/_obstore.py +++ b/packages/zarr-storage/src/zarr_storage/backends/_obstore.py @@ -29,6 +29,7 @@ from zarr_storage._byte_range import resolve from zarr_storage._errors import InvalidKeyError, KeyConflictError, UnsupportedOperationError from zarr_storage._keys import check_key, relativize +from zarr_storage._resources import Limiter, default_limiter from zarr_storage._types import ( Deleted, Found, @@ -268,17 +269,26 @@ class ObjectStoreBackend: writable Whether to accept writes. Defaults to `False` for `HTTPStore`, `True` otherwise. concurrency - The number of requests an async batch (`get_many_async`, `put_many_async`, ...) keeps in - flight. Sync batches run one request at a time. + The limit on this backend's operations in flight, sync and async, single and batched + (listings are not counted). Share one `Limiter` between backends to bound them together. + The default is a process-wide limiter for the store's class, so that every S3 backend in + the process shares one limit, as TensorStore's `s3_request_concurrency` does. Async batches + start at most `concurrency.limit` operations at once; sync batches run one at a time. """ def __init__( - self, store: ObjectStore, *, writable: bool | None = None, concurrency: int = 32 + self, + store: ObjectStore, + *, + writable: bool | None = None, + concurrency: Limiter | None = None, ) -> None: - if concurrency < 1: - raise ValueError(f"concurrency must be >= 1; got {concurrency}.") self._store = store - self._concurrency = concurrency + self._concurrency = ( + default_limiter(f"obstore.{type(store).__name__}") + if concurrency is None + else concurrency + ) self._writable = not isinstance(store, HTTPStore) if writable is None else writable # Only a filesystem refuses a path that lies under a file; elsewhere a refusal is an error. self._filesystem = isinstance(store, LocalStore) @@ -289,6 +299,11 @@ def store(self) -> ObjectStore: """The obstore store.""" return self._store + @property + def concurrency(self) -> Limiter: + """The limit on this backend's operations in flight.""" + return self._concurrency + def __repr__(self) -> str: return f"ObjectStoreBackend({self._store!r})" @@ -403,59 +418,67 @@ async def _conflicts_async(self, path: str) -> bool: # --- Readable def head(self, path: str) -> ObjectMetadata | Missing: - try: - return _metadata(obstore.head(self._store, path)) - except FileNotFoundError: - return Missing(time.time()) - except GenericError as e: - if self._refused_under_a_key(e, path): + with self._concurrency.slot(): + try: + return _metadata(obstore.head(self._store, path)) + except FileNotFoundError: return Missing(time.time()) - raise + except GenericError as e: + if self._refused_under_a_key(e, path): + return Missing(time.time()) + raise async def head_async(self, path: str) -> ObjectMetadata | Missing: - try: - return _metadata(await obstore.head_async(self._store, path)) - except FileNotFoundError: - return Missing(time.time()) - except GenericError as e: - if await self._refused_under_a_key_async(e, path): + async with self._concurrency.slot(): + try: + return _metadata(await obstore.head_async(self._store, path)) + except FileNotFoundError: return Missing(time.time()) - raise + except GenericError as e: + if await self._refused_under_a_key_async(e, path): + return Missing(time.time()) + raise def get(self, path: str, byte_range: ByteRange | None) -> Found | Missing: - return self._get(path, byte_range, {}) + with self._concurrency.slot(): + return self._get(path, byte_range, {}) async def get_async(self, path: str, byte_range: ByteRange | None) -> Found | Missing: - return await self._get_async(path, byte_range, {}) + async with self._concurrency.slot(): + return await self._get_async(path, byte_range, {}) def get_into(self, path: str, out: Buffer, byte_range: ByteRange | None) -> Copied | Missing: """A read, copied into `out`: obstore can't read into a caller's buffer.""" - view = byte_view(out, writable=True) - return copy_into(self.get(path, byte_range), view) + with self._concurrency.slot(): + view = byte_view(out, writable=True) + return copy_into(self.get(path, byte_range), view) async def get_into_async( self, path: str, out: Buffer, byte_range: ByteRange | None ) -> Copied | Missing: - view = byte_view(out, writable=True) - return copy_into(await self.get_async(path, byte_range), view) + async with self._concurrency.slot(): + view = byte_view(out, writable=True) + return copy_into(await self.get_async(path, byte_range), view) def get_if_changed( self, path: str, generation: Generation, byte_range: ByteRange | None ) -> Found | Missing | Unchanged: - try: - result = self._get(path, byte_range, _if_none_match(generation)) - except NotModifiedError: - return Unchanged(generation, time.time()) - return _unless_unchanged(result, generation) + with self._concurrency.slot(): + try: + result = self._get(path, byte_range, _if_none_match(generation)) + except NotModifiedError: + return Unchanged(generation, time.time()) + return _unless_unchanged(result, generation) async def get_if_changed_async( self, path: str, generation: Generation, byte_range: ByteRange | None ) -> Found | Missing | Unchanged: - try: - result = await self._get_async(path, byte_range, _if_none_match(generation)) - except NotModifiedError: - return Unchanged(generation, time.time()) - return _unless_unchanged(result, generation) + async with self._concurrency.slot(): + try: + result = await self._get_async(path, byte_range, _if_none_match(generation)) + except NotModifiedError: + return Unchanged(generation, time.time()) + return _unless_unchanged(result, generation) def _get(self, path: str, byte_range: ByteRange | None, options: GetOptions) -> Found | Missing: """A read with extra `options`; raises `NotModifiedError` if they include a matching @@ -486,26 +509,29 @@ async def _get_async( return _found(result, memoryview(await result.bytes_async())) def get_ranges(self, path: str, ranges: Sequence[ByteRange]) -> FoundRanges | Missing: - if len(ranges) == 0 or _is_empty_request(ranges[0]): - return self._read_whole_and_slice(path, ranges) - try: - first = obstore.get(self._store, path, options=_get_options(ranges[0])) - except FileNotFoundError: - return Missing(time.time()) - except GenericError: - # The first range starts at the end of the value; one whole read answers every range. - return self._read_whole_and_slice(path, ranges) - meta = first.meta - generation = _generation(meta["e_tag"], meta["version"]) - pinned = _pinned(generation) - if pinned is None: - return self._read_whole_and_slice(path, ranges) - try: - rest = [self._read_pinned(path, pinned, *resolve(r, meta["size"])) for r in ranges[1:]] - except (PreconditionError, FileNotFoundError): - # The value changed after the first read; one whole read is consistent by itself. - return self._read_whole_and_slice(path, ranges) - return FoundRanges((memoryview(first.bytes()), *rest), generation, time.time()) + with self._concurrency.slot(): + if len(ranges) == 0 or _is_empty_request(ranges[0]): + return self._read_whole_and_slice(path, ranges) + try: + first = obstore.get(self._store, path, options=_get_options(ranges[0])) + except FileNotFoundError: + return Missing(time.time()) + except GenericError: + # The first range starts at the end of the value; one whole read answers every range. + return self._read_whole_and_slice(path, ranges) + meta = first.meta + generation = _generation(meta["e_tag"], meta["version"]) + pinned = _pinned(generation) + if pinned is None: + return self._read_whole_and_slice(path, ranges) + try: + rest = [ + self._read_pinned(path, pinned, *resolve(r, meta["size"])) for r in ranges[1:] + ] + except (PreconditionError, FileNotFoundError): + # The value changed after the first read; one whole read is consistent by itself. + return self._read_whole_and_slice(path, ranges) + return FoundRanges((memoryview(first.bytes()), *rest), generation, time.time()) def _read_pinned(self, path: str, pinned: GetOptions, start: int, end: int) -> memoryview: if start == end: @@ -531,29 +557,32 @@ def _read_whole_and_slice( async def get_ranges_async( self, path: str, ranges: Sequence[ByteRange] ) -> FoundRanges | Missing: - if len(ranges) == 0 or _is_empty_request(ranges[0]): - return await self._read_whole_and_slice_async(path, ranges) - try: - first = await obstore.get_async(self._store, path, options=_get_options(ranges[0])) - except FileNotFoundError: - return Missing(time.time()) - except GenericError: - return await self._read_whole_and_slice_async(path, ranges) - meta = first.meta - generation = _generation(meta["e_tag"], meta["version"]) - pinned = _pinned(generation) - if pinned is None: - return await self._read_whole_and_slice_async(path, ranges) - try: - rest = await asyncio.gather( - *( - _read_pinned_async(self._store, path, pinned, *resolve(r, meta["size"])) - for r in ranges[1:] + async with self._concurrency.slot(): + if len(ranges) == 0 or _is_empty_request(ranges[0]): + return await self._read_whole_and_slice_async(path, ranges) + try: + first = await obstore.get_async(self._store, path, options=_get_options(ranges[0])) + except FileNotFoundError: + return Missing(time.time()) + except GenericError: + return await self._read_whole_and_slice_async(path, ranges) + meta = first.meta + generation = _generation(meta["e_tag"], meta["version"]) + pinned = _pinned(generation) + if pinned is None: + return await self._read_whole_and_slice_async(path, ranges) + try: + rest = await asyncio.gather( + *( + _read_pinned_async(self._store, path, pinned, *resolve(r, meta["size"])) + for r in ranges[1:] + ) ) + except (PreconditionError, FileNotFoundError): + return await self._read_whole_and_slice_async(path, ranges) + return FoundRanges( + (memoryview(await first.bytes_async()), *rest), generation, time.time() ) - except (PreconditionError, FileNotFoundError): - return await self._read_whole_and_slice_async(path, ranges) - return FoundRanges((memoryview(await first.bytes_async()), *rest), generation, time.time()) async def _read_whole_and_slice_async( self, path: str, ranges: Sequence[ByteRange] @@ -573,32 +602,38 @@ async def _read_whole_and_slice_async( # --- Writable def put(self, path: str, value: memoryview) -> Written: - return self._put(path, value, None, always) + with self._concurrency.slot(): + return self._put(path, value, None, always) async def put_async(self, path: str, value: memoryview) -> Written: - return await self._put_async(path, value, None, always) + async with self._concurrency.slot(): + return await self._put_async(path, value, None, always) def put_if_absent(self, path: str, value: memoryview) -> Written | Rejected: - return self._put(path, value, "create", if_absent) + with self._concurrency.slot(): + return self._put(path, value, "create", if_absent) async def put_if_absent_async(self, path: str, value: memoryview) -> Written | Rejected: - return await self._put_async(path, value, "create", if_absent) + async with self._concurrency.slot(): + return await self._put_async(path, value, "create", if_absent) def put_if_match( self, path: str, value: memoryview, generation: Generation ) -> Written | Rejected | Missing: - mode = _if_match_mode(generation) - if mode is None: - return _refuse(self.head(path)) - return self._put(path, value, mode, IfMatch(generation)) + with self._concurrency.slot(): + mode = _if_match_mode(generation) + if mode is None: + return _refuse(self.head(path)) + return self._put(path, value, mode, IfMatch(generation)) async def put_if_match_async( self, path: str, value: memoryview, generation: Generation ) -> Written | Rejected | Missing: - mode = _if_match_mode(generation) - if mode is None: - return _refuse(await self.head_async(path)) - return await self._put_async(path, value, mode, IfMatch(generation)) + async with self._concurrency.slot(): + mode = _if_match_mode(generation) + if mode is None: + return _refuse(await self.head_async(path)) + return await self._put_async(path, value, mode, IfMatch(generation)) def _put( self, @@ -676,67 +711,73 @@ async def _put_async( raise OSError(f"Gave up writing {path!r}: the store kept refusing it.") def delete(self, path: str) -> Deleted | Missing: - meta = self.head(path) - if isinstance(meta, Missing): - return meta - try: - obstore.delete(self._store, path) - except FileNotFoundError: - return Missing(time.time()) - return Deleted(meta.generation) + with self._concurrency.slot(): + meta = self.head(path) + if isinstance(meta, Missing): + return meta + try: + obstore.delete(self._store, path) + except FileNotFoundError: + return Missing(time.time()) + return Deleted(meta.generation) async def delete_async(self, path: str) -> Deleted | Missing: - meta = await self.head_async(path) - if isinstance(meta, Missing): - return meta - try: - await obstore.delete_async(self._store, path) - except FileNotFoundError: - return Missing(time.time()) - return Deleted(meta.generation) + async with self._concurrency.slot(): + meta = await self.head_async(path) + if isinstance(meta, Missing): + return meta + try: + await obstore.delete_async(self._store, path) + except FileNotFoundError: + return Missing(time.time()) + return Deleted(meta.generation) def delete_dir(self, directory: str) -> None: - paths = list(self._paths(directory)) - try: - if paths: - obstore.delete(self._store, paths) - except FileNotFoundError: - # A concurrent delete removed some first; delete the rest one at a time. - for path in paths: - with contextlib.suppress(FileNotFoundError): - obstore.delete(self._store, path) + with self._concurrency.slot(): + paths = list(self._paths(directory)) + try: + if paths: + obstore.delete(self._store, paths) + except FileNotFoundError: + # A concurrent delete removed some first; delete the rest one at a time. + for path in paths: + with contextlib.suppress(FileNotFoundError): + obstore.delete(self._store, path) async def delete_dir_async(self, directory: str) -> None: - paths = [path async for path in self._paths_async(directory)] - try: - if paths: - await obstore.delete_async(self._store, paths) - except FileNotFoundError: - for path in paths: - with contextlib.suppress(FileNotFoundError): - await obstore.delete_async(self._store, path) + async with self._concurrency.slot(): + paths = [path async for path in self._paths_async(directory)] + try: + if paths: + await obstore.delete_async(self._store, paths) + except FileNotFoundError: + for path in paths: + with contextlib.suppress(FileNotFoundError): + await obstore.delete_async(self._store, path) - # --- async batches: the async operations, `concurrency` at once + # --- async batches: the async operations, at most `concurrency.limit` started at once def get_many_async( self, requests: Sequence[tuple[str, ByteRange | None]] ) -> AsyncIterator[tuple[int, Found | Missing]]: - return run_many_async(self.get_async, requests, self._concurrency) + return run_many_async(self.get_async, requests, self._concurrency.limit) def get_into_many_async( self, requests: Sequence[tuple[str, Buffer, ByteRange | None]] ) -> AsyncIterator[tuple[int, Copied | Missing]]: - return run_many_async(self.get_into_async, requests, self._concurrency) + return run_many_async(self.get_into_async, requests, self._concurrency.limit) def put_many_async( self, requests: Sequence[tuple[str, memoryview]] ) -> AsyncIterator[tuple[int, Written]]: - return run_many_async(self.put_async, requests, self._concurrency) + return run_many_async(self.put_async, requests, self._concurrency.limit) def delete_many_async( self, paths: Sequence[str] ) -> AsyncIterator[tuple[int, Deleted | Missing]]: - return run_many_async(self.delete_async, [(path,) for path in paths], self._concurrency) + return run_many_async( + self.delete_async, [(path,) for path in paths], self._concurrency.limit + ) # --- Listable diff --git a/packages/zarr-storage/tests/test_batch.py b/packages/zarr-storage/tests/test_batch.py index 40b6b43c9b..c88d7918bc 100644 --- a/packages/zarr-storage/tests/test_batch.py +++ b/packages/zarr-storage/tests/test_batch.py @@ -5,13 +5,13 @@ import asyncio import threading import time -from concurrent.futures import ThreadPoolExecutor from typing import TYPE_CHECKING import pytest from zarr_storage import BatchError from zarr_storage._batch import run_many, run_many_async +from zarr_storage._resources import ThreadPool if TYPE_CHECKING: from collections.abc import AsyncIterator @@ -64,8 +64,11 @@ def _run(tracker: _Tracker, mode: str, window: int, n: int, into: list[tuple[int if mode == "serial": into.extend(run_many(tracker, requests)) elif mode == "threads": - with ThreadPoolExecutor(8) as executor: - into.extend(run_many(tracker, requests, executor, window)) + pool = ThreadPool(8) + try: + into.extend(run_many(tracker, requests, pool, window)) + finally: + pool.shutdown() else: asyncio.run(_collect(run_many_async(tracker.call_async, requests, window), into)) diff --git a/packages/zarr-storage/tests/test_local.py b/packages/zarr-storage/tests/test_local.py index a041d7a058..2699181b86 100644 --- a/packages/zarr-storage/tests/test_local.py +++ b/packages/zarr-storage/tests/test_local.py @@ -145,11 +145,6 @@ def test_overlong_key_raises(tmp_path: Path) -> None: Store(LocalBackend(tmp_path)).put("a" * 1000, b"x") -def test_zero_concurrency_raises(tmp_path: Path) -> None: - with pytest.raises(ValueError, match="concurrency"): - LocalBackend(tmp_path, concurrency=0) - - def _create( store: Store[LocalBackend], key: str, barrier: threading.Barrier, results: list[bool] ) -> None: diff --git a/packages/zarr-storage/tests/test_resources.py b/packages/zarr-storage/tests/test_resources.py new file mode 100644 index 0000000000..bf9911bd64 --- /dev/null +++ b/packages/zarr-storage/tests/test_resources.py @@ -0,0 +1,299 @@ +"""Shared resources: thread pools and limiters, alone and as backends use them.""" + +from __future__ import annotations + +import asyncio +import pickle +import threading +import time +from functools import partial +from typing import TYPE_CHECKING + +import pytest +from obstore.store import LocalStore, MemoryStore + +from zarr_storage import Copied, Deleted, Found, Limiter, Missing, Store, ThreadPool +from zarr_storage.backends import LocalBackend, ObjectStoreBackend + +if TYPE_CHECKING: + from collections.abc import Callable + from pathlib import Path + + from zarr_storage import ByteRange + + +class _Peak: + """Counts callers inside a region, and the most there ever were at once.""" + + def __init__(self) -> None: + self._lock = threading.Lock() + self._inside = 0 + self.most = 0 + self.done = 0 + + def enter(self) -> None: + with self._lock: + self._inside += 1 + self.most = max(self.most, self._inside) + + def leave(self) -> None: + with self._lock: + self._inside -= 1 + self.done += 1 + + +def _hold_sync(limiter: Limiter, peak: _Peak) -> None: + with limiter.slot(): + peak.enter() + time.sleep(0.002) + peak.leave() + + +async def _hold_async(limiter: Limiter, peak: _Peak) -> None: + async with limiter.slot(): + peak.enter() + await asyncio.sleep(0.002) + peak.leave() + + +async def _many_async(limiter: Limiter, peak: _Peak, n: int) -> None: + await asyncio.gather(*(_hold_async(limiter, peak) for _ in range(n))) + + +def _run_loop(limiter: Limiter, peak: _Peak, n: int) -> None: + asyncio.run(_many_async(limiter, peak, n)) + + +def _finishes(fn: Callable[[], object]) -> bool: + """Whether `fn()` returns within ten seconds; a deadlock fails a test instead of hanging it.""" + worker = threading.Thread(target=fn, daemon=True) + worker.start() + worker.join(10) + return not worker.is_alive() + + +# ---------------------------------------------------------------------------- Limiter + + +@pytest.mark.parametrize("limit", [1, 3]) +def test_limiter_bounds_threads_and_tasks_together(limit: int) -> None: + """Threads and tasks on two event loops never hold more than `limit` slots at once.""" + limiter, peak = Limiter(limit), _Peak() + workers = [threading.Thread(target=_hold_sync, args=(limiter, peak)) for _ in range(8)] + workers += [threading.Thread(target=_run_loop, args=(limiter, peak, 8)) for _ in range(2)] + for worker in workers: + worker.start() + for worker in workers: + worker.join() + assert (peak.done, peak.most, limiter.in_use) == (24, limit, 0) + + +async def _nested_async(limiter: Limiter) -> int: + async with limiter.slot(), limiter.slot(): + return limiter.in_use + + +def _nested_sync(limiter: Limiter, seen: list[int]) -> None: + with limiter.slot(), limiter.slot(): + seen.append(limiter.in_use) + seen.append(asyncio.run(_nested_async(limiter))) + + +def test_slot_inside_a_slot_takes_no_second_slot() -> None: + limiter, seen = Limiter(1), [] + assert _finishes(partial(_nested_sync, limiter, seen)) + assert (seen, limiter.in_use) == ([1, 1], 0) + + +async def _take_in_order(limiter: Limiter, name: str, order: list[str]) -> None: + async with limiter.slot(): + order.append(name) + + +async def _first_come_first_served() -> list[str]: + limiter, order = Limiter(1), [] + await limiter.acquire_async() + tasks = [asyncio.create_task(_take_in_order(limiter, name, order)) for name in "abc"] + await asyncio.sleep(0) + limiter.release() + await asyncio.gather(*tasks) + return order + + +def test_limiter_serves_waiters_in_order() -> None: + assert asyncio.run(_first_come_first_served()) == ["a", "b", "c"] + + +async def _cancel_while_waiting(granted_first: bool) -> int: + limiter = Limiter(1) + await limiter.acquire_async() + waiting = asyncio.create_task(limiter.acquire_async()) + await asyncio.sleep(0) + if granted_first: + limiter.release() # the slot passes to the waiter, which is then cancelled + waiting.cancel() + with pytest.raises(asyncio.CancelledError): + await waiting + if not granted_first: + limiter.release() + return limiter.in_use + + +@pytest.mark.parametrize("granted_first", [False, True]) +def test_cancelled_waiter_gives_back_its_slot(granted_first: bool) -> None: + assert asyncio.run(_cancel_while_waiting(granted_first)) == 0 + + +@pytest.mark.parametrize(("limit", "error"), [(0, ValueError), (1.5, TypeError), (True, TypeError)]) +def test_limiter_of_a_bad_limit_raises(limit: object, error: type[Exception]) -> None: + with pytest.raises(error): + Limiter(limit) # pyright: ignore[reportArgumentType] + + +def test_releasing_a_slot_not_held_raises() -> None: + with pytest.raises(RuntimeError, match="not held"): + Limiter(1).release() + + +# ---------------------------------------------------------------------------- ThreadPool + + +def _worker_view(pool: ThreadPool, other: ThreadPool) -> tuple[bool, bool]: + return pool.in_worker(), other.in_worker() + + +def test_thread_pool_knows_its_own_threads() -> None: + pool, other = ThreadPool(1), ThreadPool(1) + try: + assert not pool.in_worker() + assert pool.submit(_worker_view, pool, other).result() == (True, False) + finally: + pool.shutdown() + other.shutdown() + + +class _RecordingBackend(LocalBackend): + """A local backend that records which threads its reads run on.""" + + def __init__(self, root: Path, threads: ThreadPool) -> None: + super().__init__(root, threads=threads) + self.read_on: set[int] = set() + + def get(self, path: str, byte_range: ByteRange | None) -> Found | Missing: + self.read_on.add(threading.get_ident()) + return super().get(path, byte_range) + + +def _batch_here(store: Store[_RecordingBackend], keys: list[str]) -> int: + assert all(isinstance(result, Found) for _, result in store.get_many(keys)) + return threading.get_ident() + + +def test_batch_from_a_thread_of_its_own_pool_runs_on_that_thread(tmp_path: Path) -> None: + """A batch started on one of its pool's threads must not wait for the pool's other threads, + which may all be waiting the same way; it runs its reads on the thread that started it.""" + pool = ThreadPool(2) + backend = _RecordingBackend(tmp_path, pool) + store = Store(backend) + keys = [f"k{i}" for i in range(4)] + store.put_many((key, b"x") for key in keys) + try: + worker = pool.submit(_batch_here, store, keys).result(timeout=10) + finally: + pool.shutdown(wait=False) + assert backend.read_on == {worker} + + +@pytest.mark.parametrize(("size", "error"), [(0, ValueError), (2.0, TypeError)]) +def test_thread_pool_of_a_bad_size_raises(size: object, error: type[Exception]) -> None: + with pytest.raises(error): + ThreadPool(size) # pyright: ignore[reportArgumentType] + + +def test_shutting_down_the_default_pool_raises() -> None: + with pytest.raises(ValueError, match="shared"): + ThreadPool.default().shutdown() + + +# ---------------------------------------------------------------------------- sharing + + +def test_backends_share_resources_as_given(tmp_path: Path) -> None: + """Left out, resources are process-wide defaults; given, they are exactly what was given.""" + assert LocalBackend(tmp_path / "a").threads is ThreadPool.default() + pool = ThreadPool(2) + assert LocalBackend(tmp_path / "b", threads=pool).threads is pool + memory = [ObjectStoreBackend(MemoryStore()).concurrency for _ in range(2)] + local = ObjectStoreBackend(LocalStore(str(tmp_path))).concurrency + assert memory[0] is memory[1] + assert local is not memory[0] + limiter = Limiter(3) + assert ObjectStoreBackend(MemoryStore(), concurrency=limiter).concurrency is limiter + + +def test_pickling_keeps_resources_shared(tmp_path: Path) -> None: + """Backends that shared a resource share one after a round trip; defaults stay defaults.""" + pool, limiter = ThreadPool(2), Limiter(3) + objects = str(tmp_path) # obstore's `MemoryStore` can't be pickled; its `LocalStore` can + backends = ( + LocalBackend(tmp_path / "a", threads=pool), + LocalBackend(tmp_path / "b", threads=pool), + ObjectStoreBackend(LocalStore(objects), concurrency=limiter), + ObjectStoreBackend(LocalStore(objects), concurrency=limiter), + LocalBackend(tmp_path / "c"), + ObjectStoreBackend(LocalStore(objects)), + ) + a, b, c, d, e, f = pickle.loads(pickle.dumps(backends)) + assert a.threads is b.threads + assert a.threads is not pool + assert a.threads.max_workers == 2 + assert c.concurrency is d.concurrency + assert c.concurrency is not limiter + assert c.concurrency.limit == 3 + assert e.threads is ThreadPool.default() + assert f.concurrency is backends[5].concurrency + + +def _get_in_thread(store: Store[ObjectStoreBackend], results: list[object]) -> None: + results.append(store.get("k")) + + +async def _get_async_while_held(store: Store[ObjectStoreBackend]) -> bool: + try: + await asyncio.wait_for(store.as_async().get("k"), 0.05) + except TimeoutError: + return True + return False + + +def test_obstore_operations_wait_for_a_shared_limit() -> None: + """Every operation, sync or async, of every backend sharing a limiter waits for a slot.""" + limiter = Limiter(1) + first = Store(ObjectStoreBackend(MemoryStore(), concurrency=limiter)) + second = Store(ObjectStoreBackend(MemoryStore(), concurrency=limiter)) + second.put("k", b"x") + limiter.acquire() # another backend's operation holds the only slot + results: list[object] = [] + reader = threading.Thread(target=_get_in_thread, args=(second, results)) + reader.start() + reader.join(0.05) + assert reader.is_alive() + assert asyncio.run(_get_async_while_held(first)) + limiter.release() + reader.join(10) + assert isinstance(results[0], Found) + assert limiter.in_use == 0 + + +def _nested_operations(store: Store[ObjectStoreBackend], results: list[object]) -> None: + store.put("k", b"xy") + results.append(store.get_into("k", bytearray(2))) + results.append(store.delete("k")) + + +def test_obstore_operation_that_calls_another_takes_one_slot() -> None: + """`get_into` reads through `get` and `delete` looks with `head`: with one slot, neither waits.""" + store = Store(ObjectStoreBackend(MemoryStore(), concurrency=Limiter(1))) + results: list[object] = [] + assert _finishes(partial(_nested_operations, store, results)) + assert [type(r) for r in results] == [Copied, Deleted] From 9561df2f796ad10c28def9fcc64820dbdfb470e2 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Wed, 18 Nov 2026 11:11:47 +0100 Subject: [PATCH 2/5] docs(zarr-storage): add changelog fragment for shared resources Assisted-by: ClaudeCode:claude-opus-5-5 --- packages/zarr-storage/changes/464.feature.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 packages/zarr-storage/changes/464.feature.md diff --git a/packages/zarr-storage/changes/464.feature.md b/packages/zarr-storage/changes/464.feature.md new file mode 100644 index 0000000000..54338b5f7c --- /dev/null +++ b/packages/zarr-storage/changes/464.feature.md @@ -0,0 +1 @@ +Add `ThreadPool` and `Limiter`, resources that `zarr-storage` backends share so their threads and operations in flight are bounded together: `LocalBackend` takes `threads` and `ObjectStoreBackend` takes a `Limiter` as `concurrency`, with process-wide defaults when they are left out. From 61d6d24beb5967da4953928619d7ed53a5942c74 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Wed, 18 Nov 2026 12:00:37 +0100 Subject: [PATCH 3/5] feat(zarr-storage)!: make resource limits part of the store's context Resources move from backends to a Context that every store view carries, as in TensorStore. A Context holds a ThreadPool and a Limiter per resource key, is immutable, and resolves keys through its parents to process-wide defaults. Backends hold no resources: each names the `resource_key` its operations are limited under ("local", "memory", "obstore."), and the store gates every operation with the context's limiter for that key. Store and AsyncStore take `context=`, and gain `with_context` and `limited(n)`, a view whose operations together run at most n at once. `Limiter(n, within=parent)` holds a slot of both, so a scoped limit never escapes the one it narrows (where TensorStore's child contexts replace a resource, this nests). Slots are taken innermost first, and where the work runs: AsyncStore runs a sync-only backend on the context's thread pool and takes the slot on the pool thread, so no operation holds a slot while it waits for a thread. LocalBackend is now sync-only; AsyncStore adapts it. Batches run single operations concurrently under the context (a Store batch on the pool, an AsyncStore batch as tasks, up to the limiter's limit), which also makes sync obstore batches concurrent. Native batch methods receive the context. Slots are no longer re-entrant: re-entrancy through a context variable leaked into tasks and, on free-threaded 3.14, into new threads, letting a fan-out run unbounded under one slot. BREAKING CHANGE: LocalBackend's `threads`, its async methods and ObjectStoreBackend's `concurrency` are gone; pass resources in a Context. Backends must provide `resource_key`. Batch protocol methods take the context. Assisted-by: ClaudeCode:claude-opus-5-5 --- packages/zarr-storage/DESIGN.md | 126 +++-- packages/zarr-storage/changes/464.feature.md | 2 +- .../zarr-storage/src/zarr_storage/__init__.py | 7 +- .../src/zarr_storage/_protocols.py | 60 +- .../src/zarr_storage/_resources.py | 295 +++++++--- .../zarr-storage/src/zarr_storage/_store.py | 516 +++++++++++++----- .../src/zarr_storage/backends/_local.py | 140 +---- .../src/zarr_storage/backends/_memory.py | 4 + .../src/zarr_storage/backends/_obstore.py | 301 ++++------ .../src/zarr_storage/testing/_api.py | 14 +- packages/zarr-storage/tests/test_resources.py | 354 ++++++++---- packages/zarr-storage/tests/test_store.py | 31 +- 12 files changed, 1120 insertions(+), 730 deletions(-) diff --git a/packages/zarr-storage/DESIGN.md b/packages/zarr-storage/DESIGN.md index 62ace954e5..002192178b 100644 --- a/packages/zarr-storage/DESIGN.md +++ b/packages/zarr-storage/DESIGN.md @@ -27,8 +27,9 @@ Sources, and what each contributed: ``` Backend capability protocols over backend paths; owns shared state and lifecycle â–² -_View (backend, root, read_only, owner): validates keys, applies the root, relativizes - â–² listings, enforces read-only; `store / "a"` is a view on the same backend +_View (backend, root, read_only, context, owner): validates keys, applies the root, + â–² relativizes listings, enforces read-only, limits and schedules every operation with + its context's resources; `store / "a"` is a view on the same backend Store the operations as sync methods; `close()`, `with`; `as_async()` AsyncStore the same names as coroutines; `aclose()`, `async with`; `as_sync()` ``` @@ -37,18 +38,22 @@ A **backend** is an object that implements some capability protocols (`Readable` `Listable`, `WritableIfMatch`, `DeletableIfMatch`, and their async twins) on *backend paths*: `/`-separated strings, already joined with the root. Backends never see relative keys and never join or strip paths. They hold everything that must exist once per backend: the dict, the -filesystem root, a client, locks. Thread pools and limits on operations in flight are shared -resources the backend is given (see below). +filesystem root, a client, locks. They hold no thread pools or limits: a backend names the +`resource_key` its operations are limited under, and the store draws the resources from its +context (see "Contexts"). Backends must be thread-safe. A **`Store`** or **`AsyncStore`** is the only object users hold. Each is a view: a backend plus a -root. All key validation, root joining, listing relativization and read-only enforcement happen in -their shared base, once, so no backend can get them wrong on its own (the source of most bugs in the legacy API: #2190, #2467, #3639, #3922, #4204, +root, under a context. All key validation, root joining, listing relativization, read-only +enforcement and resource limiting happen in their shared base, once, so no backend can get them +wrong on its own (the source of most bugs in the legacy API: #2190, #2467, #3639, #3922, #4204, #4452). **Sync and async are two classes** (as zarrs has separate sync and async storage traits): `Store` -calls a backend's sync protocols and `AsyncStore` its async ones, under the same method names, so -`store.get(k)` and `await astore.get(k)` read the same way and neither class carries the other's -methods. A backend with both surfaces serves both; `store.as_async()` and `astore.as_sync()` are +calls a backend's sync protocols, and `AsyncStore` its async ones where the backend has them and +its sync ones on the context's thread pool otherwise (zarrs's `SyncToAsyncStorageAdapter`, with +the pool in place of its `spawn_blocking`), under the same method names, so `store.get(k)` and +`await astore.get(k)` read the same way and neither class carries the other's methods. A backend +that only blocks, such as a filesystem, needs no async code. Every backend serves both; `store.as_async()` and `astore.as_sync()` are views on the same backend that don't own it. Lifecycle differs too: `Store` has `close()` and `with`, `AsyncStore` has `await aclose()` and `async with`, which awaits a `ClosableAsync` backend's `aclose` (a backend whose cleanup awaits a client session) and calls `close` on any other. @@ -123,13 +128,16 @@ landed with their results and whose `remaining` holds the requests that failed o (`except*` keeps both). Conditional operations have no batch form: compare-and-swap coordinates single documents, and a batch of it belongs to transactions. -`Store` provides every batch method for any backend that can read or write, running the single -operation one at a time; a backend that has a batch capability (`ReadableMany`, -`ReadableIntoMany[O]`, `WritableMany`, and async twins) is used instead. The rule for such -fallbacks: `Store` may fall back when only speed differs, never when behavior does. A serial batch -returns what a concurrent one would; a `get_into` that read and then copied would not put the bytes -where the backend's buffer type says, which is why `get_into` has no fallback. The batch runners in -`zarr_storage._batch` implement the windowing and failure rule once, for `Store` and backends alike. +`Store` provides every batch method for any backend that can read or write, running single +operations concurrently under the view's context: `Store` on the context's thread pool, `AsyncStore` +as tasks, at most the context limiter's `limit` at once, each holding its own slot. A backend with +a batch capability (`ReadableMany`, `ReadableIntoMany[O]`, `WritableMany`, and async twins) is +used instead, for batching the store itself can do (one request for many keys, one GPU +submission); it receives the context so it can draw on the same resources. The rule for any +fallback: `Store` may fall back when only speed differs, never when behavior does. A batch of single +operations returns what a native batch would; a `get_into` that read and then copied would not put +the bytes where the backend's buffer type says, which is why `get_into` has no fallback. The batch +runners in `zarr_storage._batch` implement the windowing and failure rule once. Listing arguments are checked when the method is called; the items are produced lazily, and taking one after the store closes raises `StoreClosedError`. @@ -143,8 +151,9 @@ raises `InvalidByteRangeError`; a suffix longer than the value returns the whole ## Views, lifecycle, identity -- `store / "a"`, `store.with_read_only(True)` and `store.as_async()` (or `as_sync()`) return views - on the same backend. A view of a +- `store / "a"`, `store.with_read_only(True)`, `store.with_context(ctx)`, `store.limited(n)` and + `store.as_async()` (or `as_sync()`) return views on the same backend. A view keeps its context + unless it is given another. A view of a read-only store is read-only; `with_read_only(False)` on one raises `ReadOnlyError`. - The store the user constructed owns the backend. `close()` (`aclose()` on an `AsyncStore`) on it closes the backend; on a derived view it does nothing. After the backend closes, every view raises `StoreClosedError`. @@ -168,12 +177,10 @@ key (`a/b` when `a` is a key): that reads as absent. | `LocalBackend` | `(dev, inode, mtime_ns, size)`, with each write stamping a unique ns mtime | yes / yes, on filesystems that keep ns times | no (`KeyConflictError`) | the root | | `ObjectStoreBackend` | ETag and version (obstore's `LocalStore` ETag can repeat on ext4) | if the store supports it / not implemented | as the store does | as the store does | -Every backend has `put_if_absent` and `get_if_changed`. `LocalBackend` runs its async operations -and its batches, sync and async, on its `threads`, as many batch operations at once as the pool has -threads. `ObjectStoreBackend` holds a slot of its `concurrency` limiter for every operation, sync -and async, single and batched; async batches start at most `concurrency.limit` operations at once, -and sync batches run one at a time. `MemoryBackend` uses `Store`'s serial batches, which are as fast -for it. +Every backend has `put_if_absent` and `get_if_changed`. `LocalBackend` is sync only; `AsyncStore` +runs its operations on the context's pool. `MemoryBackend` and `ObjectStoreBackend` are also +natively async. Their resource keys are `"local"`, `"memory"`, and `"obstore."` plus the obstore +store's class name, so every S3 backend in a process shares one limit by default. `LocalBackend` stages each write in a reserved directory under the root and renames it into place, holding the lock of the key's directory; the lock lives in that directory, so overlapping roots and @@ -214,47 +221,60 @@ step, each stored key reads back as the model says, through `get` and `get_many` `get_if_changed` with its generation is `Unchanged`. The obstore tests also fail any async run that makes a blocking obstore call. -## Shared resources - -Threads and limits on operations in flight are resources that backends share, after TensorStore's -context resources (`file_io_concurrency`, `s3_request_concurrency`). zarrs has no I/O limits of its -own (its `Resources` carry only a compute target, from Rayon's global pool); it bounds object-store -requests by wrapping the store in object_store's `LimitStore` and sharing that store. - -- `ThreadPool(max_workers)` runs blocking work: `LocalBackend(root, threads=...)`. -- `Limiter(limit)` caps operations in flight, for threads and tasks on any event loop in one - first-come, first-served queue: `ObjectStoreBackend(store, concurrency=...)`. `limiter.slot()` is - a sync or async context manager. A slot is held for a whole operation, and an operation that calls - another through a limiter it already holds takes no second slot (a context variable records the - slots an operation holds), so `get_into`, which reads through `get`, can't wait on itself. -- Pass one resource to several backends to bound them together. Left out, a backend uses a - process-wide default, as TensorStore does: `ThreadPool.default()` of `max(4, cpu_count)` threads, - and one `Limiter(32)` per obstore store class, so every S3 backend in a process shares one limit - and an HTTP backend doesn't share it. A default is shared state, but it changes only speed, never - what an operation does; pass your own resource to isolate a backend. -- Whoever creates a resource owns it: a backend never shuts down a pool it was given, and closing a - `LocalBackend` refuses new operations without stopping the pool. The default pool can't be shut - down. -- Pickling a resource pickles its settings, and pickle stores each object once, so backends that - shared a resource share one new resource after a round trip; a default unpickles as the receiving - process's default. +## Contexts + +Resource limits are part of a store's context, as in TensorStore. zarrs has no I/O limits of its +own (its per-call `Resources` carry only a compute target, from Rayon's global pool); it bounds +object-store requests by wrapping the store in object_store's `LimitStore` and sharing that store. + +- A **`Context`** holds a `ThreadPool` and a `Limiter` per resource key. It is immutable; + `ctx.child(threads=..., limits={...})` overrides some resources and takes the rest from its + parent, and a context with no parent takes the rest from the process-wide defaults + (`Context.default()`): `ThreadPool.default()` of `max(4, cpu_count)` threads (TensorStore's + `file_io_concurrency` default) and one `Limiter(32)` per resource key (its + `s3_request_concurrency` default). A default is shared state, but it changes only speed, never + what an operation does. +- **Every view carries a context**, `Context.default()` unless given one (`Store(backend, + context=...)`, `store.with_context(ctx)`). Every operation through the view holds a slot of + `context.limiter(backend.resource_key)`; blocking work (a sync backend called through an + `AsyncStore`, and the operations of a `Store` batch) runs on `context.threads`. Stores sharing a + context share its limits, whatever their backends. Like TensorStore's, a context is not part of a + store's identity: `shares_backend` and `overlaps` ignore it. +- **`store.limited(n)`** is a view whose operations, together, run at most `n` at once, for a burst + of work: its context is a child whose limiter for the backend's key is `Limiter(n, within=...)` + the parent's. A limiter `within` another holds a slot of both, so a scoped limit never exceeds the + limit it narrows. Here the design departs from TensorStore, where a child context's resource + replaces the parent's: replacing would let a scoped limit escape the process-wide one. +- **Slots are taken innermost first**, so work queued on a narrow limit holds none of the wide one, + and **where the work runs**: a sync backend's operation called through an `AsyncStore` takes its + slot on the pool thread that runs it. No operation holds a slot while it waits for a thread, so + pool threads waiting for slots and slot holders waiting for threads can't block each other. +- `limiter.slot()` is a sync or async context manager, and a slot is not re-entrant, as with a lock: + stores take exactly one slot per operation, and never call each other while holding one. An + earlier draft made slots re-entrant through a context variable; that leaked into every task and + thread inheriting the context (`asyncio.gather`, and new threads on free-threaded Python 3.14), + letting a fan-out run unbounded under one slot. - A batch started on one of its pool's own threads runs on that thread, one operation at a time: waiting there for work queued on the same pool could wait forever once every thread does it. -- Thread pools for I/O and for compute should stay separate (TensorStore's `file_io_concurrency` - and `data_copy_concurrency`): compute waiting on I/O in one pool starves it. +- Whoever creates a resource owns it; the default pool can't be shut down. Pickling a context + pickles its resources' settings, and pickle stores each object once, so stores that shared a + context or a resource share one after a round trip; a default unpickles as the receiving + process's default. +- Pools for I/O and for compute should stay separate (TensorStore's `file_io_concurrency` and + `data_copy_concurrency`): compute waiting on I/O in one pool starves it. A context holds the I/O + pool; zarr-python's codec pipeline would take its own. ## Configuration No global configuration. Every knob is a constructor parameter with its default in the signature; -the shared resources above are passed the same way. +resources are passed in a store's context. ## Open - Durability: no backend syncs to disk; a `sync` option on `LocalBackend` would. - Staged files left by a crash are never read but are never removed either. - Windows rules are implemented and unit-tested, not yet run on Windows. -- Sync batches over obstore run one request at a time; a thread pool, or a native batch call, - would overlap them. +- Listings hold no slot of the context's limiter. - `get_ranges_into`, `put_from`, and a GPU direct storage backend whose `get_into_many` submits one cuFile batch. diff --git a/packages/zarr-storage/changes/464.feature.md b/packages/zarr-storage/changes/464.feature.md index 54338b5f7c..20d405e110 100644 --- a/packages/zarr-storage/changes/464.feature.md +++ b/packages/zarr-storage/changes/464.feature.md @@ -1 +1 @@ -Add `ThreadPool` and `Limiter`, resources that `zarr-storage` backends share so their threads and operations in flight are bounded together: `LocalBackend` takes `threads` and `ObjectStoreBackend` takes a `Limiter` as `concurrency`, with process-wide defaults when they are left out. +Add `Context`, the resources a `zarr-storage` store's operations draw on: a `ThreadPool` and a `Limiter` per backend resource key, with process-wide defaults. Every store view carries a context (`Store(backend, context=...)`, `with_context`), stores sharing a context share its limits, and `store.limited(n)` bounds a burst of operations within the context's limit. `AsyncStore` now runs a sync-only backend on the context's thread pool, and batches run concurrently under the context. diff --git a/packages/zarr-storage/src/zarr_storage/__init__.py b/packages/zarr-storage/src/zarr_storage/__init__.py index d18fb9ee4a..c5ac6b8a9a 100644 --- a/packages/zarr-storage/src/zarr_storage/__init__.py +++ b/packages/zarr-storage/src/zarr_storage/__init__.py @@ -3,8 +3,8 @@ A `Store` is a view of a backend at a root directory, used through sync methods; an `AsyncStore` is the same view used through coroutines, with the same method names. Backends implement capability protocols (`Readable`, `Writable`, `Listable`, their batch variants, and async twins) over backend -paths; the stores validate keys, apply their root, enforce read-only access, and are the only -objects most callers need. +paths; the stores validate keys, apply their root, enforce read-only access, schedule and limit +operations with the resources of their `Context`, and are the only objects most callers need. """ from zarr_storage._errors import ( @@ -39,7 +39,7 @@ WritableMany, WritableManyAsync, ) -from zarr_storage._resources import Limiter, ThreadPool +from zarr_storage._resources import Context, Limiter, ThreadPool from zarr_storage._store import AsyncStore, Store from zarr_storage._types import ( ByteRange, @@ -68,6 +68,7 @@ "BatchError", "ByteRange", "ClosableAsync", + "Context", "Copied", "DeletableIfMatch", "DeletableIfMatchAsync", diff --git a/packages/zarr-storage/src/zarr_storage/_protocols.py b/packages/zarr-storage/src/zarr_storage/_protocols.py index 7058b87ca7..dd06555970 100644 --- a/packages/zarr-storage/src/zarr_storage/_protocols.py +++ b/packages/zarr-storage/src/zarr_storage/_protocols.py @@ -8,15 +8,22 @@ `/`. Each capability has a sync protocol and an async twin whose methods end in `_async`. A backend -implements whichever it can. `Store` calls the sync protocols and `AsyncStore` the async ones, and -their methods are typed so that calling one the backend lacks is a type error. Compare-and-swap on a generation (`WritableIfMatch`, `DeletableIfMatch`) is a capability -of its own, since many stores can't do it; `put_if_absent` is part of `Writable`. +implements whichever it can. `Store` calls the sync protocols. `AsyncStore` calls the async twin +where the backend has it and otherwise runs the sync method on its context's thread pool, so a +backend that only blocks (a filesystem) needs no async code. Methods are typed so that calling one +the backend lacks is a type error. Compare-and-swap on a generation (`WritableIfMatch`, +`DeletableIfMatch`) is a capability of its own, since many stores can't do it; `put_if_absent` is +part of `Writable`. + +Backends hold no thread pools or limits: the store schedules and limits every operation with the +resources of its `Context`, under the backend's `resource_key`. Backends must be thread-safe, since +a store runs their sync methods on many threads at once. The batch capabilities (`ReadableMany`, `ReadableIntoMany`, `WritableMany` and their async twins) -are optional and change only speed: `Store.get_many` and its siblings work on any backend that -can read or write, one operation at a time, and use a backend's batch methods when it has them. A -batch method yields `(index, result)` pairs as operations finish and follows the failure rule of -`zarr_storage._batch`, whose runners implement it. +are for batching the store itself can do (one request for many keys, one GPU submission): without +them, `Store.get_many` and its siblings run single operations concurrently under the context's +resources. A batch method receives that context, so it can draw on the same resources; it yields +`(index, result)` pairs as operations finish and follows the failure rule of `zarr_storage._batch`. """ from __future__ import annotations @@ -26,6 +33,7 @@ if TYPE_CHECKING: from collections.abc import AsyncIterator, Iterator, Sequence + from zarr_storage._resources import Context from zarr_storage._types import ( ByteRange, Copied, @@ -70,7 +78,17 @@ class Backend(Protocol): - """What every backend provides: a lifecycle and whether it accepts writes at all.""" + """What every backend provides: a lifecycle, whether it accepts writes, and the key its + operations are limited under.""" + + @property + def resource_key(self) -> str: + """The key of the `Limiter` that stores take from their context for this backend. + + Backends that should share a limit by default share a key: every obstore S3 backend uses + `"obstore.S3Store"`, so a process's S3 requests are bounded together. + """ + ... @property def closed(self) -> bool: @@ -171,7 +189,7 @@ def delete_if_match(self, path: str, generation: Generation) -> Deleted | Reject @runtime_checkable class ReadableMany(Backend, Protocol): def get_many( - self, requests: Sequence[tuple[str, ByteRange | None]] + self, requests: Sequence[tuple[str, ByteRange | None]], context: Context ) -> Iterator[tuple[int, Found | Missing]]: """`get` for each `(path, byte_range)`, yielding `(index, result)` as each finishes.""" ... @@ -180,7 +198,7 @@ def get_many( @runtime_checkable class ReadableIntoMany(Backend, Protocol[O_contra]): def get_into_many( - self, requests: Sequence[tuple[str, O_contra, ByteRange | None]] + self, requests: Sequence[tuple[str, O_contra, ByteRange | None]], context: Context ) -> Iterator[tuple[int, Copied | Missing]]: """`get_into` for each `(path, out, byte_range)`, yielding `(index, result)`.""" ... @@ -188,11 +206,15 @@ def get_into_many( @runtime_checkable class WritableMany(Backend, Protocol): - def put_many(self, requests: Sequence[tuple[str, memoryview]]) -> Iterator[tuple[int, Written]]: + def put_many( + self, requests: Sequence[tuple[str, memoryview]], context: Context + ) -> Iterator[tuple[int, Written]]: """`put` for each `(path, value)`, yielding `(index, result)` as each finishes.""" ... - def delete_many(self, paths: Sequence[str]) -> Iterator[tuple[int, Deleted | Missing]]: + def delete_many( + self, paths: Sequence[str], context: Context + ) -> Iterator[tuple[int, Deleted | Missing]]: """`delete` for each path, yielding `(index, result)` as each finishes.""" ... @@ -207,6 +229,7 @@ def list_dir(self, directory: str) -> Iterator[ListItem]: ... +@runtime_checkable class ReadableAsync(Backend, Protocol): async def get_async(self, path: str, byte_range: ByteRange | None) -> Found | Missing: ... @@ -221,12 +244,14 @@ async def get_ranges_async( async def head_async(self, path: str) -> ObjectMetadata | Missing: ... +@runtime_checkable class ReadableIntoAsync(Backend, Protocol[O_contra]): async def get_into_async( self, path: str, out: O_contra, byte_range: ByteRange | None ) -> Copied | Missing: ... +@runtime_checkable class WritableAsync(Backend, Protocol): async def put_async(self, path: str, value: memoryview) -> Written: ... @@ -237,12 +262,14 @@ async def delete_async(self, path: str) -> Deleted | Missing: ... async def delete_dir_async(self, directory: str) -> None: ... +@runtime_checkable class WritableIfMatchAsync(Backend, Protocol): async def put_if_match_async( self, path: str, value: memoryview, generation: Generation ) -> Written | Rejected | Missing: ... +@runtime_checkable class DeletableIfMatchAsync(Backend, Protocol): async def delete_if_match_async( self, path: str, generation: Generation @@ -252,28 +279,29 @@ async def delete_if_match_async( @runtime_checkable class ReadableManyAsync(Backend, Protocol): def get_many_async( - self, requests: Sequence[tuple[str, ByteRange | None]] + self, requests: Sequence[tuple[str, ByteRange | None]], context: Context ) -> AsyncIterator[tuple[int, Found | Missing]]: ... @runtime_checkable class ReadableIntoManyAsync(Backend, Protocol[O_contra]): def get_into_many_async( - self, requests: Sequence[tuple[str, O_contra, ByteRange | None]] + self, requests: Sequence[tuple[str, O_contra, ByteRange | None]], context: Context ) -> AsyncIterator[tuple[int, Copied | Missing]]: ... @runtime_checkable class WritableManyAsync(Backend, Protocol): def put_many_async( - self, requests: Sequence[tuple[str, memoryview]] + self, requests: Sequence[tuple[str, memoryview]], context: Context ) -> AsyncIterator[tuple[int, Written]]: ... def delete_many_async( - self, paths: Sequence[str] + self, paths: Sequence[str], context: Context ) -> AsyncIterator[tuple[int, Deleted | Missing]]: ... +@runtime_checkable class ListableAsync(Backend, Protocol): def list_async(self, directory: str) -> AsyncIterator[str]: ... diff --git a/packages/zarr-storage/src/zarr_storage/_resources.py b/packages/zarr-storage/src/zarr_storage/_resources.py index 5f4671e01b..ca9c9e8ec5 100644 --- a/packages/zarr-storage/src/zarr_storage/_resources.py +++ b/packages/zarr-storage/src/zarr_storage/_resources.py @@ -1,35 +1,46 @@ -"""Resources that backends share: thread pools and limits on operations in flight. - -Modeled on TensorStore's context resources (`file_io_concurrency`, `s3_request_concurrency`): a -resource is an object you create and pass to as many backends as should share it. A backend never -closes a resource it was given. Left out, a backend uses a process-wide default, as TensorStore -does, because threads and request slots are process-wide in the end; pass your own to isolate a -backend from the rest of the process. A resource changes how fast operations run, never what they -do. - -Pickling a resource pickles its settings. Pickle stores each object once, so backends that shared -a resource before pickling share one new resource after; a default resource unpickles as the -default of the receiving process. +"""The resources a store's operations draw on: thread pools, limits on operations in flight, and the +context that holds them. + +Modeled on TensorStore's `Context`: a context maps resource keys to resource objects, a store view +carries one, and every operation through the view takes its resources from it. A backend names the +key its operations are limited under (`Backend.resource_key`) and never holds resources itself. +Resources change how fast operations run, never what they do, so a context is not part of a store's +identity. + +A context left out is `Context.default()`, which resolves every key to a process-wide default, as +TensorStore's thread pools and request limits are shared across independent contexts by default. + +Pickling a resource pickles its settings. Pickle stores each object once, so stores that shared a +resource before pickling share one new resource after; a default unpickles as the receiving +process's default. """ from __future__ import annotations import asyncio +import contextvars import os import threading from collections import deque from concurrent.futures import ThreadPoolExecutor -from contextvars import ContextVar from functools import partial +from types import MappingProxyType from typing import TYPE_CHECKING, Final, TypeVar, TypeVarTuple if TYPE_CHECKING: - from collections.abc import Callable + from collections.abc import Callable, Mapping from concurrent.futures import Future - from contextvars import Token from types import TracebackType -__all__ = ["Limiter", "Slot", "ThreadPool", "default_limiter", "default_thread_pool"] +__all__ = [ + "Context", + "Limiter", + "Slot", + "ThreadPool", + "default_context", + "default_limiter", + "default_thread_pool", +] R = TypeVar("R") Ts = TypeVarTuple("Ts") @@ -59,10 +70,11 @@ def _mark_worker(pool: ThreadPool) -> None: class ThreadPool: - """Threads that run blocking operations, such as file I/O, for the backends given it. + """Threads that run blocking operations, such as file I/O. - The threads start when first needed. Whoever creates a pool owns it and may `shutdown` it once - no backend needs it; the default pool lives as long as the process. + The threads start when first needed. Work runs in a copy of the submitter's `contextvars` + context, as `asyncio.to_thread` does. Whoever creates a pool owns it and may `shutdown` it once + nothing needs it; the default pool lives as long as the process. Parameters ---------- @@ -78,7 +90,7 @@ def __init__(self, max_workers: int) -> None: @classmethod def default(cls) -> ThreadPool: - """The process-wide pool that backends use when given none.""" + """The process-wide pool, of `DEFAULT_THREADS` threads.""" return default_thread_pool() @property @@ -106,18 +118,20 @@ def in_worker(self) -> bool: """Whether the calling thread is one of this pool's. Work started from one of the pool's own threads must not wait for other work queued on the - pool, which may never start; the batch runners run such work inline. + pool, which may never start; stores run such work inline. """ return getattr(_worker, "pool", None) is self def submit(self, fn: Callable[[*Ts], R], /, *args: *Ts) -> Future[R]: """Run `fn(*args)` on one of the pool's threads.""" - return self._pool().submit(fn, *args) + context = contextvars.copy_context() + return self._pool().submit(context.run, fn, *args) async def run(self, fn: Callable[[*Ts], R], /, *args: *Ts) -> R: """Await `fn(*args)`, run on one of the pool's threads.""" + context = contextvars.copy_context() loop = asyncio.get_running_loop() - return await loop.run_in_executor(self._pool(), partial(fn, *args)) + return await loop.run_in_executor(self._pool(), partial(context.run, fn, *args)) def shutdown(self, *, wait: bool = True) -> None: """Stop the pool's threads; with `wait`, after the work already given them. @@ -141,13 +155,13 @@ def shutdown(self, *, wait: bool = True) -> None: raise ValueError("The default thread pool is shared by the whole process.") -_default_pool_lock = threading.Lock() +_defaults_lock = threading.Lock() _default_pool: list[ThreadPool] = [] def default_thread_pool() -> ThreadPool: - """The process-wide `ThreadPool`, with `DEFAULT_THREADS` threads.""" - with _default_pool_lock: + """The process-wide `ThreadPool`, of `DEFAULT_THREADS` threads.""" + with _defaults_lock: if not _default_pool: _default_pool.append(_DefaultThreadPool(DEFAULT_THREADS)) return _default_pool[0] @@ -155,11 +169,6 @@ def default_thread_pool() -> ThreadPool: # ---------------------------------------------------------------------------- limiters -_held: ContextVar[frozenset[Limiter]] = ContextVar( - "zarr_storage_held_limiters", default=frozenset() -) -"""The limiters whose slots the current operation holds, so nested operations don't wait again.""" - class _ThreadWaiter: """A thread blocked on a limiter.""" @@ -200,20 +209,29 @@ def grant(self) -> bool: return True +def _limiter(limit: int, within: Limiter | None) -> Limiter: + return Limiter(limit, within=within) + + class Limiter: - """A limit on how many operations run at once, across every backend given it. + """A limit on how many operations run at once. - Threads and tasks on any event loop wait in one first-come, first-served queue. + Threads and tasks on any event loop wait in one first-come, first-served queue. A limiter + `within` another takes a slot of that one too, so a scoped limit can't exceed the limit it + narrows. Parameters ---------- limit The most operations that run at once. + within + A limiter whose slots this one's operations also hold. """ - def __init__(self, limit: int) -> None: + def __init__(self, limit: int, *, within: Limiter | None = None) -> None: _check_size("limit", limit) self._limit = limit + self._within = within self._lock = threading.Lock() self._in_use = 0 self._waiters: deque[_ThreadWaiter | _TaskWaiter] = deque() @@ -222,6 +240,11 @@ def __init__(self, limit: int) -> None: def limit(self) -> int: return self._limit + @property + def within(self) -> Limiter | None: + """The limiter whose slots this one's operations also hold, if any.""" + return self._within + @property def in_use(self) -> int: """How many slots are held now.""" @@ -231,14 +254,25 @@ def __repr__(self) -> str: return f"{type(self).__name__}(limit={self._limit}, in_use={self._in_use})" def __reduce__(self) -> tuple[object, ...]: - return (Limiter, (self._limit,)) + return (_limiter, (self._limit, self._within)) + + def chain(self) -> tuple[Limiter, ...]: + """This limiter and each it lies within, innermost first.""" + chain: list[Limiter] = [] + current: Limiter | None = self + while current is not None: + chain.append(current) + current = current.within + return tuple(chain) def slot(self) -> Slot: - """A context manager, sync (`with`) or async (`async with`), that holds one slot. + """A context manager, sync (`with`) or async (`async with`), holding a slot of this limiter + and of each limiter it lies within. - Prefer it to `acquire` and `release`: it releases on error, and inside an operation that - already holds a slot of this limiter it takes no second one, so an operation that calls - another can't wait on itself. + Slots are taken innermost first, so an operation waiting on a narrow limit holds none of the + wider one it narrows. A slot is not re-entrant: code holding one that starts work waiting + for another slot of the same limiter can wait forever, as with a lock. Stores take one slot + per operation; to bound a burst of operations, use a limited view (`Store.limited`). """ return Slot(self) @@ -250,7 +284,7 @@ def _take(self) -> bool: return False def acquire(self) -> None: - """Take a slot, blocking the calling thread until one is free.""" + """Take a slot of this limiter alone, blocking the calling thread until one is free.""" with self._lock: if self._take(): return @@ -259,7 +293,7 @@ def acquire(self) -> None: waiter.wait() async def acquire_async(self) -> None: - """Take a slot, waiting without blocking the event loop until one is free.""" + """Take a slot of this limiter alone, waiting without blocking the event loop.""" with self._lock: if self._take(): return @@ -278,7 +312,7 @@ async def acquire_async(self) -> None: raise def release(self) -> None: - """Give back a slot, to the longest waiter if there is one. + """Give back a slot of this limiter, to the longest waiter if there is one. Raises `RuntimeError` if no slot is held. """ @@ -291,32 +325,31 @@ def release(self) -> None: self._in_use -= 1 -class Slot: - """One use of a `Limiter`: holds a slot from entry to exit, unless the operation holds one.""" +def _release_all(limiters: list[Limiter]) -> None: + for limiter in reversed(limiters): + limiter.release() - __slots__ = ("_limiter", "_token") - def __init__(self, limiter: Limiter) -> None: - self._limiter = limiter - self._token: Token[frozenset[Limiter]] | None = None - - def _needed(self) -> bool: - return self._limiter not in _held.get() +class Slot: + """One use of a `Limiter`: holds a slot of it and of each limiter it lies within, innermost + first, from entry to exit.""" - def _mark(self) -> None: - self._token = _held.set(_held.get() | {self._limiter}) + __slots__ = ("_chain", "_taken") - def _unmark(self) -> None: - if self._token is None: - return - _held.reset(self._token) - self._token = None - self._limiter.release() + def __init__(self, limiter: Limiter) -> None: + self._chain = limiter.chain() + self._taken: list[Limiter] = [] def __enter__(self) -> None: - if self._needed(): - self._limiter.acquire() - self._mark() + taken: list[Limiter] = [] + try: + for limiter in self._chain: + limiter.acquire() + taken.append(limiter) + except BaseException: + _release_all(taken) + raise + self._taken = taken def __exit__( self, @@ -324,12 +357,19 @@ def __exit__( exc: BaseException | None, tb: TracebackType | None, ) -> None: - self._unmark() + taken, self._taken = self._taken, [] + _release_all(taken) async def __aenter__(self) -> None: - if self._needed(): - await self._limiter.acquire_async() - self._mark() + taken: list[Limiter] = [] + try: + for limiter in self._chain: + await limiter.acquire_async() + taken.append(limiter) + except BaseException: + _release_all(taken) + raise + self._taken = taken async def __aexit__( self, @@ -337,7 +377,8 @@ async def __aexit__( exc: BaseException | None, tb: TracebackType | None, ) -> None: - self._unmark() + taken, self._taken = self._taken, [] + _release_all(taken) class _DefaultLimiter(Limiter): @@ -351,14 +392,128 @@ def __reduce__(self) -> tuple[object, ...]: return (default_limiter, (self._key,)) -_default_limiters_lock = threading.Lock() _default_limiters: dict[str, Limiter] = {} def default_limiter(key: str) -> Limiter: - """The process-wide `Limiter` for `key` (such as an obstore store class), of `DEFAULT_LIMIT`.""" - with _default_limiters_lock: + """The process-wide `Limiter` for `key`, of `DEFAULT_LIMIT`.""" + with _defaults_lock: limiter = _default_limiters.get(key) if limiter is None: limiter = _default_limiters[key] = _DefaultLimiter(key) return limiter + + +# ---------------------------------------------------------------------------- contexts + + +def _context( + threads: ThreadPool | None, limits: Mapping[str, Limiter], parent: Context | None +) -> Context: + return Context(threads=threads, limits=limits, parent=parent) + + +class Context: + """The resources operations draw on: a thread pool, and a limiter per resource key. + + A context is immutable. `child` makes one that overrides some resources and takes the rest from + its parent; a context with no parent takes the rest from the process-wide defaults. Every store + view carries a context (`Store.context`); `with_context` and `limited` return views under + another one. + + Parameters + ---------- + threads + The pool that runs blocking work: a sync backend's operations called through an + `AsyncStore`, and the operations of batches. Left out, the parent's. + limits + A `Limiter` for each resource key it names (a backend's `resource_key`). Keys left out + resolve through the parent. + parent + The context to take unnamed resources from. Left out, the process-wide defaults. + """ + + __slots__ = ("_limits", "_parent", "_threads") + + def __init__( + self, + *, + threads: ThreadPool | None = None, + limits: Mapping[str, Limiter] | None = None, + parent: Context | None = None, + ) -> None: + own = {} if limits is None else dict(limits) + for key, limiter in own.items(): + if not isinstance(limiter, Limiter): # pyright: ignore[reportUnnecessaryIsInstance] + raise TypeError(f"limits[{key!r}] must be a Limiter; got {type(limiter).__name__}.") + if threads is not None and not isinstance(threads, ThreadPool): # pyright: ignore[reportUnnecessaryIsInstance] + raise TypeError(f"threads must be a ThreadPool; got {type(threads).__name__}.") + self._threads = threads + self._limits: Mapping[str, Limiter] = MappingProxyType(own) + self._parent = parent + + @classmethod + def default(cls) -> Context: + """The process-wide context: every key resolves to its process-wide default.""" + return default_context() + + @property + def parent(self) -> Context | None: + return self._parent + + @property + def threads(self) -> ThreadPool: + """The pool that runs blocking work under this context.""" + context: Context | None = self + while context is not None: + if context._threads is not None: + return context._threads + context = context._parent + return default_thread_pool() + + def limiter(self, key: str) -> Limiter: + """The limiter for resource key `key` under this context.""" + context: Context | None = self + while context is not None: + limiter = context._limits.get(key) + if limiter is not None: + return limiter + context = context._parent + return default_limiter(key) + + def child( + self, *, threads: ThreadPool | None = None, limits: Mapping[str, Limiter] | None = None + ) -> Context: + """A context with these resources, taking the rest from this one.""" + return Context(threads=threads, limits=limits, parent=self) + + def limited(self, key: str, limit: int) -> Context: + """A child whose limiter for `key` allows `limit` operations, within this context's.""" + return self.child(limits={key: Limiter(limit, within=self.limiter(key))}) + + def __repr__(self) -> str: + own = ", ".join(f"{key!r}: {limiter!r}" for key, limiter in self._limits.items()) + return f"{type(self).__name__}(threads={self._threads!r}, limits={{{own}}})" + + def __reduce__(self) -> tuple[object, ...]: + return (_context, (self._threads, dict(self._limits), self._parent)) + + +class _DefaultContext(Context): + """The process-wide context; unpickled as the receiver's own.""" + + __slots__ = () + + def __reduce__(self) -> tuple[object, ...]: + return (default_context, ()) + + +_default_context: list[Context] = [] + + +def default_context() -> Context: + """The process-wide `Context`.""" + with _defaults_lock: + if not _default_context: + _default_context.append(_DefaultContext()) + return _default_context[0] diff --git a/packages/zarr-storage/src/zarr_storage/_store.py b/packages/zarr-storage/src/zarr_storage/_store.py index de7dedc71b..6d443a84bb 100644 --- a/packages/zarr-storage/src/zarr_storage/_store.py +++ b/packages/zarr-storage/src/zarr_storage/_store.py @@ -1,13 +1,21 @@ -"""`Store` and `AsyncStore`: views of a backend at a root, through sync methods or coroutines. +"""`Store` and `AsyncStore`: views of a backend at a root, under a context, through sync methods or +coroutines. They are where the key grammar, the root, read-only access and the closed check are enforced, -once, for every backend. A backend only sees backend paths. +once, for every backend; a backend only sees backend paths. They are also where operations are +scheduled and limited: every operation holds a slot of the view's context's limiter for the +backend's `resource_key`, and blocking work runs on the context's thread pool. + +A slot is taken where the work runs. A sync backend's operation called through an `AsyncStore` +takes its slot on the pool thread that runs it, so no operation ever holds a slot while it waits +for a thread: pool threads waiting for slots and slot holders waiting for threads can't block each +other. """ from __future__ import annotations from collections.abc import AsyncGenerator, Generator -from typing import TYPE_CHECKING, Final, Generic, Self, TypeVar, cast, final +from typing import TYPE_CHECKING, Final, Generic, Self, TypeVar, TypeVarTuple, cast, final from zarr_storage._batch import run_many, run_many_async from zarr_storage._buffers import byte_view @@ -16,35 +24,37 @@ from zarr_storage._protocols import ( Backend, ClosableAsync, + DeletableIfMatchAsync, + ListableAsync, + ReadableAsync, + ReadableIntoAsync, ReadableIntoMany, ReadableIntoManyAsync, ReadableMany, ReadableManyAsync, + WritableAsync, + WritableIfMatchAsync, WritableMany, WritableManyAsync, ) +from zarr_storage._resources import default_context from zarr_storage._types import Missing if TYPE_CHECKING: - from collections.abc import AsyncIterator, Iterable, Iterator, Sequence + from collections.abc import AsyncIterator, Callable, Iterable, Iterator, Sequence from types import TracebackType from typing_extensions import Buffer from zarr_storage._protocols import ( DeletableIfMatch, - DeletableIfMatchAsync, Listable, - ListableAsync, Readable, - ReadableAsync, ReadableInto, - ReadableIntoAsync, Writable, - WritableAsync, WritableIfMatch, - WritableIfMatchAsync, ) + from zarr_storage._resources import Context, Limiter, Slot from zarr_storage._types import ( ByteRange, Copied, @@ -65,19 +75,40 @@ O = TypeVar("O") T = TypeVar("T") R = TypeVar("R") +Ts = TypeVarTuple("Ts") Q = TypeVar("Q", bound="str | tuple[str, ByteRange | None]") """A `get_many` request: a key, or a key and the part of its value to read.""" +def _limited(limiter: Limiter, fn: Callable[[*Ts], R], *args: *Ts) -> R: + """`fn(*args)`, holding a slot of `limiter`, on the calling thread.""" + with limiter.slot(): + return fn(*args) + + +def _list_all(fn: Callable[[str], Iterator[T]], directory: str) -> list[T]: + return list(fn(directory)) + + class _View(Generic[B_co]): - """What `Store` and `AsyncStore` share: a backend, a root, read-only access, ownership.""" + """What `Store` and `AsyncStore` share: a backend, a root, read-only access, a context, and + ownership.""" - __slots__ = ("_backend", "_owner", "_read_only", "_root") + __slots__ = ("_backend", "_context", "_limiter", "_owner", "_read_only", "_root") - def __init__(self, backend: B_co, *, root: str = "", read_only: bool = False) -> None: + def __init__( + self, + backend: B_co, + *, + root: str = "", + read_only: bool = False, + context: Context | None = None, + ) -> None: self._backend: Final[B_co] = backend self._root: Final[str] = normalize_directory(root) self._read_only: Final[bool] = read_only or not backend.writable + self._context: Final[Context] = default_context() if context is None else context + self._limiter: Final[Limiter] = self._context.limiter(backend.resource_key) self._owner = True # ----------------------------------------------------------------- views and identity @@ -97,20 +128,27 @@ def read_only(self) -> bool: """Whether writes through this store are refused.""" return self._read_only + @property + def context(self) -> Context: + """The resources this store's operations draw on.""" + return self._context + @property def closed(self) -> bool: """Whether the backend has been closed.""" return self._backend.closed - def _view(self, *, root: str, read_only: bool) -> Self: - view = type(self)(self._backend, root=root, read_only=read_only) + def _view(self, *, root: str, read_only: bool, context: Context) -> Self: + view = type(self)(self._backend, root=root, read_only=read_only, context=context) view._owner = False return view def __truediv__(self, directory: str) -> Self: """A view on the same backend, rooted at `directory` under this store's root.""" return self._view( - root=join(self._root, normalize_directory(directory)), read_only=self._read_only + root=join(self._root, normalize_directory(directory)), + read_only=self._read_only, + context=self._context, ) def with_read_only(self, read_only: bool = True) -> Self: @@ -121,7 +159,19 @@ def with_read_only(self, read_only: bool = True) -> Self: """ if self._read_only and not read_only: raise ReadOnlyError(f"{self!r} is read-only; its views are read-only too.") - return self._view(root=self._root, read_only=read_only) + return self._view(root=self._root, read_only=read_only, context=self._context) + + def with_context(self, context: Context) -> Self: + """A view on the same backend and root whose operations draw on `context`.""" + return self._view(root=self._root, read_only=self._read_only, context=context) + + def limited(self, limit: int) -> Self: + """A view whose operations, together, run at most `limit` at once. + + The limit narrows this view's: an operation holds a slot of both. Batches through the view + start at most `limit` operations at once. + """ + return self.with_context(self._context.limited(self._backend.resource_key, limit)) def shares_backend(self, other: _View[Backend]) -> bool: """Whether `other` reads and writes through the same backend object.""" @@ -138,7 +188,10 @@ def __repr__(self) -> str: return f"{type(self).__name__}({self._backend!r}, root={self._root!r}{flags})" def __reduce__(self) -> tuple[object, ...]: - return (_restore_store, (type(self), self._backend, self._root, self._read_only)) + return ( + _restore_store, + (type(self), self._backend, self._root, self._read_only, self._context), + ) # ----------------------------------------------------------------- path handling @@ -154,16 +207,8 @@ def _directory(self, directory: str) -> str: self._check_open() return join(self._root, normalize_directory(directory)) - def _open_items(self, items: Iterator[T]) -> Iterator[T]: - """`items`, refusing to continue once the backend closes.""" - for item in items: - self._check_open() - yield item - - async def _open_items_async(self, items: AsyncIterator[T]) -> AsyncIterator[T]: - async for item in items: - self._check_open() - yield item + def _slot(self) -> Slot: + return self._limiter.slot() def _writable_path(self, key: str) -> str: path = self._path(key) @@ -207,20 +252,33 @@ def _writable_paths(self, keys: Sequence[str]) -> list[str]: seen.add(path) return paths + def _put_requests( + self, items: Iterable[tuple[str, Buffer]] + ) -> tuple[list[str], list[tuple[str, memoryview]]]: + batch = list(items) + keys = [key for key, _ in batch] + paths = self._writable_paths(keys) + views = [byte_view(value, writable=False) for _, value in batch] + return keys, list(zip(paths, views, strict=True)) + @final class Store(_View[B_co]): - """A backend seen from a root directory, through sync methods. + """A backend seen from a root directory, under a context, through sync methods. Construct a store from a backend; the store you construct owns the backend and closes it. - `store / "a"`, `with_read_only()` and `as_async()` return views on the same backend, which - share its state and never close it. A view of a read-only store is read-only too. + `store / "a"`, `with_read_only()`, `with_context()`, `limited()` and `as_async()` return views + on the same backend, which share its state and never close it. A view of a read-only store is + read-only too. Which methods a store offers depends on its backend's capabilities: a method whose capability the backend lacks is a type error. `AsyncStore` offers the same operations as coroutines. - `backend` gives direct access to the backend, which bypasses the key grammar, the root, and - read-only and closed checks. + Every operation holds a slot of the context's limiter for the backend's `resource_key`; batches + run their operations on the context's thread pool, at most the limiter's limit at once. + + `backend` gives direct access to the backend, which bypasses the key grammar, the root, the + read-only and closed checks, and the context's limits. Stores have no value equality: `==` is identity. Use `shares_backend` and `overlaps` to compare stores. @@ -234,16 +292,27 @@ class Store(_View[B_co]): read_only Refuse writes through this store. A backend that can't write makes every store over it read-only. + context + The resources operations draw on. Left out, `Context.default()`. """ __slots__ = () - def __init__(self, backend: B_co, *, root: str = "", read_only: bool = False) -> None: - super().__init__(backend, root=root, read_only=read_only) + def __init__( + self, + backend: B_co, + *, + root: str = "", + read_only: bool = False, + context: Context | None = None, + ) -> None: + super().__init__(backend, root=root, read_only=read_only, context=context) def as_async(self) -> AsyncStore[B_co]: - """An `AsyncStore` view on the same backend and root; it doesn't own the backend.""" - view = AsyncStore(self._backend, root=self._root, read_only=self._read_only) + """An `AsyncStore` view on the same backend, root and context; it doesn't own the backend.""" + view = AsyncStore( + self._backend, root=self._root, read_only=self._read_only, context=self._context + ) view._owner = False return view @@ -265,7 +334,13 @@ def __exit__( ) -> None: self.close() - # ----------------------------------------------------------------- batch results + # ----------------------------------------------------------------- batch plumbing + + def _run_many( + self, fn: Callable[[*Ts], R], requests: Sequence[tuple[*Ts]] + ) -> Iterator[tuple[int, R]]: + """`fn` over `requests` on the context's pool, at most the limiter's limit at once.""" + return run_many(fn, requests, self._context.threads, self._limiter.limit) def _streamed( self, batch: Sequence[T], results: Iterator[tuple[int, R]] @@ -299,13 +374,33 @@ def _finished(batch: Sequence[T], results: Iterator[tuple[int, R]]) -> list[tupl raise _batch_error(group, batch, completed, completed.keys()) from None return [(request, completed[index]) for index, request in enumerate(batch)] + # ----------------------------------------------------------------- operations on backend paths + + def _get(self: Store[Readable], path: str, byte_range: ByteRange | None) -> Found | Missing: + with self._slot(): + return self._backend.get(path, byte_range) + + def _get_into( + self: Store[ReadableInto[O]], path: str, out: O, byte_range: ByteRange | None + ) -> Copied | Missing: + with self._slot(): + return self._backend.get_into(path, out, byte_range) + + def _put(self: Store[Writable], path: str, value: memoryview) -> Written: + with self._slot(): + return self._backend.put(path, value) + + def _delete(self: Store[Writable], path: str) -> Deleted | Missing: + with self._slot(): + return self._backend.delete(path) + # ----------------------------------------------------------------- reads def get( self: Store[Readable], key: str, *, byte_range: ByteRange | None = None ) -> Found | Missing: """The value at `key`, or the part `byte_range` selects.""" - return self._backend.get(self._path(key), byte_range) + return self._get(self._path(key), byte_range) def get_if_changed( self: Store[Readable], @@ -318,13 +413,17 @@ def get_if_changed( This revalidates a cached copy, as an HTTP `If-None-Match` request does. """ - return self._backend.get_if_changed(self._path(key), generation, byte_range) + path = self._path(key) + with self._slot(): + return self._backend.get_if_changed(path, generation, byte_range) def get_ranges( self: Store[Readable], key: str, ranges: Sequence[ByteRange] ) -> FoundRanges | Missing: """Several parts of the value at `key`, all from one generation.""" - return self._backend.get_ranges(self._path(key), ranges) + path = self._path(key) + with self._slot(): + return self._backend.get_ranges(path, ranges) def get_into( self: Store[ReadableInto[O]], key: str, out: O, *, byte_range: ByteRange | None = None @@ -335,11 +434,13 @@ def get_into( for the built-in backends, device memory for a GPU direct storage backend. Raises `ValueError` if `out` is shorter than the bytes to copy or can't be written. """ - return self._backend.get_into(self._path(key), out, byte_range) + return self._get_into(self._path(key), out, byte_range) def head(self: Store[Readable], key: str) -> ObjectMetadata | Missing: """The size and generation of the value at `key`.""" - return self._backend.head(self._path(key)) + path = self._path(key) + with self._slot(): + return self._backend.head(path) def exists(self: Store[Readable], key: str) -> bool: """Whether `key` holds a value.""" @@ -349,13 +450,14 @@ def exists(self: Store[Readable], key: str) -> bool: def put(self: Store[Writable], key: str, value: Buffer) -> Written: """Write `value` at `key`.""" - return self._backend.put(self._writable_path(key), byte_view(value, writable=False)) + return self._put(self._writable_path(key), byte_view(value, writable=False)) def put_if_absent(self: Store[Writable], key: str, value: Buffer) -> Written | Rejected: """Write `value` at `key` if the key holds no value; `Rejected` if it holds one.""" - return self._backend.put_if_absent( - self._writable_path(key), byte_view(value, writable=False) - ) + path = self._writable_path(key) + view = byte_view(value, writable=False) + with self._slot(): + return self._backend.put_if_absent(path, view) def put_if_match( self: Store[WritableIfMatch], key: str, value: Buffer, generation: Generation @@ -364,23 +466,28 @@ def put_if_match( `Rejected` if the key holds another generation, `Missing` if it holds no value. """ - return self._backend.put_if_match( - self._writable_path(key), byte_view(value, writable=False), generation - ) + path = self._writable_path(key) + view = byte_view(value, writable=False) + with self._slot(): + return self._backend.put_if_match(path, view, generation) def delete(self: Store[Writable], key: str) -> Deleted | Missing: """Delete the key `key`. A directory is not a key: deleting one finds it `Missing`.""" - return self._backend.delete(self._writable_path(key)) + return self._delete(self._writable_path(key)) def delete_if_match( self: Store[DeletableIfMatch], key: str, generation: Generation ) -> Deleted | Rejected | Missing: """Delete the key `key` if its value has `generation`.""" - return self._backend.delete_if_match(self._writable_path(key), generation) + path = self._writable_path(key) + with self._slot(): + return self._backend.delete_if_match(path, generation) def delete_dir(self: Store[Writable], directory: str = "") -> None: """Delete every key under `directory` (by default, every key in this store).""" - self._backend.delete_dir(self._writable_directory(directory)) + path = self._writable_directory(directory) + with self._slot(): + self._backend.delete_dir(path) # ----------------------------------------------------------------- batches @@ -391,9 +498,10 @@ def get_many( A request is a key, or a `(key, byte_range)` pair, and comes back as it was given, so `dict(store.get_many(keys))` maps keys to results. Keys are checked when `get_many` is - called; the reads happen as the pairs are taken, in no particular order. A batch is not - atomic: each read is its own `get`, from whatever generation it finds (`get_ranges` reads - several parts of one value from one generation). + called; the reads run on the context's thread pool as the pairs are taken, at most the + limiter's limit at once, in no particular order. A batch is not atomic: each read is its own + `get`, from whatever generation it finds (`get_ranges` reads several parts of one value from + one generation). If reads raise, the iterator yields the reads already running that succeed, then raises `BatchError` with the requests left in `remaining`; its `completed` is empty, since those @@ -404,8 +512,8 @@ def get_many( paths = [self._read_request(request) for request in batch] backend = self._backend if isinstance(backend, ReadableMany): - return self._streamed(batch, backend.get_many(paths)) - return self._streamed(batch, run_many(backend.get, paths)) + return self._streamed(batch, backend.get_many(paths, self._context)) + return self._streamed(batch, self._run_many(self._get, paths)) def get_into_many( self: Store[ReadableInto[O]], @@ -421,8 +529,8 @@ def get_into_many( backend = self._backend if isinstance(backend, ReadableIntoMany): many = cast("ReadableIntoMany[O]", backend) - return self._streamed(batch, many.get_into_many(paths)) - return self._streamed(batch, run_many(backend.get_into, paths)) + return self._streamed(batch, many.get_into_many(paths, self._context)) + return self._streamed(batch, self._run_many(self._get_into, paths)) def put_many( self: Store[Writable], items: Iterable[tuple[str, Buffer]] @@ -434,17 +542,11 @@ def put_many( finish, no others start, and `BatchError` reports the keys written (`completed`, with their results) and the rest (`remaining`). """ - batch = list(items) - keys = [key for key, _ in batch] - paths = self._writable_paths(keys) - requests = [ - (path, byte_view(value, writable=False)) - for path, (_, value) in zip(paths, batch, strict=True) - ] + keys, requests = self._put_requests(items) backend = self._backend if isinstance(backend, WritableMany): - return self._finished(keys, backend.put_many(requests)) - return self._finished(keys, run_many(backend.put, requests)) + return self._finished(keys, backend.put_many(requests, self._context)) + return self._finished(keys, self._run_many(self._put, requests)) def delete_many( self: Store[Writable], keys: Iterable[str] @@ -454,16 +556,22 @@ def delete_many( paths = self._writable_paths(batch) backend = self._backend if isinstance(backend, WritableMany): - return self._finished(batch, backend.delete_many(paths)) - return self._finished(batch, run_many(backend.delete, [(path,) for path in paths])) + return self._finished(batch, backend.delete_many(paths, self._context)) + return self._finished(batch, self._run_many(self._delete, [(path,) for path in paths])) # ----------------------------------------------------------------- listing + def _open_items(self, items: Iterator[T]) -> Iterator[T]: + """`items`, refusing to continue once the backend closes.""" + for item in items: + self._check_open() + yield item + def list(self: Store[Listable], directory: str = "") -> Iterator[str]: """Every key under `directory`, relative to this store's root, in no particular order. The directory is checked when `list` is called; iterating after the store closes raises - `StoreClosedError`. + `StoreClosedError`. Listings hold no slot of the context's limiter. """ paths = self._backend.list(self._directory(directory)) return self._open_items(relativize(path, self._root) for path in paths) @@ -479,12 +587,13 @@ def list_dir(self: Store[Listable], directory: str = "") -> Iterator[ListItem]: @final class AsyncStore(_View[B_co]): - """A backend seen from a root directory, through coroutines. + """A backend seen from a root directory, under a context, through coroutines. - The async twin of `Store`: the same operations under the same names, awaited, over the - backend's async capabilities (`ReadableAsync`, `WritableAsync`, ...). Listings and lazy batches - are async iterators. Construct one from a backend, or take `store.as_async()`; `as_sync()` - goes back. Close it with `await store.aclose()` or `async with`. + The async twin of `Store`: the same operations under the same names, awaited. A backend's async + methods are used where it has them; otherwise its sync methods run on the context's thread + pool, so a backend that only blocks needs no async code. Listings and lazy batches are async + iterators. Construct one from a backend, or take `store.as_async()`; `as_sync()` goes back. + Close it with `await store.aclose()` or `async with`. Parameters ---------- @@ -495,16 +604,27 @@ class AsyncStore(_View[B_co]): read_only Refuse writes through this store. A backend that can't write makes every store over it read-only. + context + The resources operations draw on. Left out, `Context.default()`. """ __slots__ = () - def __init__(self, backend: B_co, *, root: str = "", read_only: bool = False) -> None: - super().__init__(backend, root=root, read_only=read_only) + def __init__( + self, + backend: B_co, + *, + root: str = "", + read_only: bool = False, + context: Context | None = None, + ) -> None: + super().__init__(backend, root=root, read_only=read_only, context=context) def as_sync(self) -> Store[B_co]: - """A `Store` view on the same backend and root; it doesn't own the backend.""" - view = Store(self._backend, root=self._root, read_only=self._read_only) + """A `Store` view on the same backend, root and context; it doesn't own the backend.""" + view = Store( + self._backend, root=self._root, read_only=self._read_only, context=self._context + ) view._owner = False return view @@ -534,7 +654,11 @@ async def __aexit__( ) -> None: await self.aclose() - # ----------------------------------------------------------------- batch results + # ----------------------------------------------------------------- batch plumbing + + async def _in_thread(self, fn: Callable[[*Ts], R], *args: *Ts) -> R: + """`fn(*args)` on the context's pool, holding a slot taken on the pool thread.""" + return await self._context.threads.run(_limited, self._limiter, fn, *args) async def _streamed( self, batch: Sequence[T], results: AsyncIterator[tuple[int, R]] @@ -567,102 +691,195 @@ async def _finished( raise _batch_error(group, batch, completed, completed.keys()) from None return [(request, completed[index]) for index, request in enumerate(batch)] + # ----------------------------------------------------------------- operations on backend paths + + async def _get( + self: AsyncStore[ReadableAsync] | AsyncStore[Readable], + path: str, + byte_range: ByteRange | None, + ) -> Found | Missing: + backend = self._backend + if isinstance(backend, ReadableAsync): + async with self._slot(): + return await backend.get_async(path, byte_range) + return await self._in_thread(backend.get, path, byte_range) + + async def _get_into( + self: AsyncStore[ReadableIntoAsync[O]] | AsyncStore[ReadableInto[O]], + path: str, + out: O, + byte_range: ByteRange | None, + ) -> Copied | Missing: + backend = self._backend + if isinstance(backend, ReadableIntoAsync): + async with self._slot(): + return await backend.get_into_async(path, out, byte_range) + return await self._in_thread(backend.get_into, path, out, byte_range) + + async def _put( + self: AsyncStore[WritableAsync] | AsyncStore[Writable], path: str, value: memoryview + ) -> Written: + backend = self._backend + if isinstance(backend, WritableAsync): + async with self._slot(): + return await backend.put_async(path, value) + return await self._in_thread(backend.put, path, value) + + async def _delete( + self: AsyncStore[WritableAsync] | AsyncStore[Writable], path: str + ) -> Deleted | Missing: + backend = self._backend + if isinstance(backend, WritableAsync): + async with self._slot(): + return await backend.delete_async(path) + return await self._in_thread(backend.delete, path) + # ----------------------------------------------------------------- reads async def get( - self: AsyncStore[ReadableAsync], key: str, *, byte_range: ByteRange | None = None + self: AsyncStore[ReadableAsync] | AsyncStore[Readable], + key: str, + *, + byte_range: ByteRange | None = None, ) -> Found | Missing: """As `Store.get`.""" - return await self._backend.get_async(self._path(key), byte_range) + return await self._get(self._path(key), byte_range) async def get_if_changed( - self: AsyncStore[ReadableAsync], + self: AsyncStore[ReadableAsync] | AsyncStore[Readable], key: str, generation: Generation, *, byte_range: ByteRange | None = None, ) -> Found | Missing | Unchanged: """As `Store.get_if_changed`.""" - return await self._backend.get_if_changed_async(self._path(key), generation, byte_range) + path = self._path(key) + backend = self._backend + if isinstance(backend, ReadableAsync): + async with self._slot(): + return await backend.get_if_changed_async(path, generation, byte_range) + return await self._in_thread(backend.get_if_changed, path, generation, byte_range) async def get_ranges( - self: AsyncStore[ReadableAsync], key: str, ranges: Sequence[ByteRange] + self: AsyncStore[ReadableAsync] | AsyncStore[Readable], + key: str, + ranges: Sequence[ByteRange], ) -> FoundRanges | Missing: """As `Store.get_ranges`.""" - return await self._backend.get_ranges_async(self._path(key), ranges) + path = self._path(key) + backend = self._backend + if isinstance(backend, ReadableAsync): + async with self._slot(): + return await backend.get_ranges_async(path, ranges) + return await self._in_thread(backend.get_ranges, path, ranges) async def get_into( - self: AsyncStore[ReadableIntoAsync[O]], + self: AsyncStore[ReadableIntoAsync[O]] | AsyncStore[ReadableInto[O]], key: str, out: O, *, byte_range: ByteRange | None = None, ) -> Copied | Missing: """As `Store.get_into`.""" - return await self._backend.get_into_async(self._path(key), out, byte_range) + return await self._get_into(self._path(key), out, byte_range) - async def head(self: AsyncStore[ReadableAsync], key: str) -> ObjectMetadata | Missing: + async def head( + self: AsyncStore[ReadableAsync] | AsyncStore[Readable], key: str + ) -> ObjectMetadata | Missing: """As `Store.head`.""" - return await self._backend.head_async(self._path(key)) + path = self._path(key) + backend = self._backend + if isinstance(backend, ReadableAsync): + async with self._slot(): + return await backend.head_async(path) + return await self._in_thread(backend.head, path) - async def exists(self: AsyncStore[ReadableAsync], key: str) -> bool: + async def exists(self: AsyncStore[ReadableAsync] | AsyncStore[Readable], key: str) -> bool: """As `Store.exists`.""" return not isinstance(await self.head(key), Missing) # ----------------------------------------------------------------- writes - async def put(self: AsyncStore[WritableAsync], key: str, value: Buffer) -> Written: + async def put( + self: AsyncStore[WritableAsync] | AsyncStore[Writable], key: str, value: Buffer + ) -> Written: """As `Store.put`.""" - return await self._backend.put_async( - self._writable_path(key), byte_view(value, writable=False) - ) + return await self._put(self._writable_path(key), byte_view(value, writable=False)) async def put_if_absent( - self: AsyncStore[WritableAsync], key: str, value: Buffer + self: AsyncStore[WritableAsync] | AsyncStore[Writable], key: str, value: Buffer ) -> Written | Rejected: """As `Store.put_if_absent`.""" - return await self._backend.put_if_absent_async( - self._writable_path(key), byte_view(value, writable=False) - ) + path = self._writable_path(key) + view = byte_view(value, writable=False) + backend = self._backend + if isinstance(backend, WritableAsync): + async with self._slot(): + return await backend.put_if_absent_async(path, view) + return await self._in_thread(backend.put_if_absent, path, view) async def put_if_match( - self: AsyncStore[WritableIfMatchAsync], key: str, value: Buffer, generation: Generation + self: AsyncStore[WritableIfMatchAsync] | AsyncStore[WritableIfMatch], + key: str, + value: Buffer, + generation: Generation, ) -> Written | Rejected | Missing: """As `Store.put_if_match`.""" - return await self._backend.put_if_match_async( - self._writable_path(key), byte_view(value, writable=False), generation - ) - - async def delete(self: AsyncStore[WritableAsync], key: str) -> Deleted | Missing: + path = self._writable_path(key) + view = byte_view(value, writable=False) + backend = self._backend + if isinstance(backend, WritableIfMatchAsync): + async with self._slot(): + return await backend.put_if_match_async(path, view, generation) + return await self._in_thread(backend.put_if_match, path, view, generation) + + async def delete( + self: AsyncStore[WritableAsync] | AsyncStore[Writable], key: str + ) -> Deleted | Missing: """As `Store.delete`.""" - return await self._backend.delete_async(self._writable_path(key)) + return await self._delete(self._writable_path(key)) async def delete_if_match( - self: AsyncStore[DeletableIfMatchAsync], key: str, generation: Generation + self: AsyncStore[DeletableIfMatchAsync] | AsyncStore[DeletableIfMatch], + key: str, + generation: Generation, ) -> Deleted | Rejected | Missing: """As `Store.delete_if_match`.""" - return await self._backend.delete_if_match_async(self._writable_path(key), generation) + path = self._writable_path(key) + backend = self._backend + if isinstance(backend, DeletableIfMatchAsync): + async with self._slot(): + return await backend.delete_if_match_async(path, generation) + return await self._in_thread(backend.delete_if_match, path, generation) - async def delete_dir(self: AsyncStore[WritableAsync], directory: str = "") -> None: + async def delete_dir( + self: AsyncStore[WritableAsync] | AsyncStore[Writable], directory: str = "" + ) -> None: """As `Store.delete_dir`.""" - await self._backend.delete_dir_async(self._writable_directory(directory)) + path = self._writable_directory(directory) + backend = self._backend + if isinstance(backend, WritableAsync): + async with self._slot(): + await backend.delete_dir_async(path) + return + await self._in_thread(backend.delete_dir, path) # ----------------------------------------------------------------- batches def get_many( - self: AsyncStore[ReadableAsync], requests: Iterable[Q] + self: AsyncStore[ReadableAsync] | AsyncStore[Readable], requests: Iterable[Q] ) -> AsyncIterator[tuple[Q, Found | Missing]]: - """As `Store.get_many`.""" + """As `Store.get_many`; the reads run as tasks, at most the limiter's limit at once.""" batch = list(requests) self._check_open() paths = [self._read_request(request) for request in batch] backend = self._backend if isinstance(backend, ReadableManyAsync): - return self._streamed(batch, backend.get_many_async(paths)) - return self._streamed(batch, run_many_async(backend.get_async, paths)) + return self._streamed(batch, backend.get_many_async(paths, self._context)) + return self._streamed(batch, run_many_async(self._get, paths, self._limiter.limit)) def get_into_many( - self: AsyncStore[ReadableIntoAsync[O]], + self: AsyncStore[ReadableIntoAsync[O]] | AsyncStore[ReadableInto[O]], requests: Iterable[tuple[str, O] | tuple[str, O, ByteRange | None]], ) -> AsyncIterator[tuple[tuple[str, O] | tuple[str, O, ByteRange | None], Copied | Missing]]: """As `Store.get_into_many`.""" @@ -672,48 +889,69 @@ def get_into_many( backend = self._backend if isinstance(backend, ReadableIntoManyAsync): many = cast("ReadableIntoManyAsync[O]", backend) - return self._streamed(batch, many.get_into_many_async(paths)) - return self._streamed(batch, run_many_async(backend.get_into_async, paths)) + return self._streamed(batch, many.get_into_many_async(paths, self._context)) + return self._streamed(batch, run_many_async(self._get_into, paths, self._limiter.limit)) async def put_many( - self: AsyncStore[WritableAsync], items: Iterable[tuple[str, Buffer]] + self: AsyncStore[WritableAsync] | AsyncStore[Writable], + items: Iterable[tuple[str, Buffer]], ) -> list[tuple[str, Written]]: """As `Store.put_many`.""" - batch = list(items) - keys = [key for key, _ in batch] - paths = self._writable_paths(keys) - requests = [ - (path, byte_view(value, writable=False)) - for path, (_, value) in zip(paths, batch, strict=True) - ] + keys, requests = self._put_requests(items) backend = self._backend if isinstance(backend, WritableManyAsync): - return await self._finished(keys, backend.put_many_async(requests)) - return await self._finished(keys, run_many_async(backend.put_async, requests)) + return await self._finished(keys, backend.put_many_async(requests, self._context)) + return await self._finished(keys, run_many_async(self._put, requests, self._limiter.limit)) async def delete_many( - self: AsyncStore[WritableAsync], keys: Iterable[str] + self: AsyncStore[WritableAsync] | AsyncStore[Writable], keys: Iterable[str] ) -> list[tuple[str, Deleted | Missing]]: """As `Store.delete_many`.""" batch = list(keys) paths = self._writable_paths(batch) backend = self._backend if isinstance(backend, WritableManyAsync): - return await self._finished(batch, backend.delete_many_async(paths)) + return await self._finished(batch, backend.delete_many_async(paths, self._context)) + requests = [(path,) for path in paths] return await self._finished( - batch, run_many_async(backend.delete_async, [(path,) for path in paths]) + batch, run_many_async(self._delete, requests, self._limiter.limit) ) # ----------------------------------------------------------------- listing - def list(self: AsyncStore[ListableAsync], directory: str = "") -> AsyncIterator[str]: + async def _open_items(self, items: AsyncIterator[T]) -> AsyncIterator[T]: + async for item in items: + self._check_open() + yield item + + async def _listed_in_thread( + self, fn: Callable[[str], Iterator[T]], directory: str + ) -> AsyncIterator[T]: + """A sync listing, taken whole on the context's pool.""" + for item in await self._context.threads.run(_list_all, fn, directory): + yield item + + def list( + self: AsyncStore[ListableAsync] | AsyncStore[Listable], directory: str = "" + ) -> AsyncIterator[str]: """As `Store.list`.""" - paths = self._backend.list_async(self._directory(directory)) - return self._open_items_async(_relativized(paths, self._root)) + path = self._directory(directory) + backend = self._backend + if isinstance(backend, ListableAsync): + paths = backend.list_async(path) + else: + paths = self._listed_in_thread(backend.list, path) + return self._open_items(_relativized(paths, self._root)) - def list_dir(self: AsyncStore[ListableAsync], directory: str = "") -> AsyncIterator[ListItem]: + def list_dir( + self: AsyncStore[ListableAsync] | AsyncStore[Listable], directory: str = "" + ) -> AsyncIterator[ListItem]: """As `Store.list_dir`.""" - return self._open_items_async(self._backend.list_dir_async(self._directory(directory))) + path = self._directory(directory) + backend = self._backend + if isinstance(backend, ListableAsync): + return self._open_items(backend.list_dir_async(path)) + return self._open_items(self._listed_in_thread(backend.list_dir, path)) def _batch_error( @@ -737,6 +975,6 @@ async def _relativized(paths: AsyncIterator[str], root: str) -> AsyncIterator[st def _restore_store( - cls: type[_View[Backend]], backend: Backend, root: str, read_only: bool + cls: type[_View[Backend]], backend: Backend, root: str, read_only: bool, context: Context ) -> _View[Backend]: - return cls(backend, root=root, read_only=read_only) + return cls(backend, root=root, read_only=read_only, context=context) diff --git a/packages/zarr-storage/src/zarr_storage/backends/_local.py b/packages/zarr-storage/src/zarr_storage/backends/_local.py index 5545ce5f02..9521876845 100644 --- a/packages/zarr-storage/src/zarr_storage/backends/_local.py +++ b/packages/zarr-storage/src/zarr_storage/backends/_local.py @@ -12,18 +12,15 @@ import time import uuid from pathlib import Path -from typing import TYPE_CHECKING, TypedDict, TypeVar, TypeVarTuple +from typing import TYPE_CHECKING, TypedDict, TypeVar -from zarr_storage._batch import run_many, run_many_async from zarr_storage._buffers import byte_view from zarr_storage._byte_range import resolve from zarr_storage._errors import ( InvalidKeyError, KeyConflictError, - StoreClosedError, UnsupportedOperationError, ) -from zarr_storage._resources import ThreadPool, default_thread_pool from zarr_storage._types import ( Copied, Deleted, @@ -40,7 +37,7 @@ from zarr_storage.backends._common import IfMatch, always, if_absent if TYPE_CHECKING: - from collections.abc import AsyncIterator, Callable, Generator, Iterator, Sequence + from collections.abc import Callable, Generator, Iterator, Sequence from io import BufferedReader from typing_extensions import Buffer @@ -71,14 +68,12 @@ ) _WINDOWS_FORBIDDEN_CHARACTERS = frozenset('<>:"\\|?*') | frozenset(chr(i) for i in range(32)) -Ts = TypeVarTuple("Ts") R = TypeVar("R") T = TypeVar("T") class _State(TypedDict): root: str - threads: ThreadPool closed: bool @@ -394,11 +389,6 @@ class LocalBackend: create Create `root` if it doesn't exist. With `create=False`, a missing root raises `FileNotFoundError`. - threads - The threads that run the async methods' file operations and the operations of batches - (`get_many`, `put_many`, ...), which run as many at once as the pool has threads. Share one - pool between backends to bound their threads together; the default is the process-wide - `ThreadPool.default()`. The backend never shuts the pool down. """ def __init__( @@ -406,14 +396,12 @@ def __init__( root: str | os.PathLike[str], *, create: bool = True, - threads: ThreadPool | None = None, ) -> None: self._root = Path(root).resolve() if create: self._root.mkdir(parents=True, exist_ok=True) elif not self._root.is_dir(): raise FileNotFoundError(f"The root directory {str(self._root)!r} does not exist.") - self._threads = default_thread_pool() if threads is None else threads self._probe_lock = threading.Lock() self._exact_times: bool | None = None self._closed = False @@ -423,26 +411,24 @@ def root(self) -> Path: """The directory that holds the keys.""" return self._root - @property - def threads(self) -> ThreadPool: - """The threads that run the async methods and batches.""" - return self._threads - def __repr__(self) -> str: return f"LocalBackend({str(self._root)!r})" def __getstate__(self) -> _State: - return {"root": str(self._root), "threads": self._threads, "closed": self._closed} + return {"root": str(self._root), "closed": self._closed} def __setstate__(self, state: _State) -> None: self._root = Path(state["root"]) - self._threads = state["threads"] self._probe_lock = threading.Lock() self._exact_times = None self._closed = state["closed"] # --- Backend + @property + def resource_key(self) -> str: + return "local" + @property def closed(self) -> bool: return self._closed @@ -452,7 +438,7 @@ def writable(self) -> bool: return True def close(self) -> None: - """Refuse further operations. Operations already running finish; the pool stays up.""" + """Refuse further operations; operations already running finish.""" self._closed = True # --- paths @@ -706,116 +692,6 @@ def list_dir(self, directory: str) -> Iterator[ListItem]: return iter(()) return _classify(entries) - # --- batches: the operations above, as many at once as the pool has threads - - def get_many( - self, requests: Sequence[tuple[str, ByteRange | None]] - ) -> Iterator[tuple[int, Found | Missing]]: - return run_many(self.get, requests, self._open_threads(), self._threads.max_workers) - - def get_into_many( - self, requests: Sequence[tuple[str, Buffer, ByteRange | None]] - ) -> Iterator[tuple[int, Copied | Missing]]: - return run_many(self.get_into, requests, self._open_threads(), self._threads.max_workers) - - def put_many(self, requests: Sequence[tuple[str, memoryview]]) -> Iterator[tuple[int, Written]]: - return run_many(self.put, requests, self._open_threads(), self._threads.max_workers) - - def delete_many(self, paths: Sequence[str]) -> Iterator[tuple[int, Deleted | Missing]]: - return run_many( - self.delete, - [(path,) for path in paths], - self._open_threads(), - self._threads.max_workers, - ) - - # --- async twins: run the sync methods on the backend's threads - - def _open_threads(self) -> ThreadPool: - if self._closed: - raise StoreClosedError(f"{self!r} is closed.") - return self._threads - - async def _run(self, fn: Callable[[*Ts], T], *args: *Ts) -> T: - return await self._open_threads().run(fn, *args) - - async def get_async(self, path: str, byte_range: ByteRange | None) -> Found | Missing: - return await self._run(self.get, path, byte_range) - - async def get_if_changed_async( - self, path: str, generation: Generation, byte_range: ByteRange | None - ) -> Found | Missing | Unchanged: - return await self._run(self.get_if_changed, path, generation, byte_range) - - async def get_ranges_async( - self, path: str, ranges: Sequence[ByteRange] - ) -> FoundRanges | Missing: - return await self._run(self.get_ranges, path, ranges) - - async def head_async(self, path: str) -> ObjectMetadata | Missing: - return await self._run(self.head, path) - - async def get_into_async( - self, path: str, out: Buffer, byte_range: ByteRange | None - ) -> Copied | Missing: - return await self._run(self.get_into, path, out, byte_range) - - async def put_async(self, path: str, value: memoryview) -> Written: - return await self._run(self.put, path, value) - - async def put_if_absent_async(self, path: str, value: memoryview) -> Written | Rejected: - return await self._run(self.put_if_absent, path, value) - - async def put_if_match_async( - self, path: str, value: memoryview, generation: Generation - ) -> Written | Rejected | Missing: - return await self._run(self.put_if_match, path, value, generation) - - async def delete_async(self, path: str) -> Deleted | Missing: - return await self._run(self.delete, path) - - async def delete_if_match_async( - self, path: str, generation: Generation - ) -> Deleted | Rejected | Missing: - return await self._run(self.delete_if_match, path, generation) - - async def delete_dir_async(self, directory: str) -> None: - await self._run(self.delete_dir, directory) - - def get_many_async( - self, requests: Sequence[tuple[str, ByteRange | None]] - ) -> AsyncIterator[tuple[int, Found | Missing]]: - return run_many_async(self.get_async, requests, self._threads.max_workers) - - def get_into_many_async( - self, requests: Sequence[tuple[str, Buffer, ByteRange | None]] - ) -> AsyncIterator[tuple[int, Copied | Missing]]: - return run_many_async(self.get_into_async, requests, self._threads.max_workers) - - def put_many_async( - self, requests: Sequence[tuple[str, memoryview]] - ) -> AsyncIterator[tuple[int, Written]]: - return run_many_async(self.put_async, requests, self._threads.max_workers) - - def delete_many_async( - self, paths: Sequence[str] - ) -> AsyncIterator[tuple[int, Deleted | Missing]]: - return run_many_async( - self.delete_async, [(path,) for path in paths], self._threads.max_workers - ) - - async def list_async(self, directory: str) -> AsyncIterator[str]: - for path in await self._run(_materialize, self.list(directory)): - yield path - - async def list_dir_async(self, directory: str) -> AsyncIterator[ListItem]: - for item in await self._run(_materialize, self.list_dir(directory)): - yield item - - -def _materialize(items: Iterator[T]) -> list[T]: - return list(items) - def _walk(directory: Path, prefix: str) -> Iterator[str]: try: diff --git a/packages/zarr-storage/src/zarr_storage/backends/_memory.py b/packages/zarr-storage/src/zarr_storage/backends/_memory.py index e7b9f7e335..69b5f0b231 100644 --- a/packages/zarr-storage/src/zarr_storage/backends/_memory.py +++ b/packages/zarr-storage/src/zarr_storage/backends/_memory.py @@ -79,6 +79,10 @@ def __setstate__(self, state: _State) -> None: # --- Backend + @property + def resource_key(self) -> str: + return "memory" + @property def closed(self) -> bool: return self._closed diff --git a/packages/zarr-storage/src/zarr_storage/backends/_obstore.py b/packages/zarr-storage/src/zarr_storage/backends/_obstore.py index 27e7ed771f..e2038b06a3 100644 --- a/packages/zarr-storage/src/zarr_storage/backends/_obstore.py +++ b/packages/zarr-storage/src/zarr_storage/backends/_obstore.py @@ -24,12 +24,10 @@ ) from obstore.store import HTTPStore, LocalStore -from zarr_storage._batch import run_many_async from zarr_storage._buffers import byte_view, copy_into from zarr_storage._byte_range import resolve from zarr_storage._errors import InvalidKeyError, KeyConflictError, UnsupportedOperationError from zarr_storage._keys import check_key, relativize -from zarr_storage._resources import Limiter, default_limiter from zarr_storage._types import ( Deleted, Found, @@ -268,12 +266,6 @@ class ObjectStoreBackend: The obstore store. writable Whether to accept writes. Defaults to `False` for `HTTPStore`, `True` otherwise. - concurrency - The limit on this backend's operations in flight, sync and async, single and batched - (listings are not counted). Share one `Limiter` between backends to bound them together. - The default is a process-wide limiter for the store's class, so that every S3 backend in - the process shares one limit, as TensorStore's `s3_request_concurrency` does. Async batches - start at most `concurrency.limit` operations at once; sync batches run one at a time. """ def __init__( @@ -281,14 +273,8 @@ def __init__( store: ObjectStore, *, writable: bool | None = None, - concurrency: Limiter | None = None, ) -> None: self._store = store - self._concurrency = ( - default_limiter(f"obstore.{type(store).__name__}") - if concurrency is None - else concurrency - ) self._writable = not isinstance(store, HTTPStore) if writable is None else writable # Only a filesystem refuses a path that lies under a file; elsewhere a refusal is an error. self._filesystem = isinstance(store, LocalStore) @@ -300,9 +286,10 @@ def store(self) -> ObjectStore: return self._store @property - def concurrency(self) -> Limiter: - """The limit on this backend's operations in flight.""" - return self._concurrency + def resource_key(self) -> str: + """`"obstore."` and the store's class name, so that, for instance, every S3 backend shares + one limit by default, as TensorStore's `s3_request_concurrency` does.""" + return f"obstore.{type(self._store).__name__}" def __repr__(self) -> str: return f"ObjectStoreBackend({self._store!r})" @@ -418,67 +405,59 @@ async def _conflicts_async(self, path: str) -> bool: # --- Readable def head(self, path: str) -> ObjectMetadata | Missing: - with self._concurrency.slot(): - try: - return _metadata(obstore.head(self._store, path)) - except FileNotFoundError: + try: + return _metadata(obstore.head(self._store, path)) + except FileNotFoundError: + return Missing(time.time()) + except GenericError as e: + if self._refused_under_a_key(e, path): return Missing(time.time()) - except GenericError as e: - if self._refused_under_a_key(e, path): - return Missing(time.time()) - raise + raise async def head_async(self, path: str) -> ObjectMetadata | Missing: - async with self._concurrency.slot(): - try: - return _metadata(await obstore.head_async(self._store, path)) - except FileNotFoundError: + try: + return _metadata(await obstore.head_async(self._store, path)) + except FileNotFoundError: + return Missing(time.time()) + except GenericError as e: + if await self._refused_under_a_key_async(e, path): return Missing(time.time()) - except GenericError as e: - if await self._refused_under_a_key_async(e, path): - return Missing(time.time()) - raise + raise def get(self, path: str, byte_range: ByteRange | None) -> Found | Missing: - with self._concurrency.slot(): - return self._get(path, byte_range, {}) + return self._get(path, byte_range, {}) async def get_async(self, path: str, byte_range: ByteRange | None) -> Found | Missing: - async with self._concurrency.slot(): - return await self._get_async(path, byte_range, {}) + return await self._get_async(path, byte_range, {}) def get_into(self, path: str, out: Buffer, byte_range: ByteRange | None) -> Copied | Missing: """A read, copied into `out`: obstore can't read into a caller's buffer.""" - with self._concurrency.slot(): - view = byte_view(out, writable=True) - return copy_into(self.get(path, byte_range), view) + view = byte_view(out, writable=True) + return copy_into(self.get(path, byte_range), view) async def get_into_async( self, path: str, out: Buffer, byte_range: ByteRange | None ) -> Copied | Missing: - async with self._concurrency.slot(): - view = byte_view(out, writable=True) - return copy_into(await self.get_async(path, byte_range), view) + view = byte_view(out, writable=True) + return copy_into(await self.get_async(path, byte_range), view) def get_if_changed( self, path: str, generation: Generation, byte_range: ByteRange | None ) -> Found | Missing | Unchanged: - with self._concurrency.slot(): - try: - result = self._get(path, byte_range, _if_none_match(generation)) - except NotModifiedError: - return Unchanged(generation, time.time()) - return _unless_unchanged(result, generation) + try: + result = self._get(path, byte_range, _if_none_match(generation)) + except NotModifiedError: + return Unchanged(generation, time.time()) + return _unless_unchanged(result, generation) async def get_if_changed_async( self, path: str, generation: Generation, byte_range: ByteRange | None ) -> Found | Missing | Unchanged: - async with self._concurrency.slot(): - try: - result = await self._get_async(path, byte_range, _if_none_match(generation)) - except NotModifiedError: - return Unchanged(generation, time.time()) - return _unless_unchanged(result, generation) + try: + result = await self._get_async(path, byte_range, _if_none_match(generation)) + except NotModifiedError: + return Unchanged(generation, time.time()) + return _unless_unchanged(result, generation) def _get(self, path: str, byte_range: ByteRange | None, options: GetOptions) -> Found | Missing: """A read with extra `options`; raises `NotModifiedError` if they include a matching @@ -509,29 +488,26 @@ async def _get_async( return _found(result, memoryview(await result.bytes_async())) def get_ranges(self, path: str, ranges: Sequence[ByteRange]) -> FoundRanges | Missing: - with self._concurrency.slot(): - if len(ranges) == 0 or _is_empty_request(ranges[0]): - return self._read_whole_and_slice(path, ranges) - try: - first = obstore.get(self._store, path, options=_get_options(ranges[0])) - except FileNotFoundError: - return Missing(time.time()) - except GenericError: - # The first range starts at the end of the value; one whole read answers every range. - return self._read_whole_and_slice(path, ranges) - meta = first.meta - generation = _generation(meta["e_tag"], meta["version"]) - pinned = _pinned(generation) - if pinned is None: - return self._read_whole_and_slice(path, ranges) - try: - rest = [ - self._read_pinned(path, pinned, *resolve(r, meta["size"])) for r in ranges[1:] - ] - except (PreconditionError, FileNotFoundError): - # The value changed after the first read; one whole read is consistent by itself. - return self._read_whole_and_slice(path, ranges) - return FoundRanges((memoryview(first.bytes()), *rest), generation, time.time()) + if len(ranges) == 0 or _is_empty_request(ranges[0]): + return self._read_whole_and_slice(path, ranges) + try: + first = obstore.get(self._store, path, options=_get_options(ranges[0])) + except FileNotFoundError: + return Missing(time.time()) + except GenericError: + # The first range starts at the end of the value; one whole read answers every range. + return self._read_whole_and_slice(path, ranges) + meta = first.meta + generation = _generation(meta["e_tag"], meta["version"]) + pinned = _pinned(generation) + if pinned is None: + return self._read_whole_and_slice(path, ranges) + try: + rest = [self._read_pinned(path, pinned, *resolve(r, meta["size"])) for r in ranges[1:]] + except (PreconditionError, FileNotFoundError): + # The value changed after the first read; one whole read is consistent by itself. + return self._read_whole_and_slice(path, ranges) + return FoundRanges((memoryview(first.bytes()), *rest), generation, time.time()) def _read_pinned(self, path: str, pinned: GetOptions, start: int, end: int) -> memoryview: if start == end: @@ -557,32 +533,29 @@ def _read_whole_and_slice( async def get_ranges_async( self, path: str, ranges: Sequence[ByteRange] ) -> FoundRanges | Missing: - async with self._concurrency.slot(): - if len(ranges) == 0 or _is_empty_request(ranges[0]): - return await self._read_whole_and_slice_async(path, ranges) - try: - first = await obstore.get_async(self._store, path, options=_get_options(ranges[0])) - except FileNotFoundError: - return Missing(time.time()) - except GenericError: - return await self._read_whole_and_slice_async(path, ranges) - meta = first.meta - generation = _generation(meta["e_tag"], meta["version"]) - pinned = _pinned(generation) - if pinned is None: - return await self._read_whole_and_slice_async(path, ranges) - try: - rest = await asyncio.gather( - *( - _read_pinned_async(self._store, path, pinned, *resolve(r, meta["size"])) - for r in ranges[1:] - ) + if len(ranges) == 0 or _is_empty_request(ranges[0]): + return await self._read_whole_and_slice_async(path, ranges) + try: + first = await obstore.get_async(self._store, path, options=_get_options(ranges[0])) + except FileNotFoundError: + return Missing(time.time()) + except GenericError: + return await self._read_whole_and_slice_async(path, ranges) + meta = first.meta + generation = _generation(meta["e_tag"], meta["version"]) + pinned = _pinned(generation) + if pinned is None: + return await self._read_whole_and_slice_async(path, ranges) + try: + rest = await asyncio.gather( + *( + _read_pinned_async(self._store, path, pinned, *resolve(r, meta["size"])) + for r in ranges[1:] ) - except (PreconditionError, FileNotFoundError): - return await self._read_whole_and_slice_async(path, ranges) - return FoundRanges( - (memoryview(await first.bytes_async()), *rest), generation, time.time() ) + except (PreconditionError, FileNotFoundError): + return await self._read_whole_and_slice_async(path, ranges) + return FoundRanges((memoryview(await first.bytes_async()), *rest), generation, time.time()) async def _read_whole_and_slice_async( self, path: str, ranges: Sequence[ByteRange] @@ -602,38 +575,32 @@ async def _read_whole_and_slice_async( # --- Writable def put(self, path: str, value: memoryview) -> Written: - with self._concurrency.slot(): - return self._put(path, value, None, always) + return self._put(path, value, None, always) async def put_async(self, path: str, value: memoryview) -> Written: - async with self._concurrency.slot(): - return await self._put_async(path, value, None, always) + return await self._put_async(path, value, None, always) def put_if_absent(self, path: str, value: memoryview) -> Written | Rejected: - with self._concurrency.slot(): - return self._put(path, value, "create", if_absent) + return self._put(path, value, "create", if_absent) async def put_if_absent_async(self, path: str, value: memoryview) -> Written | Rejected: - async with self._concurrency.slot(): - return await self._put_async(path, value, "create", if_absent) + return await self._put_async(path, value, "create", if_absent) def put_if_match( self, path: str, value: memoryview, generation: Generation ) -> Written | Rejected | Missing: - with self._concurrency.slot(): - mode = _if_match_mode(generation) - if mode is None: - return _refuse(self.head(path)) - return self._put(path, value, mode, IfMatch(generation)) + mode = _if_match_mode(generation) + if mode is None: + return _refuse(self.head(path)) + return self._put(path, value, mode, IfMatch(generation)) async def put_if_match_async( self, path: str, value: memoryview, generation: Generation ) -> Written | Rejected | Missing: - async with self._concurrency.slot(): - mode = _if_match_mode(generation) - if mode is None: - return _refuse(await self.head_async(path)) - return await self._put_async(path, value, mode, IfMatch(generation)) + mode = _if_match_mode(generation) + if mode is None: + return _refuse(await self.head_async(path)) + return await self._put_async(path, value, mode, IfMatch(generation)) def _put( self, @@ -711,73 +678,45 @@ async def _put_async( raise OSError(f"Gave up writing {path!r}: the store kept refusing it.") def delete(self, path: str) -> Deleted | Missing: - with self._concurrency.slot(): - meta = self.head(path) - if isinstance(meta, Missing): - return meta - try: - obstore.delete(self._store, path) - except FileNotFoundError: - return Missing(time.time()) - return Deleted(meta.generation) + meta = self.head(path) + if isinstance(meta, Missing): + return meta + try: + obstore.delete(self._store, path) + except FileNotFoundError: + return Missing(time.time()) + return Deleted(meta.generation) async def delete_async(self, path: str) -> Deleted | Missing: - async with self._concurrency.slot(): - meta = await self.head_async(path) - if isinstance(meta, Missing): - return meta - try: - await obstore.delete_async(self._store, path) - except FileNotFoundError: - return Missing(time.time()) - return Deleted(meta.generation) + meta = await self.head_async(path) + if isinstance(meta, Missing): + return meta + try: + await obstore.delete_async(self._store, path) + except FileNotFoundError: + return Missing(time.time()) + return Deleted(meta.generation) def delete_dir(self, directory: str) -> None: - with self._concurrency.slot(): - paths = list(self._paths(directory)) - try: - if paths: - obstore.delete(self._store, paths) - except FileNotFoundError: - # A concurrent delete removed some first; delete the rest one at a time. - for path in paths: - with contextlib.suppress(FileNotFoundError): - obstore.delete(self._store, path) + paths = list(self._paths(directory)) + try: + if paths: + obstore.delete(self._store, paths) + except FileNotFoundError: + # A concurrent delete removed some first; delete the rest one at a time. + for path in paths: + with contextlib.suppress(FileNotFoundError): + obstore.delete(self._store, path) async def delete_dir_async(self, directory: str) -> None: - async with self._concurrency.slot(): - paths = [path async for path in self._paths_async(directory)] - try: - if paths: - await obstore.delete_async(self._store, paths) - except FileNotFoundError: - for path in paths: - with contextlib.suppress(FileNotFoundError): - await obstore.delete_async(self._store, path) - - # --- async batches: the async operations, at most `concurrency.limit` started at once - - def get_many_async( - self, requests: Sequence[tuple[str, ByteRange | None]] - ) -> AsyncIterator[tuple[int, Found | Missing]]: - return run_many_async(self.get_async, requests, self._concurrency.limit) - - def get_into_many_async( - self, requests: Sequence[tuple[str, Buffer, ByteRange | None]] - ) -> AsyncIterator[tuple[int, Copied | Missing]]: - return run_many_async(self.get_into_async, requests, self._concurrency.limit) - - def put_many_async( - self, requests: Sequence[tuple[str, memoryview]] - ) -> AsyncIterator[tuple[int, Written]]: - return run_many_async(self.put_async, requests, self._concurrency.limit) - - def delete_many_async( - self, paths: Sequence[str] - ) -> AsyncIterator[tuple[int, Deleted | Missing]]: - return run_many_async( - self.delete_async, [(path,) for path in paths], self._concurrency.limit - ) + paths = [path async for path in self._paths_async(directory)] + try: + if paths: + await obstore.delete_async(self._store, paths) + except FileNotFoundError: + for path in paths: + with contextlib.suppress(FileNotFoundError): + await obstore.delete_async(self._store, path) # --- Listable diff --git a/packages/zarr-storage/src/zarr_storage/testing/_api.py b/packages/zarr-storage/src/zarr_storage/testing/_api.py index 49ca4415e6..e4f1a5cbd4 100644 --- a/packages/zarr-storage/src/zarr_storage/testing/_api.py +++ b/packages/zarr-storage/src/zarr_storage/testing/_api.py @@ -13,15 +13,10 @@ from zarr_storage._protocols import ( Listable, - ListableAsync, Readable, - ReadableAsync, ReadableInto, - ReadableIntoAsync, Writable, - WritableAsync, WritableIfMatch, - WritableIfMatchAsync, ) from zarr_storage._store import AsyncStore @@ -59,15 +54,12 @@ class FullBackend( Writable, WritableIfMatch, Listable, - ReadableAsync, - ReadableIntoAsync[Buffer], - WritableAsync, - WritableIfMatchAsync, - ListableAsync, Protocol, ): - """A backend with every capability the suite needs of all backends, sync and async. + """A backend with every capability the suite needs of all backends. + Only the sync protocols: `AsyncStore` uses a backend's async twins where it has them and runs + its sync methods on its context's thread pool otherwise, so the suite's async run covers both. `DeletableIfMatch` is left out, since object stores lack it; `delete_if_match` reaches it. """ diff --git a/packages/zarr-storage/tests/test_resources.py b/packages/zarr-storage/tests/test_resources.py index bf9911bd64..aa6d925bb9 100644 --- a/packages/zarr-storage/tests/test_resources.py +++ b/packages/zarr-storage/tests/test_resources.py @@ -1,4 +1,4 @@ -"""Shared resources: thread pools and limiters, alone and as backends use them.""" +"""Contexts and their resources, alone and as stores draw on them.""" from __future__ import annotations @@ -10,16 +10,18 @@ from typing import TYPE_CHECKING import pytest -from obstore.store import LocalStore, MemoryStore +from obstore.store import MemoryStore -from zarr_storage import Copied, Deleted, Found, Limiter, Missing, Store, ThreadPool -from zarr_storage.backends import LocalBackend, ObjectStoreBackend +from zarr_storage import Context, Found, Limiter, Missing, Store, ThreadPool +from zarr_storage._resources import default_limiter +from zarr_storage.backends import LocalBackend, MemoryBackend, ObjectStoreBackend if TYPE_CHECKING: from collections.abc import Callable from pathlib import Path from zarr_storage import ByteRange + from zarr_storage.testing import FullBackend class _Peak: @@ -42,11 +44,13 @@ def leave(self) -> None: self.done += 1 -def _hold_sync(limiter: Limiter, peak: _Peak) -> None: +def _hold_sync(limiter: Limiter, peaks: list[_Peak]) -> None: with limiter.slot(): - peak.enter() + for peak in peaks: + peak.enter() time.sleep(0.002) - peak.leave() + for peak in peaks: + peak.leave() async def _hold_async(limiter: Limiter, peak: _Peak) -> None: @@ -72,6 +76,13 @@ def _finishes(fn: Callable[[], object]) -> bool: return not worker.is_alive() +def _join_all(threads: list[threading.Thread]) -> None: + for thread in threads: + thread.start() + for thread in threads: + thread.join() + + # ---------------------------------------------------------------------------- Limiter @@ -79,30 +90,45 @@ def _finishes(fn: Callable[[], object]) -> bool: def test_limiter_bounds_threads_and_tasks_together(limit: int) -> None: """Threads and tasks on two event loops never hold more than `limit` slots at once.""" limiter, peak = Limiter(limit), _Peak() - workers = [threading.Thread(target=_hold_sync, args=(limiter, peak)) for _ in range(8)] + workers = [threading.Thread(target=_hold_sync, args=(limiter, [peak])) for _ in range(8)] workers += [threading.Thread(target=_run_loop, args=(limiter, peak, 8)) for _ in range(2)] - for worker in workers: - worker.start() - for worker in workers: - worker.join() + _join_all(workers) assert (peak.done, peak.most, limiter.in_use) == (24, limit, 0) -async def _nested_async(limiter: Limiter) -> int: - async with limiter.slot(), limiter.slot(): - return limiter.in_use +def test_nested_limiter_holds_slots_of_both() -> None: + """A limiter `within` another is bounded by its own limit, and shares the wider one.""" + wide = Limiter(3) + narrow = Limiter(2, within=wide) + narrow_peak, wide_peak = _Peak(), _Peak() + workers = [ + threading.Thread(target=_hold_sync, args=(narrow, [narrow_peak, wide_peak])) + for _ in range(8) + ] + workers += [threading.Thread(target=_hold_sync, args=(wide, [wide_peak])) for _ in range(8)] + _join_all(workers) + assert (narrow_peak.most, wide_peak.most, wide_peak.done) == (2, 3, 16) + assert (narrow.in_use, wide.in_use) == (0, 0) -def _nested_sync(limiter: Limiter, seen: list[int]) -> None: - with limiter.slot(), limiter.slot(): - seen.append(limiter.in_use) - seen.append(asyncio.run(_nested_async(limiter))) +def _wait_for_slot(limiter: Limiter) -> None: + with limiter.slot(): + pass -def test_slot_inside_a_slot_takes_no_second_slot() -> None: - limiter, seen = Limiter(1), [] - assert _finishes(partial(_nested_sync, limiter, seen)) - assert (seen, limiter.in_use) == ([1, 1], 0) +def test_waiting_on_a_narrow_limit_holds_none_of_the_wide_one() -> None: + """Slots are taken innermost first, so a burst queued on a narrow limit leaves the wide one + free for everyone else.""" + wide = Limiter(2) + narrow = Limiter(1, within=wide) + with narrow.slot(): + waiter = threading.Thread(target=_wait_for_slot, args=(narrow,)) + waiter.start() + waiter.join(0.05) + assert waiter.is_alive() + assert wide.in_use == 1 + waiter.join(10) + assert (narrow.in_use, wide.in_use) == (0, 0) async def _take_in_order(limiter: Limiter, name: str, order: list[str]) -> None: @@ -144,6 +170,28 @@ def test_cancelled_waiter_gives_back_its_slot(granted_first: bool) -> None: assert asyncio.run(_cancel_while_waiting(granted_first)) == 0 +async def _cancel_between_slots() -> tuple[int, int]: + wide = Limiter(1) + narrow = Limiter(1, within=wide) + await wide.acquire_async() + entering = asyncio.create_task(_wait_for_slot_async(narrow)) + await asyncio.sleep(0) # holds the narrow slot, waits for the wide one + entering.cancel() + with pytest.raises(asyncio.CancelledError): + await entering + wide.release() + return narrow.in_use, wide.in_use + + +async def _wait_for_slot_async(limiter: Limiter) -> None: + async with limiter.slot(): + pass + + +def test_cancelled_while_taking_a_nested_slot_gives_back_the_inner_one() -> None: + assert asyncio.run(_cancel_between_slots()) == (0, 0) + + @pytest.mark.parametrize(("limit", "error"), [(0, ValueError), (1.5, TypeError), (True, TypeError)]) def test_limiter_of_a_bad_limit_raises(limit: object, error: type[Exception]) -> None: with pytest.raises(error): @@ -172,38 +220,6 @@ def test_thread_pool_knows_its_own_threads() -> None: other.shutdown() -class _RecordingBackend(LocalBackend): - """A local backend that records which threads its reads run on.""" - - def __init__(self, root: Path, threads: ThreadPool) -> None: - super().__init__(root, threads=threads) - self.read_on: set[int] = set() - - def get(self, path: str, byte_range: ByteRange | None) -> Found | Missing: - self.read_on.add(threading.get_ident()) - return super().get(path, byte_range) - - -def _batch_here(store: Store[_RecordingBackend], keys: list[str]) -> int: - assert all(isinstance(result, Found) for _, result in store.get_many(keys)) - return threading.get_ident() - - -def test_batch_from_a_thread_of_its_own_pool_runs_on_that_thread(tmp_path: Path) -> None: - """A batch started on one of its pool's threads must not wait for the pool's other threads, - which may all be waiting the same way; it runs its reads on the thread that started it.""" - pool = ThreadPool(2) - backend = _RecordingBackend(tmp_path, pool) - store = Store(backend) - keys = [f"k{i}" for i in range(4)] - store.put_many((key, b"x") for key in keys) - try: - worker = pool.submit(_batch_here, store, keys).result(timeout=10) - finally: - pool.shutdown(wait=False) - assert backend.read_on == {worker} - - @pytest.mark.parametrize(("size", "error"), [(0, ValueError), (2.0, TypeError)]) def test_thread_pool_of_a_bad_size_raises(size: object, error: type[Exception]) -> None: with pytest.raises(error): @@ -215,50 +231,77 @@ def test_shutting_down_the_default_pool_raises() -> None: ThreadPool.default().shutdown() -# ---------------------------------------------------------------------------- sharing - - -def test_backends_share_resources_as_given(tmp_path: Path) -> None: - """Left out, resources are process-wide defaults; given, they are exactly what was given.""" - assert LocalBackend(tmp_path / "a").threads is ThreadPool.default() - pool = ThreadPool(2) - assert LocalBackend(tmp_path / "b", threads=pool).threads is pool - memory = [ObjectStoreBackend(MemoryStore()).concurrency for _ in range(2)] - local = ObjectStoreBackend(LocalStore(str(tmp_path))).concurrency - assert memory[0] is memory[1] - assert local is not memory[0] - limiter = Limiter(3) - assert ObjectStoreBackend(MemoryStore(), concurrency=limiter).concurrency is limiter +# ---------------------------------------------------------------------------- Context + + +def test_context_resolves_resources_through_its_parents() -> None: + """Own resources first, then the parent's, then the process-wide defaults.""" + pool, a, b = ThreadPool(2), Limiter(4), Limiter(5) + default = Context.default() + assert default.threads is ThreadPool.default() + assert default.limiter("x") is default_limiter("x") + own = Context(threads=pool, limits={"a": a}) + assert (own.threads, own.limiter("a"), own.limiter("x")) == (pool, a, default_limiter("x")) + child = own.child(limits={"b": b}) + assert (child.threads, child.limiter("a"), child.limiter("b")) == (pool, a, b) + assert child.child(threads=ThreadPool.default()).threads is ThreadPool.default() + narrowed = child.limited("a", 2).limiter("a") + assert (narrowed.limit, narrowed.within) == (2, a) + + +@pytest.mark.parametrize( + "make", + [ + partial(Context, limits={"a": 4}), # pyright: ignore[reportArgumentType] + partial(Context, threads=4), # pyright: ignore[reportArgumentType] + ], + ids=["limit-not-a-limiter", "threads-not-a-pool"], +) +def test_context_of_a_bad_resource_raises(make: Callable[[], Context]) -> None: + with pytest.raises(TypeError): + make() + + +def test_views_carry_their_context() -> None: + """Views keep their context; `with_context` and `limited` change it and nothing else.""" + context = Context(limits={"memory": Limiter(8)}) + store = Store(MemoryBackend(), root="r", context=context) + for view in (store / "a", store.with_read_only(), store.as_async(), store.as_async().as_sync()): + assert view.context is context + other = Context() + assert store.with_context(other).context is other + limited = store.limited(2) + assert limited.context.parent is context + assert limited.context.limiter("memory").limit == 2 + assert limited.context.limiter("memory").within is context.limiter("memory") + assert (limited.root, limited.shares_backend(store)) == ("r", True) def test_pickling_keeps_resources_shared(tmp_path: Path) -> None: - """Backends that shared a resource share one after a round trip; defaults stay defaults.""" - pool, limiter = ThreadPool(2), Limiter(3) - objects = str(tmp_path) # obstore's `MemoryStore` can't be pickled; its `LocalStore` can - backends = ( - LocalBackend(tmp_path / "a", threads=pool), - LocalBackend(tmp_path / "b", threads=pool), - ObjectStoreBackend(LocalStore(objects), concurrency=limiter), - ObjectStoreBackend(LocalStore(objects), concurrency=limiter), - LocalBackend(tmp_path / "c"), - ObjectStoreBackend(LocalStore(objects)), + """Stores that shared a context share one after a round trip; defaults stay defaults.""" + wide = Limiter(8) + context = Context(threads=ThreadPool(2), limits={"local": Limiter(3, within=wide)}) + stores = ( + Store(LocalBackend(tmp_path / "a"), context=context), + Store(LocalBackend(tmp_path / "b"), context=context), + Store(LocalBackend(tmp_path / "c")), ) - a, b, c, d, e, f = pickle.loads(pickle.dumps(backends)) - assert a.threads is b.threads - assert a.threads is not pool - assert a.threads.max_workers == 2 - assert c.concurrency is d.concurrency - assert c.concurrency is not limiter - assert c.concurrency.limit == 3 - assert e.threads is ThreadPool.default() - assert f.concurrency is backends[5].concurrency - - -def _get_in_thread(store: Store[ObjectStoreBackend], results: list[object]) -> None: + a, b, c = pickle.loads(pickle.dumps(stores)) + assert a.context is b.context + assert a.context is not context + limiter = a.context.limiter("local") + assert (a.context.threads.max_workers, limiter.limit, limiter.within.limit) == (2, 3, 8) + assert c.context is Context.default() + + +# ---------------------------------------------------------------------------- stores under limits + + +def _get_in_thread(store: Store[FullBackend], results: list[object]) -> None: results.append(store.get("k")) -async def _get_async_while_held(store: Store[ObjectStoreBackend]) -> bool: +async def _get_async_while_held(store: Store[FullBackend]) -> bool: try: await asyncio.wait_for(store.as_async().get("k"), 0.05) except TimeoutError: @@ -266,34 +309,123 @@ async def _get_async_while_held(store: Store[ObjectStoreBackend]) -> bool: return False -def test_obstore_operations_wait_for_a_shared_limit() -> None: - """Every operation, sync or async, of every backend sharing a limiter waits for a slot.""" - limiter = Limiter(1) - first = Store(ObjectStoreBackend(MemoryStore(), concurrency=limiter)) - second = Store(ObjectStoreBackend(MemoryStore(), concurrency=limiter)) - second.put("k", b"x") - limiter.acquire() # another backend's operation holds the only slot +@pytest.mark.parametrize("backend", ["local", "obstore"]) +def test_operations_wait_for_the_contexts_limiter(tmp_path: Path, backend: str) -> None: + """Sync and async operations, on native-async and sync-only backends, wait for a slot.""" + raw: FullBackend = ( + LocalBackend(tmp_path) if backend == "local" else ObjectStoreBackend(MemoryStore()) + ) + pool, limiter = ThreadPool(2), Limiter(1) + store = Store(raw, context=Context(threads=pool, limits={raw.resource_key: limiter})) + store.put("k", b"x") + limiter.acquire() # some other operation under this context holds the only slot results: list[object] = [] - reader = threading.Thread(target=_get_in_thread, args=(second, results)) + reader = threading.Thread(target=_get_in_thread, args=(store, results)) reader.start() reader.join(0.05) assert reader.is_alive() - assert asyncio.run(_get_async_while_held(first)) + assert asyncio.run(_get_async_while_held(store)) limiter.release() reader.join(10) + pool.shutdown() # the timed-out async read's thread finishes too assert isinstance(results[0], Found) assert limiter.in_use == 0 -def _nested_operations(store: Store[ObjectStoreBackend], results: list[object]) -> None: - store.put("k", b"xy") - results.append(store.get_into("k", bytearray(2))) - results.append(store.delete("k")) +class _PeakBackend(MemoryBackend): + """A memory backend whose reads take a moment and record how many overlap.""" + def __init__(self) -> None: + super().__init__() + self.peak = _Peak() -def test_obstore_operation_that_calls_another_takes_one_slot() -> None: - """`get_into` reads through `get` and `delete` looks with `head`: with one slot, neither waits.""" - store = Store(ObjectStoreBackend(MemoryStore(), concurrency=Limiter(1))) - results: list[object] = [] - assert _finishes(partial(_nested_operations, store, results)) - assert [type(r) for r in results] == [Copied, Deleted] + def get(self, path: str, byte_range: ByteRange | None) -> Found | Missing: + self.peak.enter() + time.sleep(0.005) + self.peak.leave() + return super().get(path, byte_range) + + +def test_limited_view_bounds_its_batches() -> None: + backend = _PeakBackend() + store = Store(backend, context=Context(threads=ThreadPool(8))) + keys = [f"k{i}" for i in range(24)] + store.put_many((key, b"x") for key in keys) + assert len(list(store.limited(2).get_many(keys))) == 24 + assert backend.peak.most == 2 + backend.peak.most = 0 + assert len(list(store.get_many(keys))) == 24 + assert backend.peak.most > 2 + + +class _RecordingBackend(LocalBackend): + """A local backend that records which threads its reads run on.""" + + def __init__(self, root: Path) -> None: + super().__init__(root) + self.read_on: set[int] = set() + + def get(self, path: str, byte_range: ByteRange | None) -> Found | Missing: + self.read_on.add(threading.get_ident()) + return super().get(path, byte_range) + + +def _batch_here(store: Store[_RecordingBackend], keys: list[str]) -> int: + assert all(isinstance(result, Found) for _, result in store.get_many(keys)) + return threading.get_ident() + + +def test_batch_from_a_thread_of_its_own_pool_runs_on_that_thread(tmp_path: Path) -> None: + """A batch started on one of its pool's threads must not wait for the pool's other threads, + which may all be waiting the same way; it runs its reads on the thread that started it.""" + pool = ThreadPool(2) + backend = _RecordingBackend(tmp_path) + store = Store(backend, context=Context(threads=pool)) + keys = [f"k{i}" for i in range(4)] + store.put_many((key, b"x") for key in keys) + backend.read_on.clear() + try: + worker = pool.submit(_batch_here, store, keys).result(timeout=10) + finally: + pool.shutdown(wait=False) + assert backend.read_on == {worker} + + +def _read_batch(store: Store[LocalBackend], results: list[object]) -> None: + results.extend(result for _, result in store.get_many(["k"])) + + +def _read_async(store: Store[LocalBackend], results: list[object]) -> None: + results.append(asyncio.run(store.as_async().get("k"))) + + +def _settle() -> None: + time.sleep(0.05) + + +def test_async_read_of_a_sync_backend_holds_no_slot_while_it_waits_for_a_thread( + tmp_path: Path, +) -> None: + """One thread, one slot. A sync batch's read is queued on the thread, then an async read of the + sync backend is queued behind it. Had the async read taken the slot before its turn on the + thread, the batch's read, which runs first, would wait for that slot forever.""" + pool, limiter = ThreadPool(1), Limiter(1) + store = Store(LocalBackend(tmp_path), context=Context(threads=pool, limits={"local": limiter})) + store.put("k", b"x") + gate = threading.Event() + pool.submit(gate.wait) # keeps the only thread busy while the reads queue up + batch: list[object] = [] + single: list[object] = [] + batch_reader = threading.Thread(target=_read_batch, args=(store, batch), daemon=True) + async_reader = threading.Thread(target=_read_async, args=(store, single), daemon=True) + batch_reader.start() + _settle() # the batch's read is queued + async_reader.start() + _settle() # the async read is queued behind it + gate.set() + batch_reader.join(10) + async_reader.join(10) + assert not batch_reader.is_alive() + assert not async_reader.is_alive() + assert [type(r) for r in batch + single] == [Found, Found] + assert limiter.in_use == 0 diff --git a/packages/zarr-storage/tests/test_store.py b/packages/zarr-storage/tests/test_store.py index 63582e5066..15d1f4eaa1 100644 --- a/packages/zarr-storage/tests/test_store.py +++ b/packages/zarr-storage/tests/test_store.py @@ -29,7 +29,7 @@ if TYPE_CHECKING: from collections.abc import Iterator, Sequence - from zarr_storage import ByteRange + from zarr_storage import ByteRange, Context class _ReadOnlyBackend: @@ -38,6 +38,10 @@ class _ReadOnlyBackend: def __init__(self, values: dict[str, bytes]) -> None: self._values = values + @property + def resource_key(self) -> str: + return "test" + @property def closed(self) -> bool: return False @@ -117,17 +121,17 @@ def test_get_into_many_device_buffers() -> None: """Without a batch method of its own, the backend's `get_into` serves `get_into_many`.""" store = Store(_DeviceBackend({"a/k": b"0123", "a/j": b"45"})) / "a" whole, part = _DeviceBuffer(4), _DeviceBuffer(1) - results = [ - (request[0], result) + results = { + request[0]: result for request, result in store.get_into_many( [("k", whole), ("j", part, SuffixRequest(1)), ("missing", _DeviceBuffer(1))] ) - ] - assert results == [ - ("k", Copied(4, Generation(0))), - ("j", Copied(1, Generation(0))), - ("missing", Missing()), - ] + } + assert results == { + "k": Copied(4, Generation(0)), + "j": Copied(1, Generation(0)), + "missing": Missing(), + } assert (whole.download(4), part.download(1)) == (b"0123", b"5") @@ -139,7 +143,7 @@ def __init__(self, values: dict[str, bytes]) -> None: self.asked: list[tuple[str, ByteRange | None]] = [] def get_many( - self, requests: Sequence[tuple[str, ByteRange | None]] + self, requests: Sequence[tuple[str, ByteRange | None]], context: Context ) -> Iterator[tuple[int, Found | Missing]]: self.asked.extend(requests) for index in reversed(range(len(requests))): @@ -179,7 +183,7 @@ def get(self, path: str, byte_range: ByteRange | None) -> Found | Missing: def test_put_many_failure_stops_the_batch_and_reports_it() -> None: backend = _FailingBackend() - store = Store(backend) / "r" + store = Store(backend).limited(1) / "r" # one at a time, so the order is known with pytest.raises(BatchError) as caught: store.put_many([("ok", b"1"), ("bad", b"2"), ("later", b"3")]) error = caught.value @@ -192,7 +196,7 @@ def test_put_many_failure_stops_the_batch_and_reports_it() -> None: def test_get_many_failure_raises_after_the_reads_before_it() -> None: backend = _FailingBackend() - store = Store(backend) / "r" + store = Store(backend).limited(1) / "r" # one at a time, so the order is known store.put("ok", b"1") backend.attempted.clear() results = store.get_many(["ok", "bad", "later"]) @@ -334,4 +338,5 @@ async def _static_surface_checks( store.get_async("k") # pyright: ignore[reportAttributeAccessIssue] store.aclose() # pyright: ignore[reportAttributeAccessIssue] astore.close() # pyright: ignore[reportAttributeAccessIssue] - AsyncStore(_ReadOnlyBackend({})).get("k") # pyright: ignore[reportAttributeAccessIssue] + await AsyncStore(_ReadOnlyBackend({})).get("k") # a sync backend serves async callers + await AsyncStore(_ReadOnlyBackend({})).put("k", b"") # pyright: ignore[reportAttributeAccessIssue] From 1da64c58402870f75bd5641184fa1ef85645995e Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Wed, 18 Nov 2026 14:14:23 +0100 Subject: [PATCH 4/5] docs(zarr-storage): plan caching as a context resource over a tier store The cache is a `CachePool` in the store's context, off by default. Its bytes live in a tier, which is any writable store, so range reads, batches and the conformance suite come from the tier. An in-memory index holds what a store can't: source generations, ages, absences and eviction order. Assisted-by: ClaudeCode:claude-opus-5-5 --- packages/zarr-storage/DESIGN.md | 76 ++++++++++++++++++++++++++++++++- 1 file changed, 75 insertions(+), 1 deletion(-) diff --git a/packages/zarr-storage/DESIGN.md b/packages/zarr-storage/DESIGN.md index 002192178b..709779c62f 100644 --- a/packages/zarr-storage/DESIGN.md +++ b/packages/zarr-storage/DESIGN.md @@ -264,6 +264,76 @@ object-store requests by wrapping the store in object_store's `LimitStore` and s `data_copy_concurrency`): compute waiting on I/O in one pool starves it. A context holds the I/O pool; zarr-python's codec pipeline would take its own. +## Caching + +Planned for iteration 3; not built yet. A cache is a resource of the store's context, like +TensorStore's `cache_pool`. It is neither a store class that wraps another store (zarr-python's +`CacheStore`) nor a hook on store methods: + +- **A wrapper can't get its capabilities right.** A Python class can't have exactly the capabilities + of the backend it wraps. It would need one class per combination, or it would claim every + capability and raise at runtime, losing the typed capabilities. +- **A hook that caches is a wrapper.** To serve a hit, a hook has to skip the read and supply the + result, and then it is a wrapper. + +The rule: wrapping is for layers that change what keys mean (zip, sharding). Policies that leave +the meaning alone (limits, caching, retries) are context resources. + +- **A `CachePool(tier, total_bytes)` is a context resource**, set with `Context(cache=...)` or + `ctx.child(cache=...)`. The default pool holds 0 bytes, so caching is off until a context gives it + a budget. Stores sharing a context share its pool: one byte budget, one eviction order. +- **The cached bytes live in a store, the tier.** The tier is a memory store by default, a local + store for a disk cache, or any other writable store. Each cached value is a value in the tier, so + the cache gets the following from the tier without new code: + - byte-range reads and `get_ranges`, served from one cached whole value; + - batches, for the hits of a batch; + - thread safety and both surfaces (a memory tier answers an `AsyncStore` inline); + - the conformance suite, as the tier's tests. + + zarr-python's `CacheStore` keeps its bytes in a store too; what changes is where the cache sits. +- **A cache is more than a store.** Beside the tier, the pool keeps an index in memory for what a + store can't hold. Per entry, the index records: + - the source's generation (the tier mints its own); + - the source's `read_at`; + - the entry's size and recency; + - absences: a `Missing` from the source has no bytes to store. + + The index also runs eviction, and it combines concurrent misses on one key into one source read. +- **Tier keys are opaque and written once.** The index names each entry's bytes with a fresh key + under a random per-pool prefix. Tier keys don't come from source keys, because the key grammar + has no way to name the source backend. A refresh writes a new key and points the index at it. + Eviction removes the index entry, then deletes the key. So no reader sees a tier value change + under it, and a tier read that returns `Missing` (evicted, or a disk tier someone else cleaned) is + a miss. Pools sharing a disk tier don't collide. +- **Entries are keyed by backend instance and path**, the identity `shares_backend` uses. Closing a + backend drops its entries. +- **Each view decides how stale a read may be**, with `store.with_max_age(seconds)`, default `0`. + TensorStore makes the same split (`recheck_cached_metadata`, `recheck_cached_data`), because + metadata and chunks need different answers. + - A hit younger than `max_age` returns without touching the source. + - An older hit revalidates with `get_if_changed(key, generation)`. On object stores that's a + conditional GET: one round trip, no bytes. + - `math.inf` trusts the cache forever, for values that are never rewritten. + - An absence ages like a value, but only a full read can revalidate it, so caching absences pays + only when `max_age > 0`: zarr's metadata probes for format versions, for example. +- **Writes keep a pool coherent.** A write or delete through a view whose context holds the pool + drops that key's entry, and `delete_dir` drops every entry under its prefix. Writes also bump a + per-key epoch, and a read that started before a write to its key returns its result but + doesn't fill the cache. So a view + sees its pool's own writes whatever its `max_age`. Writes from other processes, other pools, or + straight to the backend show up only through revalidation, as in TensorStore. +- **Hits hold no source slot.** The tier runs under a context of the pool's own, so a hit waits on + neither the source's limiter nor the process-wide `memory` limiter. A revalidation or a miss holds + a source slot like any other read. +- Listings are never cached, and neither is a value larger than the budget. +- Pickling a pool pickles its budget and its tier's configuration, never its contents. The + receiver starts with an empty index and an empty tier. +- `get_into` into a buffer type the tier can't write (device memory, with a host tier) bypasses + the pool. +- Every `StoreSuite` runs again through a cached view (a pool with a budget, `max_age=0`), where the + cache must be invisible. The state machine adds writes made straight to the backend, which only + revalidation can catch. + ## Configuration No global configuration. Every knob is a constructor parameter with its default in the signature; @@ -277,11 +347,15 @@ resources are passed in a store's context. - Listings hold no slot of the context's limiter. - `get_ranges_into`, `put_from`, and a GPU direct storage backend whose `get_into_many` submits one cuFile batch. +- Caching: range reads of values the pool doesn't hold whole go to the source. Caching ranges (shard + indexes) needs entries keyed by key, range and generation. +- Caching: a disk tier starts cold in every process, because the index lives in memory. +- Caching: a pool over a memory backend copies values for no gain; nothing stops one yet. ## Iterations 1. **Core:** types, `Store`, the memory and local backends, the conformance suite in `zarr_storage.testing`, pickling. 2. **Remote and archive backends:** obstore (S3, GCS, Azure, HTTP; async-native), fsspec, zip. -3. **Composition:** range coalescing, caching, retry, a sync bridge for async-only backends, URL specs. +3. **Composition:** range coalescing, caching (see Caching), retry, a sync bridge for async-only backends, URL specs. 4. **Transactions:** staged writes committed with generations, memory first. From 422dee44397404e587bac61869f780a52a5227e3 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Wed, 18 Nov 2026 14:18:14 +0100 Subject: [PATCH 5/5] docs(zarr-storage): leave shard index caching to the codec pipeline Assisted-by: ClaudeCode:claude-opus-5-5 --- packages/zarr-storage/DESIGN.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/packages/zarr-storage/DESIGN.md b/packages/zarr-storage/DESIGN.md index 709779c62f..0867b09478 100644 --- a/packages/zarr-storage/DESIGN.md +++ b/packages/zarr-storage/DESIGN.md @@ -347,8 +347,11 @@ resources are passed in a store's context. - Listings hold no slot of the context's limiter. - `get_ranges_into`, `put_from`, and a GPU direct storage backend whose `get_into_many` submits one cuFile batch. -- Caching: range reads of values the pool doesn't hold whole go to the source. Caching ranges (shard - indexes) needs entries keyed by key, range and generation. +- Caching: range reads of values the pool doesn't hold whole go to the source. Shard indexes are the + case that matters, and they belong to the codec pipeline, not here: the sharding codec knows where + an index is and can cache it decoded, keyed by store, key and generation. The storage layer gives + it what it needs: the generation of every read, and `get_if_changed` with a byte range to + revalidate one. - Caching: a disk tier starts cold in every process, because the index lives in memory. - Caching: a pool over a memory backend copies values for no gain; nothing stops one yet.