Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
ff36c01
test: check the Codec contract for every codec and add static type tests
d-v-b Nov 21, 2026
145994e
fix: return the caller's out from decode(buf, out=out) in every codec
d-v-b Nov 21, 2026
52be3f5
fix(typing): correct registry, compat and NDArrayLike annotations and…
d-v-b Nov 21, 2026
c26d438
feat(abc): annotate the Codec base class and add structural codec pro…
d-v-b Nov 21, 2026
3220705
feat(typing): ship .pyi stubs for the Cython extension modules
d-v-b Nov 21, 2026
2dcbbcd
feat(typing): annotate the compressor codecs with per-codec config Ty…
d-v-b Nov 21, 2026
d5b3d9b
feat(typing): annotate the checksum codecs and make JenkinsLookup3 co…
d-v-b Nov 21, 2026
d8a20b3
feat(typing): annotate the filter codecs
d-v-b Nov 21, 2026
d664d8b
feat(typing): annotate the object and array-to-bytes codecs
d-v-b Nov 21, 2026
39f3692
style: sort test imports now that the Cython modules have stubs
d-v-b Nov 21, 2026
25a5ec9
Merge branch 'typing/05-cython-stubs' into typing/06a-compressors
d-v-b Nov 21, 2026
2587eb9
ci: allow the PEP 705 keys stubtest reports on the Cython config Type…
d-v-b Nov 21, 2026
d84c7e1
Merge branch 'typing/06a-compressors' into typing/06b-checksums
d-v-b Nov 21, 2026
accbed5
Merge branch 'typing/06b-checksums' into typing/06c-filters
d-v-b Nov 21, 2026
3dd78eb
Merge branch 'typing/06c-filters' into typing/06d-object-codecs
d-v-b Nov 21, 2026
cffb3e4
Merge remote-tracking branch 'upstream/main' into typing/02-out-contract
d-v-b Nov 21, 2026
f3ff244
Merge branch 'typing/02-out-contract' into typing/03-fix-annotations
d-v-b Nov 21, 2026
8b4d3b1
Merge branch 'typing/03-fix-annotations' into typing/04-base-class
d-v-b Nov 21, 2026
50a5b27
Merge branch 'typing/04-base-class' into typing/05-cython-stubs
d-v-b Nov 21, 2026
bf3d071
Merge branch 'typing/05-cython-stubs' into typing/06a-compressors
d-v-b Nov 21, 2026
832b3a4
Merge branch 'typing/06a-compressors' into typing/06b-checksums
d-v-b Nov 21, 2026
867b068
Merge branch 'typing/06b-checksums' into typing/06c-filters
d-v-b Nov 21, 2026
cc8b7ce
Merge branch 'typing/06c-filters' into typing/06d-object-codecs
d-v-b Nov 21, 2026
e3b9d2d
Merge remote-tracking branch 'upstream/main' into typing/03-fix-annot…
d-v-b Nov 21, 2026
05dabce
Merge branch 'typing/03-fix-annotations' into typing/04-base-class
d-v-b Nov 21, 2026
889670a
Merge branch 'typing/04-base-class' into typing/05-cython-stubs
d-v-b Nov 21, 2026
bc90f92
Merge branch 'typing/05-cython-stubs' into typing/06a-compressors
d-v-b Nov 21, 2026
26b856b
Merge branch 'typing/06a-compressors' into typing/06b-checksums
d-v-b Nov 21, 2026
fc88f40
Merge branch 'typing/06b-checksums' into typing/06c-filters
d-v-b Nov 21, 2026
c53d97c
Merge branch 'typing/06c-filters' into typing/06d-object-codecs
d-v-b Nov 21, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -162,6 +162,17 @@ jobs:
- name: pyright (strict type tests)
run: pyright -p tests/typing

- name: stubtest (Cython stubs against the built extensions)
# Read the stubs from src/: mypy ignores the installed copies until numcodecs
# ships a py.typed marker.
env:
MYPYPATH: src
run: |
python -m mypy.stubtest --mypy-config-file pyproject.toml \
--allowlist tests/typing/stubtest-allowlist.txt \
numcodecs.blosc numcodecs.zstd numcodecs.lz4 numcodecs.vlen \
numcodecs.fletcher32 numcodecs.jenkins numcodecs.compat_ext numcodecs._shuffle

test-zarr:
runs-on: ubuntu-latest
strategy:
Expand Down
12 changes: 12 additions & 0 deletions docs/abc.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,4 +10,16 @@
.. automethod:: decode
.. automethod:: get_config
.. automethod:: from_config

Protocols
---------

These describe codecs structurally, for code that accepts any object implementing the
interface whether or not it subclasses :class:`Codec`. The three kinds mirror the codec
kinds of the Zarr version 3 specification.

.. autoclass:: CodecProtocol
.. autoclass:: BytesBytesCodec
.. autoclass:: ArrayArrayCodec
.. autoclass:: ArrayBytesCodec
```
50 changes: 50 additions & 0 deletions docs/release.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,58 @@
already written into `out`; only the returned object changes.
* `JenkinsLookup3.decode` accepts any contiguous writable buffer as `out`, not only an
ndarray, and raises `ValueError` when `out` has the wrong size.
* `JenkinsLookup3.get_config()` reports `prefix` as a list of ints, so a configuration
with a prefix can be serialized to JSON and passed back to `get_codec`. It held a numpy
array, which `json.dumps` rejected and `get_codec` could not compare. The constructor
accepts a list of ints as well as a bytes-like `prefix`.
* CRC32, Adler32 and CRC32C check the byte length of the input to `decode`, not its
length along the first axis, before looking for the checksum.
* `Fletcher32.decode` without `out` returns a `uint8` ndarray, like the other checksum
codecs, instead of a Cython memoryview slice.
* Correct type annotations that described something other than what the code does:
`numcodecs.registry.codec_registry` holds codec classes (`dict[str, type[Codec]]`) and
`entries` holds single entry points (`dict[str, EntryPoint]`). `numcodecs.compat`
functions now annotate their parameters, and `ndarray_copy` returns the type of `dst`
(or of `src` when `dst` is `None`). The `NDArrayLike`, `DType` and `FlagsObj` protocols
declare read-only properties, so `numpy.ndarray` now satisfies `NDArrayLike` for type
checkers, and `numcodecs.ndarray_like.BufferLike` names "a buffer or an ndarray-like".
* `numcodecs` defines `__all__`, so type checkers treat its re-exported codec classes as
public. `from numcodecs import *` now imports only those names.
* `register_codec` raises `ValueError` for a class without a `codec_id` when no id is
given, instead of registering it under `None`.
* `Codec.codec_id` no longer defaults to `None`. Abstract intermediate classes such as
`Checksum32` leave it unset, so reading it raises `AttributeError`; every concrete codec
sets it.

### Enhancements

* Annotate `numcodecs.abc.Codec`. `encode` and `decode` take and return
`numcodecs.ndarray_like.BufferLike` (an object exporting the buffer protocol, or an
ndarray-like), with `buf` positional-only, and `decode(buf, out=out)` is typed to return
the type of `out`. `get_config` returns `Mapping[str, Any]` and `from_config` returns
`Self`. Previously pyright inferred that `encode` and `decode` return `None`.
* Add structural protocols `numcodecs.abc.CodecProtocol`, `BytesBytesCodec`,
`ArrayArrayCodec` and `ArrayBytesCodec` for code that accepts any object implementing the
codec interface.
* Annotate the concrete codecs. Each `get_config` returns a TypedDict naming the codec's
parameters (`ZlibConfig`, `BloscConfig`, ...), all subclasses of
`numcodecs.abc.CodecConfig`, which carries the `id` and declares any further keys as
read-only JSON values (PEP 728 `extra_items`). `numcodecs.abc.JSON` names a
JSON-serializable value. Compressors: Zlib, GZip, BZ2, LZMA, Base64, Blosc, Zstd and LZ4.
Checksums: CRC32, Adler32, CRC32C, JenkinsLookup3 and Fletcher32. Filters: AsType,
Delta, Quantize, FixedScaleOffset, BitRound, PackBits, Shuffle and Categorize. Object
and array-to-bytes codecs: JSON, MsgPack, Pickle (whose `decode` returns `Any`), the VLen
codecs, ZFPY and PCodec.
`typing_extensions` >= 4.13 is now required.
* Ship `.pyi` type stubs for the Cython extension modules (`blosc`, `zstd`, `lz4`, `vlen`,
`fletcher32`, `jenkins`, `compat_ext` and `_shuffle`), checked against the compiled
modules with `mypy.stubtest` in CI. Type checkers previously saw the Blosc, Zstd, LZ4,
Fletcher32 and VLen codecs as unknown.

### Deprecations

* Loading a codec through the `numcodecs.codecs` entry point group whose class does not
subclass `numcodecs.abc.Codec` now emits a `DeprecationWarning`.

(release_0.17.1)=

Expand Down
3 changes: 2 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ description = """
A Python package providing buffer compression and transformation codecs \
for use in data storage and communication applications."""
readme = "README.md"
dependencies = ["numpy>=2.0", "packaging", "typing_extensions"]
dependencies = ["numpy>=2.0", "packaging", "typing_extensions>=4.13"]
requires-python = ">=3.12"
dynamic = [
"version",
Expand Down Expand Up @@ -245,6 +245,7 @@ quote-style = "preserve"
[tool.mypy]
ignore_errors = false
ignore_missing_imports = true
python_version = "3.12"
enable_error_code = ["ignore-without-code", "redundant-expr", "truthy-bool"]
# TODO: set options below to true
strict = false
Expand Down
40 changes: 40 additions & 0 deletions src/numcodecs/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,42 @@
from numcodecs.registry import get_codec as get_codec
from numcodecs.registry import register_codec

# Names listed here are the public API re-exported from submodules. Optional codecs are
# appended below when their dependency imports.
__all__ = [
"BZ2",
"CRC32",
"JSON",
"LZ4",
"LZMA",
"Adler32",
"AsType",
"Base64",
"BitRound",
"Blosc",
"Categorize",
"Delta",
"FixedScaleOffset",
"Fletcher32",
"GZip",
"JenkinsLookup3",
"PackBits",
"Pickle",
"Quantize",
"Shuffle",
"VLenArray",
"VLenBytes",
"VLenUTF8",
"Zlib",
"Zstd",
"blosc",
"get_codec",
"lz4",
"register_codec",
"vlen",
"zstd",
]

__version__: str = _version("numcodecs")
from numcodecs.zlib import Zlib

Expand Down Expand Up @@ -131,18 +167,22 @@
from numcodecs.zfpy import ZFPY

register_codec(ZFPY)
__all__ += ["ZFPY"]

with suppress(ImportError):
from numcodecs.msgpacks import MsgPack

register_codec(MsgPack)
__all__ += ["MsgPack"]

with suppress(ImportError):
from numcodecs.checksum32 import CRC32C

register_codec(CRC32C)
__all__ += ["CRC32C"]

with suppress(ImportError):
from numcodecs.pcodec import PCodec

register_codec(PCodec)
__all__ += ["PCodec"]
4 changes: 4 additions & 0 deletions src/numcodecs/_shuffle.pyi
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
from collections.abc import Buffer

def _doShuffle(src: Buffer, des: Buffer, element_size: int) -> None: ...
def _doUnshuffle(src: Buffer, des: Buffer, element_size: int) -> None: ...
111 changes: 98 additions & 13 deletions src/numcodecs/abc.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,17 +29,55 @@
"""

from abc import ABC, abstractmethod
from collections.abc import Buffer, Mapping, Sequence
from typing import Any, Protocol, Self, overload

from typing_extensions import ReadOnly, TypedDict

from .ndarray_like import BufferLike, NDArrayLike

__all__ = [
"JSON",
"ArrayArrayCodec",
"ArrayBytesCodec",
"BytesBytesCodec",
"Codec",
"CodecConfig",
"CodecProtocol",
]

type JSON = int | float | bool | str | Sequence[JSON] | Mapping[str, JSON] | None
"""A JSON-serializable value. `Sequence` and `Mapping` rather than `list` and `dict`, so
that tuples, TypedDicts and lists of narrower types count (the same definition as
`zarr_metadata`'s `JSONValue`)."""


# `extra_items` (PEP 728) says that any further keys hold JSON values and are read-only. It
# lets type checkers that implement PEP 728 (pyright) treat every codec configuration as a
# `Mapping[str, JSON]`, and as Zarr v2 codec metadata. mypy does not implement PEP 728 yet,
# so it rejects the keyword and treats `CodecConfig` as a plain TypedDict.
class CodecConfig(TypedDict, extra_items=ReadOnly[JSON]): # type: ignore[call-arg]
"""The configuration every codec returns from `Codec.get_config`: its `id`, plus any
further JSON-serializable parameters.

Concrete codecs return a subclass that declares their own parameters. This is the shape
of a Zarr version 2 compressor or filter.
"""

id: str


class Codec(ABC):
"""Codec abstract base class."""

# override in sub-class
codec_id: str | None = None
"""Codec identifier."""
# Declared without a value (abstract intermediate classes leave it unset) and without
# `ClassVar`, so that `Codec` instances match protocols that declare `codec_id` either
# as a plain attribute or as a read-only property.
codec_id: str
"""Codec identifier. Every concrete codec class sets it as a class attribute."""

@abstractmethod
def encode(self, buf): # pragma: no cover
def encode(self, buf: BufferLike, /) -> BufferLike: # pragma: no cover
"""Encode data in `buf`.

Parameters
Expand All @@ -55,8 +93,14 @@ def encode(self, buf): # pragma: no cover
protocol.
"""

@overload
def decode(self, buf: BufferLike, /, out: None = None) -> BufferLike: ...
@overload
def decode[OutT: BufferLike](self, buf: BufferLike, /, out: OutT) -> OutT: ...
@abstractmethod
def decode(self, buf, out=None): # pragma: no cover
def decode(
self, buf: BufferLike, /, out: BufferLike | None = None
) -> BufferLike: # pragma: no cover
"""Decode data in `buf`.

Parameters
Expand All @@ -72,18 +116,18 @@ def decode(self, buf, out=None): # pragma: no cover
-------
dec : buffer-like
Decoded data. May be any object supporting the new-style
buffer protocol.
buffer protocol. If `out` was provided, `out` itself.
"""

def get_config(self):
def get_config(self) -> Mapping[str, Any]:
"""Return a dictionary holding configuration parameters for this
codec. Must include an 'id' field with the codec identifier. All
values must be compatible with JSON encoding."""

# override in sub-class if need special encoding of config values

# setup config object
config = {'id': self.codec_id}
config: dict[str, Any] = {'id': self.codec_id}

# by default, assume all non-private members are configuration
# parameters - override this in sub-class if not the case
Expand All @@ -94,8 +138,8 @@ def get_config(self):
return config

@classmethod
def from_config(cls, config):
"""Instantiate codec from a configuration object."""
def from_config(cls, config: Mapping[str, Any], /) -> Self:
"""Instantiate codec from a configuration object, without its 'id' field."""
# N.B., assume at this point the 'id' field has been removed from
# the config object

Expand All @@ -105,14 +149,14 @@ def from_config(cls, config):
# keyword arguments without any special decoding
return cls(**config)

def __eq__(self, other):
def __eq__(self, other: object) -> bool:
# override in sub-class if need special equality comparison
try:
return self.get_config() == other.get_config()
return bool(self.get_config() == other.get_config()) # type: ignore[attr-defined]
except AttributeError:
return False

def __repr__(self):
def __repr__(self) -> str:
# override in sub-class if need special representation

# by default, assume all non-private members are configuration
Expand All @@ -124,3 +168,44 @@ def __repr__(self):
]
r += ', '.join(params) + ')'
return r


class CodecProtocol(Protocol):
"""The structural interface of a codec, for code that accepts any object implementing
it, whether or not it subclasses `Codec`."""

@property
def codec_id(self) -> str: ... # pragma: no cover

def encode(self, buf: BufferLike, /) -> BufferLike: ... # pragma: no cover

def decode(self, buf: BufferLike, /, out: Any = None) -> BufferLike: ... # pragma: no cover

def get_config(self) -> Mapping[str, Any]: ... # pragma: no cover

@classmethod
def from_config(cls, config: Mapping[str, Any], /) -> Self: ... # pragma: no cover


class BytesBytesCodec(CodecProtocol, Protocol):
"""A codec that maps bytes to bytes, such as a compressor or a checksum."""

def encode(self, buf: BufferLike, /) -> Buffer: ... # pragma: no cover

def decode(self, buf: BufferLike, /, out: Any = None) -> Buffer: ... # pragma: no cover


class ArrayArrayCodec(CodecProtocol, Protocol):
"""A codec that maps an array to an array, such as a filter."""

def encode(self, buf: BufferLike, /) -> NDArrayLike: ... # pragma: no cover

def decode(self, buf: BufferLike, /, out: Any = None) -> NDArrayLike: ... # pragma: no cover


class ArrayBytesCodec(CodecProtocol, Protocol):
"""A codec that serializes an array to bytes and back."""

def encode(self, buf: BufferLike, /) -> Buffer: ... # pragma: no cover

def decode(self, buf: BufferLike, /, out: Any = None) -> NDArrayLike: ... # pragma: no cover
Loading
Loading