Skip to content

feat(typing): ship py.typed and check numcodecs with strict mypy - #915

Merged
d-v-b merged 35 commits into
zarr-developers:mainfrom
d-v-b:typing/07-py-typed
Nov 21, 2026
Merged

d-v-b merged 35 commits into
zarr-developers:mainfrom
d-v-b:typing/07-py-typed

Conversation

@d-v-b

@d-v-b d-v-b commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

🤖 AI text below 🤖

numcodecs now ships a py.typed marker (PEP 561), so type checkers use its annotations and stubs instead of treating the package as untyped. mypy runs in strict mode over src, in CI and in pre-commit (now mypy 2.4.1); third-party codecs that subclass Codec are type-checked against it for the first time.

Summary

Effect, measured against a wheel built from this branch:

  • Before: a downstream file typed the way zarr-python's numcodecs wrappers are got Any everywhere from mypy (with import-untyped errors unless ignore_missing_imports was set), and None or Unknown from pyright.
  • Now: both checkers, in strict mode, give it real types for get_codec, the registry, Codec methods including decode(out=...), every concrete codec, and the compat helpers.
  • pyright --verifytypes: completeness goes from 22.1% to 99.3% (699 known, 5 ambiguous, 0 unknown symbols).

How:

  • The remaining strict-mode errors in src are fixed:
    • values from untyped optional dependencies (zfpy, pcodec, msgpack, google_crc32c) are bound to annotated names;
    • the NDArrayLike metaclass subclasses typing._ProtocolMeta, which typeshed declares, instead of Protocol.__class__. Type checkers see the latter as Any, which made every BufferLike annotation "partially unknown".
  • Instance attributes on the filter codecs and ZFPY are annotated.
  • Codec.from_config's config is no longer positional-only, so Codec matches protocols that declare it as a regular parameter, such as zarr-python's Numcodec.
  • mypy configuration: the repository's config is strict, with mypy_path = "src" and ignore_missing_imports only for the untyped optional dependencies. Unannotated test functions are still not checked, as before.
  • pre-commit runs mypy 2.4.1, matching the pinned CI version. This overlaps chore: update pre-commit hook mypy #835.
  • CI checks that the installed package carries py.typed and the stubs.

Notes for downstream projects:

  • Plugins (e.g. imagecodecs) may hit two one-line fixes:
    • a # type: ignore[misc] on class X(Codec) becomes unused;
    • a from_config(cls, config: dict[...]) override must accept a Mapping.
  • zarr-python can drop ignore_missing_imports for numcodecs and its two numcodecs-related type: ignores. Under pyright, its Numcodec protocol declares buf as a keyword parameter, which Codec's positional-only buf does not satisfy; adding / there, or using numcodecs.abc.CodecProtocol, fixes that.
Tests

Full suite passes locally (1016 passed). mypy strict over src and tests, mypy and pyright strict over tests/typing, and stubtest against the installed wheel are all clean, and the wheel contains py.typed and the stubs.

Stack and merge order

The agent split honest type annotations for numcodecs into a stack of 10 PRs, following an audit of the existing annotations and of the earlier attempts (#443, #698, #701, #702). Each PR is opened from the fork against main, so until the PRs before it merge, its diff also contains their commits. The change specific to this PR starts at commit 4fcbc9a; later commits on its branch are follow-up fixes and merges of the PR below it. Merge in this order:

  1. test: check the Codec contract for every codec and add static type tests #906 typing/01-harness: test: check the Codec contract for every codec and add static type tests
  2. fix: return the caller's out from decode(buf, out=out) in every codec #907 typing/02-out-contract: fix: return the caller's out from decode(buf, out=out) in every codec
  3. fix(typing): correct registry, compat and NDArrayLike annotations and add __all__ #908 typing/03-fix-annotations: fix(typing): correct registry, compat and NDArrayLike annotations and add all
  4. feat(abc): annotate the Codec base class and add structural codec protocols #909 typing/04-base-class: feat(abc): annotate the Codec base class and add structural codec protocols
  5. feat(typing): ship .pyi stubs for the Cython extension modules #910 typing/05-cython-stubs: feat(typing): ship .pyi stubs for the Cython extension modules
  6. feat(typing): annotate the compressor codecs with per-codec config TypedDicts #911 typing/06a-compressors: feat(typing): annotate the compressor codecs with per-codec config TypedDicts
  7. feat(typing): annotate the checksum codecs and make JenkinsLookup3 configs JSON #912 typing/06b-checksums: feat(typing): annotate the checksum codecs and make JenkinsLookup3 configs JSON
  8. feat(typing): annotate the filter codecs #913 typing/06c-filters: feat(typing): annotate the filter codecs
  9. feat(typing): annotate the object and array-to-bytes codecs #914 typing/06d-object-codecs: feat(typing): annotate the object and array-to-bytes codecs
  10. feat(typing): ship py.typed and check numcodecs with strict mypy #915 typing/07-py-typed: feat(typing): ship py.typed and check numcodecs with strict mypy ← this PR

🤖 Generated with Claude Code

d-v-b added 10 commits November 21, 2026 13:48
Add tests/test_codec_contract.py, which runs every registered codec through
the behavior the Codec base class promises: encode and decode return buffers,
decode(out=...) returns the caller's out, and get_config round-trips through
from_config. Known violations are strict xfails, so fixing one forces its
entry to be removed.

Add tests/typing, where assert_type checks (verified by mypy and pyright in
strict mode) sit next to runtime assertions (verified by pytest), and a
`typing` CI job that runs both checkers plus the repository's own mypy
configuration. The type checkers are pinned in a new `typing` dependency
group.

Assisted-by: ClaudeCode:claude-opus-5-5
The Codec docstring says decode writes into `out`, and callers expect `out`
back. Ten codecs (Zlib, BZ2, LZMA, Base64, Shuffle, ZFPY, PCodec and the
three VLen codecs) rebound `out` to a flattened view and returned that, and
numcodecs.compat.ndarray_copy did the same for any `out` that was not an
ndarray, which every filter and checksum codec inherited. The decoded bytes
were already in the caller's buffer; only the returned object changes.

JenkinsLookup3.decode now accepts any contiguous writable buffer as `out`
and checks its size, and Fletcher32.decode returns a uint8 ndarray instead of
a Cython memoryview slice that no type annotation could name.

The strict xfails in tests/test_codec_contract.py are removed.

Assisted-by: ClaudeCode:claude-opus-5-5
… add __all__

Several annotations described something other than what the code does:

- codec_registry holds classes: dict[str, type[Codec]], not dict[str, Codec].
- entries holds single entry points: dict[str, EntryPoint].
- ndarray_copy returned NDArrayLike, but hands back `src` unchanged (often
  bytes) when `dst` is None. It is now overloaded on `dst`.
- NDArrayLike, DType and FlagsObj declared settable attributes, so no
  numpy.ndarray satisfied NDArrayLike under mypy or pyright. They now declare
  read-only properties. The isinstance cache moves to module level, because
  a protocol without annotations of its own would otherwise pick up the
  metaclass's annotation as a required member.

The compat functions annotate their parameters with a new BufferLike alias
(collections.abc.Buffer | NDArrayLike), is_ndarray_like returns TypeIs, and
get_codec/register_codec are annotated. register_codec now refuses a class
without a codec_id instead of registering it under None, and get_codec warns
(DeprecationWarning) when an entry point loads a class that does not
subclass Codec, so that `-> Codec` can become true.

numcodecs/__init__.py defines __all__, so that type checkers treat the
re-exported codec classes as public once numcodecs ships py.typed.

mypy now checks against Python 3.12, the oldest supported version; doing so
caught `typing.TypeIs`, which only exists from 3.13 and now comes from
typing_extensions.

Assisted-by: ClaudeCode:claude-opus-5-5
…tocols

Codec.encode and Codec.decode were unannotated with docstring-only bodies,
so mypy saw them as Any and pyright inferred that both return None, which
made every typed subclass an "incompatible override" under pyright.

- encode/decode take and return BufferLike (collections.abc.Buffer |
  NDArrayLike), with `buf` positional-only so that subclasses may rename it.
- decode is overloaded so that decode(buf, out=out) returns the type of
  `out`, which every codec now honors.
- codec_id is declared `str` without a default instead of `str | None =
  None`. It is not a ClassVar: with ClassVar, Codec instances do not match
  protocols that declare codec_id as a plain attribute (zarr-python's
  Numcodec) or as a property, under mypy or pyright.
- get_config returns Mapping[str, Any], so that concrete codecs can later
  return per-codec TypedDicts, and from_config returns Self.

numcodecs.abc also exports CodecProtocol and the three Zarr v3 codec kinds
(BytesBytesCodec, ArrayArrayCodec, ArrayBytesCodec) as structural
protocols, so consumers can type code that accepts any codec-like object.

Assisted-by: ClaudeCode:claude-opus-5-5
blosc, zstd, lz4, vlen, fletcher32, jenkins, compat_ext and _shuffle had no
stubs, so type checkers saw Blosc, Zstd, LZ4, Fletcher32 and the VLen codecs
as unknown, and numcodecs' own mypy run silently treated them as Any. Each
module now has a stub next to its .pyx, installed by meson.

The stubs follow probed behavior: the compressors return bytes, or `out`
itself when it is given; Fletcher32.decode returns a uint8 ndarray; the VLen
codecs encode to bytearray and decode to object arrays. VLenArray.encode
takes an object array only, since numpy turns a list of equal-length arrays
into a 2-D array.

The typing CI job runs mypy.stubtest against the installed extensions, with
an allowlist for Cython artifacts (__test__, __pyx_capi__, and the `(self, /)`
signature Cython reports for argument-less module functions).

Checksum32.checksum is declared positional-only, so that JenkinsLookup3 can
use jenkins_lookup3 (whose parameters are named differently) as its checksum;
it is wrapped in staticmethod so type checkers do not bind it as a method.
It was never bound at runtime, being a builtin function.

Assisted-by: ClaudeCode:claude-opus-5-5
…pedDicts

Zlib, GZip, BZ2, LZMA and Base64 annotate __init__, encode (-> bytes) and
decode (overloaded: bytes without `out`, the type of `out` with it), and
Blosc, Zstd and LZ4 get the same through their stubs. Each get_config
returns a TypedDict of the codec's parameters; for codecs that inherit the
base implementation it is a cast of the inherited dict, so subclasses that
add attributes keep them in their configuration as before.

numcodecs.abc.CodecConfig is the common base: `id: str`, with PEP 728
`extra_items=ReadOnly[JSON]`, which lets pyright treat any codec
configuration as a Mapping[str, JSON] (and as Zarr v2 codec metadata).
mypy does not implement PEP 728 yet and sees a plain TypedDict. JSON uses
Sequence/Mapping, matching zarr_metadata's JSONValue. `extra_items` needs
typing_extensions >= 4.13 at runtime.

The Cython modules define their config TypedDicts with the functional syntax,
which Cython compiles; the stubs declare the same types with the class
syntax.

Assisted-by: ClaudeCode:claude-opus-5-5
…nfigs JSON

CRC32, Adler32, CRC32C, JenkinsLookup3 and Fletcher32 are annotated, and
their configurations are typed: Checksum32Config has an optional `location`
(present only when passed to the constructor, which matches the Zarr v3
crc32c codec having no configuration), and JenkinsLookup3Config adds
`initval` and `prefix`.

Typing the JenkinsLookup3 configuration exposed a bug: with a prefix,
get_config() held a numpy array, so the configuration could neither be
serialized to JSON nor passed back to get_codec. It now reports the prefix
as a list of ints, which the constructor accepts.

Checksum32.encode and decode are annotated with the union of what the
family returns (uint8 ndarrays for CRC32/Adler32/CRC32C; bytes and a
memoryview for JenkinsLookup3, which overrides both). Changing
JenkinsLookup3's return types instead would break callers that compare its
output with bytes, as its own tests do.

Checksum32.decode checks the byte length of its input instead of len(buf),
which counted only the first axis of a multidimensional array. This
overlaps upstream PR zarr-developers#891.

Assisted-by: ClaudeCode:claude-opus-5-5
AsType, Delta, Quantize, FixedScaleOffset, BitRound, PackBits, Shuffle and
Categorize annotate their constructors, encode/decode (with the `out`
overloads) and get_config, each returning its own CodecConfig subclass.

- The numeric filters return ndarrays of the codec's dtypes, which numpy's
  types cannot express from a runtime dtype, so they are npt.NDArray[Any].
- BitRound returns NDArrayLike: it works on ndarray-likes such as cupy arrays
  and may return its input unchanged.
- PackBits encodes to uint8 and decodes to bool; Shuffle works on uint8
  bytes, and its encode also takes `out`.
- FixedScaleOffset's `offset` and `scale` are `float`, which covers the ints
  its configuration echoes back when ints were passed.
- Categorize accepts a plain iterable of strings when its dtype is object.

Every filter satisfies numcodecs.abc.ArrayArrayCodec.

Assisted-by: ClaudeCode:claude-opus-5-5
JSON, MsgPack, Pickle, VLenUTF8, VLenBytes, VLenArray, ZFPY and PCodec are
annotated, with config TypedDicts.

- JSON and MsgPack encode any array-like and decode to an ndarray whose dtype
  and shape come from the payload.
- Pickle encodes any object and decodes to Any, which is the truth: it
  returns whatever was pickled.
- ZFPY keeps the base `BufferLike` parameter, raising at runtime for anything
  but a C-contiguous numpy array, rather than narrowing the parameter type and
  breaking substitutability.
- PCodec's spec literals get names (ModeSpecName, DeltaSpecName,
  PagingSpecName) shared by its constructor and configuration.

The object codecs now pass `out` through ensure_ndarray before np.copyto,
which returns the same array for an ndarray `out` and lets the call
type-check against BufferLike.

Assisted-by: ClaudeCode:claude-opus-5-5
numcodecs now ships a PEP 561 py.typed marker, installed by meson, so type
checkers read its annotations and the Cython stubs instead of treating the
package as untyped.

Against a built wheel, a downstream consumer under mypy --strict and pyright
strict now gets real types for get_codec, the registry, Codec methods
(including decode(out=...)), every concrete codec and the compat helpers,
where it got Any (mypy) or None and Unknown (pyright) before. pyright's
--verifytypes completeness goes from 22.1% to 99.3% (699 known, 5 ambiguous,
0 unknown symbols).

To get there:

- The remaining strict-mode errors in src are fixed: values from untyped
  optional dependencies (zfpy, pcodec, msgpack, google_crc32c) are bound to
  annotated names, and the NDArrayLike metaclass subclasses
  typing._ProtocolMeta, which typeshed declares, instead of
  Protocol.__class__, which type checkers see as Any and which made every
  BufferLike annotation "partially unknown".
- Instance attributes on the filter codecs and ZFPY are annotated.
- Codec.from_config's `config` is no longer positional-only, so Codec matches
  protocols that declare it as a regular parameter (zarr-python's Numcodec).

The repository's mypy configuration is strict, with mypy_path = src and
ignore_missing_imports only for the untyped optional dependencies. Tests keep
their previous treatment (unannotated test functions are not checked).
pre-commit runs mypy 2.4.1, matching the pinned CI version. The typing CI
job checks that the installed package carries py.typed and the stubs.

Assisted-by: ClaudeCode:claude-opus-5-5
@codecov

codecov Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (1d2229f) to head (18192cc).

Additional details and impacted files
@@            Coverage Diff             @@
##              main      #915    +/-   ##
==========================================
  Coverage   100.00%   100.00%            
==========================================
  Files           27        27            
  Lines          922      1196   +274     
==========================================
+ Hits           922      1196   +274     
Files with missing lines Coverage Δ
src/numcodecs/__init__.py 100.00% <100.00%> (ø)
src/numcodecs/abc.py 100.00% <100.00%> (ø)
src/numcodecs/astype.py 100.00% <100.00%> (ø)
src/numcodecs/base64.py 100.00% <100.00%> (ø)
src/numcodecs/bitround.py 100.00% <100.00%> (ø)
src/numcodecs/bz2.py 100.00% <100.00%> (ø)
src/numcodecs/categorize.py 100.00% <100.00%> (ø)
src/numcodecs/checksum32.py 100.00% <100.00%> (ø)
src/numcodecs/compat.py 100.00% <100.00%> (ø)
src/numcodecs/delta.py 100.00% <100.00%> (ø)
... and 15 more
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

d-v-b added 10 commits November 21, 2026 17:04
With .pyi stubs next to the .pyx sources, ruff resolves numcodecs.blosc,
numcodecs.zstd, numcodecs.fletcher32 and numcodecs.jenkins as first-party
modules and wants them grouped with the other numcodecs imports.

Assisted-by: ClaudeCode:claude-opus-5-5
…dDicts

The Cython modules create their config TypedDicts at runtime with
typing.TypedDict, which adds __mutable_keys__ and __readonly_keys__;
stubtest does not expect them on the stubs' class-syntax TypedDicts.
The allowlist entry was only added in the py.typed change, but it is
needed from the commit that introduced the TypedDicts.

Assisted-by: ClaudeCode:claude-opus-5-5
The typing CI job and pre-commit run mypy with a different set of packages
installed than a development environment:

- setuptools_scm (build-time, used by _build_utils/gitversion.py) and
  importlib_metadata (test-only backport) are not installed in either, so
  they join the untyped optional dependencies whose missing imports are
  ignored.
- pre-commit also checks docs/conf.py, whose Mock.__getattr__ needed
  annotations under strict mode.
- pre-commit installs zarr, and tests/test_zarr3.py only imported the
  top-level package under TYPE_CHECKING, so zarr.codecs.numcodecs was
  not a known attribute. It now imports the submodule.

Assisted-by: ClaudeCode:claude-opus-5-5
@d-v-b
d-v-b merged commit dd52bc0 into zarr-developers:main Nov 21, 2026
36 checks passed
d-v-b added a commit to d-v-b/numcodecs that referenced this pull request Nov 21, 2026
Resolve the conflict in src/numcodecs/pickles.py by keeping the typed
signatures from zarr-developers#915 and the deprecation warning in Pickle.__init__.

Two tests added upstream construct the Pickle codec and now see the
deprecation warning: test_config_round_trip[pickle] (zarr-developers#906), which
rebuilds the codec from its config, and the static type test
test_pickle (zarr-developers#915). Both get a filterwarnings mark scoped to the
deprecation message.

Assisted-by: ClaudeCode:claude-opus-5.5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant