Skip to content

fix(typing): correct registry, compat and NDArrayLike annotations and add __all__ - #908

Open
d-v-b wants to merge 6 commits into
zarr-developers:mainfrom
d-v-b:typing/03-fix-annotations
Open

d-v-b wants to merge 6 commits into
zarr-developers:mainfrom
d-v-b:typing/03-fix-annotations

Conversation

@d-v-b

@d-v-b d-v-b commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

🤖 AI text below 🤖

Correct type annotations that described something other than what the code does, and define __all__ in numcodecs so type checkers treat its re-exported codec classes as public. register_codec now raises for a class without a codec_id, and loading a non-Codec class through the numcodecs.codecs entry point group emits a DeprecationWarning.

Summary

Annotations that were wrong:

  • codec_registry was annotated dict[str, Codec] but holds classes; it is now dict[str, type[Codec]].
  • entries was annotated dict[str, EntryPoints] but holds single EntryPoints.
  • ndarray_copy claimed to return NDArrayLike but returns src unchanged (often bytes) when dst is None. It is now overloaded on dst.
  • NDArrayLike, DType and FlagsObj declared settable attributes, so no numpy.ndarray satisfied NDArrayLike under mypy or pyright. They now declare read-only properties.
    • The isinstance cache moves to module level. A protocol with no annotations of its own otherwise inherits the metaclass's __annotations__, and runtime_checkable then demands the cache as a member.

Other changes:

  • The compat functions annotate their parameters with a new numcodecs.ndarray_like.BufferLike alias (collections.abc.Buffer | NDArrayLike).
  • is_ndarray_like returns TypeIs.
  • get_codec and register_codec are annotated.
  • __all__ in numcodecs/__init__.py: without it, numcodecs.Zlib and friends count as private re-exports once the package ships py.typed. As a side effect, from numcodecs import * now imports only the public names.
  • mypy now targets Python 3.12, the oldest supported version. Doing so caught typing.TypeIs, which only exists from 3.13 and now comes from typing_extensions.
Tests

Full suite passes locally; there are new tests for register_codec without an id and for the entry-point deprecation warning, plus assert_type tests for compat and registry.

Stack and merge order

The agent split honest type annotations for numcodecs into a stack of 10 PRs, following an audit of the existing annotations and of the earlier attempts (#443, #698, #701, #702). Each PR is opened from the fork against main, so until the PRs before it merge, its diff also contains their commits. The change specific to this PR starts at commit 52be3f5; later commits on its branch are follow-up fixes and merges of the PR below it. Merge in this order:

  1. test: check the Codec contract for every codec and add static type tests #906 typing/01-harness: test: check the Codec contract for every codec and add static type tests
  2. fix: return the caller's out from decode(buf, out=out) in every codec #907 typing/02-out-contract: fix: return the caller's out from decode(buf, out=out) in every codec
  3. fix(typing): correct registry, compat and NDArrayLike annotations and add __all__ #908 typing/03-fix-annotations: fix(typing): correct registry, compat and NDArrayLike annotations and add all ← this PR
  4. feat(abc): annotate the Codec base class and add structural codec protocols #909 typing/04-base-class: feat(abc): annotate the Codec base class and add structural codec protocols
  5. feat(typing): ship .pyi stubs for the Cython extension modules #910 typing/05-cython-stubs: feat(typing): ship .pyi stubs for the Cython extension modules
  6. feat(typing): annotate the compressor codecs with per-codec config TypedDicts #911 typing/06a-compressors: feat(typing): annotate the compressor codecs with per-codec config TypedDicts
  7. feat(typing): annotate the checksum codecs and make JenkinsLookup3 configs JSON #912 typing/06b-checksums: feat(typing): annotate the checksum codecs and make JenkinsLookup3 configs JSON
  8. feat(typing): annotate the filter codecs #913 typing/06c-filters: feat(typing): annotate the filter codecs
  9. feat(typing): annotate the object and array-to-bytes codecs #914 typing/06d-object-codecs: feat(typing): annotate the object and array-to-bytes codecs
  10. feat(typing): ship py.typed and check numcodecs with strict mypy #915 typing/07-py-typed: feat(typing): ship py.typed and check numcodecs with strict mypy

🤖 Generated with Claude Code

Add tests/test_codec_contract.py, which runs every registered codec through
the behavior the Codec base class promises: encode and decode return buffers,
decode(out=...) returns the caller's out, and get_config round-trips through
from_config. Known violations are strict xfails, so fixing one forces its
entry to be removed.

Add tests/typing, where assert_type checks (verified by mypy and pyright in
strict mode) sit next to runtime assertions (verified by pytest), and a
`typing` CI job that runs both checkers plus the repository's own mypy
configuration. The type checkers are pinned in a new `typing` dependency
group.

Assisted-by: ClaudeCode:claude-opus-5-5
The Codec docstring says decode writes into `out`, and callers expect `out`
back. Ten codecs (Zlib, BZ2, LZMA, Base64, Shuffle, ZFPY, PCodec and the
three VLen codecs) rebound `out` to a flattened view and returned that, and
numcodecs.compat.ndarray_copy did the same for any `out` that was not an
ndarray, which every filter and checksum codec inherited. The decoded bytes
were already in the caller's buffer; only the returned object changes.

JenkinsLookup3.decode now accepts any contiguous writable buffer as `out`
and checks its size, and Fletcher32.decode returns a uint8 ndarray instead of
a Cython memoryview slice that no type annotation could name.

The strict xfails in tests/test_codec_contract.py are removed.

Assisted-by: ClaudeCode:claude-opus-5-5
… add __all__

Several annotations described something other than what the code does:

- codec_registry holds classes: dict[str, type[Codec]], not dict[str, Codec].
- entries holds single entry points: dict[str, EntryPoint].
- ndarray_copy returned NDArrayLike, but hands back `src` unchanged (often
  bytes) when `dst` is None. It is now overloaded on `dst`.
- NDArrayLike, DType and FlagsObj declared settable attributes, so no
  numpy.ndarray satisfied NDArrayLike under mypy or pyright. They now declare
  read-only properties. The isinstance cache moves to module level, because
  a protocol without annotations of its own would otherwise pick up the
  metaclass's annotation as a required member.

The compat functions annotate their parameters with a new BufferLike alias
(collections.abc.Buffer | NDArrayLike), is_ndarray_like returns TypeIs, and
get_codec/register_codec are annotated. register_codec now refuses a class
without a codec_id instead of registering it under None, and get_codec warns
(DeprecationWarning) when an entry point loads a class that does not
subclass Codec, so that `-> Codec` can become true.

numcodecs/__init__.py defines __all__, so that type checkers treat the
re-exported codec classes as public once numcodecs ships py.typed.

mypy now checks against Python 3.12, the oldest supported version; doing so
caught `typing.TypeIs`, which only exists from 3.13 and now comes from
typing_extensions.

Assisted-by: ClaudeCode:claude-opus-5-5
@codecov

codecov Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (1d2229f) to head (e3b9d2d).

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #908   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           27        27           
  Lines          922       939   +17     
=========================================
+ Hits           922       939   +17     
Files with missing lines Coverage Δ
src/numcodecs/__init__.py 100.00% <100.00%> (ø)
src/numcodecs/compat.py 100.00% <100.00%> (ø)
src/numcodecs/ndarray_like.py 100.00% <100.00%> (ø)
src/numcodecs/registry.py 100.00% <100.00%> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant