Skip to content

feat(pickle): deprecate the Pickle codec, warning on construction - #905

Open
d-v-b wants to merge 2 commits into
zarr-developers:mainfrom
d-v-b:feat/deprecate-pickle
Open

d-v-b wants to merge 2 commits into
zarr-developers:mainfrom
d-v-b:feat/deprecate-pickle

Conversation

@d-v-b

@d-v-b d-v-b commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

🤖 AI text below 🤖

The Pickle codec is deprecated and will be removed in numcodecs 0.19.0. Constructing it, directly or through numcodecs.get_codec({"id": "pickle"}), emits a DeprecationWarning. Encoding and decoding still work until removal, and decoding still requires the opt-in described in GHSA-jqc8-qjfv-5m6r. To encode arrays of Python objects, use VLenUTF8, VLenBytes, VLenArray, JSON or MsgPack instead. Refs #706.

Design
  • Where it warns: Pickle.__init__ only. Every route to the codec goes through the constructor: Pickle(...), numcodecs.Pickle, Pickle.from_config, get_codec({"id": "pickle"}), and any downstream library that resolves the codec from array metadata. That makes construction the single place to warn, and it fires once per codec instance, when the array is opened or the codec is chosen, rather than once per chunk. The agent did not add a warning to encode/decode: it would repeat per chunk, decode is already gated by the advisory's opt-in, and the warning would arrive after the codec was already selected.
  • Reads of existing data are not broken. The warning does not change behavior unless the caller escalates DeprecationWarning to an error. The decode gate from GHSA-jqc8-qjfv-5m6r is untouched.
  • Warning class: DeprecationWarning, matching every earlier numcodecs deprecation: the numcodecs.zarr3 module, crc32c, PCodec(delta_spec="none"), and the blosc helper functions (via deprecated). Python hides DeprecationWarning by default unless it is attributed to __main__, so the warning uses skip_file_prefixes (Python >= 3.12, the minimum supported) to attribute it to the first frame outside numcodecs. A user calling get_codec({"id": "pickle"}) from a script sees it, and downstream test suites with filterwarnings = error see it at their own call site. A plain stacklevel=2 would attribute the get_codec path to numcodecs/abc.py, where it is hidden. FutureWarning would be visible to every end user, but it would break with precedent; reviewers may prefer it given the security angle.
  • Removal target: 0.19.0. The blosc helpers were deprecated in 0.15.0 and removed in 0.16.0, so the precedent is one minor release. The agent assumed the next release is 0.18.0, since Unreleased already adds a dependency (packaging). If this ships in a 0.17.x patch, the 0.19.0 target still leaves at least one minor release. The version lives in _REMOVAL_VERSION, used by the message, and in the docstring, codec docs page and release note.
  • Message. It describes the current state and the alternatives: VLenUTF8 for strings, VLenBytes for bytes, VLenArray for variable-length numeric arrays, and JSON or MsgPack for mixed objects. It also states why: decoding can execute arbitrary code, and pickled data can only be read by Python.
Changes
  • src/numcodecs/pickles.py: adds the warning in __init__, a deprecation paragraph and a numpydoc Warns section in the class docstring, and lists the alternative codecs in See Also. The doctest example builds the codec inside warnings.catch_warnings(), because --doctest-modules runs with warnings as errors.
  • docs/other/pickles.md: adds a "Deprecated" admonition, above the existing security warning, that links to the alternative codecs.
  • docs/release.md: adds an Unreleased → Deprecations entry.
  • tests/test_pickles.py:
    • test_construction_warns is parametrized over Pickle, numcodecs.Pickle, Pickle.from_config, get_codec and get_codec with a protocol. For each, it asserts that exactly one warning is raised, that the warning is attributed to the test file rather than to numcodecs internals, that the message names the removal version and every alternative, and that encode/decode still round-trip with no further warnings.
    • test_construction_raises_when_deprecation_warnings_are_errors covers the one error case the warning adds.
    • test_decode_refused_without_opt_in is kept unchanged.
    • The tests that predate the deprecation ignore the warning through a module-level pytestmark. The import-time codec list is built inside catch_warnings.
    • After merging main, two newer tests that construct Pickle get the same filter, scoped to the deprecation message: test_config_round_trip[pickle] in tests/test_codec_contract.py (test: check the Codec contract for every codec and add static type tests #906) and test_pickle in tests/typing/test_object_codec_types.py (feat(typing): ship py.typed and check numcodecs with strict mypy #915).
    • The agent checked that the attribution assertion matters: with stacklevel=2 in place of skip_file_prefixes, the three from_config/get_codec cases fail.
Verification
  • After merging main (through feat(typing): ship py.typed and check numcodecs with strict mypy #915), the agent ran the full suite with doctests (pytest tests src/numcodecs) on Python 3.13, with msgpack, pcodec, zfpy, zarr 3.4.2, crc32c and google-crc32c installed: 1055 passed, 21 skipped. The skips are cupy, free-threading-only and legacy-crc32c-only tests.
  • The CI typing checks pass locally: mypy --config-file pyproject.toml src tests, mypy --config-file tests/typing/mypy.ini tests/typing, and pyright -p tests/typing.
  • SKIP=mypy pre-commit run --all-files passes. The mypy hook was skipped because zfpy does not build on the pre-commit interpreter (CPython 3.15t); the mypy runs above cover it.
  • A Sphinx HTML build of the docs gives no warnings for the changed pages, and the admonition links resolve.
Downstream impact (zarr-python)

The agent checked zarr-python by reading hand-written v2 arrays with each released version. zarr derives object_codec_id from the first entry in filters + compressor whose id is in zarr.core.dtype.common.OBJECT_CODEC_IDS, and from 3.1.0 on, only vlen-utf8 and vlen-bytes resolve to a data type. Two layouts behave differently:

Layout zarr 3.0.x zarr 3.1.0 to 3.4.1 zarr 3.4.2
filters=[pickle], dtype="|O" (what zarr 2's object_codec=Pickle() wrote) opens; decode needs the numcodecs opt-in ValueError: No Zarr data type found at open same as 3.1
filters=[vlen-utf8], compressor=pickle opens; decode needs the numcodecs opt-in opens; decode needs the numcodecs opt-in opens; decode needs both array.allow_unsafe_pickle_decode (zarr-developers/zarr-python#4465) and the numcodecs opt-in
  • Where the array opens, zarr builds the codec through numcodecs.get_codec, so this warning fires once at open, attributed to zarr/core/metadata/v2.py. Python's default filters hide it there; downstream test suites that turn warnings into errors see it.
  • The second layout is unusual for real data, but it is the one crafted metadata would use to reach Pickle.decode, so both gates still matter.
  • zarr-python 2 imports numcodecs.blosc.cbuffer_sizes at import time, which was removed in numcodecs 0.16 (checked for 2.18.2, 2.18.4 and 2.18.7), and 2.18.7 pins numcodecs<0.16. So zarr 2 never runs with a numcodecs that has the decode gate or this warning.
  • There is no zarr.codecs.numcodecs.Pickle wrapper.

zarr-python follow-ups, not part of this PR:

  1. Needed before this ships: zarr 3.4.2's tests/test_v2.py constructs Pickle() (the #4465 tests), and zarr runs pytest with filterwarnings = ["error", ...]. Once this release is out, those tests fail on the deprecation warning. They need @pytest.mark.filterwarnings("ignore:The Pickle codec is deprecated:DeprecationWarning") or a matching entry in pyproject.toml.
  2. Give a targeted error when v2 metadata resolves object_codec_id to pickle, in place of the generic "No Zarr data type found" message.
  3. Remove "pickle" from ObjectCodecID / OBJECT_CODEC_IDS, and remove the array.allow_unsafe_pickle_decode gate, once numcodecs removes the codec (0.19.0).

🤖 Generated with Claude Code

Constructing Pickle, directly or through get_codec({"id": "pickle"}),
now emits a DeprecationWarning naming the planned removal in 0.19.0
and the codecs to use for object data instead (VLenUTF8, VLenBytes,
VLenArray, JSON, MsgPack). The warning is attributed to the first
caller outside numcodecs, so it points at user code or at the library
that selected the codec from array metadata. Encoding and decoding are
unchanged, and decoding still requires the opt-in from
GHSA-jqc8-qjfv-5m6r.

Refs: zarr-developers#706

Assisted-by: ClaudeCode:claude-opus-5.5
@codecov

codecov Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (dd52bc0) to head (6dd335d).

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #905   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           27        27           
  Lines         1196      1201    +5     
=========================================
+ Hits          1196      1201    +5     
Files with missing lines Coverage Δ
src/numcodecs/pickles.py 100.00% <100.00%> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Resolve the conflict in src/numcodecs/pickles.py by keeping the typed
signatures from zarr-developers#915 and the deprecation warning in Pickle.__init__.

Two tests added upstream construct the Pickle codec and now see the
deprecation warning: test_config_round_trip[pickle] (zarr-developers#906), which
rebuilds the codec from its config, and the static type test
test_pickle (zarr-developers#915). Both get a filterwarnings mark scoped to the
deprecation message.

Assisted-by: ClaudeCode:claude-opus-5.5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant