Repository navigation
Conversation
Constructing Pickle, directly or through get_codec({"id": "pickle"}),
now emits a DeprecationWarning naming the planned removal in 0.19.0
and the codecs to use for object data instead (VLenUTF8, VLenBytes,
VLenArray, JSON, MsgPack). The warning is attributed to the first
caller outside numcodecs, so it points at user code or at the library
that selected the codec from array metadata. Encoding and decoding are
unchanged, and decoding still requires the opt-in from
GHSA-jqc8-qjfv-5m6r.
Refs: zarr-developers#706
Assisted-by: ClaudeCode:claude-opus-5.5
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #905 +/- ##
=========================================
Coverage 100.00% 100.00%
=========================================
Files 27 27
Lines 1196 1201 +5
=========================================
+ Hits 1196 1201 +5
🚀 New features to boost your workflow:
|
This was referenced Nov 21, 2026
Resolve the conflict in src/numcodecs/pickles.py by keeping the typed signatures from zarr-developers#915 and the deprecation warning in Pickle.__init__. Two tests added upstream construct the Pickle codec and now see the deprecation warning: test_config_round_trip[pickle] (zarr-developers#906), which rebuilds the codec from its config, and the static type test test_pickle (zarr-developers#915). Both get a filterwarnings mark scoped to the deprecation message. Assisted-by: ClaudeCode:claude-opus-5.5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
The
Picklecodec is deprecated and will be removed in numcodecs 0.19.0. Constructing it, directly or throughnumcodecs.get_codec({"id": "pickle"}), emits aDeprecationWarning. Encoding and decoding still work until removal, and decoding still requires the opt-in described in GHSA-jqc8-qjfv-5m6r. To encode arrays of Python objects, useVLenUTF8,VLenBytes,VLenArray,JSONorMsgPackinstead. Refs #706.Design
Pickle.__init__only. Every route to the codec goes through the constructor:Pickle(...),numcodecs.Pickle,Pickle.from_config,get_codec({"id": "pickle"}), and any downstream library that resolves the codec from array metadata. That makes construction the single place to warn, and it fires once per codec instance, when the array is opened or the codec is chosen, rather than once per chunk. The agent did not add a warning toencode/decode: it would repeat per chunk,decodeis already gated by the advisory's opt-in, and the warning would arrive after the codec was already selected.DeprecationWarningto an error. The decode gate from GHSA-jqc8-qjfv-5m6r is untouched.DeprecationWarning, matching every earlier numcodecs deprecation: thenumcodecs.zarr3module,crc32c,PCodec(delta_spec="none"), and the blosc helper functions (viadeprecated). Python hidesDeprecationWarningby default unless it is attributed to__main__, so the warning usesskip_file_prefixes(Python >= 3.12, the minimum supported) to attribute it to the first frame outside numcodecs. A user callingget_codec({"id": "pickle"})from a script sees it, and downstream test suites withfilterwarnings = errorsee it at their own call site. A plainstacklevel=2would attribute theget_codecpath tonumcodecs/abc.py, where it is hidden.FutureWarningwould be visible to every end user, but it would break with precedent; reviewers may prefer it given the security angle.packaging). If this ships in a 0.17.x patch, the 0.19.0 target still leaves at least one minor release. The version lives in_REMOVAL_VERSION, used by the message, and in the docstring, codec docs page and release note.Changes
src/numcodecs/pickles.py: adds the warning in__init__, a deprecation paragraph and a numpydocWarnssection in the class docstring, and lists the alternative codecs in See Also. The doctest example builds the codec insidewarnings.catch_warnings(), because--doctest-modulesruns with warnings as errors.docs/other/pickles.md: adds a "Deprecated" admonition, above the existing security warning, that links to the alternative codecs.docs/release.md: adds an Unreleased → Deprecations entry.tests/test_pickles.py:test_construction_warnsis parametrized overPickle,numcodecs.Pickle,Pickle.from_config,get_codecandget_codecwith a protocol. For each, it asserts that exactly one warning is raised, that the warning is attributed to the test file rather than to numcodecs internals, that the message names the removal version and every alternative, and that encode/decode still round-trip with no further warnings.test_construction_raises_when_deprecation_warnings_are_errorscovers the one error case the warning adds.test_decode_refused_without_opt_inis kept unchanged.pytestmark. The import-time codec list is built insidecatch_warnings.main, two newer tests that constructPickleget the same filter, scoped to the deprecation message:test_config_round_trip[pickle]intests/test_codec_contract.py(test: check the Codec contract for every codec and add static type tests #906) andtest_pickleintests/typing/test_object_codec_types.py(feat(typing): ship py.typed and check numcodecs with strict mypy #915).stacklevel=2in place ofskip_file_prefixes, the threefrom_config/get_codeccases fail.Verification
main(through feat(typing): ship py.typed and check numcodecs with strict mypy #915), the agent ran the full suite with doctests (pytest tests src/numcodecs) on Python 3.13, with msgpack, pcodec, zfpy, zarr 3.4.2, crc32c and google-crc32c installed: 1055 passed, 21 skipped. The skips are cupy, free-threading-only and legacy-crc32c-only tests.mypy --config-file pyproject.toml src tests,mypy --config-file tests/typing/mypy.ini tests/typing, andpyright -p tests/typing.SKIP=mypy pre-commit run --all-filespasses. The mypy hook was skipped because zfpy does not build on the pre-commit interpreter (CPython 3.15t); the mypy runs above cover it.Downstream impact (zarr-python)
The agent checked zarr-python by reading hand-written v2 arrays with each released version. zarr derives
object_codec_idfrom the first entry infilters+compressorwhose id is inzarr.core.dtype.common.OBJECT_CODEC_IDS, and from 3.1.0 on, onlyvlen-utf8andvlen-bytesresolve to a data type. Two layouts behave differently:filters=[pickle],dtype="|O"(what zarr 2'sobject_codec=Pickle()wrote)ValueError: No Zarr data type foundat openfilters=[vlen-utf8],compressor=picklearray.allow_unsafe_pickle_decode(zarr-developers/zarr-python#4465) and the numcodecs opt-innumcodecs.get_codec, so this warning fires once at open, attributed tozarr/core/metadata/v2.py. Python's default filters hide it there; downstream test suites that turn warnings into errors see it.Pickle.decode, so both gates still matter.numcodecs.blosc.cbuffer_sizesat import time, which was removed in numcodecs 0.16 (checked for 2.18.2, 2.18.4 and 2.18.7), and 2.18.7 pinsnumcodecs<0.16. So zarr 2 never runs with a numcodecs that has the decode gate or this warning.zarr.codecs.numcodecs.Picklewrapper.zarr-python follow-ups, not part of this PR:
tests/test_v2.pyconstructsPickle()(the #4465 tests), and zarr runs pytest withfilterwarnings = ["error", ...]. Once this release is out, those tests fail on the deprecation warning. They need@pytest.mark.filterwarnings("ignore:The Pickle codec is deprecated:DeprecationWarning")or a matching entry inpyproject.toml.object_codec_idtopickle, in place of the generic "No Zarr data type found" message."pickle"fromObjectCodecID/OBJECT_CODEC_IDS, and remove thearray.allow_unsafe_pickle_decodegate, once numcodecs removes the codec (0.19.0).🤖 Generated with Claude Code