Repository navigation
python(feat): support bit field definitions on import configs - #861
Merged
Merged
Conversation
Contributor
|
Python docs preview: https://sift-stack.github.io/sift/python/pr-861/ Deployed from |
This reverts commit bc54be6.
alden-sift
marked this pull request as ready for review
November 4, 2026 00:45
alexluck-sift
left a comment
Collaborator
There was a problem hiding this comment.
Not blocking: do you expect users to need units, a description or signedness per element? Elements only carry a name, index and width, so that would be a proto change to ChannelBitFieldElement. Worth tracking separately if so?
Co-authored-by: Alex Luck <luck@siftstack.com>
alexluck-sift
previously approved these changes
Nov 9, 2026
Removed detailed explanation about bit field values and import requirements from the changelog. See relevant docstrings instead
alexluck-sift
approved these changes
Nov 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was changed
CSV, Parquet, TDMS and HDF5 import configs can now define bit field channels. Before this,
DataColumnBasehad nobit_field_elementsfield, so a config fromdetect_configcouldn't describe aBIT_FIELDchannel.What's a bit field
A bit field is one integer that packs several small values into fixed bit ranges. Hardware status words are the common case. In this 8-bit status value, bit 0 says whether the vehicle is armed, bits 1 to 3 hold an error code, and bits 4 to 7 hold a mode:
The raw value
53(0b0011_0101) decodes toarmed=1,error_code=2,mode=3. Sift splits aBIT_FIELDchannel into one sub-channel per element (status.armed,status.error_code,status.mode), but it needs the layout to do that. Each element is aChannelBitFieldElementwith aname, a starting bitindexand abit_countwidth.detect_configcan't infer the layout, so the user supplies it:Value formats
In CSV and Parquet, a bit field value must be raw bytes, a string of 0s and 1s, or URL-safe base64. Bytes are little-endian. Integers aren't accepted, and Parquet has no bit field type, so the client honors the config and expects the data in one of those formats. It doesn't rewrite files.
TDMS and HDF5 store typed integers, and an integer channel imports as its little-endian bytes, so those files need no conversion.
import_from_pathchecks CSV and ParquetBIT_FIELDcolumns before uploading, so a wrong format fails before the file is sent:10(bit string) or1234(base64), since the client can't tell those from intended input.Other changes
DataColumnBasegetsbit_field_elements. A validator requires it whendata_typeisBIT_FIELDand rejects it otherwise. It also rejects duplicate names, negative indexes, zero-width elements and overlapping ranges.BIT_FIELD, since those imports don't support bit field overrides. The field is onDataColumnBase, soUlogDataColumnandMcapDataColumnopt out through a_supports_bit_fieldsclass flag, and the validator raises when they're set toBIT_FIELD._channel_config_proto()writes the elements, and every_from_protoreads them back, including the embedded Parquet config path._channel_config_proto()re-runs the enum and bit field checks. Pydantic doesn't validate on assignment, so an in-place edit like the one above could otherwise send aBIT_FIELDchannel with no elements.enum_types. MCAP built its channel config by hand. It now uses the shared helper and reads enums back.Multi-channel single-channel-per-row Parquet still doesn't support bit fields, same as enums, because that layout derives channel configs per row.
Testing
BIT_FIELDat construction and on in-place edits, each validation error, in-place edits caught at import, and the pre-upload checks: accepted and rejected Parquet types, nested fields, sampled CSV and Parquet string values, unpadded bit strings and digit-only base64 passing, and a failing check stopping the import before anything is created or uploaded.binary,fixed_size_binary, bit string and base64 string columns, a 16-bit Parquet status word, and CSV bit strings and base64, and TDMS and HDF5uint8,uint16andint16channels all import and decode to the expected element values. The 16-bit cases decode0x1234tolow=0x34,high=0x12, so multi-byte values are little-endian.Verification on a local stack
Each case wrote a synthetic file, imported it into a local stack and read the elements back:
Accepted formats. All six imports succeeded and created a
BIT_FIELDchannel with the three elements.binaryfixed_size_binary[1]string, bitsstring, base640x350x3500110101NQ==00110101NQ==0x000x0000000000AA==00000000AA==0xff0xff11111111_w==11111111_w==0x010x0100000001AQ==00000001AQ==All six returned identical values, matching the hand-decoded expectation.
16-bit status word.
0x123434 120x00ffff 000xff0000 ff0x000101 00TDMS and HDF5 integer channels. Each file held one integer
statuschannel, set toBIT_FIELDafterdetect_config:TDMS and HDF5 returned identical values:
uint8uint160x1234,0x00ff,0xff00,0x0001int160x1234, -1, -256, 1Rejected before upload. Each raised a
ValueErrorbefore an import was created. A bit field value is never read as a decimal integer, so a CSV cell53isn't a bit string or valid base64. It has to be written as00110101,NQ==or the byte0x35.uint8list<uint8>Parquet
uint8:Parquet
list<uint8>:CSV: