Skip to content

all(feat): support custom time formats on CSV and Parquet imports - #884

Merged
omar-sift merged 4 commits into
mainfrom
sift/support-custom-date-formats
Nov 18, 2026
Merged

omar-sift merged 4 commits into
mainfrom
sift/support-custom-date-formats

Conversation

@omar-sift

@omar-sift omar-sift commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Summary

CSV and Parquet imports can now read timestamps in a custom Unix date format. The built-in formats only cover fixed layouts, so a file with month-first timestamps like 10-29-2026 16:53:45 could not be imported.

CsvTimeColumn(column=4, format=TimeFormat.CUSTOM, custom_format="%m-%d-%Y %H:%M:%S")
  • TimeFormat.CUSTOM and custom_format: new on CsvTimeColumn and ParquetTimeColumn. custom_format defaults to "", so existing configs are unchanged, and it is kept when a config is read back from the server. Importing with TimeFormat.CUSTOM and an empty custom_format raises ValueError before upload. Times without an offset are read as UTC; zone names (%Z) are rejected, so use %z or %:z.
  • sift-cli import csv and sift-cli import parquet (flat-dataset, cpr single, cpr multi): --time-format custom with --time-format-str "%m-%d-%Y %H:%M:%S". custom without --time-format-str fails before upload.
  • TDMS and HDF5: TimeFormat.CUSTOM and --time-format custom are rejected, since those configs have no format string field.
  • Protos: TIME_FORMAT_CUSTOM and custom_format come from the public protos update in sift(chore): update public protos #890. This PR does not change protos or generated code.

Merge after the server side of this change is deployed. Older servers ignore custom_format.

Merge after the server side (azimuth #16601, merged) is deployed. Older servers ignore custom_format.

Verification

  • sift_client: test_data_imports.py, 167 passed against a local server running azimuth main, including CSV and Parquet round trips, the empty default, CUSTOM without custom_format rejected at import, and TDMS and HDF5 rejecting CUSTOM. ruff and pyright pass.
  • sift-cli: cargo test -p sift_cli, 289 passed; clippy clean. Ran the built binary with --preview: CSV, Parquet flat-dataset and cpr single accept custom with --time-format-str and error without it; import tdms and import hdf5 reject custom.
  • Against a local server running azimuth main: a CSV import with month-first timestamps and custom_format="%m-%d-%Y %H:%M:%S" read back 120 points starting at 2026-10-29T16:53:45Z. Parquet imports (flat dataset and single channel per row) with the same format were verified before the field was renamed to custom_format; the field number is unchanged, so the wire format is the same. A mismatched custom_format fails with an error naming the value and format, and existing formats import unchanged.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Python docs preview: https://sift-stack.github.io/sift/python/pr-884/

Deployed from 3a87876. The link may take up to a minute to become live as GitHub Pages propagates.

@omar-sift omar-sift changed the title sift(chore): Support custom date formats all(feat): support custom time formats on CSV and Parquet imports Nov 12, 2026
@omar-sift
omar-sift force-pushed the sift/support-custom-date-formats branch 5 times, most recently from 75c960c to a2c6a8d Compare November 12, 2026 19:48
@omar-sift
omar-sift requested a review from wei-sift November 12, 2026 21:06
@omar-sift
omar-sift marked this pull request as ready for review November 12, 2026 21:06
@omar-sift
omar-sift force-pushed the sift/support-custom-date-formats branch from a2c6a8d to 8e3b491 Compare November 12, 2026 21:47
@omar-sift
omar-sift marked this pull request as draft November 13, 2026 16:53
@omar-sift
omar-sift force-pushed the sift/support-custom-date-formats branch from 8e3b491 to 968af4a Compare November 14, 2026 01:10
@omar-sift
omar-sift marked this pull request as ready for review November 14, 2026 01:31
@omar-sift
omar-sift force-pushed the sift/support-custom-date-formats branch from 968af4a to 86b6ceb Compare November 16, 2026 23:47
@omar-sift
omar-sift requested a review from wei-sift November 17, 2026 16:47
wei-sift
wei-sift previously approved these changes Nov 17, 2026

@wei-sift wei-sift left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

@omar-sift
omar-sift merged commit 4e6880e into main Nov 18, 2026
27 checks passed
@omar-sift
omar-sift deleted the sift/support-custom-date-formats branch November 18, 2026 16:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants