Skip to content

python(feat): keep a run open across file imports - #848

Merged
ian-sift merged 10 commits into
mainfrom
file-upload-leave-run-open-flag
Nov 16, 2026
Merged

ian-sift merged 10 commits into
mainfrom
file-upload-leave-run-open-flag

Conversation

@ian-sift

@ian-sift ian-sift commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Add leave_run_open to CreateDataImportFromUrl and CreateDataImportFromUpload. Omitting it keeps the current behavior: a finished import stops its run and fires run-stop automations.
  • import_from_path and Run.import_data take leave_run_open (default false). Set it when more files are still coming, then call run.stop() after the last import.
  • Those methods also take a list of paths. When the files are available together, pass them in one call. That upload is one import into one run.
  • Pass a mapping of path to config to set a file's config yourself. A key is a path from the list, or that file's name. Files you leave out are detected. One config still applies to a single path.
  • The CLI flag is in rust(feat): keep a run open across file imports #889, stacked on this pull request, so the Python and Rust releases stay separate.

Test plan

  • Python data-import unit tests, including the flag forwarded onto the upload request
  • A list of paths uploads one zip. A mapping supplies one file's config and the other file is still detected. A single path still takes one config.
  • Import two files into one run with the flag on the first and confirm the run stays open until the second import finishes
  • Manually upload two CSV files in one import_from_path call to the local backend

Backend verification

Uploaded a.csv and b.csv together with import_from_path against the local backend (localhost:50051 / localhost:8080). The client created one import, the upload returned HTTP 200, and the worker read both files (voltage_a and voltage_b) and created the run.

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Python docs preview: https://sift-stack.github.io/sift/python/pr-848/

Deployed from fc949f8. The link may take up to a minute to become live as GitHub Pages propagates.

Several files can land in one run before run-stop automations fire. The Python upload helpers take leave_run_open, and sift-cli import commands take --leave-run-open.
The sync DataImportAPI type comes from the generated stub. Without leave_run_open there, mypy rejects the argument at the Run.import_data call site.
@ian-sift
ian-sift force-pushed the file-upload-leave-run-open-flag branch from 341d776 to e886247 Compare November 12, 2026 19:31
…-open-flag

# Conflicts:
#	rust/crates/sift_cli/CHANGELOG.md
Callers can pass several paths to one import so those files share a run.
Both import paths resolved the run the same way. The batch tests asserted
on the typed client methods, which pyright rejects.
Callers who already have the files were told to upload them one at a time.
Pass the paths together instead.
Comment thread rust/crates/sift_cli/CHANGELOG.md Resolved
A list of paths still uploads as one zip. A mapping of path to config
overrides detection for the files it names. The batch upload also drops
the manual temp-directory cleanup and the data-type cast.
Python and the CLI release separately. The flag ships in a pull request
stacked on this one.
@ian-sift
ian-sift enabled auto-merge (squash) November 13, 2026 23:37
Config matching and the zip live with the other import helpers, so tests
call them without a client. One file and several files share config,
asset, and run resolution, and differ only at the upload.
@ian-sift
ian-sift merged commit 9184035 into main Nov 16, 2026
27 checks passed
@ian-sift
ian-sift deleted the file-upload-leave-run-open-flag branch November 16, 2026 23:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants