Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions rust/crates/sift_cli/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,13 @@ This project adheres to [Semantic Versioning](http://semver.org/).

### What's New

- `import` accepts `--leave-run-open`. A finished import marks its run stopped
and fires run-stop automations unless the flag is set. Stop the run after
the last import. The flag does nothing when the import has no run.
- `import csv` accepts `--run-id`, the id of an existing run. `--run` creates
a new run each time, because run names are not unique. Use `--run-id` to
add a file to a run left open with `--leave-run-open`.

## [v0.9.0] - November 16, 2026

### What's New
Expand Down
3 changes: 3 additions & 0 deletions rust/crates/sift_cli/assets/docs/src/data/import-csv.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,9 @@ sift-cli import csv ./telemetry.csv \
--wait
```

If you have more than one file for that run, import them together. See
[Several files in one run](./importing.md#several-files-in-one-run).

## Pointing at the right rows and time column

If your file has metadata rows above the header, or the timestamp is not the
Expand Down
3 changes: 3 additions & 0 deletions rust/crates/sift_cli/assets/docs/src/data/import-mcap.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,9 @@ sift-cli import mcap ./recording.mcap \
--wait
```

If you have more than one file for that run, import them together. See
[Several files in one run](./importing.md#several-files-in-one-run).

## Previewing the detected channels

`--preview` (`-p`) parses the file locally and prints the channels the import
Expand Down
3 changes: 3 additions & 0 deletions rust/crates/sift_cli/assets/docs/src/data/import-tdms.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,9 @@ sift-cli import tdms ./measurement.tdms \
--wait
```

If you have more than one file for that run, import them together. See
[Several files in one run](./importing.md#several-files-in-one-run).

## Channels with missing timing

When a channel lacks timing metadata, choose how the CLI should respond with
Expand Down
3 changes: 3 additions & 0 deletions rust/crates/sift_cli/assets/docs/src/data/import-ulog.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@ sift-cli import ulog ./flight.ulg \
--wait
```

If you have more than one file for that run, import them together. See
[Several files in one run](./importing.md#several-files-in-one-run).

## Anchoring timestamps

ULog records boot-relative timestamps. The import anchors them to UTC using
Expand Down
32 changes: 32 additions & 0 deletions rust/crates/sift_cli/assets/docs/src/data/importing.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,38 @@ queryable immediately afterward.
sift-cli import csv ./telemetry.csv --asset rover-1 --wait
```

### Several files in one run

If the files are available together, import them in one call from the Python
client. Pass the paths to `client.data_import.import_from_path` or
`Run.import_data`. That upload is one import into one run.

```python
client.data_import.import_from_path(
["part-1.csv", "part-2.csv"],
asset="rover-1",
run_name="sortie-4",
)
```

Pass a mapping of path to config when a file needs a column mapping or
channel list you set. Files you leave out of the mapping are detected.

The CLI imports one file per command. `--run` creates a new run each time,
because run names are not unique. A second command with the same `--run`
does not add data to the first run. Pass `--run-id` with the id of the
open run.

When a later file is not available yet, pass `--leave-run-open` on the
first import. A finished import then does not mark the run stopped and
does not fire run-stop automations. Omit the flag on the last import.
The flag does nothing when the import has no run.

```sh
sift-cli import csv ./part-1.csv --asset rover-1 --run sortie-4 --leave-run-open
sift-cli import csv ./part-2.csv --asset rover-1 --run-id <run-id>
```

### Time formats

Imports that parse a time column accept a `--time-format` describing how
Expand Down
25 changes: 25 additions & 0 deletions rust/crates/sift_cli/src/cli/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -433,6 +433,13 @@ pub struct ImportCsvArgs {
#[arg(short, long)]
pub run: Option<String>,

/// The id of an existing run to add this data to. Mutually exclusive with --run.
///
/// `--run` creates a new run each time, because run names are not unique.
/// Use this to add a file to a run left open with `--leave-run-open`.
#[arg(long, conflicts_with = "run")]
pub run_id: Option<String>,

/// Row number containing column headers (1-based)
#[arg(long, default_value_t = 1, help_heading = "Layout")]
pub header_row: usize,
Expand Down Expand Up @@ -486,6 +493,15 @@ pub struct ImportCsvArgs {
/// Preview the parsed schema without uploading
#[arg(short, long)]
pub preview: bool,

/// Leave the run open when the import finishes.
///
/// By default a finished import marks its run stopped, which fires
/// run-stop automations. Set this when more files are coming to the run,
/// then stop the run after the last import. Ignored when the import has
/// no run.
#[arg(long)]
pub leave_run_open: bool,
}

#[derive(clap::Args)]
Expand Down Expand Up @@ -544,6 +560,15 @@ pub struct CommonImportArgs {
/// Preview the parsed schema without uploading
#[arg(short, long)]
pub preview: bool,

/// Leave the run open when the import finishes.
///
/// By default a finished import marks its run stopped, which fires
/// run-stop automations. Set this when more files are coming to the run,
/// then stop the run after the last import. Ignored when the import has
/// no run.
#[arg(long)]
pub leave_run_open: bool,
}

#[derive(Subcommand)]
Expand Down
51 changes: 51 additions & 0 deletions rust/crates/sift_cli/src/cmd/import/csv.rs
Original file line number Diff line number Diff line change
Expand Up @@ -338,6 +338,7 @@ fn create_data_import_request<R: io::Read>(
data_columns,
first_data_row: args.first_data_row as u32,
run_name: args.run.clone().unwrap_or_default(),
run_id: args.run_id.clone().unwrap_or_default(),
time_column: Some(CsvTimeColumn {
relative_start_time,
column_number: args.time_column as u32,
Expand All @@ -346,6 +347,7 @@ fn create_data_import_request<R: io::Read>(
}),
..Default::default()
}),
leave_run_open: args.leave_run_open,
..Default::default()
})
}
Expand Down Expand Up @@ -378,6 +380,7 @@ mod test_create_data_import_request {
path: PathBuf::default(),
asset: "test_asset".into(),
run: None,
run_id: None,
header_row: 1,
first_data_row: 2,
channel_column: Vec::default(),
Expand All @@ -391,10 +394,12 @@ mod test_create_data_import_request {
relative_start_time: None,
wait: false,
preview: false,
leave_run_open: true,
},
)
.expect("expected Result::Ok");

assert!(req.leave_run_open);
let csv_config = req.csv_config.expect("expected Option::Some");
assert_eq!(String::from("test_asset"), csv_config.asset_name);
assert!(csv_config.run_id.is_empty());
Expand All @@ -412,6 +417,46 @@ mod test_create_data_import_request {
assert_eq!(String::from("channel"), config.name);
}

#[test]
fn run_id_is_set_on_the_request() {
let test_csv = indoc! {"
time,channel
2025-10-04T21:58:13Z,1.0
"};
let csv_reader = csv::ReaderBuilder::new()
.has_headers(false)
.from_reader(test_csv.as_bytes());

let req = create_data_import_request(
csv_reader,
&ImportCsvArgs {
path: PathBuf::default(),
asset: "test_asset".into(),
run: None,
run_id: Some("run-1".into()),
header_row: 1,
first_data_row: 2,
channel_column: Vec::default(),
data_type: Vec::default(),
unit: Vec::default(),
description: Vec::default(),
enum_config: Vec::default(),
bit_field_config: Vec::default(),
time_column: 1,
time_format: TimeFormat::default(),
relative_start_time: None,
wait: false,
preview: false,
leave_run_open: false,
},
)
.expect("expected Result::Ok");

let csv_config = req.csv_config.expect("expected Option::Some");
assert_eq!(csv_config.run_id, "run-1");
assert!(csv_config.run_name.is_empty());
}

#[test]
fn simple_type_override() {
let test_csv = indoc! {"
Expand All @@ -428,6 +473,7 @@ mod test_create_data_import_request {
path: PathBuf::default(),
asset: "test_asset".into(),
run: Some("test_run".into()),
run_id: None,
header_row: 1,
first_data_row: 2,
channel_column: vec![2],
Expand All @@ -441,6 +487,7 @@ mod test_create_data_import_request {
relative_start_time: None,
wait: false,
preview: false,
leave_run_open: false,
},
)
.expect("expected Result::Ok");
Expand Down Expand Up @@ -472,6 +519,7 @@ mod test_create_data_import_request {
path: PathBuf::default(),
asset: "test_asset".into(),
run: Some("test_run".into()),
run_id: None,
header_row: 1,
first_data_row: 2,
channel_column: vec![2],
Expand All @@ -485,6 +533,7 @@ mod test_create_data_import_request {
relative_start_time: None,
wait: false,
preview: false,
leave_run_open: false,
},
)
.expect("expected Result::Ok");
Expand Down Expand Up @@ -532,6 +581,7 @@ mod test_create_data_import_request {
path: PathBuf::default(),
asset: "test_asset".into(),
run: Some("test_run".into()),
run_id: None,
header_row: 1,
first_data_row: 2,
channel_column: vec![2, 3],
Expand All @@ -545,6 +595,7 @@ mod test_create_data_import_request {
relative_start_time: None,
wait: false,
preview: false,
leave_run_open: false,
},
)
.expect("expected Result::Ok");
Expand Down
10 changes: 7 additions & 3 deletions rust/crates/sift_cli/src/cmd/import/hdf5/import.rs
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ use sift_rs::{
};

use crate::{
cli::ImportHdf5Args,
cli::{CommonImportArgs, ImportHdf5Args},
cmd::{
Context,
import::{
Expand Down Expand Up @@ -98,7 +98,7 @@ pub async fn run(ctx: Context, args: ImportHdf5Args) -> Result<ExitCode> {

let file = File::open(&args.common.path).context("failed to open hdf5 file")?;

let create_data_import_req = create_data_import_request(hdf5_config);
let create_data_import_req = create_data_import_request(hdf5_config, &args.common);

let CreateDataImportFromUploadResponse { upload_url, .. } = data_imports_client
.create_data_import_from_upload(create_data_import_req)
Expand Down Expand Up @@ -154,9 +154,13 @@ pub fn build_hdf5_config(args: &ImportHdf5Args) -> Result<Hdf5Config> {
})
}

fn create_data_import_request(config: Hdf5Config) -> CreateDataImportFromUploadRequest {
pub(crate) fn create_data_import_request(
config: Hdf5Config,
common: &CommonImportArgs,
) -> CreateDataImportFromUploadRequest {
CreateDataImportFromUploadRequest {
hdf5_config: Some(config),
leave_run_open: common.leave_run_open,
..Default::default()
}
}
13 changes: 12 additions & 1 deletion rust/crates/sift_cli/src/cmd/import/hdf5/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,9 @@ use crate::cmd::import::utils::group_path_to_channel_name;
use chrono::DateTime;
use hdf5::types::{EnumMember, EnumType, FloatSize, IntSize, TypeDescriptor};
use sift_rs::common::r#type::v1::{ChannelConfig, ChannelDataType};
use sift_rs::data_imports::v2::TimeFormat as ProtoTimeFormat;
use sift_rs::data_imports::v2::{Hdf5Config, TimeFormat as ProtoTimeFormat};

use crate::cmd::import::hdf5::import::create_data_import_request;

fn make_args() -> ImportHdf5Args {
ImportHdf5Args {
Expand All @@ -23,6 +25,7 @@ fn make_args() -> ImportHdf5Args {
run_id: None,
wait: false,
preview: false,
leave_run_open: false,
Comment thread
alexluck-sift marked this conversation as resolved.
Resolved
},
schema: Hdf5Schema::OneD,
time_format: Some(TimeFormat::AbsoluteRfc3339),
Expand Down Expand Up @@ -462,3 +465,11 @@ fn weave_empty_returns_empty() {
let woven = weave_time_channel_rows(&[], &[]);
assert!(woven.is_empty());
}

#[test]
fn leave_run_open_is_set_on_the_request() {
let mut args = make_args();
args.common.leave_run_open = true;
let req = create_data_import_request(Hdf5Config::default(), &args.common);
assert!(req.leave_run_open);
}
10 changes: 7 additions & 3 deletions rust/crates/sift_cli/src/cmd/import/mcap/import.rs
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ use sift_rs::{
};

use crate::{
cli::{ImportMcapArgs, mcap::McapComplexTypesMode},
cli::{CommonImportArgs, ImportMcapArgs, mcap::McapComplexTypesMode},
cmd::{
Context,
import::{
Expand Down Expand Up @@ -72,7 +72,7 @@ pub async fn run(ctx: Context, args: ImportMcapArgs) -> Result<ExitCode> {

let file = File::open(&args.common.path).context("failed to open mcap file")?;

let create_data_import_req = create_data_import_request(mcap_config);
let create_data_import_req = create_data_import_request(mcap_config, &args.common);

let CreateDataImportFromUploadResponse { upload_url, .. } = data_imports_client
.create_data_import_from_upload(create_data_import_req)
Expand Down Expand Up @@ -179,9 +179,13 @@ pub fn channel_configs(
configs
}

fn create_data_import_request(config: McapConfig) -> CreateDataImportFromUploadRequest {
pub(crate) fn create_data_import_request(
config: McapConfig,
common: &CommonImportArgs,
) -> CreateDataImportFromUploadRequest {
CreateDataImportFromUploadRequest {
mcap_config: Some(config),
leave_run_open: common.leave_run_open,
..Default::default()
}
}
Loading
Loading