Repository navigation
feat(cli): --format csv, stdin streams via '-', and telecommand inspection - #7
mysma-9403 wants to merge 1 commit into
Conversation
…ction Telemetry decoding often feeds analysis pipelines, spreadsheets, and stream processors, where plain text table and JSON lines require extra parsing or do not stream into CSV consumers cleanly. xtce was also strictly file-bound, failing if piped input used the standard '-' convention. * `--format csv` outputs tabular parameter values per packet: `packet,container,parameter,eng` (and an optional `,raw` column with `--raw`). RFC 4180 escaping is applied when strings contain commas, double quotes, or newlines. * `--no-summary` suppresses the trailing timing and packet count lines, leaving stdout clean for piping directly into downstream tools (`xsv`, `jq`, Pandas). * Reading from standard input (`-`) is now supported across the CLI: `xtce decode` accepts `-` for either the packet stream or the definition, and `xtce info`, `xtce codegen` and `xtce bench` accept `-` for definition files. * `xtce info --verbose` now lists the telecommands section, showing each command, inheritance, backing container, and declared argument types. Tests added for CSV string escaping, float formatting, and tabular output. All 9 differential cases match the reference, fmt and clippy clean.
There was a problem hiding this comment.
🪶 Altanix Code Review & Security Gate
This PR introduces support for reading XTCE definitions and packet streams from standard input (stdin) across all CLI commands. It also adds a new CSV output format for packet decoding and improves the CLI reporting capabilities by including telecommands in the info command.
✨ Co wprowadza (Key Additions):
- Added CSV output format for packet decoding with support for raw values.
- Implemented stdin support for 'info', 'decode', 'codegen', and 'bench' commands.
- Added telecommand reporting to the 'info' command output.
- Added a summary toggle ('--no-summary') for the decode command.
🔄 Co zmienia / poprawia (Modifications & Fixes):
- Refactored file loading logic into a centralized 'load_db' helper function.
- Updated CLI argument descriptions to reflect stdin support.
- Improved error handling and output formatting for the decode command, including redirecting non-table output to stderr.
- Refactored 'run' function in main.rs to use dedicated runner functions for better maintainability.
Risk Score: 9/10 | Security Passed: ✅ YES
📁 Zmiany w plikach:
| Zmieniony Plik | Opis Zmiany |
|---|---|
crates/xtce-cli/src/decode.rs |
Implements the new CSV serialization logic, summary suppression, and stdin reading for packet streams. |
crates/xtce-cli/src/main.rs |
Contains the core logic for command routing, stdin handling, and the new centralized database loading mechanism. |
crates/xtce-cli/src/info.rs |
Adds telecommand reporting and updates source display logic for stdin. |
🧪 Wygenerowane Testy Jednostkowe & Regresyjne (AI Tester)
EDGE_CASE: DiffParser::parse (tests/test_diff_parser.rs)
Verifies that the parser does not panic on empty git diff input.
#[test]
fn test_empty_diff_handling() {
let files = DiffParser::parse("");
assert!(files.is_empty());
}
🛡️ Zweryfikowano autonomicznie przez Altanix — Architektoniczny Code Review & Live AppSec Gatekeeper.
✨ Chcesz chronić swoje repozytoria przed lukami CVE i wyścigami danych? Wypróbuj Altanix za darmo (14 dni) • 🚀 Odblokuj Plan Pro ($20/dev)
| fn load_stream(packets: &Path) -> std::io::Result<Vec<u8>> { | ||
| if packets.as_os_str() == "-" { | ||
| let mut bytes = Vec::new(); | ||
| std::io::stdin().read_to_end(&mut bytes)?; | ||
| Ok(bytes) | ||
| } else { | ||
| std::fs::read(packets) | ||
| } | ||
| } |
There was a problem hiding this comment.
Loading the entire packet stream into memory via read_to_end is a DATA_SCALE_RISK. If the input file or stdin stream is large, this will cause an OOM error. Consider processing the stream in chunks or using a buffered reader.
| fn load_stream(packets: &Path) -> std::io::Result<Vec<u8>> { | |
| if packets.as_os_str() == "-" { | |
| let mut bytes = Vec::new(); | |
| std::io::stdin().read_to_end(&mut bytes)?; | |
| Ok(bytes) | |
| } else { | |
| std::fs::read(packets) | |
| } | |
| } | |
| fn load_stream(packets: &Path) -> std::io::Result<Box<dyn std::io::Read>> { | |
| if packets.as_os_str() == "-" { | |
| Ok(Box::new(std::io::stdin())) | |
| } else { | |
| Ok(Box::new(std::fs::File::open(packets)?)) | |
| } | |
| } |
There was a problem hiding this comment.
🪶 Altanix Code Review & Security Gate
This PR introduces support for reading XTCE definitions and packet streams from standard input (stdin) across all CLI commands. Additionally, it adds a new CSV output format for the 'decode' command and improves the 'info' command by displaying telecommands.
✨ Key Changes & Capabilities:
- Added CSV output format to the 'decode' command with support for raw values.
- Added telecommand reporting to the 'info' command.
- Implemented stdin support for 'info', 'decode', 'codegen', and 'bench' commands.
🔄 Modifications & Fixes:
- Refactored CLI command execution into dedicated functions (run_info, run_codegen, run_bench) for better maintainability.
- Updated 'decode' command to allow suppressing the summary output.
- Improved error handling and input stream management for file-based operations.
- Added unit tests for CSV formatting, including string escaping and float handling.
Risk Score: 9/10 | Security: ✅ PASSED
📁 File Changes:
| Modified File | Description |
|---|---|
crates/xtce-cli/src/decode.rs |
Implements the new CSV output logic, including custom string escaping and float formatting, which are critical for data integrity. |
crates/xtce-cli/src/main.rs |
Contains the main CLI entry point changes and the refactoring of command execution logic to support stdin. |
crates/xtce-cli/src/info.rs |
Adds the new telecommand reporting feature to the info command. |
🧪 Generated Unit & Regression Tests (AI Tester)
EDGE_CASE: write_csv_string (crates/xtce-cli/src/decode.rs)
Verifies that CSV string escaping correctly handles commas and double quotes to prevent malformed CSV output.
#[test]
fn test_write_csv_string_escaping() {
let mut out = Vec::new();
// Test escaping of quotes and commas
super::write_csv_string(&mut out, "data, \"quoted\"").unwrap();
assert_eq!(String::from_utf8(out).unwrap(), "\"data, \"\"quoted\"\"\"");
let mut out = Vec::new();
// Test no escaping needed
super::write_csv_string(&mut out, "simple").unwrap();
assert_eq!(String::from_utf8(out).unwrap(), "simple");
}ERROR_HANDLING: run (crates/xtce-cli/src/decode.rs)
Ensures that the CLI correctly rejects configurations where both definition and packets are requested from stdin simultaneously.
#[test]
fn test_run_stdin_conflict_error() {
let options = super::Options {
definition: std::path::Path::new("-"),
packets: std::path::Path::new("-"),
root: None,
skip_header_bytes: 0,
limit: None,
only: &[],
format: super::Format::Csv,
show_raw: false,
no_summary: false,
};
let result = super::run(&options);
assert!(result.is_err());
assert_eq!(result.unwrap_err().to_string(), "cannot read both definition and packets from standard input");
}EDGE_CASE: write_csv_float (crates/xtce-cli/src/decode.rs)
Validates that floating point special values (NaN, Infinity) are serialized into the expected CSV format.
#[test]
fn test_write_csv_float_special_values() {
let mut out = Vec::new();
super::write_csv_float(&mut out, f64::NEG_INFINITY).unwrap();
assert_eq!(String::from_utf8(out).unwrap(), "-Infinity");
let mut out = Vec::new();
super::write_csv_float(&mut out, f64::NAN).unwrap();
assert_eq!(String::from_utf8(out).unwrap(), "NaN");
}🛡️ Autonomously verified by Altanix — Architectural Code Review & Live AppSec Gatekeeper.
✨ Want to protect your repositories from CVEs and data races? Try Altanix for free (14-day trial) • 🚀 Upgrade to Pro ($20/dev)
| fn load_stream(packets: &Path) -> std::io::Result<Vec<u8>> { | ||
| if packets.as_os_str() == "-" { | ||
| let mut bytes = Vec::new(); | ||
| std::io::stdin().read_to_end(&mut bytes)?; | ||
| Ok(bytes) | ||
| } else { | ||
| std::fs::read(packets) | ||
| } | ||
| } |
There was a problem hiding this comment.
Loading the entire packet stream into memory via read_to_end is a risk for large files. Consider using a buffered reader and processing packets in a streaming fashion to avoid OOM errors.
| fn load_stream(packets: &Path) -> std::io::Result<Vec<u8>> { | |
| if packets.as_os_str() == "-" { | |
| let mut bytes = Vec::new(); | |
| std::io::stdin().read_to_end(&mut bytes)?; | |
| Ok(bytes) | |
| } else { | |
| std::fs::read(packets) | |
| } | |
| } | |
| fn load_stream(packets: &Path) -> std::io::Result<Box<dyn std::io::Read>> { | |
| if packets.as_os_str() == "-" { | |
| Ok(Box::new(std::io::stdin())) | |
| } else { | |
| Ok(Box::new(std::fs::File::open(packets)?)) | |
| } | |
| } |
There was a problem hiding this comment.
🪶 Altanix Code Review & Security Gate
This PR enhances the
xtce-clitool by adding support for reading XTCE definitions and packet streams from standard input ('-'). It also introduces a CSV output format for decoded telemetry and adds support for displaying telecommand metadata in the 'info' command.
✨ Key Changes & Capabilities:
- Added CSV output format to 'xtce decode' with support for raw/engineering values.
- Added support for reading XTCE definitions and binary packet streams from stdin.
- Implemented telecommand metadata reporting in 'xtce info'.
- Added a '--no-summary' flag to suppress timing and packet count output in 'xtce decode'.
🔄 Modifications & Fixes:
- Refactored CLI command execution into separate functions (run_info, run_codegen, run_bench) for better maintainability.
- Updated 'xtce decode' to handle errors and non-table output formats by printing to stderr instead of stdout.
- Improved 'xtce info' to display telecommand structures including arguments and bit-widths.
- Standardized database loading logic to support both file paths and stdin.
Risk Score: 7/10 | Security: ✅ PASSED
📁 File Changes:
| Modified File | Description |
|---|---|
crates/xtce-cli/src/decode.rs |
Implements the new CSV output logic, stdin reading, and modifies the core decoding loop and summary reporting. |
crates/xtce-cli/src/main.rs |
Updates CLI argument parsing and refactors command execution logic to support stdin and new flags. |
crates/xtce-cli/src/info.rs |
Adds telecommand reporting functionality to the info command. |
🧪 Generated Unit & Regression Tests (AI Tester)
HAPPY_PATH: write_csv (crates/xtce-cli/src/decode.rs)
Verifies that the CSV formatting logic correctly concatenates fields and respects the expected CSV structure.
#[test]
fn test_write_csv_output_format() {
use super::{write_csv, Options, Format};
// Mocking minimal structure for DecodedPacket is complex,
// so we verify the logic of CSV row generation via the helper functions.
let mut out = Vec::new();
let container_name = "TelemetryContainer";
let param_name = "Voltage";
let eng_value = "12.5";
// Simulate row: index,container,parameter,eng
write!(out, "0,").unwrap();
super::write_csv_string(&mut out, container_name).unwrap();
write!(out, ",").unwrap();
super::write_csv_string(&mut out, param_name).unwrap();
write!(out, ",").unwrap();
write!(out, "{}", eng_value).unwrap();
let result = String::from_utf8(out).unwrap();
assert_eq!(result, "0,TelemetryContainer,Voltage,12.5");
}EDGE_CASE: write_csv_string (crates/xtce-cli/src/decode.rs)
Ensures that CSV string escaping correctly handles commas, double quotes, and newlines to prevent malformed CSV output.
#[test]
fn test_write_csv_string_escaping_edge_cases() {
let mut out = Vec::new();
// Test input containing all special characters that trigger quoting
super::write_csv_string(&mut out, "data,with\"quotes\"\nnewline").unwrap();
let result = String::from_utf8(out).unwrap();
// Expected: "data,with""quotes""
newline"
assert!(result.starts_with('"'));
assert!(result.ends_with('"'));
assert!(result.contains(""""));
}ERROR_HANDLING: run (crates/xtce-cli/src/decode.rs)
Verifies that the CLI correctly identifies and rejects the invalid configuration where both definition and packets are sourced from stdin.
#[test]
fn test_run_stdin_conflict_error() {
use super::{run, Options, Format};
use std::path::Path;
let options = Options {
definition: Path::new("-"),
packets: Path::new("-"),
only: &[],
format: Format::Table,
show_raw: false,
no_summary: false,
};
let result = run(&options);
assert!(result.is_err());
assert_eq!(result.unwrap_err().to_string(), "cannot read both definition and packets from standard input");
}🛡️ Autonomously verified by Altanix — Architectural Code Review & Live AppSec Gatekeeper.
✨ Want to protect your repositories from CVEs and data races? Try Altanix for free (14-day trial) • 🚀 Upgrade to Pro ($20/dev)
| if options.definition.as_os_str() == "-" && options.packets.as_os_str() == "-" { | ||
| return Err("cannot read both definition and packets from standard input".into()); | ||
| } |
There was a problem hiding this comment.
While this logic works for standard CLI usage, consider that if a file is actually named - on the filesystem, this will incorrectly trigger the error. Given the context of xtce-cli, this is likely acceptable, but ensure that Path::new("-") is the intended way to handle stdin across all platforms.
| } | ||
|
|
||
| let decode_start = Instant::now(); | ||
| for (index, framed) in PacketIter::new(&stream, options.skip_header_bytes).enumerate() { |
There was a problem hiding this comment.
The current implementation loads the entire packet stream into memory (Vec<u8>). While this is efficient for small-to-medium files, it poses a DATA_SCALE_RISK for massive telemetry dumps. Since the current architecture relies on &[u8] slices for zero-copy parsing, this is acceptable for now, but be aware that processing multi-gigabyte files will require a memory-mapped file (memmap2) approach rather than std::fs::read.
Telemetry decoding often feeds analysis pipelines, spreadsheets, and stream processors, where table and JSON lines require extra parsing or do not stream into tabular consumers cleanly.
xtcewas also strictly file-bound, failing if piped input used the standard-convention.Changes
--format csv: outputs tabular parameter values per packet in standard format:packet,container,parameter,eng(and an optional,rawcolumn with--raw).RFC 4180 escaping is applied when names or strings contain commas, double quotes, or newlines.
--no-summary: suppresses the trailing timing and packet count lines, leaving stdout clean for piping directly into downstream tools (xsv,jq, Pandas).-) support:xtce decodeaccepts-for either the packet stream or the definition.xtce info,xtce codegenandxtce benchaccept-for definition files.-for both definition and packets indecodereturns an error cleanly.xtce info --verbose: lists the telecommands section, showing each command, inheritance, backing container, and declared argument types.Verification
xtce-cli(including CSV string escaping, special float formatting, and tabular output).cargo xtask diff).cargo fmt --checkandcargo clippy --workspace --all-targets -- -D warningsclean.