Skip to content

data-import.md

Latest commit

 

History

History
185 lines (150 loc) · 8.28 KB

data-import.md

File metadata and controls

185 lines (150 loc) · 8.28 KB
Raw

Data Import

Import measurement data as DataCreationDTO into OpenSILEX via the AddList API (POST /core/data), driven by a CSV file and a YAML configuration. Data is sent in batches (default 5000) and a full report is produced for every row.

Overview

The data import script processes a CSV file to create OpenSILEX data points. For each row, it:

  1. Parses the CSV — auto-detects the header row and comment lines, applies the configured encoding
  2. Maps columns — resolves CSV columns to DataCreationDTO fields (date, value, variable, target, device)
  3. Resolves URIs — looks up variable, device and target URIs from the name → URI mappings (or uses URIs as-is)
  4. Builds DTOs — one DataCreationDTO per measurement (long format) or per measurement column (wide format)
  5. Sends in batches — POSTs batches to POST /core/data, re-sending failed batches row by row
  6. Writes a report — one report line per produced DTO, plus an errors file for failed/skipped rows

Usage

# download the example files (config + CSV, long and wide formats)
opensilex-py-download-data-config-example --dest ./data_examples

# import data
opensilex-py-run-data-import \
    --host http://localhost:8666/rest \
    --identifier admin@opensilex.org \
    --password secret \
    --csv data.csv \
    --config config.yaml \
    --verbose

Quick test with the wide-format fixtures downloaded by the command above (test_data_wide.csv + test_config_wide.yaml in the current directory):

opensilex-py-download-data-config-example
opensilex-py-run-data-import \
    --host http://localhost:8666/rest \
    --identifier admin@opensilex.org \
    --password admin \
    --csv test_data_wide.csv \
    --config test_config_wide.yaml \
    --verbose

CLI Arguments

opensilex-py-run-data-import

Argument Meaning Default value Notes
--host Base URL of the OpenSILEX REST API (must include /rest). http://localhost:8666/rest Also via OPENSILEX_HOST.
--identifier Email of the OpenSILEX account. admin@opensilex.org Also via OPENSILEX_IDENTIFIER.
--password Password of the OpenSILEX account. None (required) Also via OPENSILEX_PASSWORD. Avoid passing the password in clear text on the command line; prefer the environment variable.
--csv Path to the CSV input file. – Required.
--config Path to the YAML configuration file. – Required.
--verbose / -v Print detailed logs and tracebacks. False

opensilex-py-download-data-config-example

Argument Meaning Default value Notes
--dest Destination directory where example files will be copied. . (current directory) Directory is created automatically if it does not exist; can be relative or absolute path. 
# default – copies files into the current directory
opensilex-py-download-data-config-example

# specify a custom destination
opensilex-py-download-data-config-example --dest ./data_examples

opensilex-py-generate-data-config

Generate a starter YAML configuration from a CSV file. The header row and comment lines are auto-detected, column roles (date, value, variable, target, device) are guessed from the column names, and the distinct values found in the variable/target/device columns are pre-filled as mapping keys (the URIs are left empty for the user to fill in).

# long format (default) – 1 row = 1 measurement
opensilex-py-generate-data-config --csv data.csv

# wide format – 1 row = N measurements (one column per measurement)
opensilex-py-generate-data-config --csv data.csv --format wide

# explicit output path and header line
opensilex-py-generate-data-config --csv data.csv --config config.yaml --header-row 4

# overwrite an existing config without being prompted
opensilex-py-generate-data-config --csv data.csv --force
Argument Meaning Default value Notes
--csv Path to the CSV file to analyze. – Required – the file must exist and be readable.
--config Output YAML path. <csv_stem>_config.yaml next to the CSV
--format Data format: long (1 row = 1 measurement) or wide (1 row = N measurements). long
--header-row 1-indexed header line. auto-detected
--force Overwrite an existing config without confirmation. False Otherwise a prompt asks before overwriting.

YAML Configuration

The config maps CSV columns to the DataCreationDTO fields, including the nested provenance sub-object. Three name → existing URI mappings are provided (variable_mapping, device_mapping, target_mapping); a value that is not in the mapping but already looks like a URI is used as-is.

data:
  format: long            # long (1 line = 1 measurement) | wide (1 line = N measurements)
  batch_size: 5000
  csv:
    encoding: utf-8-sig
    header_row: 4         # 1-indexed header line (InfluxDB files start with '#' lines)
    skip_comment_rows: true
    metadata_rows:        # optional: extract global constants from lines above the header
      - { row: 1, column: 2, pattern: "Nom de la parcelle:{name}", assign_to: target }
  variable_mapping:
    soil_water_potential: "http://.../id/variable/soil_water_potential"
  device_mapping:
    WM_09: "https://.../device/WM_09"
  target_mapping:
    AVD: "opensilex-sandbox:id/scientific-object/so-vitadapt"
  fields:
    date: { column: "_time" }
    value: { column: "_value", type: auto }   # auto | float | int | bool | str
    variable: { column: "_field" }
    target: { column: "site" }
  provenance:
    uri: "https://.../provenance/WM_09_Tensiometer"   # constant
    experiments: ["http://.../experiment/crops-life-2026"]   # constant list
    prov_was_associated_with:
      - { column: "device" }        # per-row device -> device_mapping
      # - { value: "HSTI" }         # or a constant device
  # wide format only:
  # date_column: "Date et Heure"
  # measurements: { hum1: "soil_humidity_1", temp1: "soil_temperature_1" }
  • format: long – one CSV line produces one DataCreationDTO.
  • format: wide – one CSV line (a timestamp) is melted into N DataCreationDTO, one per measurements column (each with its own variable).
  • The device rdf_type is resolved from the OpenSILEX API (OntologyApi.get_uri_types, POST /ontology/uri_types) in batch; if it cannot be resolved it falls back to vocabulary:SensingDevice.
  • Rows that fail validation (missing value, unknown name, invalid date…) are not sent and are reported as skipped.

Report

A report CSV <csv>_data_report.csv is written next to the input file with one line per produced DTO: row (1-based data row, excluding header/comment lines), variable, status (success | failed | skipped), reason, error, and timestamp (ISO 8601 UTC time the import was run). When there are any failed or skipped rows, a second CSV <csv>_data_errors.csv containing only those rows (with the same timestamp) is written next to the input file and its path is printed to the console.

Failure isolation. When a whole batch is rejected, it is re-sent row by row so each failing row is identified individually. When a batch is only partially created, the already-created rows are not re-sent (to avoid duplicates): if the API returns an ordered list of created items, the remaining rows are re-sent row by row; if it returns an unordered count, the unconfirmed rows are reported as failed (not re-sent) so they can be re-imported safely. A timestamped log file is also written in the CSV directory.

Size limit. The CSV is fully loaded into memory (a pandas DataFrame) before processing. This is fine for typical InfluxDB exports (tens of thousands of rows) but for very large exports (millions of rows / several GB) consider splitting the file into smaller chunks.

Programmatic API

from opensilex_python_client.auth import connect
from opensilex_python_client.data import import_from_csv

client = connect.connect_to_opensilex({
    "host": "http://localhost:8666/rest",
    "identifier": "admin@opensilex.org",
    "password": "secret",
})

import_from_csv.run(client, "data.csv", "config.yaml", debug=True)