diff --git a/README.md b/README.md index 4b69453..eed05da 100644 --- a/README.md +++ b/README.md @@ -2,8 +2,8 @@ Command-line and Python client for downloading and deploying datasets on DBpedia Databus. - ## Table of Contents + - [Quickstart](#quickstart) - [Python](#python) - [Docker](#docker) @@ -19,25 +19,21 @@ Command-line and Python client for downloading and deploying datasets on DBpedia - [Deploy](#cli-deploy) - [Delete](#cli-delete) - [Manifest](#cli-manifest) - - [Replay](#cli-manifest-replay) - - [Summary](#cli-manifest-summary) - [Workflow](#cli-workflow) - [Module Usage](#module-usage) - - [Deploy](#module-deploy) - [Development & Contributing](#development--contributing) - [Linting](#linting) - [Testing](#testing) - ## Quickstart -The client supports two main workflows: downloading datasets from the Databus and deploying datasets to the Databus. Below you can choose how to run it (Python or Docker), then follow the sections on [DBpedia downloads](#dbpedia-knowledge-graphs), [CLI usage](#cli-usage), or [module usage](#module-usage). +The client supports two main workflows: downloading datasets from the Databus and deploying datasets to the Databus. You can use Python or Docker, then follow the sections on [DBpedia downloads](#dbpedia-knowledge-graphs), [CLI usage](#cli-usage), or [module usage](#module-usage). You can use either **Python** or **Docker**. Both methods support all client features. The Docker image is available at [dbpedia/databus-python-client](https://hub.docker.com/r/dbpedia/databus-python-client). ### Python -Requirements: [Python 3.11+](https://www.python.org/downloads/) and [pip](https://pip.pypa.io/en/stable/installation/) +Requirements: [Python 3.11+](https://www.python.org/downloads/) and [pip](https://pip.pypa.io/en/stable/installation/). Before using the client, install it via pip: @@ -55,21 +51,19 @@ You can then use the client in the command line: ```bash databusclient --help -databusclient deploy --help -databusclient delete --help -databusclient download --help +databusclient [delete|deploy|download|manifest|workflow] --help ``` ### Docker -Requirements: [Docker](https://docs.docker.com/get-docker/) +Requirements: [Docker](https://docs.docker.com/get-docker/). ```bash docker run --rm -v $(pwd):/data dbpedia/databus-python-client --help -docker run --rm -v $(pwd):/data dbpedia/databus-python-client deploy --help -docker run --rm -v $(pwd):/data dbpedia/databus-python-client download --help ``` +The same Docker invocation pattern can be used for the commands documented in [CLI Usage](doc/cli-usage.md). + ## DBpedia Commands to download the [DBpedia Knowledge Graphs](#dbpedia-knowledge-graphs) generated by Live Fusion. DBpedia Live Fusion publishes two kinds of KGs: @@ -81,831 +75,78 @@ Commands to download the [DBpedia Knowledge Graphs](#dbpedia-knowledge-graphs) g To download BUSL 1.1 licensed datasets, you need to register and get an access token. -1. If you do not have a DBpedia Account yet (Forum/Databus), please register at [https://account.dbpedia.org](https://account.dbpedia.org) +1. If you do not have a DBpedia Account yet (Forum/Databus), please register at [https://account.dbpedia.org](https://account.dbpedia.org). 2. Log in at [https://account.dbpedia.org](https://account.dbpedia.org) and create your token. 3. Save the token to a file, e.g. `vault-token.dat`. ### DBpedia Knowledge Graphs #### Download Live Fusion KG Dump (BUSL 1.1, registration needed) -High-frequency, conflict-resolved knowledge graph that merges Live Wikipedia and Wikidata signals into a single, queryable dump for enterprise consumption. [More information](https://databus.dbpedia.org/dbpedia-enterprise/live-fusion-kg-dump) + +High-frequency, conflict-resolved knowledge graph that merges Live Wikipedia and Wikidata signals into a single, queryable dump for enterprise consumption. [More information](https://databus.dbpedia.org/dbpedia-enterprise/live-fusion-kg-dump). + ```bash -# Python databusclient download https://databus.dbpedia.org/dbpedia-enterprise/live-fusion-kg-dump --vault-token vault-token.dat -# Docker -docker run --rm -v $(pwd):/data dbpedia/databus-python-client download https://databus.dbpedia.org/dbpedia-enterprise/live-fusion-kg-dump --vault-token vault-token.dat ``` #### Download Enriched Knowledge Graphs (BUSL 1.1, registration needed) **DBpedia Wikipedia Extraction Enriched** -DBpedia-based enrichment of structured Wikipedia extractions (currently EN DBpedia only). [More information](https://databus.dbpedia.org/dbpedia-enterprise/dbpedia-wikipedia-kg-enriched-dump) +DBpedia-based enrichment of structured Wikipedia extractions, currently EN DBpedia only. [More information](https://databus.dbpedia.org/dbpedia-enterprise/dbpedia-wikipedia-kg-enriched-dump). ```bash -# Python databusclient download https://databus.dbpedia.org/dbpedia-enterprise/dbpedia-wikipedia-kg-enriched-dump --vault-token vault-token.dat -# Docker -docker run --rm -v $(pwd):/data dbpedia/databus-python-client download https://databus.dbpedia.org/dbpedia-enterprise/dbpedia-wikipedia-kg-enriched-dump --vault-token vault-token.dat ``` #### Download DBpedia Wikipedia Knowledge Graphs (CC-BY-SA, no registration needed) -Original extraction of structured Wikipedia data before enrichment. [More information](https://databus.dbpedia.org/dbpedia/dbpedia-wikipedia-kg-dump) +Original extraction of structured Wikipedia data before enrichment. [More information](https://databus.dbpedia.org/dbpedia/dbpedia-wikipedia-kg-dump). ```bash -# Python databusclient download https://databus.dbpedia.org/dbpedia/dbpedia-wikipedia-kg-dump -# Docker -docker run --rm -v $(pwd):/data dbpedia/databus-python-client download https://databus.dbpedia.org/dbpedia/dbpedia-wikipedia-kg-dump ``` #### Download DBpedia Wikidata Knowledge Graphs (CC-BY-SA, no registration needed) -Original extraction of structured Wikidata data before enrichment. [More information](https://databus.dbpedia.org/dbpedia/dbpedia-wikidata-kg-dump) +Original extraction of structured Wikidata data before enrichment. [More information](https://databus.dbpedia.org/dbpedia/dbpedia-wikidata-kg-dump). ```bash -# Python databusclient download https://databus.dbpedia.org/dbpedia/dbpedia-wikidata-kg-dump -# Docker -docker run --rm -v $(pwd):/data dbpedia/databus-python-client download https://databus.dbpedia.org/dbpedia/dbpedia-wikidata-kg-dump ``` ## CLI Usage -To get started with the command-line interface (CLI) of the databus-python-client, you can use either the Python installation or the Docker image. The examples below show both methods. - -**Help and further general information:** - -```bash -# Python -databusclient --help -# Docker -docker run --rm -v $(pwd):/data dbpedia/databus-python-client --help - -# Output: -Usage: databusclient [OPTIONS] COMMAND [ARGS]... - - Databus Client CLI - -Options: - --help Show this message and exit. - -Commands: - delete Delete a dataset from the databus. - deploy Flexible deploy to Databus command supporting three modes: - download Download datasets from databus, optionally using vault access... - manifest Manifest utilities. -``` +The command-line interface provides commands for downloading, deploying, and deleting datasets, as well as recording manifests and running declarative workflows. Detailed command documentation, options, examples, manifest operations, and workflow syntax are available in [CLI Usage](doc/cli-usage.md). ### Download -With the download command, you can download datasets or parts thereof from the Databus. The download command expects one or more Databus URIs or a SPARQL query as arguments. The URIs can point to files, versions, artifacts, groups, or collections. If a SPARQL query is provided, the query must return download URLs from the Databus which will be downloaded. - -```bash -# Python -databusclient download $DOWNLOADTARGET -# Docker -docker run --rm -v $(pwd):/data dbpedia/databus-python-client download $DOWNLOADTARGET -``` - -- `$DOWNLOADTARGET` - - Can be any Databus URI including collections OR SPARQL query (or several thereof). -- `--localdir` - - If no `--localdir` is provided, the current working directory is used as base directory. The downloaded files will be stored in the working directory in a folder structure according to the Databus layout, i.e. `./$ACCOUNT/$GROUP/$ARTIFACT/$VERSION/`. -- `--vault-token` - - If the dataset/files to be downloaded require vault authentication, you need to provide a vault token with `--vault-token /path/to/vault-token.dat`. See [Registration (Access Token)](#registration-access-token) for details on how to get a vault token. - - Note: Vault tokens are only required for certain protected Databus hosts (for example: `data.dbpedia.io`, `data.dev.dbpedia.link`). The client now detects those hosts and will fail early with a clear message if a token is required but not provided. Do not pass `--vault-token` for public downloads. -- `--databus-key` - - If the databus is protected and needs API key authentication, you can provide the API key with `--databus-key YOUR_API_KEY`. -- `--compression` - - Enables on-the-fly compression format conversion during download. Supported formats: `bz2`, `gz`, `xz`, `none`. The source compression is auto-detected from the file extension. Use `none` to decompress files without recompressing. Example: `--compression gz` converts all downloaded compressed files to gzip format. -- `--format` - - Enables on-the-fly RDF and tabular format conversion during download (Layer 2 and Layer 3). Supported formats: `ntriples` (`nt`), `turtle` (`ttl`), `rdf-xml` (`rdf`, `xml`), `nquads` (`nq`), `trig`, `trix`, `json-ld` (`jsonld`), `csv`, `tsv`. Short aliases shown in brackets. Only the converted output file is kept — the original is deleted after successful conversion. Within the same equivalence class (e.g. turtle to ntriples) conversion is lossless. Across classes (e.g. RDF to CSV) some flags below may be required. -- `--graph-name` - - Required when converting RDF triples to a quad format (e.g. turtle to nquads). Assigns all triples to the specified named graph URI. Example: `--format nquads --graph-name https://example.org/mygraph`. -- `--base-uri` - - Required when converting CSV/TSV to RDF triples. Used as the base for constructing subject URIs from CSV row identifiers. Example: `--format ntriples --base-uri https://example.org/data/`. -- `--validate-checksum` - - Validates the checksums of downloaded files against the checksums provided by the Databus. If a checksum does not match, an error is raised and the file is deleted. - -**Help and further information on download command:** -```bash -# Python -databusclient download --help -# Docker -docker run --rm -v $(pwd):/data dbpedia/databus-python-client download --help - -# Output: -Usage: databusclient download [OPTIONS] DATABUSURIS... - - Download datasets from databus, optionally using vault access if vault - options are provided. Supports on-the-fly compression format conversion - using --convert-to and --convert-from options. - -Options: - --localdir TEXT Local databus folder (if not given, databus - folder structure is created in current working - directory) - --databus TEXT Databus URL (if not given, inferred from - databusuri, e.g. - https://databus.dbpedia.org/sparql) - --vault-token TEXT Path to Vault refresh token file - --databus-key TEXT Databus API key to download from protected - databus - --all-versions When downloading artifacts, download all - versions instead of only the latest - --authurl TEXT Keycloak token endpoint URL [default: https://a - uth.dbpedia.org/realms/dbpedia/protocol/openid- - connect/token] - --clientid TEXT Client ID for token exchange [default: vault- - token-exchange] - --convert-to [bz2|gz|xz] Target compression format for on-the-fly - conversion during download (supported: bz2, gz, - xz) - --convert-from [bz2|gz|xz] Source compression format to convert from - (optional filter). Only files with this - compression will be converted. - --validate-checksum Validate checksums of downloaded files - --help Show this message and exit. -``` - -#### Examples of using the download command - -**Download File**: download of a single file -```bash -# Python -databusclient download https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01/mappingbased-literals_lang=az.ttl.bz2 -# Docker -docker run --rm -v $(pwd):/data dbpedia/databus-python-client download https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01/mappingbased-literals_lang=az.ttl.bz2 -``` - -**Download Version**: download of all files of a specific version -```bash -# Python -databusclient download https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01 -# Docker -docker run --rm -v $(pwd):/data dbpedia/databus-python-client download https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01 -``` - -**Download Artifact**: download of all files with the latest version of an artifact -```bash -# Python -databusclient download https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals -# Docker -docker run --rm -v $(pwd):/data dbpedia/databus-python-client download https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals -``` - -**Download Group**: download of all files with the latest version of all artifacts of a group -```bash -# Python -databusclient download https://databus.dbpedia.org/dbpedia/mappings -# Docker -docker run --rm -v $(pwd):/data dbpedia/databus-python-client download https://databus.dbpedia.org/dbpedia/mappings -``` - -**Download Collection**: download of all files within a collection -```bash -# Python -databusclient download https://databus.dbpedia.org/dbpedia/collections/dbpedia-snapshot-2022-12 -# Docker -docker run --rm -v $(pwd):/data dbpedia/databus-python-client download https://databus.dbpedia.org/dbpedia/collections/dbpedia-snapshot-2022-12 -``` - -**Download Query**: download of all files returned by a query (SPARQL endpoint must be provided with `--databus`) -```bash -# Python -databusclient download 'PREFIX dcat: SELECT ?x WHERE { ?sub dcat:downloadURL ?x . } LIMIT 10' --databus https://databus.dbpedia.org/sparql -# Docker -docker run --rm -v $(pwd):/data dbpedia/databus-python-client download 'PREFIX dcat: SELECT ?x WHERE { ?sub dcat:downloadURL ?x . } LIMIT 10' --databus https://databus.dbpedia.org/sparql -``` - -**Download with Compression Conversion**: download files and convert compression format on-the-fly. Source compression is auto-detected from the file extension. -```bash -# Convert all compressed files to gzip format -databusclient download https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01 --compression gz - -# Download a collection and unify all files to bz2 format -databusclient download https://databus.dbpedia.org/dbpedia/collections/dbpedia-snapshot-2022-12 --compression bz2 - -# Decompress files without recompressing -databusclient download https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01/mappingbased-literals_lang=az.ttl.bz2 --compression none -``` - -**Download with Format Conversion**: download files and convert RDF or tabular format on-the-fly. Only the converted output file is kept. -```bash -# Convert RDF/XML to Turtle -databusclient download https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01/mappingbased-literals_lang=az.ttl.bz2 --format turtle - -# Convert N-Quads to TriG (within quad equivalence class) -databusclient download https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01 --format trig - -# Convert RDF to CSV (cross-class, produces companion .meta.json) -databusclient download https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01/mappingbased-literals_lang=az.ttl.bz2 --format csv - -# Combine format conversion and compression -databusclient download https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01/mappingbased-literals_lang=az.ttl.bz2 --format ntriples --compression gz -``` - -**Download with Mapping Conversion (Layer 3)**: convert across format classes — between RDF triples, RDF quads, and tabular data. -```bash -# RDF Triples -> RDF Quads (requires --graph-name) -databusclient download https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01/mappingbased-literals_lang=az.ttl.bz2 --format nquads --graph-name https://example.org/mygraph - -# RDF Quads -> RDF Triples (splits into one file per named graph, in a subdirectory) -databusclient download https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01/mappingbased-literals_lang=az.nq --format turtle - -# RDF Triples -> CSV (produces a companion .meta.json preserving datatypes/language tags) -databusclient download https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01/mappingbased-literals_lang=az.ttl.bz2 --format csv - -# CSV -> RDF Triples (requires --base-uri; lossless if companion .meta.json is present) -databusclient download https://databus.dbpedia.org/dbpedia/some-tabular-dataset/2022.12.01/data.csv --format ntriples --base-uri https://example.org/data/ - -# RDF Quads -> CSV (adds a 'graph' column) -databusclient download https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01/mappingbased-literals_lang=az.nq --format csv -``` +The `download` command retrieves Databus files, versions, artifacts, groups, collections, or SPARQL query results. It supports authentication, checksum validation, compression conversion, and RDF or tabular format conversion. See the [download docs](doc/cli-usage.md#cli-download). ### Deploy -With the deploy command, you can deploy datasets to the Databus. The deploy command supports three modes: -1. Classic dataset deployment via list of distributions -2. Metadata-based deployment via metadata JSON file -3. Upload & deploy via Nextcloud/WebDAV - -```bash -# Python -databusclient deploy [OPTIONS] [DISTRIBUTIONS]... -# Docker -docker run --rm -v $(pwd):/data dbpedia/databus-python-client deploy [OPTIONS] [DISTRIBUTIONS]... -``` - -**Help and further information on deploy command:** -```bash -# Python -databusclient deploy --help -# Docker -docker run --rm -v $(pwd):/data dbpedia/databus-python-client deploy --help - -# Output: -Usage: databusclient deploy [OPTIONS] [DISTRIBUTIONS]... - - Flexible deploy to Databus command supporting three modes: - - - Classic deploy (distributions as arguments) - - - Metadata-based deploy (--metadata ) - - - Upload & deploy via Nextcloud (--webdav-url, --remote, --path) - -Options: - --version-id TEXT Target databus version/dataset identifier of the form [required] - --title TEXT Artifact & Version Title: used for BOTH artifact and - version. Keep stable across releases; identifies the - data series. [required] - --abstract TEXT Artifact & Version Abstract: used for BOTH artifact and - version (max 200 chars). Updating it changes both - artifact and version metadata. [required] - --description TEXT Artifact & Version Description: used for BOTH artifact - and version. Supports Markdown. Updating it changes both - artifact and version metadata. [required] - --license TEXT License (see dalicc.net) [required] - --apikey TEXT API key [required] - --metadata PATH Path to metadata JSON file (for metadata mode) - --webdav-url TEXT WebDAV URL (e.g., - https://cloud.example.com/remote.php/webdav) - --remote TEXT rclone remote name (e.g., 'nextcloud') - --path TEXT Remote path on Nextcloud (e.g., 'datasets/mydataset') - --help Show this message and exit. -``` - -### Mode 1: Classic Deploy (Distributions) - -```bash -# Python -databusclient deploy \ ---version-id https://databus.dbpedia.org/user1/group1/artifact1/2022-05-18 \ ---title "Client Testing" \ ---abstract "Testing the client...." \ ---description "Testing the client...." \ ---license http://dalicc.net/licenselibrary/AdaptivePublicLicense10 \ ---apikey MYSTERIOUS \ -'https://raw.githubusercontent.com/dbpedia/databus/master/server/app/api/swagger.yml|type=swagger' -# Docker -docker run --rm -v $(pwd):/data dbpedia/databus-python-client deploy \ ---version-id https://databus.dbpedia.org/user1/group1/artifact1/2022-05-18 \ ---title "Client Testing" \ ---abstract "Testing the client...." \ ---description "Testing the client...." \ ---license http://dalicc.net/licenselibrary/AdaptivePublicLicense10 \ ---apikey MYSTERIOUS \ -'https://raw.githubusercontent.com/dbpedia/databus/master/server/app/api/swagger.yml|type=swagger -``` -A few more notes for CLI usage: - -- The content variants can be left out ONLY IF there is just one distribution - - For complete inferred: Just use the URL with `https://raw.githubusercontent.com/dbpedia/databus/master/server/app/api/swagger.yml` - - If other parameters are used, you need to leave them empty like `https://raw.githubusercontent.com/dbpedia/databus/master/server/app/api/swagger.yml||yml|7a751b6dd5eb8d73d97793c3c564c71ab7b565fa4ba619e4a8fd05a6f80ff653:367116` - - -### Mode 2: Deploy with Metadata File - -Use a JSON metadata file to define all distributions. -The metadata.json should list all distributions and their metadata. -All files referenced there will be registered on the Databus. -```bash -# Python -databusclient deploy \ - --metadata ./metadata.json \ - --version-id https://databus.dbpedia.org/user1/group1/artifact1/1.0 \ - --title "Metadata Deploy Example" \ - --abstract "This is a short abstract of the dataset." \ - --description "This dataset was uploaded using metadata.json." \ - --license https://dalicc.net/licenselibrary/Apache-2.0 \ - --apikey "API-KEY" -# Docker -docker run --rm -v $(pwd):/data dbpedia/databus-python-client deploy \ - --metadata ./metadata.json \ - --version-id https://databus.dbpedia.org/user1/group1/artifact1/1.0 \ - --title "Metadata Deploy Example" \ - --abstract "This is a short abstract of the dataset." \ - --description "This dataset was uploaded using metadata.json." \ - --license https://dalicc.net/licenselibrary/Apache-2.0 \ - --apikey "API-KEY" -``` -Example `metadata.json` metadata file structure (`file_format` and `compression` are optional): -```json -[ - { - "checksum": "0929436d44bba110fc7578c138ed770ae9f548e195d19c2f00d813cca24b9f39", - "size": 12345, - "url": "https://cloud.example.com/remote.php/webdav/datasets/mydataset/example.ttl", - "file_format": "ttl" - }, - { - "checksum": "2238acdd7cf6bc8d9c9963a9f6014051c754bf8a04aacc5cb10448e2da72c537", - "size": 54321, - "url": "https://cloud.example.com/remote.php/webdav/datasets/mydataset/example.csv.gz", - "file_format": "csv", - "compression": "gz" - } -] -``` - -### Mode 3: Upload & Deploy via Nextcloud - -Upload local files or folders to a WebDAV/Nextcloud instance and automatically deploy to DBpedia Databus. [Rclone](https://rclone.org/) is required. - -```bash -# Python -databusclient deploy \ - --webdav-url https://cloud.example.com/remote.php/webdav \ - --remote nextcloud \ - --path datasets/mydataset \ - --version-id https://databus.dbpedia.org/user1/group1/artifact1/1.0 \ - --title "Test Dataset" \ - --abstract "Short abstract of dataset" \ - --description "This dataset was uploaded for testing the Nextcloud → Databus pipeline." \ - --license https://dalicc.net/licenselibrary/Apache-2.0 \ - --apikey "API-KEY" \ - ./localfile1.ttl \ - ./data_folder -# Docker -docker run --rm -v $(pwd):/data dbpedia/databus-python-client deploy \ - --webdav-url https://cloud.example.com/remote.php/webdav \ - --remote nextcloud \ - --path datasets/mydataset \ - --version-id https://databus.dbpedia.org/user1/group1/artifact1/1.0 \ - --title "Test Dataset" \ - --abstract "Short abstract of dataset" \ - --description "This dataset was uploaded for testing the Nextcloud → Databus pipeline." \ - --license https://dalicc.net/licenselibrary/Apache-2.0 \ - --apikey "API-KEY" \ - ./localfile1.ttl \ - ./data_folder -``` +The `deploy` command publishes datasets using distribution arguments, metadata JSON files, or WebDAV/Nextcloud uploads. See the [deploy docs](doc/cli-usage.md#cli-deploy). ### Delete -With the delete command you can delete collections, groups, artifacts, and versions from the Databus. Deleting files is not supported via API. - -**Note**: Deleting datasets will recursively delete all data associated with the dataset below the specified level. Please use this command with caution. As security measure, the delete command will prompt you for confirmation before proceeding with any deletion. - -```bash -# Python -databusclient delete [OPTIONS] DATABUSURIS... -# Docker -docker run --rm -v $(pwd):/data dbpedia/databus-python-client delete [OPTIONS] DATABUSURIS... -``` - -**Help and further information on delete command:** -```bash -# Python -databusclient delete --help -# Docker -docker run --rm -v $(pwd):/data dbpedia/databus-python-client delete --help - -# Output: -Usage: databusclient delete [OPTIONS] DATABUSURIS... - - Delete a dataset from the databus. - - Delete a group, artifact, or version identified by the given databus URI. - Will recursively delete all data associated with the dataset. - -Options: - --databus-key TEXT Databus API key to access protected databus [required] - --dry-run Perform a dry run without actual deletion - --force Force deletion without confirmation prompt - --help Show this message and exit. -``` - -To authenticate the delete request, you need to provide an API key with `--databus-key YOUR_API_KEY`. - -If you want to perform a dry run without actual deletion, use the `--dry-run` option. This will show you what would be deleted without making any changes. - -As security measure, the delete command will prompt you for confirmation before proceeding with the deletion. If you want to skip this prompt, you can use the `--force` option. - -#### Examples of using the delete command - -**Delete Version**: delete a specific version -```bash -# Python -databusclient delete https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01 --databus-key YOUR_API_KEY -# Docker -docker run --rm -v $(pwd):/data dbpedia/databus-python-client delete https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01 --databus-key YOUR_API_KEY -``` - -**Delete Artifact**: delete an artifact and all its versions -```bash -# Python -databusclient delete https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals --databus-key YOUR_API_KEY -# Docker -docker run --rm -v $(pwd):/data dbpedia/databus-python-client delete https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals --databus-key YOUR_API_KEY -``` - -**Delete Group**: delete a group and all its artifacts and versions -```bash -# Python -databusclient delete https://databus.dbpedia.org/dbpedia/mappings --databus-key YOUR_API_KEY -# Docker -docker run --rm -v $(pwd):/data dbpedia/databus-python-client delete https://databus.dbpedia.org/dbpedia/mappings --databus-key YOUR_API_KEY -``` - -**Delete Collection**: delete collection -```bash -# Python -databusclient delete https://databus.dbpedia.org/dbpedia/collections/dbpedia-snapshot-2022-12 --databus-key YOUR_API_KEY -# Docker -docker run --rm -v $(pwd):/data dbpedia/databus-python-client delete https://databus.dbpedia.org/dbpedia/collections/dbpedia-snapshot-2022-12 --databus-key YOUR_API_KEY -``` +The `delete` command removes Databus versions, artifacts, groups, or collections and provides dry-run and confirmation safeguards. See the [delete docs](doc/cli-usage.md#cli-delete). ### Manifest -All three commands support an optional `--manifest` flag that writes a structured JSON-LD record of the operation to disk: - -**Download** -```bash -# Python -databusclient download https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01/mappingbased-literals_lang=az.ttl.bz2 --manifest ./manifests/download-run.jsonld -# Docker -docker run --rm -v $(pwd):/data dbpedia/databus-python-client download https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01/mappingbased-literals_lang=az.ttl.bz2 --manifest ./manifests/download-run.jsonld -``` - -**Deploy** -```bash -# Python -databusclient deploy \ - --version-id https://databus.dbpedia.org/user1/group1/artifact1/2022-05-18 \ - --title "Client Testing" --abstract "Testing the client...." \ - --description "Testing the client...." \ - --license http://dalicc.net/licenselibrary/AdaptivePublicLicense10 \ - --apikey YOUR_KEY --manifest ./manifests/deploy-run.jsonld \ - 'https://raw.githubusercontent.com/dbpedia/databus/master/server/app/api/swagger.yml|type=swagger' -# Docker -docker run --rm -v $(pwd):/data dbpedia/databus-python-client deploy \ - --version-id https://databus.dbpedia.org/user1/group1/artifact1/2022-05-18 \ - --title "Client Testing" --abstract "Testing the client...." \ - --description "Testing the client...." \ - --license http://dalicc.net/licenselibrary/AdaptivePublicLicense10 \ - --apikey YOUR_KEY --manifest ./manifests/deploy-run.jsonld \ - 'https://raw.githubusercontent.com/dbpedia/databus/master/server/app/api/swagger.yml|type=swagger' -``` -**Delete** -```bash -# Python -databusclient delete https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01 --databus-key YOUR_API_KEY --manifest ./manifests/delete-run.jsonld -# Docker -docker run --rm -v $(pwd):/data dbpedia/databus-python-client delete https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01 --databus-key YOUR_API_KEY --manifest ./manifests/delete-run.jsonld -``` - -The manifest records input parameters, per-file URLs, checksums, byte sizes, timestamps, and success/failure status for each file. It uses the DataID vocabulary and is versioned via `dbus:schemaVersion`. - -- If the target path already exists, the manifest is written to an auto-suffixed path (e.g. `run_1.jsonld`) with a warning. -- Sensitive fields (API keys, vault tokens) are never written. -- If manifest writing fails, a warning is printed and the exit code reflects the actual operation result. -- If the operation itself fails, a `dbus:operationError` block is recorded in the manifest capturing the error type, message, and traceback. - -Refer [examples/reproducible-download.md](examples/reproducible-download.md) for a full walkthrough. - - -#### Replay - -Any manifest written with `--manifest` can be replayed later using `databusclient manifest replay `. Replay re-executes the original operation using the parameters recorded in the manifest — you don't need to remember or retype the original command. - -```bash -# Python -databusclient manifest replay [OPTIONS] MANIFEST_PATH -# Docker -docker run --rm -v $(pwd):/data dbpedia/databus-python-client manifest replay [OPTIONS] MANIFEST_PATH -``` - -**Important:** credentials are never stored in the manifest and must always be supplied fresh at replay time — `--vault-token`, `--databus-key`, and `--apikey` behave exactly as they do on the original commands. - -```bash -databusclient manifest replay --help - -# Output: -Usage: databusclient manifest replay [OPTIONS] MANIFEST_PATH - - Replay a previously recorded manifest operation. - - Currently supports replay of download, delete, and deploy manifests. - For delete manifests, an interactive confirmation is required by - default -- use --force to skip it for scripted/unattended use, or - --dry-run to preview without prompting or deleting. - -Options: - --localdir TEXT Override local output directory for download replay. - --databus TEXT Override Databus endpoint for replay. - --vault-token TEXT Vault token file path, required if manifest auth - method is vault_token. - --databus-key TEXT Databus API key, required if manifest auth method is - databus_key. Also required for delete replay. - --apikey TEXT Databus API key, required for deploy replay. - --force For delete replay: skip the interactive confirmation - prompt. Required for unattended/scripted replay. - --dry-run For delete replay: force a dry-run preview even if - the original operation wasn't one. - --help Show this message and exit. -``` - -**Replaying a download:** -```bash -databusclient manifest replay ./manifests/download-run.jsonld --localdir ./replayed-data -``` -If `--localdir` is omitted, replay falls back to the same auto-computed folder structure a fresh download would use — this is not necessarily the same folder the original download used, since the original folder location itself is never stored in the manifest. - -**Replaying a delete:** by default, replay asks for confirmation before deleting, exactly like a normal `delete` call: -```bash -databusclient manifest replay ./manifests/delete-run.jsonld --databus-key YOUR_API_KEY -# About to replay a DELETE operation for the following 1 URI(s): -# - https://databus.dbpedia.org/... -# This is irreversible. Proceed? [y/N]: -``` -For unattended/scripted use (e.g. CI/CD), skip the prompt with `--force`: -```bash -databusclient manifest replay ./manifests/delete-run.jsonld --databus-key YOUR_API_KEY --force -``` -If the original delete was run with `--dry-run --manifest ...`, replay automatically previews without deleting — no flag needed. You can also force a preview on a manifest that wasn't originally a dry run: -```bash -databusclient manifest replay ./manifests/delete-run.jsonld --databus-key YOUR_API_KEY --dry-run -``` - -**Replaying a deploy:** supported for classic (distributions-as-arguments) and metadata-file deploys. The manifest stores fully-resolved deployment metadata (checksums, sizes, formats already computed), so replay never re-downloads or re-hashes the original files: -```bash -databusclient manifest replay ./manifests/deploy-run.jsonld --apikey YOUR_API_KEY -``` -Replaying redeploys the same version — if it already exists on Databus, it is updated. WebDAV/Nextcloud deploys cannot be replayed, since the originally uploaded local files may no longer exist at their original paths by the time replay runs. - - -#### Summary - -Print a readable summary of any recorded manifest without replaying it: - -```bash -# Python -databusclient manifest summary ./manifests/download-run.jsonld -# Docker -docker run --rm -v $(pwd):/data dbpedia/databus-python-client manifest summary ./manifests/download-run.jsonld -``` - -Example output: - -``` -Command : download -Executed : 2024-03-24T10:02:49.500418+00:00 -Endpoint : https://databus.dbpedia.org/sparql -Auth : vault_token -Files : 1 succeeded · 0 failed -Total : 100.0 MB -Status : completed -``` - -Only existing data already stored in the manifest is read — no new files are downloaded or written, and no network access happens. +The manifest options record operation parameters, file outcomes, checksums, byte sizes, and execution summaries in JSON-LD. Manifests can also be replayed or summarized. See the [manifest docs](doc/cli-usage.md#cli-manifest). ### Workflow -The workflow command runs a multi-step pipeline of `download`, `deploy`, and `delete` operations defined in a YAML file. Steps run in order, and a later step can use the output of an earlier step — for example, deploying the exact file a previous step just downloaded. - -```bash -# Python -databusclient workflow run [OPTIONS] WORKFLOW_PATH -# Docker -docker run --rm -v $(pwd):/data dbpedia/databus-python-client workflow run [OPTIONS] WORKFLOW_PATH -``` - -**Help and further information on the workflow command:** -```bash -# Python -databusclient workflow run --help -# Docker -docker run --rm -v $(pwd):/data dbpedia/databus-python-client workflow run --help - -# Output: -Usage: databusclient workflow run [OPTIONS] WORKFLOW_PATH - - Run a declarative workflow pipeline from a YAML file. - - Executes each step in order, chaining outputs between steps via - ${steps.name.output_files}-style references, and applying each step's - on_error behavior (fail/continue/retry). - -Options: - --help Show this message and exit. -``` - -#### Workflow YAML format - -A workflow file has a top-level `steps:` list. Each step needs a unique `name` and a `command` (`download`, `deploy`, or `delete`), plus fields specific to that command. - -```yaml -steps: - - name: fetch_dataset - command: download - uri: https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01/mappingbased-literals_lang=az.ttl.bz2 - localdir: ./data - - - name: publish_dataset - command: deploy - version_id: https://databus.dbpedia.org/myaccount/research/labels/2024.01 - title: "Processed Labels" - abstract: "Processed from DBpedia 2023.12.01" - description: "Converted and redeployed labels dataset" - license: https://creativecommons.org/licenses/by-sa/3.0/ - api_key: ${DATABUS_API_KEY} - files: ${steps.fetch_dataset.output_urls} - on_error: fail -``` - -**Environment variables:** any value written as `${VARIABLE_NAME}` is resolved from the environment when the workflow starts. If the variable is not set, the workflow fails immediately with a clear error before any step runs — credentials should always be passed this way, never written directly in the file. - -**Step chaining:** a step's outputs can be referenced by later steps using `${steps.step_name.output_key}`: -- `${steps.name.output_files}` — local file paths produced by a `download` step. -- `${steps.name.output_urls}` — the actual, redirect-resolved source URL(s) the file was downloaded from, useful for redeploying an unmodified file via classic deploy mode. - -#### Deploy step modes within a workflow - -A `deploy` step supports the same modes as the `deploy` CLI command: - -- **Classic mode** (`files:` is a list of URLs) — use `${steps.name.output_urls}` to redeploy a file exactly as it was downloaded, unmodified. Classic mode does not accept local file paths; if `files:` contains anything other than a `http://`/`https://` URL, the step fails with a clear error rather than crashing. -- **WebDAV mode** (`webdav_url:`, `remote:`, `path:` all provided) — use `${steps.name.output_files}` (local paths) here. The step uploads the local files to the WebDAV server first, then deploys the resulting URLs. This is the only way to deploy a file that was locally modified during the workflow (e.g. via `--format`/`--compression` on the download step), since only WebDAV mode re-establishes a real, fetchable URL for locally changed content. - -```yaml - - name: publish_converted_dataset - command: deploy - version_id: https://databus.dbpedia.org/myaccount/research/labels/2024.01 - title: "Processed Labels" - abstract: "Processed from DBpedia 2023.12.01" - description: "Converted and redeployed labels dataset" - license: https://creativecommons.org/licenses/by-sa/3.0/ - api_key: ${DATABUS_API_KEY} - webdav_url: https://cloud.example.com/remote.php/webdav - remote: nextcloud - path: datasets/mydataset - files: ${steps.fetch_dataset.output_files} -``` - -#### Error handling - -Each step declares an `on_error` behavior (defaults to `fail` if not set): - -| Mode | Behavior | -|---|---| -| `fail` | Stop the entire workflow immediately if this step fails. | -| `continue` | Log the failure and move on to the next step anyway. | -| `retry` | Retry the step up to `max_attempts` times, waiting `delay_seconds` between attempts. If all attempts fail, the workflow stops. | - -```yaml - - name: fetch_dataset - command: download - uri: https://databus.dbpedia.org/... - on_error: retry - retry: - max_attempts: 3 - delay_seconds: 5 -``` - -A retry re-runs the entire step from scratch, not just the part that failed. - -**Delete steps never prompt for confirmation inside a workflow** — since workflows are meant to run unattended, a `delete` step always behaves as if `--force` was passed. - -#### Examples - -Full working example files are available under [`examples/workflows/`](examples/workflows/): -- `download-deploy.yml` — download a file, then redeploy it (classic mode). -- `download-delete.yml` — download a file, then delete an old version. -- `full-pipeline.yml` — download, deploy, and delete chained together in one run. - -```bash -databusclient workflow run examples/workflows/download-deploy.yml -``` +The `workflow` command runs declarative download, deploy, and delete pipelines from YAML files, with step chaining and per-step error handling. See the [workflow docs](doc/cli-usage.md#cli-workflow). ## Module Usage - -### Deploy - -#### Step 1: Create lists of distributions for the dataset - -```python -from databusclient import create_distribution - -# create a list -distributions = [] - -# minimal requirements -# compression and filetype will be inferred from the path -# this will trigger the download of the file to evaluate the shasum and content length -distributions.append( - create_distribution(url="https://raw.githubusercontent.com/dbpedia/databus/master/server/app/api/swagger.yml", cvs={"type": "swagger"}) -) - -# full parameters -# will just place parameters correctly, nothing will be downloaded or inferred -distributions.append( - create_distribution( - url="https://example.org/some/random/file.csv.bz2", - cvs={"type": "example", "realfile": "false"}, - file_format="csv", - compression="bz2", - sha256_length_tuple=("7a751b6dd5eb8d73d97793c3c564c71ab7b565fa4ba619e4a8fd05a6f80ff653", 367116) - ) -) -``` - -A few notes: - -* The dict for content variants can be empty ONLY IF there is just one distribution -* There can be no compression if there is no file format - -#### Step 2: Create dataset - -```python -from databusclient import create_dataset - -# minimal way -dataset = create_dataset( - version_id="https://dev.databus.dbpedia.org/denis/group1/artifact1/2022-05-18", - title="Client Testing", - abstract="Testing the client....", - description="Testing the client....", - license_url="http://dalicc.net/licenselibrary/AdaptivePublicLicense10", - distributions=distributions, -) - -# with group metadata -dataset = create_dataset( - version_id="https://dev.databus.dbpedia.org/denis/group1/artifact1/2022-05-18", - title="Client Testing", - abstract="Testing the client....", - description="Testing the client....", - license_url="http://dalicc.net/licenselibrary/AdaptivePublicLicense10", - distributions=distributions, - group_title="Title of group1", - group_abstract="Abstract of group1", - group_description="Description of group1" -) -``` - -NOTE: Group metadata is applied only if all group parameters are set. - -#### Step 3: Deploy to Databus - -```python -from databusclient import deploy - -# to deploy something you just need the dataset from the previous step and an API key -# API key can be found (or generated) at https://$$DATABUS_BASE$$/$$USER$$#settings -deploy(dataset, "mysterious API key") -``` +The Python API exposes helpers for creating distributions and datasets and for deploying them programmatically. See the [module usage docs](doc/module-usage.md). ## Development & Contributing diff --git a/databusclient/cli.py b/databusclient/cli.py index ca3e71f..d96d325 100644 --- a/databusclient/cli.py +++ b/databusclient/cli.py @@ -333,7 +333,7 @@ def download( ): """ Download datasets from databus, optionally using vault access if vault options are provided. - Supports on-the-fly compression format conversion using --convert-to and --convert-from options. + Supports on-the-fly compression format conversion using the --compression option. """ # Determine auth method for manifest (never store the token itself) auth_method = None diff --git a/doc/README.md b/doc/README.md new file mode 100644 index 0000000..101768f --- /dev/null +++ b/doc/README.md @@ -0,0 +1,7 @@ +# Documentation + +- [CLI Usage](cli-usage.md) - Complete command-line documentation for download, deploy, delete, manifests, and workflows. +- [Module Usage](module-usage.md) - Python API examples for creating distributions, datasets, and deployments. +- [Reproducible Download](examples/reproducible-download.md) - Record and replay a download with a JSON-LD manifest. +- [Workflow Examples](examples/workflows/README.md) - Example download, deploy, delete, and manifest workflows. +- [GSoC 2026](gsoc-2026/README.md) - Project overview and proposal. \ No newline at end of file diff --git a/doc/cli-usage.md b/doc/cli-usage.md new file mode 100644 index 0000000..a30cbf1 --- /dev/null +++ b/doc/cli-usage.md @@ -0,0 +1,253 @@ +## CLI Usage + +To get started with the command-line interface (CLI) of the databus-python-client, you can use either the Python installation or the Docker image. The examples below show both methods. + +**Help and further general information:** + +```bash +databusclient --help +databusclient download --help +``` + +#### Examples of using the download command + +**Download File**: download of a single file +```bash +databusclient download https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01/mappingbased-literals_lang=az.ttl.bz2 +``` + +**Download Version**: download of all files of a specific version +```bash +databusclient download https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01 +``` + +**Download Artifact**: download of all files with the latest version of an artifact +```bash +databusclient download https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals +``` + +**Download Group**: download of all files with the latest version of all artifacts of a group +```bash +databusclient download https://databus.dbpedia.org/dbpedia/mappings +``` + +**Download Collection**: download of all files within a collection +```bash +databusclient download https://databus.dbpedia.org/dbpedia/collections/dbpedia-snapshot-2022-12 +``` + +**Download Query**: download of all files returned by a query (SPARQL endpoint must be provided with `--databus`) +```bash +databusclient download 'PREFIX dcat: SELECT ?x WHERE { ?sub dcat:downloadURL ?x . } LIMIT 10' --databus https://databus.dbpedia.org/sparql +``` + +**Download with Compression Conversion**: download files and convert them to a different compression format on-the-fly +```bash +# Convert all compressed files to gzip format +databusclient download https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01 --compression gz + +# Decompress files without recompressing +databusclient download https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals --compression none + +# Download a collection and unify all files to bz2 format +databusclient download https://databus.dbpedia.org/dbpedia/collections/dbpedia-snapshot-2022-12 --compression bz2 +``` + + +### Deploy + +With the deploy command, you can deploy datasets to the Databus. The deploy command supports three modes: +1. Classic dataset deployment via list of distributions +2. Metadata-based deployment via metadata JSON file +3. Upload & deploy via Nextcloud/WebDAV + +```bash +databusclient deploy [OPTIONS] [DISTRIBUTIONS]... +``` + +**Help and further information on deploy command:** +```bash +databusclient deploy --help +``` + +### Mode 1: Classic Deploy (Distributions) + +```bash +databusclient deploy \ +--version-id https://databus.dbpedia.org/user1/group1/artifact1/2022-05-18 \ +--title "Client Testing" \ +--abstract "Testing the client...." \ +--description "Testing the client...." \ +--license http://dalicc.net/licenselibrary/AdaptivePublicLicense10 \ +--apikey MYSTERIOUS \ +'https://raw.githubusercontent.com/dbpedia/databus/master/server/app/api/swagger.yml|type=swagger' +``` +A few more notes for CLI usage: + +- The content variants can be left out ONLY IF there is just one distribution + - For complete inferred: Just use the URL with `https://raw.githubusercontent.com/dbpedia/databus/master/server/app/api/swagger.yml` + - If other parameters are used, you need to leave them empty like `https://raw.githubusercontent.com/dbpedia/databus/master/server/app/api/swagger.yml||yml|7a751b6dd5eb8d73d97793c3c564c71ab7b565fa4ba619e4a8fd05a6f80ff653:367116` + + +### Mode 2: Deploy with Metadata File + +Use a JSON metadata file to define all distributions. +The metadata.json should list all distributions and their metadata. +All files referenced there will be registered on the Databus. +```bash +databusclient deploy \ + --metadata ./metadata.json \ + --version-id https://databus.dbpedia.org/user1/group1/artifact1/1.0 \ + --title "Metadata Deploy Example" \ + --abstract "This is a short abstract of the dataset." \ + --description "This dataset was uploaded using metadata.json." \ + --license https://dalicc.net/licenselibrary/Apache-2.0 \ + --apikey "API-KEY" +``` +Example `metadata.json` metadata file structure (`file_format` and `compression` are optional): +```json +[ + { + "checksum": "0929436d44bba110fc7578c138ed770ae9f548e195d19c2f00d813cca24b9f39", + "size": 12345, + "url": "https://cloud.example.com/remote.php/webdav/datasets/mydataset/example.ttl", + "file_format": "ttl" + }, + { + "checksum": "2238acdd7cf6bc8d9c9963a9f6014051c754bf8a04aacc5cb10448e2da72c537", + "size": 54321, + "url": "https://cloud.example.com/remote.php/webdav/datasets/mydataset/example.csv.gz", + "file_format": "csv", + "compression": "gz" + } +] +``` + +### Mode 3: Upload & Deploy via Nextcloud + +Upload local files or folders to a WebDAV/Nextcloud instance and automatically deploy to DBpedia Databus. [Rclone](https://rclone.org/) is required. + +```bash +databusclient deploy \ + --webdav-url https://cloud.example.com/remote.php/webdav \ + --remote nextcloud \ + --path datasets/mydataset \ + --version-id https://databus.dbpedia.org/user1/group1/artifact1/1.0 \ + --title "Test Dataset" \ + --abstract "Short abstract of dataset" \ + --description "This dataset was uploaded for testing the Nextcloud → Databus pipeline." \ + --license https://dalicc.net/licenselibrary/Apache-2.0 \ + --apikey "API-KEY" \ + ./localfile1.ttl \ + ./data_folder +``` + + +### Delete + +With the delete command you can delete collections, groups, artifacts, and versions from the Databus. Deleting files is not supported via API. + +**Note**: Deleting datasets will recursively delete all data associated with the dataset below the specified level. Please use this command with caution. As security measure, the delete command will prompt you for confirmation before proceeding with any deletion. + +```bash +databusclient delete [OPTIONS] DATABUSURIS... +``` + +**Help and further information on delete command:** +```bash +databusclient delete --help +``` + +To authenticate the delete request, you need to provide an API key with `--databus-key YOUR_API_KEY`. + +If you want to perform a dry run without actual deletion, use the `--dry-run` option. This will show you what would be deleted without making any changes. + +As security measure, the delete command will prompt you for confirmation before proceeding with the deletion. If you want to skip this prompt, you can use the `--force` option. + +#### Examples of using the delete command + +**Delete Version**: delete a specific version +```bash +databusclient delete https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01 --databus-key YOUR_API_KEY +``` + +**Delete Artifact**: delete an artifact and all its versions +```bash +databusclient delete https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals --databus-key YOUR_API_KEY +``` + +**Delete Group**: delete a group and all its artifacts and versions +```bash +databusclient delete https://databus.dbpedia.org/dbpedia/mappings --databus-key YOUR_API_KEY +``` + +**Delete Collection**: delete collection +```bash +databusclient delete https://databus.dbpedia.org/dbpedia/collections/dbpedia-snapshot-2022-12 --databus-key YOUR_API_KEY +``` + + +### Manifest + +The `download`, `deploy`, and `delete` commands accept `--manifest PATH` to write a structured JSON-LD record of the operation. + +```bash +databusclient download https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01 --manifest ./manifests/download-run.jsonld +``` + +The manifest records input parameters, file URLs, checksums, byte sizes, timestamps, and success or failure status. API keys and vault tokens are not stored. If the operation fails, the manifest includes a `dbus:operationError` record with the error type and message. Existing target paths are preserved by writing an auto-suffixed manifest path. + +See [Reproducible Download](examples/reproducible-download.md) for a complete download recording and replay example. + + +#### Replay + +Replay a recorded download, deploy, or delete operation with: + +```bash +databusclient manifest replay [OPTIONS] MANIFEST_PATH +``` + +Credentials are never stored in manifests and must be supplied again when needed. Use `--localdir` for download output, `--databus` for the endpoint, `--vault-token`, `--databus-key`, or `--apikey` for authentication, and `--force` or `--dry-run` for delete replay. Deploy replay supports classic and metadata-file deployments, but not WebDAV deployments because their original local files may no longer exist. + +```bash +databusclient manifest replay ./manifests/download-run.jsonld --localdir ./replayed-data +``` + + +#### Summary + +Print the stored results from a manifest without replaying the operation or accessing the network: + +```bash +databusclient manifest summary ./manifests/download-run.jsonld +``` + +The summary displays the command, execution time, file counts, byte total when available, overall status, operation errors, and individual failed files. + + +### Workflow + +Run a multi-step `download`, `deploy`, and `delete` pipeline from YAML: + +```bash +databusclient workflow run [OPTIONS] WORKFLOW_PATH +``` + +Each workflow has a top-level `steps` list. Steps run in order, can reference earlier outputs using `${steps.step_name.output_files}` or `${steps.step_name.output_urls}`, and support `on_error: fail`, `continue`, or `retry`. Retry settings use `max_attempts` and `delay_seconds`. + +```yaml +steps: + - name: fetch_dataset + command: download + uri: https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01/mappingbased-literals_lang=az.ttl.bz2 + localdir: ./data + - name: publish_dataset + command: deploy + files: ${steps.fetch_dataset.output_urls} + on_error: fail +``` + +Workflow runs always produce a console summary. Pass `--manifest PATH`, or set the top-level YAML `manifest` value, to also write one unified JSON-LD manifest covering the complete workflow and each step. + +See [Workflow Examples](examples/workflows/README.md) for ready-to-use download, deploy, delete, and failure-handling workflows. diff --git a/examples/reproducible-download.md b/doc/examples/reproducible-download.md similarity index 100% rename from examples/reproducible-download.md rename to doc/examples/reproducible-download.md diff --git a/examples/workflows/README.md b/doc/examples/workflows/README.md similarity index 93% rename from examples/workflows/README.md rename to doc/examples/workflows/README.md index ccc7c8f..c879d9e 100644 --- a/examples/workflows/README.md +++ b/doc/examples/workflows/README.md @@ -25,4 +25,4 @@ databusclient workflow run download-deploy.yml Workflows can write a unified manifest covering every step in two ways: pass `--manifest path.jsonld` on the command line, or set a top-level `manifest:` key inside the YAML file itself (the command-line flag takes priority if both are given). Several of the examples above use the YAML key. Every manifest file entry that came from a workflow step is tagged with `dbus:stepName`, so a multi-step run stays traceable to which step produced or failed on which file. -See the main [README's Workflow section](../../README.md#cli-workflow) for the full YAML format, step chaining, error handling, and WebDAV deploy mode documentation. +See the [CLI usage documentation](../../cli-usage.md#workflow) for the full YAML format, step chaining, error handling, and WebDAV deploy mode documentation. diff --git a/examples/workflows/batch-deployment-with-retry.yml b/doc/examples/workflows/batch-deployment-with-retry.yml similarity index 100% rename from examples/workflows/batch-deployment-with-retry.yml rename to doc/examples/workflows/batch-deployment-with-retry.yml diff --git a/examples/workflows/ci-cd-integration.yml b/doc/examples/workflows/ci-cd-integration.yml similarity index 100% rename from examples/workflows/ci-cd-integration.yml rename to doc/examples/workflows/ci-cd-integration.yml diff --git a/examples/workflows/download-delete.yml b/doc/examples/workflows/download-delete.yml similarity index 100% rename from examples/workflows/download-delete.yml rename to doc/examples/workflows/download-delete.yml diff --git a/examples/workflows/download-deploy.yml b/doc/examples/workflows/download-deploy.yml similarity index 100% rename from examples/workflows/download-deploy.yml rename to doc/examples/workflows/download-deploy.yml diff --git a/examples/workflows/failure-debugging.yml b/doc/examples/workflows/failure-debugging.yml similarity index 100% rename from examples/workflows/failure-debugging.yml rename to doc/examples/workflows/failure-debugging.yml diff --git a/examples/workflows/full-pipeline.yml b/doc/examples/workflows/full-pipeline.yml similarity index 100% rename from examples/workflows/full-pipeline.yml rename to doc/examples/workflows/full-pipeline.yml diff --git a/examples/workflows/nightly-publishing-pipeline.yml b/doc/examples/workflows/nightly-publishing-pipeline.yml similarity index 100% rename from examples/workflows/nightly-publishing-pipeline.yml rename to doc/examples/workflows/nightly-publishing-pipeline.yml diff --git a/examples/workflows/reproducible-research-download.yml b/doc/examples/workflows/reproducible-research-download.yml similarity index 100% rename from examples/workflows/reproducible-research-download.yml rename to doc/examples/workflows/reproducible-research-download.yml diff --git a/doc/gsoc-2026/README.md b/doc/gsoc-2026/README.md new file mode 100644 index 0000000..4c47ec1 --- /dev/null +++ b/doc/gsoc-2026/README.md @@ -0,0 +1,15 @@ +# GSoC 2026 + +Hi, I'm Dhanashree Petare ([GitHub](https://github.com/DhanashreePetare)), and I contributed to this project as part of Google Summer of Code 2026, under the DBpedia organization. + +My project extended the Databus Python Client with reproducible, workflow-aware data operations, delivered across five milestones: + +1. **Format and Mapping Conversion Layer** - RDF triple, RDF quad, and tabular format conversion during download, bringing the Python client to feature parity with the Java client. [CLI usage](../cli-usage.md). +2. **Structured Run Manifest System** - JSON-LD manifests recording operation parameters, file metadata, checksums, and execution results for every `download`, `deploy`, and `delete` run. [CLI usage](../cli-usage.md). +3. **Manifest Replay and Summary** - re-executing a past operation from its saved manifest, and printing a readable console summary of any manifest. [CLI usage](../cli-usage.md). +4. **Declarative Workflow Engine** - YAML-defined pipelines chaining `download`/`deploy`/`delete` steps, with step-to-step output chaining and per-step error handling (`fail`/`continue`/`retry`). [CLI usage](../cli-usage.md). +5. **Workflow-Manifest Integration and Example Workflows** - a unified manifest covering an entire workflow run, an automatic console summary, and eight example workflows. [Workflow examples](../examples/). + +My project proposal is available [here](proposal_DhanashreePetare.pdf). + +Thank you. \ No newline at end of file diff --git a/doc/gsoc-2026/proposal_DhanashreePetare.pdf b/doc/gsoc-2026/proposal_DhanashreePetare.pdf new file mode 100644 index 0000000..ff6037e Binary files /dev/null and b/doc/gsoc-2026/proposal_DhanashreePetare.pdf differ diff --git a/doc/module-usage.md b/doc/module-usage.md new file mode 100644 index 0000000..9521b1d --- /dev/null +++ b/doc/module-usage.md @@ -0,0 +1,79 @@ +# Module Usage + +The client exposes Python functions for creating distributions and datasets and deploying them programmatically. + + +### Deploy + +#### Step 1: Create lists of distributions for the dataset + +```python +from databusclient import create_distribution + +# create a list +distributions = [] + +# minimal requirements +# compression and filetype will be inferred from the path +# this will trigger the download of the file to evaluate the shasum and content length +distributions.append( + create_distribution(url="https://raw.githubusercontent.com/dbpedia/databus/master/server/app/api/swagger.yml", cvs={"type": "swagger"}) +) + +# full parameters +# will just place parameters correctly, nothing will be downloaded or inferred +distributions.append( + create_distribution( + url="https://example.org/some/random/file.csv.bz2", + cvs={"type": "example", "realfile": "false"}, + file_format="csv", + compression="bz2", + sha256_length_tuple=("7a751b6dd5eb8d73d97793c3c564c71ab7b565fa4ba619e4a8fd05a6f80ff653", 367116) + ) +) +``` + +Compression and file type can be inferred from the path. A checksum and content length can be supplied when they are already known. The content variants dictionary may be empty only when there is one distribution. Compression cannot be used without a file format. + +## Create dataset + +```python +from databusclient import create_dataset + +# minimal way +dataset = create_dataset( + version_id="https://dev.databus.dbpedia.org/denis/group1/artifact1/2022-05-18", + title="Client Testing", + abstract="Testing the client....", + description="Testing the client....", + license_url="http://dalicc.net/licenselibrary/AdaptivePublicLicense10", + distributions=distributions, +) + +# with group metadata +dataset = create_dataset( + version_id="https://dev.databus.dbpedia.org/denis/group1/artifact1/2022-05-18", + title="Client Testing", + abstract="Testing the client....", + description="Testing the client....", + license_url="http://dalicc.net/licenselibrary/AdaptivePublicLicense10", + distributions=distributions, + group_title="Title of group1", + group_abstract="Abstract of group1", + group_description="Description of group1" +) +``` + +Group metadata can be supplied with `group_title`, `group_abstract`, and `group_description`. It is applied only when all group parameters are provided. + +## Deploy a dataset + +```python +from databusclient import deploy + +# to deploy something you just need the dataset from the previous step and an API key +# API key can be found (or generated) at https://$$DATABUS_BASE$$/$$USER$$#settings +deploy(dataset, "mysterious API key") +``` + +The API key can be found or generated in the Databus account settings.