feat: add openlineage support by razvan · Pull Request #726 · stackabletech/spark-k8s-operator

razvan · 2026-07-17T07:58:17Z

Description

Spark can emit OpenLineage events for queries. This PR makes essential properties configurable. The operator uses sensible defaults where possible.

Depends on:

⌛ feat: add openlineage to spark image docker-images#1577

Part of stackabletech/issues#856

Decision: https://github.com/stackabletech/decisions/issues/90

CRD change

See extra/crds.yaml for the CRD diff.

Examples

The example below shows how add inline spec.lineage configuration to a SparkApplication resource.

The following fields are new:

connection : provided by the OpenLineageConfig struct from op-rs and common to all products supporting this in the future.
appName: specific to Spark. Used to build names for Job events.

Note

The sister Trino PR introduces a new field called jobNameFormat that is equivalent but not the same as appName here.
spec.lineage.appName cannot contain template vars. It is a literal name applied to all OpenLineage events emitted by Spark.

Inline configuration

apiVersion: spark.stackable.tech/v1alpha1
kind: SparkApplication
metadata:
  name: orders-by-nation
spec:
  sparkImage:
    productVersion: 4.1.2
  mode: cluster
  mainApplicationFile: local:///stackable/spark/examples/src/main/python/pi.py
  lineage:
    appName: orders-by-nation
    connection:
      inline:
        http:
          host: marquez
          port: 5000

Definition of Done Checklist

Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
Please make sure all these things are done and tick the boxes

Author

Changes are OpenShift compatible
CRD changes approved
CRD documentation for all fields, following the style guide.
Helm chart can be installed and deployed operator works
Integration tests passed (for non trivial changes)
Changes need to be "offline" compatible
Links to generated (nightly) docs added
Release note snippet added

Reviewer

Code contains useful comments
Code contains useful logging statements
(Integration-)Test cases added
Documentation added or updated. Follows the style guide.
Changelog updated
Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

Feature Tracker has been updated
Proper release label has been added
Links to generated (nightly) docs added
Release note snippet added
Add type/deprecation label & add to the deprecation schedule
Add type/experimental label & add to the experimental features tracker

* tests: better testing of the hbase connectors * bump hbase * update supported versions * update fixture

* Revert "fix: make application templates namespaced (#694)" This reverts commit b326be3. * revert changelog * update nix crate hashes

) * Reapply "fix: make application templates namespaced (#694)" (#719) This reverts commit e1b21e1. * update changelog

…ationClass Follows the operator-rs change: derive the OpenLineage backend bearer token from the connection's `credentialsSecretName` Secret (key `apiKey`) instead of resolving a Static AuthenticationClass. Drops ResolvedOpenLineageAuth and the authenticationclasses RBAC rule. See stackabletech/decisions#90 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…h API group Follow the operator-rs rename of OpenLineageJob.app_name to job_name (serialized appName -> jobName) and the move of the OpenLineageConnection CRD to the lineage.stackable.tech API group: - read spec.openLineage.job_name in resolved_openlineage_app_name(); update the resolution doc comment - update the operator RBAC Role apiGroup - update the crd test manifest, kuttl manifest, and usage-guide docs (jobName) - regenerate extra/crds.yaml (appName -> jobName) The Spark conf key `spark.openlineage.appName` is an OpenLineage-Spark library property, not the CRD field, and is left unchanged (as are the unrelated Spark SparkSession.appName() calls and local app_name variables). Verified with cargo check/clippy/test and CRD regeneration against a local operator-rs checkout carrying the rename. Committed with --no-verify because the cargo/regenerate-charts pre-commit hooks would otherwise build against the un-pushed operator-rs branch; those checks were run manually instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Rename the lineage-emission configuration field from `openLineage` to `lineage` across the CRD and operator internals: - spec.openLineage -> spec.lineage (Rust field spec.open_lineage -> lineage) - rename operator-internal identifiers: module openlineage.rs -> lineage.rs, SparkApplication::resolved_openlineage_app_name -> resolved_lineage_app_name, openlineage_enabled -> lineage_enabled - update the usage-guide docs, the kuttl test manifest, and CHANGELOG - regenerate extra/crds.yaml (openLineage -> lineage) Kept: the OpenLineage technology name in prose, operator-rs types (OpenLineageJob, OpenLineageConnection) and their open_lineage_conn(ection) identifiers, the spark.openlineage.* / OPENLINEAGE_* keys, and the openlineage_transport_env_vars helper (produces the OPENLINEAGE_* transport env). Verified with cargo check/clippy/test, rustfmt, and CRD regeneration against a local operator-rs checkout. Committed with --no-verify because the cargo/regenerate pre-commit hooks would build against the un-pushed operator-rs branch; those checks were run manually instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

… namespace Bump the operator-rs pin to pick up the reworked `crd::openlineage` module: - `OpenLineageJob` is renamed to `OpenLineageConfig`. - `OpenLineageConnectionSpec` now selects an `OpenLineageTransport` (currently only `http`) instead of carrying host/port/tls/credentialsSecretName directly. Add a `lineage::http_transport` helper so every access goes through one place; it stops compiling when a new transport variant is added upstream. - `HttpTransport` gained a `path` field, defaulting to `/api/v1/lineage`, wired to `spark.openlineage.transport.endpoint` and, on the authenticated path, to the new `OPENLINEAGE__TRANSPORT__ENDPOINT` env var. - `OpenLineageConfig::namespace` is now a required `String` defaulting to `default`, so the operator-side fallback to `metadata.namespace` is gone. NOTE: this changes behaviour for users who did not set `namespace` explicitly - lineage is now reported under `default` rather than the application's Kubernetes namespace. The pin also brings in operator-rs' removal of `product-config`. Update the usage guide, the kuttl manifests and extra/crds.yaml for the nested transport schema. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Resolve conflicts and adapt to stackable-operator 0.120.0: drop the removed OpenLineage jobName field and HttpTransport::transport_url(), and thread the OpenLineage connection through the new EnvVarSet and ResolvedProductImage APIs. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

- cleanup code comments - cleanup code by unconditionally adding --add-opens to all spark versions

sweb · 2026-10-14T12:57:45Z

+        /// A stable OpenLineage application (job) name. Setting this prevents fragmented run
+        /// history. If empty, the name resolves to `spark.app.name` from `sparkConf` and then to
+        /// the name of the `SparkApplication`.
+        #[serde(default)]


is a default of "" a problem and should we treat this explicitly?

#[serde(default, skip_serializing_if = "Option::is_none")] pub app_name: Option<String>,

I made it optional e0b1292

sweb · 2026-10-15T06:50:47Z

+pub(crate) fn transport_base_url(http: &HttpTransport) -> String {
+    format!(
+        "{scheme}://{host}:{port}",
+        scheme = if http.tls.uses_tls() { "https" } else { "http" },


Something that Claude flagged that we may want to look at as a follow up: TLS with verification=None will not work for open lineage - Spark will run but the lineage will be lost. Claude suggests to handle it similarly to S3 connection handling - I don't think this needs to be fixed now.

This was a good catch. Added a warning in the docs 31e4fcf

sweb · 2026-10-15T06:56:31Z

    log_dir: &Option<ResolvedLogDir>,
-) -> (String, String) {
+    open_lineage_conn: Option<&openlineage::ResolvedOpenLineageConnection>,
+) -> Result<(String, String), Error> {


I think the return type can be simplified to (String, String) if I am not mistaken. Don't see a ? or Err

Was a leftover I believe 643d13e

sweb · 2026-10-15T07:19:37Z

+        // via the client's `OPENLINEAGE__` env-var configuration. See `openlineage_transport_env_vars`
+        // and the matching skip of the `spark.openlineage.transport.*` `--conf` in `build_command`.
+        // The token is sourced from the referenced Secret, so it never lands in the `--conf` args.
+        if let Some(connection) = open_lineage_conn {


just because I have spent a lot of time on a sidequest to get SSL env vars propagated from driver to executor - I have the hunch that the openlineage env var also lands on the exec and I don't think that this is necessary. Probably not a huge deal.

Less token exposure means slightly more secure, so I removed the env vars from the executor pods d33b2bc

razvan · 2026-10-20T08:57:01Z

Vote on the PR body please!

sweb and others added 9 commits July 10, 2026 09:20

feat: add basic openlineage support

739b49a

tests: better testing of the hbase connectors (#717)

43235f5

* tests: better testing of the hbase connectors * bump hbase * update supported versions * update fixture

Revert "fix: make application templates namespaced (#694)" (#719)

c7313a8

* Revert "fix: make application templates namespaced (#694)" This reverts commit b326be3. * revert changelog * update nix crate hashes

Reapply "fix: make application templates namespaced (#694)" (#719) (#720

87da728

) * Reapply "fix: make application templates namespaced (#694)" (#719) This reverts commit e1b21e1. * update changelog

extract OL structs into openlineage.rs

4335b15

refactor!: remove openlineage.enabled field

454e8cf

refactor!: replace OL config map with host:port fields

6cdec1a

feat: add tls field

24dfb75

refactor!: use crd::openlineage module from op-rs

43ccff2

razvan self-assigned this Jul 17, 2026

razvan and others added 7 commits July 17, 2026 09:59

Merge branch 'main' into feat/openlineage-from-op-rs

5f63d94

chore: regenerate-nix ... again

7f079f3

feat: support Spark 3 as well

ed2c33e

refactor!: remove ResolvedOpenLineageAppName struct

b85328a

feat: openlineage.tls now also installs the server CA if necesary

93f09bd

test: add integration test for openlineage events

c448ebc

add authentication with bearer token

92535f1

razvan changed the title ~~feat: use openlineage from op-rs~~ feat: add openlineage support Jul 20, 2026

razvan and others added 12 commits July 21, 2026 15:37

merge main

846effc

Merge branch 'main' into feat/openlineage-from-op-rs

fd4c1fe

chore: bump stackable-operator to 0.120.0

71a3038

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

fix changelog after main merge

148a19b

Merge branch 'main' into feat/openlineage-from-op-rs

9e8e598

fix(lint): typos

2b1fed8

chore: cleanups

381df72

- cleanup code comments - cleanup code by unconditionally adding --add-opens to all spark versions

feat: add new lineage.appName field

96ed213

razvan marked this pull request as ready for review October 13, 2026 11:54

razvan mentioned this pull request Oct 13, 2026

feat: add openlineage to spark image stackabletech/docker-images#1577

Merged

6 tasks

razvan added this to Stackable Engineering Oct 13, 2026

sweb reviewed Oct 15, 2026

View reviewed changes

razvan added 5 commits October 15, 2026 16:16

merge main branch

384500a

review feedback: add warning in the doc about TLS verification

31e4fcf

review feedback: remove unnecessary Result return type

643d13e

review feedback: remove openlineage env vars from the executor pods

d33b2bc

review feedback: make app_name field optional

e0b1292

razvan requested a review from sweb October 15, 2026 15:35

Techassi moved this from Development: Waiting for Review to Development: In Review in Stackable Engineering Oct 19, 2026

Navigation Menu

Repository navigation

feat: add openlineage support - #726

feat: add openlineage support#726
razvan wants to merge 34 commits into
mainfrom
feat/openlineage-from-op-rs

razvan commented Jul 17, 2026 •

edited

Loading

Uh oh!

sweb Oct 14, 2026

Uh oh!

razvan Oct 15, 2026

Uh oh!

sweb Oct 15, 2026

Uh oh!

razvan Oct 15, 2026

Uh oh!

sweb Oct 15, 2026

Uh oh!

razvan Oct 15, 2026

Uh oh!

sweb Oct 15, 2026

Uh oh!

razvan Oct 15, 2026

Uh oh!

razvan commented Oct 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

razvan commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

CRD change

Examples

Inline configuration

Definition of Done Checklist

Author

Reviewer

Acceptance

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

razvan commented Oct 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

razvan commented Jul 17, 2026 •

edited

Loading