feat: add openlineage support

CHANGELOG.md

@@ -6,6 +6,7 @@ All notable changes to this project will be documented in this file.
 
 ### Added
 
+- Add `spec.clusterConfig.lineage` to emit [OpenLineage](https://openlineage.io/) lineage events ([#914]).
 - Support floating tags for product images via the new `spec.image.stackableVersionPolicy` field
   ([#944]).
 - Add `/ready` endpoint to the operator Deployment, which reports the CRD installation status ([#949]).
@@ -70,6 +71,7 @@ All notable changes to this project will be documented in this file.
 
 [#909]: https://github.com/stackabletech/trino-operator/pull/909
 [#913]: https://github.com/stackabletech/trino-operator/pull/913
+[#914]: https://github.com/stackabletech/trino-operator/pull/914
 [#918]: https://github.com/stackabletech/trino-operator/pull/918
 [#923]: https://github.com/stackabletech/trino-operator/pull/923
 [#924]: https://github.com/stackabletech/trino-operator/pull/924

Cargo.toml

@@ -30,5 +30,8 @@ tracing = "0.1"
 yaml_serde = "0.10"
 
 [patch."https://github.com/stackabletech/operator-rs.git"]
-# stackable-operator = { git = "https://github.com/stackabletech//operator-rs.git", branch = "main"}
+# TODO: Switch back to a release tag once operator-rs#1250 (the `crd::openlineage`
+# module) has merged and a new stackable-operator release has been published.
+# See <https://github.com/stackabletech/operator-rs/pull/1250>.
+# stackable-operator = { git = "https://github.com/stackabletech//operator-rs.git", branch = "main" }
 # stackable-operator = { path = "../operator-rs/crates/stackable-operator" }

deploy/helm/trino-operator/templates/clusterrole-operator.yaml

@@ -171,3 +171,12 @@ rules:
       - get
       - list
       - watch
+  # Read OpenLineage connection configuration referenced in the TrinoCluster spec.
+  - apiGroups:
+      - lineage.stackable.tech
+    resources:
+      - openlineageconnections
+    verbs:
+      - get
+      - list
+      - watch

docs/modules/trino/pages/usage-guide/openlineage.adoc

@@ -0,0 +1,149 @@
+= OpenLineage
+:description: Emit OpenLineage data lineage events from Trino queries to an OpenLineage-compatible backend such as Marquez.
+:keywords: OpenLineage, data lineage, event listener, Marquez, Trino
+
+https://openlineage.io/[OpenLineage] is an open standard for data lineage collection.
+Trino ships an https://trino.io/docs/current/admin/event-listeners-openlineage.html[OpenLineage event listener] as a core plugin, which emits lineage events for the queries run on the cluster.
+The `spec.clusterConfig.lineage` block in the TrinoCluster custom resource configures the OpenLineage event listener and its connection to a backend.
+
+When configured, the operator:
+
+* writes an `event-listener.properties` file on the *coordinator* selecting the OpenLineage plugin and its HTTP transport,
+* points the transport at the backend endpoint (`<scheme>://<host>:<port>`),
+* records this cluster's `trino.uri` and a lineage namespace, and
+* (optionally) imports a backend CA certificate into the coordinator trust store and injects a bearer token for authenticated backends.
+
+The event listener runs on the coordinator only; workers are not affected.
+
+[IMPORTANT]
+====
+By default, `SELECT` queries do *not* emit any lineage events, so a cluster that only reads data produces no lineage at all.
+To also capture reads, set `openlineage-event-listener.trino.include-query-types` as shown in <<include-select-queries>>.
+====
+
+== Enabling OpenLineage
+
+OpenLineage is configured through `spec.clusterConfig.lineage.connection`, which either inlines an OpenLineage connection or references a standalone `OpenLineageConnection` resource by name:
+
+[source,yaml]
+----
+apiVersion: trino.stackable.tech/v1alpha1
+kind: TrinoCluster
+metadata:
+  name: trino
+spec:
+  clusterConfig:
+    lineage: # <1>
+      connection: # <2>
+        inline:
+          http:
+            host: marquez
+            port: 5000
+            # path: /api/v1/lineage # <4>
+            # tls:
+            #   verification:
+            #     server:
+            #       caCert:
+            #         secretClass: marquez-ca
+      # namespace: trino-lineage # <3>
+      # jobNameFormat: "$QUERY_ID" # <5>
+      # datasetNamespaceUri: https://trino-prod # <6>
+  ...
+----
+<1> Adding the `lineage` block enables OpenLineage event emission. Omit it entirely to leave OpenLineage off (the default).
+<2> Required. The OpenLineage backend connection. Use `inline` (shown here) to define the connection directly, or `reference: <name>` to point at an `OpenLineageConnection` resource in the same namespace.
+<3> Optional. The OpenLineage namespace events are reported under. Defaults to `default`.
+<4> Optional. URL path of the endpoint events are POSTed to. Defaults to `/api/v1/lineage`.
+<5> Optional. Sets the OpenLineage job name format. Accepts an arbitrary string with optional `$QUERY_ID`, `$USER`, `$SOURCE` and `$CLIENT_IP` substitution variables. Defaults to `$QUERY_ID`. See the link:https://trino.io/docs/current/admin/event-listeners-openlineage.html[Trino OpenLineage event listener documentation {external-link-icon}^].
+<6> Optional. Sets `openlineage-event-listener.trino.uri`, from which Trino derives the namespace of every dataset it reports. The URI is only an identifier and is never connected to. Defaults to `https://<name of the TrinoCluster>.<its namespace>` when empty or unset. Set it explicitly to keep dataset identities stable across Kubernetes clusters or namespace moves.
+
+The `OpenLineageConnection` resource (`lineage.stackable.tech`) is a reusable definition of a connection to an OpenLineage backend, provided by the Stackable operator library.
+Reference it to share one backend definition across multiple clusters:
+
+[source,yaml]
+----
+apiVersion: lineage.stackable.tech/v1alpha1
+kind: OpenLineageConnection
+metadata:
+  name: marquez
+spec:
+  http:
+    host: marquez
+    port: 5000
+----
+
+[IMPORTANT]
+====
+With TLS server verification, the `host` must match a Subject Alternative Name in the backend's certificate. Use the exact hostname the certificate was issued for (for an in-cluster backend, typically its full service FQDN such as `marquez.<namespace>.svc.cluster.local`), otherwise the TLS handshake fails and no events are delivered.
+====
+
+[#authentication]
+== Authentication
+
+Backends that require authentication (for example DataHub, or a Marquez instance behind a gateway) expect a bearer token on each request.
+Configure this by setting `credentialsSecretName` on the connection to the name of a Secret holding the token:
+
+[source,yaml]
+----
+apiVersion: trino.stackable.tech/v1alpha1
+kind: TrinoCluster
+metadata:
+  name: trino
+spec:
+  clusterConfig:
+    lineage:
+      connection:
+        inline:
+          http:
+            host: marquez
+            port: 5000
+            credentialsSecretName: openlineage-token # <1>
+  ...
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: openlineage-token
+  namespace: default # <2>
+stringData:
+  apiKey: "s3cr3t-bearer-token" # <3>
+----
+<1> Name of the Secret holding the token. If omitted, no authentication is used. The Secret must exist in the same namespace as the `TrinoCluster`.
+<2> The Secret must be in the cluster's namespace so it can be projected into the coordinator.
+<3> The token must be stored under the fixed key `apiKey`. The token is mounted as a file into the coordinator and resolved into the event listener configuration at startup, so it never appears in the ConfigMap or the pod's environment.
+
+The operator translates this into the OpenLineage HTTP transport's `openlineage-event-listener.transport.api-key` (bearer) authentication.
+Combine it with the `tls` block above when the backend also needs TLS server verification.
+
+== Overriding the generated configuration
+
+The generated `event-listener.properties` can be extended or overridden per role group through xref:usage-guide/overrides.adoc[config overrides], using the `event-listener.properties` file name.
+Overrides are merged last and therefore take precedence over the operator-generated values:
+
+[source,yaml]
+----
+spec:
+  coordinators:
+    configOverrides:
+      event-listener.properties:
+        openlineage-event-listener.transport.timeout: "10s"
+----
+
+Event listeners run on the coordinator, so `event-listener.properties` overrides only take effect on the coordinator role.
+
+[#include-select-queries]
+=== Emitting lineage for `SELECT` queries
+
+`openlineage-event-listener.trino.include-query-types` controls which query types emit lineage events.
+It replaces the default list, so repeat the default types when adding `SELECT`:
+
+[source,yaml]
+----
+spec:
+  coordinators:
+    configOverrides:
+      event-listener.properties:
+        openlineage-event-listener.trino.include-query-types: "SELECT,INSERT,DELETE,MERGE,UPDATE,ALTER_TABLE_EXECUTE,DATA_DEFINITION"
+----
+
+Every `SELECT` then produces a `START` and a `COMPLETE` (or `FAIL`) event, which can noticeably increase the load on the lineage backend for clusters serving many interactive or dashboard queries.