From 3713343d55f4e312af7ee20a9c2555f9752d3d95 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?St=C3=A9phane=20Manciot?= Date: Fri, 4 Sep 2026 06:54:21 +0200 Subject: [PATCH 1/2] =?UTF-8?q?docs:=20BI=20docs=20twins=20=E2=80=94=20pub?= =?UTF-8?q?lish=20the=20structural=20gaps,=20retire=20the=20Custom=20SQL?= =?UTF-8?q?=20workaround?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Story 20.6, the dual-docs half of the website's BI connection-guide corrections (SOFTNETWORK-APP/softclient4es-web#43). - `client/bi_tools.md`: Power BI and Metabase lose the "Compatible … JDBC" tier — there is no JDBC path to be compatible with. Both, plus Looker and dbt, move to an explicit "no supported connection path" row carrying its mechanism: no generic JDBC database type (Metabase), no JDBC connector in Power Query, Looker-maintained drivers with a per-dialect parameter allowlist, a mandatory adapter plugin (dbt). Looker's gap is structural, not commercial — a licence would not close it. The intro no longer promises a full connect/browse/JOIN guide for tools that cannot connect. - `sql/known_limitations.md`: the Tableau "use Extract mode, or write Custom SQL" workaround is removed. Tableau documents that it "must wrap the custom SQL statement within a select statement", producing a derived table — which this same file lists as not in this release. Looker stops being handed a subquery workaround for a tool that cannot connect at all. Claims stay in lockstep with the website. No tier moves: Tableau stays Compatible, Looker gets none. Closed Issue #273 --- documentation/client/bi_tools.md | 27 ++++++++++---- documentation/sql/known_limitations.md | 51 +++++++++++++++++++++----- 2 files changed, 61 insertions(+), 17 deletions(-) diff --git a/documentation/client/bi_tools.md b/documentation/client/bi_tools.md index 381d6818..8017e0a9 100644 --- a/documentation/client/bi_tools.md +++ b/documentation/client/bi_tools.md @@ -2,21 +2,34 @@ SoftClient4ES connects to common BI and SQL tools through the **JDBC driver** (`jdbc:elastic://host:port`, driver class `app.softnetwork.elastic.jdbc.ElasticDriver`) or the **Arrow Flight SQL** server (`grpc://host:32010`). -Full, step-by-step guides — connect, browse your indices, and run a cross-index JOIN — live on the website. This page is an index; the website carries the screenshots and per-tool detail. +Where a supported connection path exists, a full step-by-step guide — connect, browse your indices, and run a cross-index JOIN — lives on the website. This page is an index; the website carries the screenshots and per-tool detail, and states the blocker for the tools that cannot connect. -## Tested vs Compatible +## Tested, Compatible, and no supported path -**Tested** tools are exercised against SoftClient4ES. **Compatible** tools speak a working protocol but have not been through formal regression (best-effort). +**Tested** tools are exercised against SoftClient4ES. **Compatible** tools speak a working protocol but +have not been through formal regression (best-effort). A third group cannot connect at all, for reasons +on the tool's own side. -| Tool | Tier | Path | Guide | +| Tool | Status | Path | Guide | |---|---|---|---| | Apache Superset | Tested (dedicated dialect) | Arrow Flight SQL | https://softclient4es.dev/integrations/superset/ | | DBeaver | Tested | JDBC or Arrow Flight SQL | https://softclient4es.dev/integrations/dbeaver/ | | Grafana | Tested (via Arrow Flight SQL) | Arrow Flight SQL | https://softclient4es.dev/integrations/grafana/ | | Tableau | Compatible (not formally tested) | JDBC | https://softclient4es.dev/integrations/tableau/ | -| Power BI | Compatible (not formally tested) | JDBC | https://softclient4es.dev/integrations/power-bi/ | -| Metabase | Compatible (not formally tested) | JDBC | https://softclient4es.dev/integrations/metabase/ | +| Metabase | **No supported connection path** — Metabase has no generic JDBC database type; anything else needs a community driver plugin | — | https://softclient4es.dev/integrations/metabase/ | +| Power BI | **No supported connection path** — Power Query has no JDBC connector (its generic connectors are ODBC and OData); the one candidate, a generic Arrow Flight SQL ODBC driver, is unproven | — | https://softclient4es.dev/integrations/power-bi/ | +| Looker | **No supported connection path** — Looker-maintained drivers only, with a per-dialect JDBC parameter allowlist. Structural, not commercial: a licence would not close it | — | — | +| dbt | **No supported connection path** — a dedicated adapter plugin is mandatory; no generic JDBC/ODBC adapter exists and no SoftClient4ES adapter exists | — | — | + +*(Each "no supported connection path" blocker was checked against the vendor's own connection +documentation — Metabase, Microsoft Power Query, Looker and dbt — on 2026-08-31 and 2026-09-01.)* ## Honest-gap note -Every tool runs the superpower of this release — a **cross-index JOIN** that Elasticsearch can't do — best through explicit `JOIN … ON …` SQL. Some BI tools auto-generate nested subqueries (Tableau live connections, Power BI DirectQuery relationships, the Metabase GUI Question builder); subqueries and CTEs are not in this release yet. Use Extract / Import / Native-SQL mode with explicit JOINs as the workaround. Full BI-tool subquery / CTE support is coming in the next release (Quarter 4 2026). See the website's Known Limitations page for the full picture. +The superpower of this release is a **cross-index JOIN** that Elasticsearch can't do, and it runs best +through explicit `JOIN … ON …` SQL — from any tool where you control the statement that is sent (Superset +SQL Lab, DBeaver, Grafana). Some BI tools compose SQL for you: subqueries and CTEs are not in this release +yet, and neither is the quoted, fully-qualified identifier form Tableau generates. Tableau's Custom SQL is +not a way around that — Tableau wraps a custom query inside a `SELECT … FROM ( … )`, which is a derived +table (Tableau's Custom SQL documentation, checked 2026-09-01). Full BI-tool subquery / CTE support is coming in the next release (Quarter 4 2026). See the +website's Known Limitations page for the full picture. diff --git a/documentation/sql/known_limitations.md b/documentation/sql/known_limitations.md index 9dbb6a3d..33c756bb 100644 --- a/documentation/sql/known_limitations.md +++ b/documentation/sql/known_limitations.md @@ -8,16 +8,47 @@ SoftClient4ES runs a large, practical subset of ANSI SQL on Elasticsearch — in ## Using a BI tool? Read this first -If your BI tool just failed on a subquery or a CTE, you're in the right place. Some BI tools auto-generate nested SQL (subqueries / derived tables) even when your logical query has none. Until the next release lands full subquery support, send **explicit JOIN SQL** instead of letting the tool compose nested queries: - -- **Tableau** — Tableau **live** connections can auto-generate subqueries. Use **Extract** mode (Tableau runs the extract locally, subquery-free), or write **Custom SQL** with explicit JOINs instead of letting Tableau compose the query. -- **Power BI** — DirectQuery / query folding can compose nested SQL. Prefer **Import** mode (folds locally, nothing nested is pushed), or author explicit-JOIN queries; avoid relationships that force generated subqueries until the next release. -- **Looker** — BI tools that build **derived tables / measures** (e.g. Looker) can compose subqueries. Model **explicit JOINs** in the SQL the tool sends rather than relying on tool-composed derived tables; avoid symmetric-aggregate measures that force derived tables until the next release. -- **Metabase** — the GUI Question builder can emit subqueries for multi-stage questions. Use **Native (SQL)** queries with explicit `JOIN … ON …` instead of the visual builder for any query that would otherwise nest. - -> **General rule:** prefer **explicit JOIN SQL** over tool-generated nested SQL. If you control the query, a cross-index JOIN is fully supported in the current release. - -Tableau, Power BI, and Metabase are **Compatible** (work via the JDBC/ADBC spec, not formally tested by us). **Apache Superset** (dedicated dialect), **DBeaver**, and **Grafana** (via Arrow Flight SQL) are **Tested**. +Two different things can stop a BI tool here, and it is worth separating them. + +### Some tools cannot connect at all — and that is on their side + +- **Metabase** — no generic JDBC database type exists; anything not on Metabase's own driver list needs a + community driver plugin, which is code nobody has written for SoftClient4ES. See [Metabase](../client/bi_tools.md). +- **Power BI** — Power Query has no JDBC connector; its generic connectors are ODBC and OData. The only + candidate path is a generic Arrow Flight SQL ODBC driver, which is unproven. See [Power BI](../client/bi_tools.md). +- **Looker** — Looker connects only through drivers it maintains itself, and it allowlists JDBC parameters + per dialect, so a customer-supplied driver cannot be introduced. This gap is **structural, not + commercial** — a licence would not close it. +- **dbt** — dbt requires a dedicated adapter plugin per platform. There is no generic JDBC or ODBC adapter, + and no SoftClient4ES adapter. + +None of these is a gap we can close from our side: each one needs either a change by the vendor or a driver +or adapter plugin that nobody has written. + +*(Each blocker checked against the vendor's own connection documentation — Metabase, Microsoft Power Query, +Looker and dbt — on 2026-08-31 and 2026-09-01.)* + +### Tools that connect, but generate SQL we do not accept yet + +Some BI tools auto-generate nested SQL (subqueries / derived tables) even when your logical query has none. +Until the next release lands full subquery support, send **explicit JOIN SQL** instead of letting the tool +compose nested queries — where the tool lets you: + +- **Apache Superset / DBeaver / Grafana** — you control the SQL. Write explicit JOINs for anything that + would otherwise nest, and everything in **Works in this release** below is available to you. +- **Tableau** — connecting and browsing work; queries are the constrained part. Drag-and-drop worksheets + quote and fully qualify every identifier, a form we do not accept yet, and **Custom SQL is not a way + around it**: Tableau documents that it *"must wrap the custom SQL statement within a select statement"* (Tableau's Custom SQL + documentation, checked 2026-09-01), + which turns your query into a derived table. **Extract** mode narrows the exposure but does not remove + it — the extract is still built by querying the source. + See [Tableau](../client/bi_tools.md). + +> **General rule:** prefer **explicit JOIN SQL** over tool-generated nested SQL. If you control the query, a +> cross-index JOIN is fully supported in the current release. + +**Apache Superset** (dedicated dialect), **DBeaver**, and **Grafana** (via Arrow Flight SQL) are **Tested**. +**Tableau** is **Compatible** — the connection path works, but it is not yet in our formal regression suite. ## Works in this release From 61ac3baf9bbd487e51b0dd7bb5f42bd85868ad43 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?St=C3=A9phane=20Manciot?= Date: Sat, 5 Sep 2026 11:12:11 +0200 Subject: [PATCH 2/2] docs: four status tiers; Power BI is Unproven, not blocked; drop dbt Lead + PM review of the tool table. Power BI moves from "No supported connection path" to a new "Unproven" tier. The old label was inaccurate and understated us: Epic 19 established a credible path -- a generic Arrow Flight SQL ODBC driver whose default port 32010 is the sidecar's. Nobody has connected it, so it is a lead to follow, not a route we support. "Compatible" would have been the wrong home: that tier means the protocol demonstrably works, which is exactly what is unknown here. dbt removed. It is a transformation framework, not a BI tool, so it is a category error on this page; and its blocker ("an adapter is mandatory and none exists") is true of every platform dbt does not support -- publishing it reads as an apology for a gap nobody asked about. Looker KEPT, deliberately. Enterprise prospects ask, the answer is non-obvious, and "structural, not commercial -- a licence would not close it" ends that conversation early instead of via support. Removing the row does not remove the question. Metabase stays in the no-path group, with softer wording. It was proposed as "Custom JDBC (not tested)" -- NOT taken: Epic 19 verified live that Metabase has no generic JDBC database type, and the existing "choose the generic JDBC database type" line in metabase.mdx was itself logged as a customer-facing doc defect. Labelling it Custom JDBC would have re-shipped that defect in the very PR that removes it. Connecting needs a community driver plugin we do not ship. Doc twins move together (bi_tools.md + known_limitations.md here; the web PR carries bi-connectivity.astro). Verified on the RENDERED output, not just source: "Unproven (ODBC)" present, "No supported path" gone; npm run build green, 46 pages. Co-Authored-By: Claude Opus 5 --- documentation/client/bi_tools.md | 23 +++++++++++++---------- documentation/sql/known_limitations.md | 19 +++++++++++-------- 2 files changed, 24 insertions(+), 18 deletions(-) diff --git a/documentation/client/bi_tools.md b/documentation/client/bi_tools.md index 8017e0a9..cb4479b4 100644 --- a/documentation/client/bi_tools.md +++ b/documentation/client/bi_tools.md @@ -4,11 +4,15 @@ SoftClient4ES connects to common BI and SQL tools through the **JDBC driver** (` Where a supported connection path exists, a full step-by-step guide — connect, browse your indices, and run a cross-index JOIN — lives on the website. This page is an index; the website carries the screenshots and per-tool detail, and states the blocker for the tools that cannot connect. -## Tested, Compatible, and no supported path +## Tested, Compatible, Unproven, and no path today -**Tested** tools are exercised against SoftClient4ES. **Compatible** tools speak a working protocol but -have not been through formal regression (best-effort). A third group cannot connect at all, for reasons -on the tool's own side. +Four tiers, each a different claim: + +- **Tested** — exercised against SoftClient4ES. +- **Compatible** — speaks a working protocol, but has not been through formal regression (best-effort). +- **Unproven** — a connection path exists on paper, but nobody has connected it yet. Not a promise. +- **No path today** — cannot connect without software that does not exist, for reasons on the tool's + own side. | Tool | Status | Path | Guide | |---|---|---|---| @@ -16,13 +20,12 @@ on the tool's own side. | DBeaver | Tested | JDBC or Arrow Flight SQL | https://softclient4es.dev/integrations/dbeaver/ | | Grafana | Tested (via Arrow Flight SQL) | Arrow Flight SQL | https://softclient4es.dev/integrations/grafana/ | | Tableau | Compatible (not formally tested) | JDBC | https://softclient4es.dev/integrations/tableau/ | -| Metabase | **No supported connection path** — Metabase has no generic JDBC database type; anything else needs a community driver plugin | — | https://softclient4es.dev/integrations/metabase/ | -| Power BI | **No supported connection path** — Power Query has no JDBC connector (its generic connectors are ODBC and OData); the one candidate, a generic Arrow Flight SQL ODBC driver, is unproven | — | https://softclient4es.dev/integrations/power-bi/ | -| Looker | **No supported connection path** — Looker-maintained drivers only, with a per-dialect JDBC parameter allowlist. Structural, not commercial: a licence would not close it | — | — | -| dbt | **No supported connection path** — a dedicated adapter plugin is mandatory; no generic JDBC/ODBC adapter exists and no SoftClient4ES adapter exists | — | — | +| Power BI | **Unproven** — Power Query's generic connectors are ODBC and OData, never JDBC. The one candidate path is a generic Arrow Flight SQL ODBC driver pointed at the sidecar; it has not been connected yet | ODBC (unproven) | https://softclient4es.dev/integrations/power-bi/ | +| Metabase | **No path today** — Metabase has no generic JDBC database type. Connecting needs a community driver plugin, and we do not ship one | — | https://softclient4es.dev/integrations/metabase/ | +| Looker | **No path today** — Looker-maintained drivers only, with a per-dialect JDBC parameter allowlist. Structural, not commercial: a licence would not close it | — | — | -*(Each "no supported connection path" blocker was checked against the vendor's own connection -documentation — Metabase, Microsoft Power Query, Looker and dbt — on 2026-08-31 and 2026-09-01.)* +*(Each blocker was checked against the vendor's own connection documentation — Metabase, Microsoft +Power Query and Looker — on 2026-08-31 and 2026-09-01.)* ## Honest-gap note diff --git a/documentation/sql/known_limitations.md b/documentation/sql/known_limitations.md index 33c756bb..ee72efe1 100644 --- a/documentation/sql/known_limitations.md +++ b/documentation/sql/known_limitations.md @@ -10,23 +10,26 @@ SoftClient4ES runs a large, practical subset of ANSI SQL on Elasticsearch — in Two different things can stop a BI tool here, and it is worth separating them. +### One tool has a path nobody has walked yet + +- **Power BI** — Power Query's generic connectors are ODBC and OData, never JDBC. The one candidate path + is a generic Arrow Flight SQL ODBC driver pointed at the sidecar. It is **unproven**: nobody has + connected it, so treat it as a lead to follow rather than a supported route. + See [Power BI](../client/bi_tools.md). + ### Some tools cannot connect at all — and that is on their side - **Metabase** — no generic JDBC database type exists; anything not on Metabase's own driver list needs a community driver plugin, which is code nobody has written for SoftClient4ES. See [Metabase](../client/bi_tools.md). -- **Power BI** — Power Query has no JDBC connector; its generic connectors are ODBC and OData. The only - candidate path is a generic Arrow Flight SQL ODBC driver, which is unproven. See [Power BI](../client/bi_tools.md). - **Looker** — Looker connects only through drivers it maintains itself, and it allowlists JDBC parameters per dialect, so a customer-supplied driver cannot be introduced. This gap is **structural, not commercial** — a licence would not close it. -- **dbt** — dbt requires a dedicated adapter plugin per platform. There is no generic JDBC or ODBC adapter, - and no SoftClient4ES adapter. -None of these is a gap we can close from our side: each one needs either a change by the vendor or a driver -or adapter plugin that nobody has written. +Neither is a gap we can close from our side: each needs either a change by the vendor or a driver plugin +that nobody has written. -*(Each blocker checked against the vendor's own connection documentation — Metabase, Microsoft Power Query, -Looker and dbt — on 2026-08-31 and 2026-09-01.)* +*(Each blocker checked against the vendor's own connection documentation — Metabase, Microsoft Power Query +and Looker — on 2026-08-31 and 2026-09-01.)* ### Tools that connect, but generate SQL we do not accept yet