Skip to content

kernel: distribute the kernel via per-platform bindings modules (go get, no build step, all 5 platforms) - #440

Open
msrathore-db wants to merge 6 commits into
mainfrom
kernel-nested-modules
Open

kernel: distribute the kernel via per-platform bindings modules (go get, no build step, all 5 platforms)#440
msrathore-db wants to merge 6 commits into
mainfrom
kernel-nested-modules

Conversation

@msrathore-db

@msrathore-db msrathore-db commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

What

Reworks the kernel/SEA backend's native-library delivery so a kernel opt-in go get build links a prebuilt archive with no Rust toolchain and no make kernel-lib step, on all 5 supported platforms.

  • Consumes the separate databricks/databricks-sql-kernel-bindings repo (one nested Go module per platform, lib/<os>_<arch>), replacing the old in-tree kernellib/ modules. go.mod requires the 5 bindings modules; go.sum pinned.
  • Per-platform cgo shims (cgo_<os>[_<arch>].go) blank-import the matching bindings module for its #cgo LDFLAGS link side-effect.
  • Adds darwin/amd64 and linux/arm64 (previously unsupported) → now supports linux amd64/arm64, darwin arm64/amd64, windows amd64. cgo_unsupported.go updated to match.
  • ABI-version handshake: calls kernel_abi_version() (databricks/databricks-sql-kernel#285) once at OpenSession, failing with a clear "C ABI mismatch" error if the pinned bindings archive doesn't match this driver build — instead of crashing on a mismatched struct. Since the archive is now separately versioned, this is the safety check that catches a stale/wrong pin.
  • Docs: docs/RELEASING.md (versioning + path-prefixed tags) and README "Cloning" (the committed-binary + --filter=blob:none guidance) — plus the committed C header the cgo layer includes at compile time.

CUJ

  • Thrift (default): unchanged — pure Go, CGO_ENABLED=0, no kernel bytes fetched.
  • Kernel/SEA (opt-in): go get + CGO_ENABLED=1 go build -tags databricks_kernel — pulls only the target platform's archive at the pinned version. No build step, no Rust.

Verification

Runtime-verified end-to-end (real SELECT 1 + full data-type suite through the kernel against a live warehouse) on all 5 platforms:

  • darwin/arm64 (native Mac), darwin/amd64 (Rosetta), linux/amd64 (native), linux/arm64 (qemu), windows/amd64 (native Windows VM).
  • Built from the real bindings modules via go get (the true consumer path); the ABI check logs kernel ABI version ok: 1 and connect proceeds.

Dependencies

  • Requires databricks/databricks-sql-kernel#285 (kernel_abi_version()) merged, and the bindings archives rebuilt from that merged kernel main. Current bindings v0.1.0 are built from the kernel branch carrying that commit; they'll be rebuilt from kernel main once Prepare for v1.9.0 release #285 lands.

This pull request and its description were written by Isaac.

Commit the prebuilt darwin/arm64 kernel static archive in a nested per-platform
Go module (internal/backend/kernel/kernellib/darwin_arm64) plus the C header, so
a kernel opt-in build works from 'go get' with no 'make kernel-lib' step. The
nested module is imported for its cgo link side-effect only under
darwin && arm64 && databricks_kernel, so a pure-Go Thrift build or a non-darwin
build never compiles or downloads it.

Co-authored-by: Isaac

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium

Distribution-model change (nested per-platform kernel module) looks sound for the in-tree build: build constraints on cgo_darwin.go and link.go match, the default Thrift build stays CGO-free, and .gitignore/header handling is consistent. One Medium concern: the require v0.0.0 + local replace won't resolve for external go get consumers once a release is cut, since replace isn't transitive and module-graph resolution ignores build tags — the release must be gated on publishing/tagging the nested module.

Comment thread go.mod Outdated
// only the archive for the platform it targets (and nothing at all for a
// pure-Go Thrift build). The replace pins them to the in-tree directories; when
// published, the require versions are what a `go get` consumer resolves.
require github.com/databricks/databricks-sql-go/internal/backend/kernel/kernellib/darwin_arm64 v0.0.0

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — The nested module is wired with require .../kernellib/darwin_arm64 v0.0.0 + a local replace. This works for in-tree builds, but note two things that undercut the PR's "works straight from go get" goal for external consumers:

  1. replace is not transitive. A downstream project that does go get github.com/databricks/databricks-sql-go ignores this repo's replace directive entirely (replace is honored only in the main module). It sees only the bare require .../darwin_arm64 v0.0.0.

  2. Module-graph resolution is build-tag-independent. MVS must load the go.mod of every required module to build the graph, even for a pure-Thrift (CGO_ENABLED=0, no tag) build that never compiles a file from the nested module. Because v0.0.0 is not a published/tagged version of the nested module, that resolution would fail for all consumers — not just kernel builds — with an "unknown revision" error, once a release of this repo is cut carrying this go.mod.

The PR description acknowledges this ("when published, the require versions are what a go get consumer resolves"), so this is a known follow-up rather than a defect in the in-tree workflow. Flagging so the release that publishes this is gated on: tagging the nested module at a real version and updating the require to match. Until then, external go get of a tagged release would break even for Thrift-only users.

Add sync-kernel-libs.yml: a manually-triggered workflow that downloads the
per-platform libdatabricks_sql_kernel.a archives built by the kernel repo's
build-c-abi-libs workflow for a chosen kernel release tag, verifies each
checksum, places them into the nested internal/backend/kernel/kernellib/<platform>
modules, moves KERNEL_REV to the tag's commit, and opens a sync PR.

This is the consumer half of the committed-per-platform kernel distribution: it
keeps the committed archives (and header + KERNEL_REV) in lockstep with a kernel
release without a manual copy, while leaving the actual bump as a reviewed PR.

Co-authored-by: Isaac

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 High · 1 Low

Solid, well-documented distribution change; the nested-module wiring is correct for in-repo builds. Main concern (High): the unconditional require ... v0.0.0 + local replace in go.mod resolves in-repo but not for external go get consumers (replace is ignored downstream), which would break the default build until the nested module is actually published/tagged — contradicting the README's "works straight from go get" claim. One Low note on the sync workflow swallowing go mod verify failures. Nit: the verify step's comment says it "type-checks the darwin build" but the command builds linux/amd64.

Comment thread go.mod Outdated
// only the archive for the platform it targets (and nothing at all for a
// pure-Go Thrift build). The replace pins them to the in-tree directories; when
// published, the require versions are what a `go get` consumer resolves.
require github.com/databricks/databricks-sql-go/internal/backend/kernel/kernellib/darwin_arm64 v0.0.0

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — The nested module is pulled in with an unconditional require ... v0.0.0 plus a replace => ./internal/backend/kernel/kernellib/darwin_arm64. This resolves correctly for in-repo builds (make test, go build ./..., CI), because replace short-circuits the placeholder v0.0.0.

But replace directives are ignored in any module other than the main module (Go modules reference). So for an external consumer that does go get github.com/databricks/databricks-sql-go, the replace does not apply and Go must resolve .../kernellib/darwin_arm64@v0.0.0 from the proxy/VCS. Since the top-level require is unconditional, this module's go.mod is loaded for every consumer build graph — including a pure-Thrift, CGO_ENABLED=0 build that never compiles a kernellib file. Unless the tag internal/backend/kernel/kernellib/darwin_arm64/v0.0.0 actually exists in the published repo, go get/go build fails at module resolution (unknown revision v0.0.0) for all consumers, not just kernel users.

This directly contradicts the README section added in this PR ("works straight from go get with no build step") and the PR's headline claim. The PR text acknowledges "when published, the require versions are what a go get consumer resolves" — so the gap is known, but as it stands the merged+tagged state would break the default build for downstream consumers. Please confirm the publish/tagging plan makes v0.0.0 (or the eventual pinned version) resolvable before the README advertises frictionless go get, or gate the require so a Thrift build doesn't force resolution.

Comment thread .github/workflows/sync-kernel-libs.yml Outdated
branch="sync/kernel-${KERNEL_TAG}"
git config user.name "databricks-ci"
git config user.email "ci@databricks.com"
git checkout -b "$branch"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — go mod verify || true swallows any failure, so a genuinely corrupt/tampered module cache would not fail this "verify module graph resolves" step — only the subsequent go build gates the job. The comment frames this step as catching a broken go.mod/missing module; the || true defeats the go mod verify half of that intent. Consider dropping || true (or logging explicitly why a failure is tolerated) so the checksum verification of the freshly-committed archive can actually fail the sync PR.

Remove sync-kernel-libs.yml. The kernel repo's build-c-abi-libs workflow now
PUSHES the compiled per-platform archives into this repo and opens the sync PR
(see databricks-sql-kernel#244). Push keeps the cross-repo credential in the
private kernel repo (a token with PR-write on this public driver only), instead
of putting a kernel-source-read token in this public repo — which the pull model
would have required.

Co-authored-by: Isaac

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium

Solid, well-documented build/distribution change; the build constraints on cgo_darwin.go and the nested link.go match, and the Thrift-build exclusion reasoning is sound. One medium concern: the require v0.0.0 + local replace only makes the kernel backend build in-tree — replace is ignored downstream and v0.0.0 isn't resolvable, so the "works from go get" claim isn't met for external consumers until the nested module is published at a real version.

Comment thread go.mod Outdated
// only the archive for the platform it targets (and nothing at all for a
// pure-Go Thrift build). The replace pins them to the in-tree directories; when
// published, the require versions are what a `go get` consumer resolves.
require github.com/databricks/databricks-sql-go/internal/backend/kernel/kernellib/darwin_arm64 v0.0.0

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — The nested-module wiring here works only for in-tree builds — which is all the PR's verification actually exercised (go build ./... run from within this repo, where the replace applies). It does not deliver the headline "works straight from go get, no build step" for external consumers:

  • replace directives are ignored for dependencies. When someone adds databricks-sql-go as a dependency, only their main module's replace directives take effect; the replace on line 62 of this repo's go.mod is dropped. So a downstream go build -tags databricks_kernel for darwin/arm64 will try to resolve .../kernellib/darwin_arm64 v0.0.0 from the module proxy, not from ./internal/....
  • v0.0.0 is not a resolvable version. There is no internal/backend/kernel/kernellib/darwin_arm64/v0.0.0 tag, so that resolution fails for a consumer. go-duckdb's model (cited in the PR) requires the nested modules to be published as real tagged versions that the parent requires directly — not a v0.0.0 placeholder held together by an in-tree replace.

Net effect: the go get path the PR is built around is currently only proven for builds run inside this checkout. Recommend either (a) tempering the README/PR claim to "builds from a repo checkout" until the nested modules are published and required at real versions, or (b) documenting the publish+version-bump step as a hard prerequisite before the kernel backend is advertised as go-get-installable. The Thrift-build verification is unaffected (the import is build-tag-excluded), so this only concerns the kernel-tag darwin/arm64 consumer path.

…elease docs)

Rework the darwin/arm64 nested-module distribution so a consumer's versioned
'go get' resolves the kernel archive correctly:

- driver go.mod now requires the nested module at a REAL version (v0.0.1) with a
  replace pointing at the in-tree source. Per the Go module spec a dependency's
  replace is ignored by consumers, so it is safe to ship: local/dev builds use
  the committed archive, while a downstream 'go get' resolves the published
  version from the proxy. (The earlier v0.0.0 pseudo-version could not be
  resolved by a consumer.)
- add docs/RELEASING.md documenting path-prefixed-tag publishing
  (internal/backend/kernel/kernellib/<platform>/vX.Y.Z), how the driver require
  version pins the kernel per driver release, and the go.sum refresh step.
- README: note per-platform + versioned resolution and link RELEASING.md.

Verified: thrift + kernel build; TestKernelE2ESelect1 and TestKernelE2EDataTypes
pass on pecotesting; a proxy simulation confirms a consumer resolves the nested
module at the pinned version pulling only one platform (~17MB), and that a
dependency's replace does not leak to consumers.

Co-authored-by: Isaac

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good — clean, well-documented distribution change with one low-severity note. The nested-module wiring is correct: build tags on cgo_darwin.go and link.go match exactly (cgo && databricks_kernel && darwin && arm64), the blank import collects the archive's LDFLAGS at link, the committed header sits in the driver module's own zip while the 59 MB .a lives in the excluded nested-module subtree, and .gitignore correctly un-ignores include/ without matching the still-ignored lib/. The only concern is the placeholder require v0.0.1 versus the documented lockstep-version model (F1).

Comment thread go.mod Outdated
// applies when this repo is the main module — so it is safe to ship: it never
// affects a downstream `go get`, which always resolves the published version.
require github.com/databricks/databricks-sql-go/internal/backend/kernel/kernellib/darwin_arm64 v0.0.1

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — The require pins the nested module at v0.0.1, but the adjacent comment (and docs/RELEASING.md) assert these versions are "REAL published versions ... bumped in lockstep with each driver release," so that go get databricks-sql-go@vX.Y.Z transitively pins the matching kernel archive and "upgrading the driver is what moves the kernel version."

Right now v0.0.1 is a placeholder that only builds because the local replace shadows it in this repo. That's fine for this development-phase PR, but the code contradicts the documented model, and it is load-bearing at release time: if a driver release tag ships this go.mod verbatim, a downstream -tags databricks_kernel build on darwin/arm64 will (a) require a published internal/backend/kernel/kernellib/darwin_arm64/v0.0.1 tag to exist on the proxy, and (b) freeze the kernel at v0.0.1 regardless of driver version — defeating the lockstep pinning the comment promises. Worth a TODO/note that this must be bumped and tagged (RELEASING.md steps 2–3) before it goes out in a tagged release, so the placeholder isn't shipped by accident.

Rework the kernel backend's native-lib delivery to the separate
github.com/databricks/databricks-sql-kernel-bindings repo (one nested Go module
per platform, lib/<os>_<arch>), replacing the in-tree kernellib/ nested modules.

- go.mod requires the 5 bindings modules at v0.1.0 (darwin arm64/amd64, linux
  amd64/arm64, windows amd64); go.sum pinned.
- Per-platform cgo shims (cgo_<os>[_<arch>].go) blank-import the matching
  bindings module for its #cgo LDFLAGS link side-effect. Adds darwin_amd64 and
  linux_arm64 (previously unsupported).
- cgo_unsupported.go now excludes all 5 supported targets.
- Removed the in-tree internal/backend/kernel/kernellib/ modules + committed .a.

Verified on darwin/arm64: thrift build clean (no kernel fetch); kernel build
links the external bindings module; TestKernelE2ESelect1 + TestKernelE2EDataTypes
pass on pecotesting with the driver built from FETCHED bindings (no local
replace) — the real consumer path.

Bindings archives currently built from kernel PoC branch ff1e965 (has
kernel_abi_version) for testing; shipping revision follows via build-c-abi-libs CI.

Co-authored-by: Isaac

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 2 High · 1 Medium · 1 Low

The code side is coherent — the kernel shims blank-import the external databricks-sql-kernel-bindings/lib/<platform> modules and go.mod/go.sum match. But the PR's new prose (README "Cloning the repository", docs/RELEASING.md, and the .gitignore comment) documents a different, unshipped design: in-repo nested modules under internal/backend/kernel/kernellib/<platform> with committed .a files and replace directives — none of which exist (verified: no such dir, no replace in go.mod). Two high-severity doc/impl divergences (F1, F2), plus an orphan !databricks_kernel_dynlib build tag on darwin/arm64 only that can produce an opaque link failure (F3). The PR title/description itself ("nested per-platform module committed in-repo") reflects the abandoned design and should be reconciled with the external-repo reality.

Comment thread docs/RELEASING.md

```
github.com/databricks/databricks-sql-go (the driver module)
└── internal/backend/kernel/kernellib/<platform>/ (one NESTED module per platform)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — This new release doc describes a distribution model that does not match the shipped code. It documents in-repo nested modules at internal/backend/kernel/kernellib/<platform>/ with committed .a files, and a require + replace ... => ./internal/backend/kernel/kernellib/<platform> pair in the driver go.mod.

But the actual implementation (go.mod:56-72, and every cgo_*.go shim) depends on the external, separate repo github.com/databricks/databricks-sql-kernel-bindings/lib/<platform> — there is no internal/backend/kernel/kernellib/ directory, no committed archive in this repo, and no replace directive in go.mod (verified: grep '^replace' go.mod returns nothing).

So the entire "module layout", "How versioning works" (the replace lines), and "Publishing: path-prefixed tags" sections instruct a release process against modules that live in a different repository. Step 2 ("keep the matching replace lines") and step 3 (tagging internal/backend/kernel/kernellib/<platform>/vX.Y.Z in this repo) would be actively wrong. Please rewrite this doc around the external databricks-sql-kernel-bindings model that go.mod actually uses, or split out which steps happen in which repo.

Comment thread README.md

This repo commits a small number of **prebuilt kernel binaries** (per-platform
`libdatabricks_sql_kernel.a`, ~62 MB each, each in its own nested module under
`internal/backend/kernel/kernellib/<platform>`) so that the SEA/kernel backend

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — The new "Cloning the repository" section states this repo "commits a small number of prebuilt kernel binaries (per-platform libdatabricks_sql_kernel.a, ~62 MB each, each in its own nested module under internal/backend/kernel/kernellib/<platform>)" and gives a git sparse-checkout example selecting internal/backend/kernel/kernellib/darwin_arm64.

None of that exists in the tree: the archives live in the external databricks-sql-kernel-bindings repo (go.mod:68-72), not in-repo, and there is no kernellib/ path to sparse-checkout here. The --filter=blob:none rationale ("skip the committed-archive history") therefore doesn't apply to this repo as shipped — a direct clone of this repo pulls no large kernel .a at all. This section will mislead contributors into a sparse-checkout that keeps a nonexistent path. Align it with the external-module reality, or drop the committed-binary framing.

@@ -1,20 +1,11 @@
//go:build cgo && databricks_kernel && darwin && arm64
//go:build cgo && databricks_kernel && !databricks_kernel_dynlib && darwin && arm64

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — This shim's build constraint gained && !databricks_kernel_dynlib, but none of the other four shims (cgo_darwin_amd64.go, cgo_linux.go, cgo_linux_arm64.go, cgo_windows.go) carry it, and grep 'dynlib' finds no complementary dynlib shim anywhere. The comment still claims "Same build constraint as that module's prebuilt.go" — which now can't be true if only darwin/arm64 has this extra term.

Concrete consequence: building darwin/arm64 with -tags databricks_kernel,databricks_kernel_dynlib excludes this file (via !databricks_kernel_dynlib), while cgo_unsupported.go also excludes darwin/arm64 (its !(darwin && arm64) term). So no link shim is compiled, the external module is never blank-imported, and cgo.go's kernel_* C symbols fail at final link with the exact opaque "undefined reference" dump the unsupported-guard exists to pre-empt at compile time. This looks like a stray/incomplete edit (a planned dynlib variant with no landing file). Either drop the !databricks_kernel_dynlib term, or add the dynlib shim + extend cgo_unsupported.go so the guard still fires.

Comment thread .gitignore
#
# The paths below remain ignored: they are scratch dirs still used by
# `make kernel-lib` for platforms not yet committed (linux/windows) and for
# source builds — never committed.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — This comment says the committed distribution model "commits ... each platform's prebuilt archive under kernellib// (nested modules), so go get needs no build step." That describes the same in-repo model that isn't implemented (see F1/F2) — nothing is committed under kernellib/ here. The actual ignore-rule change (un-ignoring include/ so the committed header lands) is correct; only the explanatory comment is inaccurate. Trim it to reference the committed header + the external bindings repo.

(Anchored to the nearest changed line — see the description for the exact location.)

The driver links a PREBUILT per-platform kernel archive delivered by the
separate databricks-sql-kernel-bindings modules, so the archive and this driver
build can drift (a stale/wrong pinned bindings version). Call kernel_abi_version()
(added in databricks-sql-kernel#285) once at OpenSession and fail with a clear
'C ABI mismatch' error instead of crashing on the first mismatched
struct/function. expectedKernelABIVersion (=1) is bumped in lockstep with the
pinned bindings version on any breaking C ABI change.

cgo-only (gated by the databricks_kernel tag); the pure-Go Thrift build is
unaffected. Verified on pecotesting: connect logs 'kernel ABI version ok: 1' and
TestKernelE2ESelect1 passes.

Co-authored-by: Isaac
@msrathore-db msrathore-db changed the title kernel: distribute darwin/arm64 lib as nested per-platform module (go get, no build step) kernel: distribute the kernel via per-platform bindings modules (go get, no build step, all 5 platforms) Aug 23, 2026

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 3 Medium · 1 Low

Mechanically the cgo rewiring (blank-import the external bindings module per platform, ABI-version handshake at OpenSession) is sound, and the committed header + ABI check are reasonable. The main issue is that the PR pivoted the distribution model to the external databricks-sql-kernel-bindings modules (go.mod requires them, with no replace and no in-tree kernellib/), but the new docs — all of docs/RELEASING.md and the README "Cloning" section — still describe the abandoned in-tree kernellib/<platform> nested-module model, which would misdirect a release (F1/F2). One stray build-tag (!databricks_kernel_dynlib on darwin/arm64 only, with no matching dynlib file or guard update) can produce the exact opaque linker error the unsupported guard is meant to prevent (F3). Note also the cgo.go package doc comment (unchanged, so not inline-anchorable) still describes the make kernel-lib / ".gitignore'd; nothing kernel-built is com

[...truncated to keep verdict scannable]

Comment thread docs/RELEASING.md
also carries a `replace` pointing at the in-tree source:

```
require github.com/databricks/databricks-sql-go/internal/backend/kernel/kernellib/darwin_arm64 v1.2.3

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — This entire release doc describes an in-tree module layout that the shipped code does not use. It says the driver is a multi-module repo with one nested module per platform at internal/backend/kernel/kernellib/<platform>/, that go.mod carries a require plus a replace pointing at the in-tree source, and that release means tagging internal/backend/kernel/kernellib/<platform>/vX.Y.Z.

But the actual go.mod in this PR requires the external github.com/databricks/databricks-sql-kernel-bindings/lib/<platform> modules, with no replace directives and no in-tree kernellib/ modules anywhere in the repo (grep confirms kernellib appears only in docs/comments). The PR description itself says these external bindings modules are "replacing the old in-tree kernellib/ modules."

As written, RELEASING.md is instructions for the previous design: step 2 ("keep the matching replace lines") and step 3 ("tag internal/backend/kernel/kernellib/<platform>/vX.Y.Z") would both fail — there are no replace lines and no such modules to tag. A maintainer following this doc would mis-release. Rewrite it around the external bindings repo (versions pinned via the require on databricks-sql-kernel-bindings/lib/*; publishing/tagging happens in that repo).

Comment thread README.md

This repo commits a small number of **prebuilt kernel binaries** (per-platform
`libdatabricks_sql_kernel.a`, ~62 MB each, each in its own nested module under
`internal/backend/kernel/kernellib/<platform>`) so that the SEA/kernel backend

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — The new "Cloning the repository" section states this repo commits per-platform prebuilt binaries (libdatabricks_sql_kernel.a, ~62 MB each) "in its own nested module under internal/backend/kernel/kernellib/<platform>", and the --filter=blob:none / --sparse guidance selects paths under internal/backend/kernel/kernellib/....

That is no longer true for the model this PR ships: the archives now live in the external databricks-sql-kernel-bindings repo (pulled by go get), not committed here — there is no kernellib/ tree in this repo. So the stated motivation ("committed binaries cannot be delta-compressed... a full clone accumulates their whole history") and the sparse-checkout example paths point at directories that don't exist in this repo. This will mislead contributors. Update the section to reflect that the kernel archives are consumed as external modules, and drop/revise the committed-binary and sparse-checkout guidance accordingly.

@@ -1,20 +1,11 @@
//go:build cgo && databricks_kernel && darwin && arm64
//go:build cgo && databricks_kernel && !databricks_kernel_dynlib && darwin && arm64

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — This shim's build constraint adds !databricks_kernel_dynlib, but that tag appears nowhere else in the repo: there is no databricks_kernel_dynlib-gated link file, the other four shims (cgo_darwin_amd64.go, cgo_linux.go, cgo_linux_arm64.go, cgo_windows.go) do not carry the negation, and cgo_unsupported.go's exclusion list is not dynlib-aware.

Consequence: building darwin/arm64 with -tags 'databricks_kernel databricks_kernel_dynlib' excludes this file (its !databricks_kernel_dynlib), while cgo_unsupported.go is still excluded too (its !(darwin && arm64) is false on that target). So no shim compiles — the bindings module is never blank-imported, its #cgo LDFLAGS are never collected, and the link fails with the opaque undefined reference to kernel_* that cgo_unsupported.go exists precisely to prevent. Either drop this stray !databricks_kernel_dynlib negation, or complete the dynlib variant consistently across all platforms and teach the unsupported guard about it.

Comment thread .gitignore
#
# The paths below remain ignored: they are scratch dirs still used by
# `make kernel-lib` for platforms not yet committed (linux/windows) and for
# source builds — never committed.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — This comment now describes a committed-archive layout ("each platform's prebuilt archive under kernellib// (nested modules)... intentionally NOT ignored") that the shipped model abandons — the archives live in the external bindings repo, and there is no kernellib/ tree here. The include/ un-ignore (committed header) is still correct, but the kernellib/<platform>/ prose is stale. Trim the comment to match reality (committed header only).

(Anchored to the nearest changed line — see the description for the exact location.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant