Repository navigation
feat(odbc): ODBC in the release wheels, typed cursor bounds, live driver tests on Linux and Windows - #198
Open
kvlonge wants to merge 32 commits into
Open
feat(odbc): ODBC in the release wheels, typed cursor bounds, live driver tests on Linux and Windows#198kvlonge wants to merge 32 commits into
kvlonge wants to merge 32 commits into
Conversation
Build sqlload and sqlload-cli wheels for Windows x86_64 (MSVC) and macOS x86_64 (macos-15-intel), and test Windows in CI. - librdkafka builds with CMake on Windows, where its configure script and make are missing. - Local paths: drive paths (C:\data), UNC shares (\\server\share), \\?\ paths and file:///C:/ and file://server/share URLs are local folders on Windows; Unix paths are unchanged. - A postgres_cdc replication session's relay on Windows is a 127.0.0.1 port, since the replication client cannot reach a socket file there. It relays only the connection whose startup message carries a random per-session secret, drops every other connection before a byte reaches the server, gives each 5 seconds, and never forwards the secret. - Verify gains a Windows job: unit tests, then the executable end to end over drive paths, a UNC share, psqlODBC against the runner's Postgres, and a postgres_cdc load streamed through the relay. - Publish builds and smoke-tests every wheel on pull requests that change packaging, and publishes only when dispatched.
…nto session_relay A relay refusal no longer disables tokio::select!'s join arm, which could leave an admitted client unread until the relay's deadline when two refusals finished together; a test now opens refused connections first. The Unix socket-folder and Windows loopback relays live in session_relay. On Windows, folder names under a filesystem connection match ignoring case, and file://localhost/C:/ names the drive. Publish's PR trigger also covers the workspace manifest and lockfile.
russh reaches ssh-agent through SSH_AUTH_SOCK's Unix socket only on Unix; on Windows it now opens the pipe SSH_AUTH_SOCK names, or else the OpenSSH agent service's. The test kit sets Unix key-file permissions only on Unix.
…and isn't tested there
…erals
An ODBC source's date and wall-clock timestamp cursor bounds are now {d '...'}
and {ts 'YYYY-MM-DD HH:MM:SS[.fff]'} escape literals, which each driver turns
into its database's own syntax (OpenEdge compares no string with a timestamp).
Cursor flags and cursor_start also accept those literals.
WIP for CHI-1236: driver-manager linkage, wheels and live driver tests follow.
…lds; releases never cancel
… folder-swap gap Windows has A Windows temp path in a TOML basic string reads \U as a unicode escape, so the command tests now quote the root with single quotes. Windows can't swap two folders in one rename, so a reader may briefly see no folder there, as the files target docs now say.
…sions, list tables without schemas MySQL Connector/ODBC reports a backtick as its identifier quote even when the session reads ANSI double quotes, so the connection now asks the database once. A driver without schemas is asked for its tables with no schema pattern, which MySQL's NO_SCHEMA default otherwise refuses. Adds MySQL and SQL Server containers to the testkit and a script installing the three drivers. WIP for CHI-1236: container tests follow.
…on a cp1252 console
…, and test SQL Server, MySQL and Postgres drivers
An ODBC source's date and wall-clock timestamp cursor bounds are now bound as
ODBC DATE and TIMESTAMP parameters (seven fractional digits) instead of {d}/{ts}
escapes: SQL Server reads {ts} as a datetime, which rounds milliseconds, so a
cursor resuming from .997 read that row again. Timestamps a driver describes
with fewer fractional digits than it returns (MySQL Connector/ODBC describes
DATETIME(n) with none) are now fetched in microseconds instead of truncated.
Container tests load a millisecond timestamp cursor and a date cursor through
Microsoft's ODBC Driver 18, MySQL Connector/ODBC and psqlODBC, and check the next
load reads exactly the rows after each boundary. The Verify container job
installs the three drivers with .github/scripts/install_odbc_drivers.sh.
WIP for CHI-1236: driver-manager linkage, wheels, Windows drivers and docs follow.
…the column's type SQL Server compares a legacy datetime column with a datetime2 parameter at the column's exact 1/300 s ticks, which its driver reads rounded to milliseconds, so a cursor resuming from .123 (stored .12333) read that row again, and an inclusive bound at .997 (stored .99667) missed it. No TIMESTAMP bind precision avoids this: the driver sends datetime2 for every one. When SQL Server's driver describes the parameter with one to three fractional digits, the bound is now text with that many digits, which SQL Server converts to the column's own type. Every other timestamp bound stays a TIMESTAMP(7) parameter, exact against every precision on all three live drivers; a lower declared precision than the value holds fails with 22008, so it is not derived from the column. WIP for CHI-1236.
…er's A Windows main thread, sqlload.exe's or python.exe's, has 1 MB of stack, which a debug build's copy into a files target outgrew (0xC00000FD, reproduced on Linux with ulimit -s 1024). Every command now runs on a scoped thread with a Linux main thread's default 8 MiB, whoever calls it.
… 18 on LocalDB, MySQL Connector/ODBC, psqlODBC The Windows job installs the three drivers from pinned, SHA-256-checked MSIs (.github/scripts/install_odbc_drivers.ps1), starts the image's SQL Server LocalDB and a MySQL 8.0 server, and windows_e2e.py --cursor-drivers runs the container tests' cursor scenario through each driver: a datetime2(3) and a legacy datetime table on LocalDB, DATETIME(3) on MySQL and timestamp(3) on Postgres, the next load reading exactly the rows after each boundary. WIP for CHI-1236.
…w cursor bounds are bound WIP for CHI-1236: the release version is checked before the pull request.
…ested wheel A table function yields rows and then blocks; the load gets a real Ctrl-C (SIGINT, or a console Ctrl-C event on Windows) and must end at once as the native executable does, with no traceback and nothing committed, and the next load must write every row once. Commands now run on a thread of their own rather than Python's main thread, so this checks Ctrl-C still reaches them.
object_store keeps a local store's root as a file:// URL, and the url crate reads
file://localhost/share/... back as a drive path, so every file on a \\localhost\ share failed to
convert ("Unable to convert URL ... to filesystem path"). The store now opens such a share as
\\127.0.0.1\share, the same server; what SQLLoad prints keeps the path as written. The Windows
end-to-end checks also write through the machine's own name, as a share on another server is named.
…istribution The generic glibc tarball loads its own newer libssl.so.3, which fails with OPENSSL_3.2.0 not found in a process where msodbcsql18 or psqlODBC already loaded Ubuntu 24.04's OpenSSL 3.0, as the container tests do. MySQL's package for the release links the system OpenSSL. The docs say so for users, and the site's check for unshipped systems no longer reads the ODBC page, which names the drivers SQLLoad is tested with.
# Conflicts: # crates/sqlload/tests/loader/test_types.rs
…ve console does, with a control The Windows wheel's Ctrl-C check found the load still running 15 s after the event. A plain load installs no Ctrl-C handler of its own, and a Windows child inherits its parent's "ignore Ctrl-C" flag, which the runner's processes may carry. Each interrupted child now clears that flag first, as a process in an interactive console has it clear. Before the load, a plain Python process with Ctrl-C's default action gets the same event: once as the runner starts it (printed) and once with the flag cleared (it must end), so a failing load is the load's fault, not the check's.
…y on time cursors, literals for pinned cursors
- A SQL Server legacy datetime bound sent as text now uses the T separator. With a space, SQL Server
reads month and day by the login's language, so a british session read a 2026-01-05 maximum as
1 May and the next load silently skipped every row up to it. The live legacy-datetime case now
connects with Language=British and a 5 January maximum, and fails with the old format.
- When the driver can't describe a SQL Server timestamp parameter, the bound goes as seven-digit ISO
text: exact for datetime2, refused by datetime, rather than a datetime2 bind that misreads it.
- {ts '...'} and {d '...'} are read only for timestamp and date cursors; a text cursor keeps such a
value as it is.
- A cursor column pinned to a type compares with a quoted literal again, as the pinned type may not
be the source column's own.
- The catalog asks for every schema and falls back to no schema pattern only when that is refused
and the driver reports no schema name length, so a driver reporting 0 as no maximum still lists
its schemas.
- Linux installs msodbcsql18 18.7.1.1, the release Windows installs, and registers MySQL's driver
file by its exact name.
…its step The runner ends the processes a step starts when the step ends, so the mysqld started with Start-Process was gone by the end-to-end step (connection refused on 3307). It now runs as the sqlload-mysql service from an option file. The image's MySQL is 8.4, not 8.0; the docs say so.
…nd macOS Both wheels now build the odbc feature. On Linux and macOS they bundle the unixODBC 2.3.14 driver manager, built from its pinned source release (SHA-256 checked) with libltdl built in, linked dynamically and installed as a separate, replaceable shared library by maturin's repair. It reads odbcinst.ini and odbc.ini in /etc on Linux and Homebrew's etc on macOS, or the folder ODBCSYSINI names, so SQLLoad needs no system unixODBC. Windows uses odbc32.dll. Every wheel carries unixODBC's LGPL-2.1 text and a notice naming its exact source, and a check asserts it, and that each Linux and macOS wheel bundles exactly one libodbc. Each wheel is smoke-tested against a made-up driver: on Linux in a container with no unixODBC, on macOS after removing Homebrew's copy (registered under ODBCSYSINI and in the default odbcinst.ini), and on Windows through odbc32.dll. macOS uses maturin's repair rather than delocate, which points an installed sqlload-cli binary at a folder that doesn't exist.
…n the sdists, keep sdists free of ODBC - The bundled unixODBC is configured with a driver search path (Debian's multiarch odbc folders, /usr/lib64 and Homebrew's lib), as Debian's own build is: Debian and Ubuntu driver packages register a file name alone (Driver=psqlodbcw.so), which it could not load. The Linux smoke test now loads a library registered by file name in /usr/lib/<arch>-linux-gnu/odbc; it fails with a build without the path. - Both sdists carry unixODBC-2.3.14.tar.gz, SHA-checked, as the corresponding source of the library the Linux and macOS wheels bundle (LGPL-2.1 section 4); the NOTICE says so, and the sdist and PyPI jobs check it is there, unchanged. The download and its pin live in unixodbc_source.sh. - The odbc feature moves from the pyprojects to the wheel builds' --features (and make test-python), so building an sdist needs no driver manager. - A cursor pinned to text or varchar binds as text again; only a cursor pinned to another type falls back to a literal. The backslash test pins one of its text cursors and fails without this. - The catalog's retry without a schema pattern keeps the original error when it too fails, and returns it outright when the schema name length can't be read.
# Conflicts: # .github/scripts/windows_e2e.py # .github/workflows/publish.yml # .github/workflows/verify.yml
Deploying sqlload with
|
| Latest commit: |
116ef18
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://ba245106.sqlload.pages.dev |
| Branch Preview URL: | https://wip-chi-1236-odbc.sqlload.pages.dev |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CHI-1236. ODBC ships in the
sqlloadandsqlload-cliwheels on every platform, cursor bounds bind as typed parameters, and three drivers are tested live on Linux and Windows.The wheels
*.libs/libodbc-*.so.2,*.dylibs/libodbc-*.2.dylib). It is linked dynamically and built from its source release, pinned by SHA-256 (.github/scripts/unixodbc_source.sh,build_unixodbc.sh), with libltdl built in.odbcinst.iniandodbc.iniin/etcon Linux and in Homebrew'setcon macOS, or in the folderODBCSYSINInames.Driver=psqlodbcw.so), are looked for in/usr/lib/<arch>-linux-gnu/odbc,/usr/lib/<arch>-linux-gnu,/usr/lib64/odbcand/usr/lib64, and on macOS in Homebrew'slib.odbc32.dll.--auditwheel=repair) on both Linux and macOS. On a Mac, delocate pointed an installedsqlload-clibinary at a folder that doesn't exist.odbcfeature is set by the wheel builds (--features odbc), not the pyprojects, so building an sdist needs no driver manager.unixODBC-2.3.14.tar.gzas the corresponding source, from the same place on PyPI as the wheels.assert_wheel_notices.sh: every wheel has the notice; each Linux and macOS wheel bundles exactly one libodbc, Windows wheels none. Runs in the build job and again before upload.assert_sdist_sources.sh: each sdist carries the tarball with the pinned hash. Runs in the sdist job and before upload.smoke_odbc_wheel.py, against made-up drivers:ODBCSYSINI, through/etc/odbcinst.ini, and by bare file name.Cursor bounds
?parameters of the cursor's type: a timestamp without a time zone asTIMESTAMPwith seven fractional digits, a date asDATE, text as text. No database's literal rules (backslash escapes,{ts}rounding, OpenEdge's refusal to compare a string with a timestamp) can change a bound. There is no OpenEdge-only rendering.datetime: it stores 1/300 s ticks, which the driver reads rounded to milliseconds. Adatetime2bind would re-read or skip the boundary row, so its bound goes as ISO 8601 text (2026-01-31T23:59:59.123) in the column's precision, which SQL Server reads alike under every login language. The live case runs withLanguage=Britishand a 5 January maximum. With a space instead ofT, it read 1 May and skipped rows.--start-cursor,--end-cursorandcursor_startalso take{ts '...'}and{d '...'}for timestamp and date cursors.DATETIME(n)with no digits). ANSI quoting is probed when a driver reports another quote character, and the catalog retries without a schema pattern only when a driver refuses one and reports no schema name length.Bind precision (D1064)
Probed live with four values per column, each bound with
>and>=:DATETIME,DATETIME(3),DATETIME(6),TIMESTAMP(3)timestamp(0),(3),(6)smalldatetime,datetime2(0),(3),(7)VARCHAR(255)) or psqlODBC (TIMESTAMP(-1)).datetime: the one type wrong with any TIMESTAMP bind, which is why it gets the text path above.Drivers tested
datetime2(3), legacydatetime(British login)Each live case loads twice and checks that the second load reads exactly the rows after each boundary: a millisecond maximum (
.997, or.123for legacydatetime), rows on both sides of it, and a date cursor. On Linux, MySQL Connector/ODBC comes from MySQL's.debfor the distribution, pinned by SHA-256. The generic tarball's private OpenSSL fails to load once another driver has loaded the system one.OpenEdge is not live-tested. No live server was used, by decision. Its
{ts}parsing and timestamp binding are unit-tested; live confirmation is left to the first real user.Checks run locally (Linux)
test_odbc_driversandtest_odbc_to_duckdb: SQL Server 2022 (both cursor types), MySQL 8.4, Postgres 17.Tin the legacy-datetimetext, the British case skips rows. With a pinned text cursor falling back to a literal, the backslash case fails.cargo test --workspace --lib --features sqlload/odbc: 269 passed.make test-python(with ODBC): 366 passed, 1 skipped.make check-ciconfigurations, fensu (0 faults), and the site's content check.sqlload-cliwheel built with--auditwheel=repair --features odbcpassesassert_wheel_notices.shand the ODBC smoke test in python:3.12-slim with no unixODBC. Both sdists passassert_sdist_sources.sh.Reviewed once independently, then a follow-up check of the fixes and the linkage. Their findings are fixed: a language-dependent
datetimetext bound,{ts}unwrapping on text cursors, pinned cursors, the bundled driver manager's driver search path, sdists needing a driver manager, and the catalog fallback's error.