diff --git a/CHANGELOG.md b/CHANGELOG.md index 759fd7698..78b8e7f15 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,2 +1,1624 @@ +## 0.11.0-rc1 + +[Release Migration Guide](docs/releases/0_11_0.md) +[Migration Helpers](migration-helpers) - small code helpers to convert old configuration to a new one. + +### Breaking Changes + +- **[client-v2]** `com.clickhouse.client.api.observability.SpanSupport` now uses `QUERY` and `INSERT` operation + constants (`QUERY ` and `INSERT .` span names). `db.operation.name` attribute is set to + `INSERT` for insert operations and left unset for queries because SQL statements are not parsed on the client. + +- **[client-v2]** `com.clickhouse.client.api.metrics.OperationMetrics` now has a single constructor, + `OperationMetrics(ClientStatisticsHolder, OperationType)`; the constructor without an operation type was removed. + Metrics are created by the client, which always knows the kind of the operation it runs, and the constructor takes + an internal type (`com.clickhouse.client.api.internal.ClientStatisticsHolder`), so application code is not expected + to call it. (https://github.com/ClickHouse/clickhouse-java/issues/2974) + +- **[client-v2, jdbc-v2]** The socket buffer options `socket_rcvbuf` and `socket_sndbuf` have no default value anymore. + They were set to 804800 bytes for every connection, which replaced the buffer size the operating system would choose + and disabled its auto-tuning; the value was also capped by the operating system limits, so on many systems it had no + effect. Now the options are applied only when the application sets them (`Client.Builder#setSocketRcvbuf`, + `Client.Builder#setSocketSndbuf`, or the properties of the same name), and the operating system sizes and tunes the + buffers otherwise. Setting them is not recommended unless a measurement shows a benefit. An application that relies on + the previous size can restore it by setting both options to 804800. + (https://github.com/ClickHouse/clickhouse-java/issues/3121) + +### New Features + +- **[client-v2]** Added an experimental streaming RowBinary reader and writer + (`com.clickhouse.client.api.data_formats.stream`). Nested values used to be written only after the whole + container existed in memory, because `Array`, `Map`, unflattened `Nested`, and `QBit` put a length in front of + their elements, and they were read only after the whole container had been allocated. `RowBinaryStreamWriter` + and `RowBinaryStreamReader` treat a row and each nested value as a scope that is opened and closed explicitly. + A container opened with a known size writes that length immediately and sends the element bytes straight to the + stream; a container opened without a size buffers the encoded bytes and writes the length when the scope closes. + `writeString(byte[])` keeps bytes that are not valid UTF-8. The writer accepts `RowBinary` and + `RowBinaryWithDefaults`. The reader accepts `RowBinary` (schema supplied by the caller), `RowBinaryWithNames`, + and `RowBinaryWithNamesAndTypes` (schema taken from the header). A column that was not written is an error. + `RowBinaryFormatWriter`, POJO insert, and JDBC are unchanged, and `Native` is not supported on this path. + A `JSON` column can be written as a binary object with `beginJson`: each path is a name plus a value, a nested + object is dotted paths, a JSON null is `writeNull(path)`, and an array of objects is an `Array(JSON)` path whose + elements are themselves JSON documents. `writeString` without a path is still the string form. + See `docs/reader_writer_api.md`. + +- **[migration-helpers]** Added `migration-helpers` module containing `ConfigurationMigrationHelper` and + `ConfigPropertyCache` to convert configuration properties and connection URLs from v1 (0.7.1) format to v2 (0.9.8+) + format (automatically prefixing ClickHouse server settings with `clickhouse_setting_`, custom headers with + `http_header_`, and mapping renamed property keys). + +- **[client-v2, jdbc-v2]** Added support for the `MultiPoint` geo data type (ClickHouse `26.8+`). Previously the type was + unknown to the client, so reading or writing a `MultiPoint` column failed with `Unknown data type: MultiPoint`, and a + `MultiPoint` value inside a `Geometry` column failed with an out-of-range variant discriminator. `MultiPoint` is + `Array(Point)` on the wire, exactly like `Ring` and `LineString`, so it is read and written as `double[][]` through + generic records, binary readers, POJO binding, and SQL parameter formatting, and is read from `Dynamic` columns. In the + JDBC driver (`jdbc-v2`) `MultiPoint` maps + to `java.sql.Types.ARRAY`, is returned as `double[][]` from `getObject` and as a `java.sql.Array` from `getArray`, and is + reported by `ResultSetMetaData` and `DatabaseMetaData`. ClickHouse `26.8` also adds `MultiPoint` to the `Geometry` + variant; the server appends it after the existing six variants instead of ordering it by type name, so the client now + keeps that order and decodes a `MultiPoint` held in a `Geometry` column. Because `MultiPoint` shares its Java + representation (`double[][]`) with `Ring` and `LineString`, it is not selectable through the shape-based `Geometry` + write path — a 2D value keeps resolving to `Ring` as before, and writing `MultiPoint` requires a concrete `MultiPoint` + column. (https://github.com/ClickHouse/clickhouse-java/issues/3048) +- **[client-v2, jdbc-v2]** Added a Micrometer implementation of the metrics SPI. + `Client.Builder.setMetricsRecorder(new MicrometerMetricsRecorder(meterRegistry))` reports the metrics of every client + operation to a Micrometer `MeterRegistry`: a timer `db.client.operation.duration` per completed operation, a timer + `clickhouse.client.operation.serialization.duration` when the client measured the serialization step, a counter + `clickhouse.client.operation.count` per completed operation, and a counter `clickhouse.client.operation.retries` per + retried attempt. Previously the client could bind only its connection-pool gauges to Micrometer, so exporting the + metrics of the operations themselves was left to the application. Meter names, units, descriptions and tag keys are + the standard ones of the SPI - the recorder derives them through `MetricsSupport`, so they are the names of + `MetricName` and the keys of `MetricAttribute` and mean the same as for every other recorder. A successful operation + carries no `error.type` tag and a failed one does, so the outcomes are separate time series of the same meter and a + failure the server reported also carries `db.response.status_code`; a duration the client did not measure is not + recorded, so no operation is reported with a made-up duration. The seconds of the SPI are handed to the registry as + nanoseconds, because a Micrometer timer keeps its own time unit, so a backend publishes the duration in the unit it + expects. The no-argument constructor reports to `Metrics.globalRegistry`, which is what the jdbc-v2 + `jdbc_metrics_recorder` property needs, so a JDBC connection exports its metrics to Micrometer by naming the class - + `jdbc_metrics_recorder=com.clickhouse.client.api.observability.micrometer.MicrometerMetricsRecorder` - without + application code. `micrometer-core` stays an optional dependency of `client-v2` and is not shaded into the `all` + artifacts, so a client that does not use this recorder needs no Micrometer on the classpath. + (https://github.com/ClickHouse/clickhouse-java/issues/2975) +- **[client-v2, jdbc-v2]** Added a metrics SPI that lets an application export the metrics of client operations to any + metrics backend. `Client.Builder.setMetricsRecorder(MetricsRecorder)` registers a backend-agnostic recorder from the + `com.clickhouse.client.api.observability` package, and the jdbc-v2 property `jdbc_metrics_recorder` names the recorder + class a connection registers with its own client. Previously the client collected operation metrics but only returned + them to the caller, so exporting them was left to the application. Each completed operation reports exactly one + success or one failure event, and each retried attempt reports a retry event, which gives the operation duration, the + serialization duration, the number of operations by outcome and the number of retries. The SPI follows the pattern of + the span SPI: an implementation extends the `DefaultMetricsRecorder` base class and overrides only what it cares + about, so it keeps working when the client starts reporting an event it does not know about, and the reusable + `MetricsSupport` class derives the standard values from the same structures, so its logic is opt-in and overridable. + Metric names, units and attribute keys follow the OpenTelemetry semantic conventions for database clients where a + convention exists and are placed under `clickhouse.` where it does not; they are defined by the `MetricName` and + `MetricAttribute` enums, durations are reported in seconds, and a duration the client did not measure is reported as + `MetricsSupport.DURATION_UNKNOWN` instead of a made-up value. The metric attributes are deliberately a smaller set + than the span attributes, because an attribute of a metric becomes a time series: the statement text, the query id and + the statement parameters stay on spans. Nothing is recorded and no metrics-related work is done when no recorder is + registered. (https://github.com/ClickHouse/clickhouse-java/issues/2975) +- **[client-v2]** Added an OpenTelemetry implementation of the observability SPI. + `Client.Builder.setSpanRecorder(new OpenTelemetrySpanRecorder(openTelemetry))` + reports every client operation and every transport request as an OpenTelemetry `CLIENT` span: an operation span is + started as a child of the current OpenTelemetry context, so it joins the application's own trace, and each request + span - including one per retry - is a child of its operation span. Span names and attribute keys are the standard + ones of the SPI (the recorder derives them through `SpanSupport`), every value is recorded with the OpenTelemetry + attribute type that matches it, and a failure sets the span status to `ERROR` and is recorded as an OpenTelemetry + exception event next to the `error.type` and `db.response.status_code` attributes. The recorder reports to a + supplied `OpenTelemetry` instance, to a `Tracer` given to `new OpenTelemetrySpanRecorder(Tracer)`, or to + `GlobalOpenTelemetry` - read when a span is started - when constructed without arguments. Previously an application that wanted + OpenTelemetry spans had to write that mapping itself. The OpenTelemetry API is a compile-only dependency of + `client-v2`: the recorder is used only by an application that already provides `opentelemetry-api` at runtime, so + nothing is added to the classpath of a client that does not use it. + (https://github.com/ClickHouse/clickhouse-java/issues/2974) +- **[client-v2]** Added an observability SPI that lets an application observe client operations as spans. + `Client.Builder.setSpanRecorder(SpanRecorder)` registers a backend-agnostic recorder from the new + `com.clickhouse.client.api.observability` package: each operation (a query, a command or an insert - including + the `ping` and `getTableSchema` calls, which run a query) starts one operation span, and every transport + request made for it - including each retry - starts a child request span. `SpanRecorder` and `Span` are plain interfaces; an implementation extends the + `DefaultSpanRecorder` base class and overrides only what it cares about, so it keeps working when + the client starts a kind of span it does not know about. The registered recorder is called first and receives + everything the client knows about the operation (its `QuerySettings`/`InsertSettings`, the statement, the target + table, the batch size, the endpoint, the metrics of the completed operation and the failure), so it is free to + record whatever it needs and in whatever form; the reusable `SpanSupport` class derives the standard span names + and attribute values from those same structures and is called by a recorder implementation that wants them, so + its logic is opt-in and overridable. Span names and attribute keys follow the OpenTelemetry semantic conventions for + database and HTTP client spans; the keys are defined by the `SpanAttribute` enum and the values are derived by + `SpanSupport`, so all recorders that use it report the same information (statement text, target database and table, query id, + statement parameters, batch size, the first configured endpoint on the operation span and the per-attempt + server address and port on the request spans, HTTP status, returned rows, and the error type and ClickHouse + error code on failure). The outcome of a completed operation is reported per operation kind - `recordQuerySuccess` + for a read and `recordInsertSuccess` for an insert - because the metrics that describe a read are not the ones that + describe a write: a query reports `db.response.returned_rows`, `clickhouse.response.read_rows` and + `clickhouse.response.read_bytes`, an insert reports `clickhouse.response.written_rows` and + `clickhouse.response.written_bytes`. The same distinction is available on the metrics themselves through the new + `OperationMetrics#getOperationType()`, which returns the new `com.clickhouse.client.api.metrics.OperationType` - + the kind of the call the application made, so a command that writes is reported as a query. + An operation span is started on the calling thread, so it joins + the caller's ambient trace even when the operation runs on the client's executor, and it is ended exactly once + for every operation that starts. Previously the client exposed no hook for tracing, so an + application could not attribute a query or a retried request to its own trace. When no recorder is registered + nothing is recorded and no span-related work is done, so the default path is unchanged. An OpenTelemetry + implementation of the SPI is available as `OpenTelemetrySpanRecorder`. + (https://github.com/ClickHouse/clickhouse-java/issues/2974) +- **[client-v2, jdbc-v2]** Added support for the `BFloat16` data type (ClickHouse `24.11+`). `BFloat16` columns are read as + Java `float` values (widening is lossless) and written from `float`/`Float` values, including through generic records, POJO + binding, `Nullable(BFloat16)`, and `BFloat16` values held in `Dynamic`/`Variant` columns. On write the client keeps the + high 16 bits of the `float`, matching the ClickHouse server's own `Float32` → `BFloat16` conversion. In the JDBC driver + (`jdbc-v2`) `BFloat16` maps to `java.sql.Types.FLOAT` / `java.lang.Float` and is read and written through the standard + `getFloat`/`setFloat` and `getObject` accessors, and reported as such by `ResultSetMetaData` and `DatabaseMetaData`. + Previously reading or writing a `BFloat16` column failed with an + unsupported-data-type error. (https://github.com/ClickHouse/clickhouse-java/issues/2279) +- **[client-v2, jdbc-v2]** Added support for the experimental `QBit(element_type, dimension[, stride])` vector data type + (ClickHouse `25.10+`; the `allow_experimental_qbit_type` server setting is required to create a column). The type-name + parser accepts two or three parameters (the optional third is the stride) and recognizes the documented element types + `Int8`, `BFloat16`, `Float32`, and `Float64`; an element type outside that set is parsed with a warning rather than + rejected, so a newer server-side element type keeps parsing. A `QBit` value is transmitted over `RowBinary` exactly like + `Array(element_type)`, so it is read and written as a Java array of the element type (`float[]` for + `BFloat16`/`Float32`, `double[]` for `Float64`) through generic records, binary readers, and POJO binding, via a + dedicated `QBit` read/serialize path. A `QBit` held inside a `Dynamic`/`Variant`/`JSON` column is also decoded (its + binary type encoding is read back to the concrete `QBit(...)` type). In the + JDBC driver (`jdbc-v2`) `QBit` maps to `java.sql.Types.ARRAY` and is returned as a `java.sql.Array` from + `getObject`/`getArray`. Previously `QBit` was an unimplemented type constant and reading or writing such a column + failed. A plain top-level `QBit` column with a `Float32`, `Float64`, or `BFloat16` element type is also read through + the `Native` output format: there the server transmits it using its internal bit-plane-transposed + `Tuple(FixedString(...))` layout, and the client reverses that transposition to reconstruct the same + `float[]`/`double[]` vector as `RowBinary`. A `QBit` that is strided (`QBit(element_type, dimension, stride)`), + wrapped in `Nullable`/`LowCardinality`, nested inside another type (e.g. `Array`/`Tuple`/`Map(String, QBit(...))`), + or carrying any other element type is not yet decoded over `Native` and fails fast with a clear error directing you + to a `RowBinary` format such as `RowBinaryWithNamesAndTypes`. + (https://github.com/ClickHouse/clickhouse-java/issues/2610) +- **[client-v2, jdbc-v2]** Added TLS cipher suite selection. `Client.Builder.setSSLCipherSuites(String...)` (client-v2) + and the comma-separated `ssl_cipher_suites` connection property (client-v2 and jdbc-v2) restrict the cipher suites + enabled on secure connections; when unset, the transport defaults are used. Cipher-suite selection is independent of the + trust configuration and `ssl_mode`. (https://github.com/ClickHouse/clickhouse-java/issues/2882) +- **[client-v2, jdbc-v2]** Added support for un-flattened `Nested(...)` columns (tables created with + `flatten_nested = 0`). Previously the `RowBinary` writer threw `UnsupportedOperationException: Unsupported + data type: Nested` when inserting such a column. `client-v2` now serializes a `Nested(f1 T1, ..., fN TN)` + column the same way it is read — identically to `Array(Tuple(T1, ..., TN))` (a var-uint row count followed + by one tuple per nested row) — so it can be written through the insert path / `RowBinaryFormatWriter`. In + `jdbc-v2` an un-flattened `Nested(...)` column is exposed as a JDBC `ARRAY` whose element type is + `Tuple(f1 T1, ..., fN TN)`: it can be inserted through `Connection#createArrayOf`/`setArray` or `setObject` + and read back through `getArray`/`getObject`, and `java.sql.Array#getResultSet()` iterates the nested rows + as `(INDEX, VALUE)` pairs where each `VALUE` is the tuple. (https://github.com/ClickHouse/clickhouse-java/issues/2477) +- **[client-v2, jdbc-v2]** Added logging on previously-silent error and diagnostic paths (no functional or + public-API change). (https://github.com/ClickHouse/clickhouse-java/issues/2969) + +### Bug Fixes + +- **[jdbc-v2]** Added the non-reserved keywords `AGGREGATE`, `BOUNDED`, `EXTEND`, `HANDLER`, `IDLE`, `PROTOCOL`, + `RECENT`, `TIMEOUT` and `UNORDERED` (ClickHouse `26.8+`; `IDLE`, `TIMEOUT` and `RECENT` come from the multi-word + keywords `IDLE TIMEOUT` and `RECENT SAMPLES`) to the list of keywords allowed in identifier positions. The server + accepts all of them as a column or table alias, so a query using one of them as an identifier must parse. + (https://github.com/ClickHouse/clickhouse-java/issues/3113) +- **[client-v2]** Fixed a query with statement parameters sent in the request body + (`client.http.use_form_request_for_query=true`) failing with `LZ4 decompression failed ... (LZ4_DECODER_FAILED)` + when client request compression and HTTP compression were both enabled. The multipart body is always sent + uncompressed, but the request still declared `Content-Encoding: lz4`; ClickHouse `26.8+` honours that header for + multipart requests and tried to decompress a plain body. The header is now omitted for multipart requests, like + the `decompress` query parameter already was. Response compression (`Accept-Encoding`, + `enable_http_compression`) is unchanged. (https://github.com/ClickHouse/clickhouse-java/issues/3075) +- **[client-v1]** Fixed the `DateTime64` case of `testReadWriteSimpleTypes` failing against ClickHouse 26.8. From 26.8 + an unquoted number written to a `DateTime64` column in the `Values`/`Quoted` and `JSON` paths is a Unix timestamp in + seconds instead of the raw scaled value - the server setting `input_format_read_datetime_number_as_raw_value` + changed its default from `1` to `0` - so the `insert into ... values(1)` of the test stored `1970-01-01 00:00:01` + instead of the expected `1970-01-01 00:00:00.001`. The test now writes a quoted date-time literal for `DateTime64`, + as it already does for `FixedString` and `UUID`, so the written value means the same on every server version and the + sub-second round-trip stays covered. No client code is affected: both clients quote a date-time value in a text + statement or send it in `RowBinary`. (https://github.com/ClickHouse/clickhouse-java/issues/3114) +- **[jdbc-v2, client-v2]** Fixes issue with `FORMAT` in query unable to override format set by client when used with + ClickHouse 26.8+. Default format is `RowBinaryWithNamesAndTypes` set at client level. For JDBC, recommend using + `format=JSONEachRow` to query JSON. Setting `format=` (empty or `null`) omits the format request header so explicit + query `FORMAT` clauses take effect; note that on JDBC any statement without a `FORMAT` clause will fail because the + server falls back to `default_format` (`TabSeparated`). `DatabaseMetaData` is unaffected: every statement it runs + internally pins `RowBinaryWithNamesAndTypes` in its own settings, so metadata keeps working regardless of the + connection's `format` property. (https://github.com/ClickHouse/clickhouse-java/issues/3086) +- **[jdbc-v2]** Fixed `Connection#prepareStatement` and `PreparedStatement#addBatch` throwing + `StringIndexOutOfBoundsException` for an `INSERT ... VALUES (...)` statement containing a JDBC escape sequence + (`{d '...'}`, `{ts '...'}`, ...) or a ClickHouse query parameter whose name starts with `d`/`t` (e.g. `{d:Int32}`). + The default `JAVACC` parser records the values list positions as offsets into the SQL it rebuilds from the token + stream, where such sequences are rewritten or dropped, while the driver slices the original SQL with them — so the + slice was taken at the wrong offsets or past the end of the statement. The positions are now checked against the + original SQL and discarded when they do not address its values list, in which case the driver falls back to its + generic parameter substitution path. Such a statement is now prepared without error; the escape sequence itself is + still sent to the server unchanged. The `ANTLR4` parser backends were not affected. + (https://github.com/ClickHouse/clickhouse-java/issues/3017) +- **[jdbc-v2]** Fixed a `?` inside a `//` line comment or inside a heredoc (dollar quoted string, e.g. `$$...$$` or + `$tag$...$tag$`) being counted as a `PreparedStatement` parameter. Such a statement expected a value the application + could not supply, so `executeQuery()` failed with `Parameter at position 'N' is not set` for a query the server + executes fine. The placeholder scan now skips both token kinds, like the server lexer does; a `$` that does not open a + heredoc is still treated as an ordinary character (it is a valid identifier character). + (https://github.com/ClickHouse/clickhouse-java/issues/3009) +- **[jdbc-v2]** Fixed `INSERT INTO [TABLE] FUNCTION f(...) VALUES (?)` failing with + `Code: 60 ... does not exist. (UNKNOWN_TABLE)` when the `beta.row_binary_for_simple_insert` feature was + enabled. Neither SQL parser reported a table-function insert target as a function, so the statement was + routed to the `RowBinary` writer, which looked the function name (or a placeholder such as `unknown`) up as + a table. Both parsers now report such a statement as using a function, so it stays on the regular SQL path; + additionally the JavaCC grammar no longer mis-parses `INSERT INTO TABLE FUNCTION f(...)` by consuming + `FUNCTION` as the table name. Inserts into a plain table are unaffected and still use the `RowBinary` + writer. (https://github.com/ClickHouse/clickhouse-java/issues/3015) +- **[jdbc-v2]** Fixed `Connection#prepareStatement` throwing a `NullPointerException` for an + `INSERT ... VALUES (...)` statement whose values list the default JavaCC parser cannot parse — most commonly one + containing a heredoc string (`$$...$$`), which the grammar has no token for, but also any other unparsable token + inside the list. The parser's error recovery left the values list's start position recorded without its matching end + position, which was then unboxed unguarded. Both positions are now dropped together, so the driver falls back to its + generic parameter-substitution path and such statements are prepared and executed successfully. The `ANTLR4` + parser backends were not affected. (https://github.com/ClickHouse/clickhouse-java/issues/3013) +- **[client-v2]** Fixed reading a `JSON` or named `Tuple` value nested in a `Dynamic` column when a typed path or + element name requires quoting (it contains a space, a comma or a bracket). Names read from the binary type encoding + were appended to the reconstructed type name unquoted, so e.g. ``JSON(`a b` Int64)`` inside a `Dynamic` column + produced a malformed type name and the whole query failed with `IllegalArgumentException: Unknown data type: b Int64`. + Such names are now back-quoted (with inner back-quotes escaped) exactly as the server renders them, and `JSON` skip + paths and path regexps are emitted with their `SKIP` / `SKIP REGEXP` markers. Names that need no quoting are + rendered as before. Top-level `JSON` columns and `JSON` nested in `Map`/`Tuple`/`Array` were + not affected — their type comes from the `RowBinaryWithNamesAndTypes` header, which the server already quotes. + (https://github.com/ClickHouse/clickhouse-java/issues/3001) +- **[client-v2]** Fixed reading a `Variant`, `Nested`, `Decimal` or `Enum` value held in a `Dynamic` column. The + concrete type rebuilt from the binary type encoding did not match what the server encoded: `Variant` was wrapped + twice (so the discriminator selected the wrong element), `Nested` read only the element names and left the element + type encodings in the stream, and `Decimal`/`Enum` lost their precision and scale / their constants whenever the + value sat inside another type, so a decimal read back unscaled (`1.2500` as `12500`) and every enum value read back + as ``. The constant width of an enum is now taken from the type tag rather than from the number of + constants, which also fixes reading an `Enum16` with fewer than 128 constants and negative `Enum8` constants. + (https://github.com/ClickHouse/clickhouse-java/issues/3003) +- **[client-v2]** Fixed a `Nullable(T)` column bound to a **primitive** POJO field silently corrupting a row on the + POJO read path. The compiled setter went straight to a primitive read method without consuming the `Nullable` + null-marker byte, which is on the wire for every value of a nullable column regardless of the value, so the stream + stayed shifted by one byte per row and the nullable column and every column after it decoded from the wrong offset + without any error being raised. The generated setter now consumes the marker; a value that is actually `NULL` cannot + be held by a primitive field and is reported with a `NullValueException`. Boxed POJO fields are unaffected. + (https://github.com/ClickHouse/clickhouse-java/issues/2993) +- **[client-v2]** Fixed the `Native` format reader (`NativeFormatReader`) misreading `Array` columns in multi-row + results whose rows have different lengths. Native encodes an array column as cumulative row offsets followed by the + flattened elements, but the reader used the first row's offset as the element count for every row — truncating later + rows and desyncing the columns that follow the array in the same block. Each row's length is now derived from the + difference between consecutive offsets, and empty array rows (`len == 0`) no longer read a phantom element. Results + with uniform array lengths were unaffected. (https://github.com/ClickHouse/clickhouse-java/issues/2955) +- **[jdbc-v2]** Fixed `SQLException#getSQLState()` returning the generic data-exception state `22000` + when ClickHouse reports an unknown table. The driver now returns `42S02` (base table or view not found) while + preserving the ClickHouse error code and original exception. (https://github.com/ClickHouse/clickhouse-java/issues/3104) +- **[client-v2]** Fixed truncated LZ4 stream errors reporting literal `{0}` and `{1}` placeholders instead of the + number of bytes read and expected. (https://github.com/ClickHouse/clickhouse-java/issues/3108) +- **[jdbc-v2]** Fixed `DatabaseMetaData#getTables` reporting `TABLE_TYPE = TABLE` for a table with the `BigQuery` + engine (present in `system.table_engines` since ClickHouse `26.8`). The engine was missing from the + engine-to-table-type mapping, so it fell back to the default `TABLE`, and `getTables(..., types = {"REMOTE TABLE"})` + returned no row for such a table. `BigQuery` is now mapped to `REMOTE TABLE`, like the other external-storage + engines. (https://github.com/ClickHouse/clickhouse-java/issues/3049) +- **[jdbc-v2]** Fixed `PreparedStatement.getMetaData()` losing the result-set schema for a statement whose SQL + contains a comment. The `DESCRIBE` query used to resolve the metadata was built by re-scanning the SQL with a + regex that knew only quoted tokens, so a `?` inside a `--` / `#` / `/* */` comment was rewritten to `NULL` and + an odd `'` inside a comment mis-paired the quote alternative, leaving a real placeholder unreplaced — the + `DESCRIBE` then failed and the driver silently returned untyped metadata. The metadata query is now built from + the placeholder positions the statement parser already computed, so it always matches the SQL that a + parameterized execution produces. (https://github.com/ClickHouse/clickhouse-java/issues/3011) +- **[client-v2]** Fixed reading a `UInt64` column into a **primitive** POJO field (e.g. `long`) always failing with + `ClassCastException: BinaryStreamReader cannot be cast to java.math.BigInteger`. The compiled setter for that + combination never emitted a read call, so it cast the reader itself instead of a value and consumed nothing from the + stream, which made every primitive field bound to a `UInt64` column unusable. The value is now read and converted to + the target primitive with the matching `Number` accessor, which narrows a value that does not fit the same way a Java + narrowing conversion does (a `long` holds every `UInt64` value bit-for-bit and can be read back with + `Long.toUnsignedString(long)`; a `boolean` is `true` for any non-zero value). Boxed fields (`BigInteger`, `Long`) are + unaffected. (https://github.com/ClickHouse/clickhouse-java/issues/2996) +- **[jdbc-v2]** Fixed `ResultSetMetaData.getPrecision()` and `getScale()` returning `0` for columns wrapped in + `SimpleAggregateFunction(func, T)`. The wrapper is transparent on the read path (values are read as plain `T`), but + both accessors described the wrapper itself, which carries no precision or scale — so a + `SimpleAggregateFunction(sum, Decimal(18, 4))` column looked like a scale-0 value and + `SimpleAggregateFunction(any, DateTime64(3, tz))` looked like second precision. They now describe the nested type. + `AggregateFunction` columns are unchanged, since their values are aggregation states rather than values of the + nested type. (https://github.com/ClickHouse/clickhouse-java/issues/3042) +- **[clickhouse-jdbc]** Fixed `Connection#prepareStatement` throwing a `NullPointerException` for an + `INSERT ... VALUES (...)` statement whose values list the JavaCC parser cannot parse — most commonly one + containing a heredoc string (`$$...$$`), which the grammar has no token for, but also any other unparsable token + inside the list. The parser's error recovery left the values list's start position recorded without its matching end + position, which was then unboxed unguarded. Both positions are now dropped together, so the driver falls back to its + generic parameter-substitution path instead of failing, and a statement such as + `insert into t values ($$a@b$$, ?)` is prepared and executed successfully. + (https://github.com/ClickHouse/clickhouse-java/issues/3033) +- **[jdbc-v2]** Fixed the ANTLR4 lexer not nesting `/* */` block comments. ClickHouse (and the JavaCC parser backend) + raise the nesting level on an inner `/*` and close the comment only at the matching `*/`, while the ANTLR4 lexer ended + the comment at the first `*/` and lexed the rest of it as SQL. With the `ANTLR4` / `ANTLR4_PARAMS_PARSER` backends this + made statements the server accepts (e.g. `SELECT 1 /* ) /* ) */ ) */, 2`) report syntax errors, and made + `ANTLR4_PARAMS_PARSER` count a `?` inside the nested part of a comment as a bind parameter. Comments that do not nest + are unaffected; an unterminated block comment is now skipped to the end of the statement instead of being lexed as + stray tokens. (https://github.com/ClickHouse/clickhouse-java/issues/3021) +- **[client-v2]** Fixed reading a `SimpleAggregateFunction(func, T)` value held in a `Dynamic` column. The binary type + encoding of such a value (`0x2E `) was not consumed + at all, so the read failed with `IndexOutOfBoundsException`, and the unconsumed encoding bytes would otherwise have + been interpreted as row data and desynchronized the rest of the `RowBinary` stream. The concrete type is now + reconstructed from the encoding and the value is read as its argument type `T`, so it reads exactly like the same + value in a plain `SimpleAggregateFunction` column. (https://github.com/ClickHouse/clickhouse-java/issues/3005) +- **[jdbc-v2]** Fixed `PreparedStatement#executeBatch` sending a syntactically broken `INSERT` when an `ANTLR4` parser + backend is selected (`jdbc_sql_parser=ANTLR4` / `ANTLR4_PARAMS_PARSER`) and the values list contains a value + expression the bundled grammar cannot parse - a JDBC escape sequence (`{d '...'}`), or valid ClickHouse syntax the + grammar does not cover such as a hex string literal (`hex(x'AB')`). Such a statement is still given a parse tree, + completed by error recovery, and the values list positions and the value group count were read from it: the values + list was reported to stop at the closing parenthesis of a nested function call, so the batch template lost its own + closing parenthesis, and a two-group values list could be reported as a single group. Both are now discarded when the + statement could not be parsed without errors, so the driver uses its generic parameter substitution path instead - and, + with the beta `RowBinary` writer enabled, such a statement is no longer routed to it. The default `JAVACC` backend is + not affected by this. + (https://github.com/ClickHouse/clickhouse-java/issues/3019) +- **[jdbc-v2]** Fixed the ANTLR4 lexer rejecting `//` line comments, which the ClickHouse server and the driver's + JavaCC grammar both accept. Because `/` is also the division operator, `// comment` was lexed as two operator + tokens, so a statement containing a `//` comment was reported as a syntax error by the ANTLR4-based parser + backends (`ANTLR4`, `ANTLR4_PARAMS_PARSER`), and an `INSERT` preceded by such a comment was misclassified as a + statement with a result set. `//` is now skipped like `--`, `#` and `#!`; a single `/` and `//` inside a string + literal or a quoted identifier are unaffected. Placeholder counting inside `//` comments for the backends that + scan the raw SQL separately (`JAVACC`, `ANTLR4`) is fixed by + https://github.com/ClickHouse/clickhouse-java/issues/3009. + (https://github.com/ClickHouse/clickhouse-java/issues/3023) +- **[jdbc-v2]** Fixed `?` parameter placeholders being lost when `jdbc_sql_parser=ANTLR4_PARAMS_PARSER` is selected and + the bundled grammar cannot match part of the statement - a JDBC escape sequence (`{d '...'}`), or valid ClickHouse + syntax the grammar does not cover such as a hex string literal (`hex(x'AB')`). That backend read the placeholders only + from the parse tree, and the tokens error recovery skips are not part of it, so a placeholder inside such an expression + was dropped: `getParameterMetaData().getParameterCount()` was too low, `setXxx` for a dropped placeholder failed, and + the remaining values were substituted at the wrong offsets. The placeholders are now re-derived from the original SQL + when the statement could not be parsed without errors, as the other two backends always do. + (https://github.com/ClickHouse/clickhouse-java/issues/3025) +- **[client-v2, jdbc-v2]** Fixed `Client.getTableSchema(...)`, `Client.getTableSchemaFromQuery(...)` and `ping()` + failing against ClickHouse `26.8+`, where the `X-ClickHouse-Format` header the client sends wins over a `FORMAT` + clause in the query. These internal queries now set their format in the settings instead of a `FORMAT` clause. + (https://github.com/ClickHouse/clickhouse-java/issues/3068) +- **[jdbc-v2]** Fixed an `INSERT` whose values list holds a function call the bundled `ANTLR4` grammar cannot match - + such as `hex(x'AB')`, valid ClickHouse the grammar has no hex string literal for - being reported to hold no function + call when an `ANTLR4` parser backend is selected (`jdbc_sql_parser=ANTLR4` / `ANTLR4_PARAMS_PARSER`). Function calls in + a values list are reported by a callback on the parse tree, and such a statement is still given a parse tree, completed + by error recovery, which skips the tokens the parser recovered on - the function call among them. With the beta + `RowBinary` writer enabled (`beta.row_binary_for_simple_insert=true`) the statement was then routed to it, where a + literal function-call column cannot be written; it now takes the generic parameter substitution path, as it already did + for a function call the grammar matches. Since such a parse tree cannot tell, any insert that could not be parsed + without errors is now assumed to hold a function call in its values list, so none of them is written with the + `RowBinary` writer. The default `JAVACC` backend is not affected by this. + (https://github.com/ClickHouse/clickhouse-java/issues/3027) +- **[clickhouse-client]** Fixed JPMS/module-path service loading for `ClickHouseRequestManager` by loading client + services from the `com.clickhouse.client` module, which declares the required `uses` directives. This avoids + `ServiceConfigurationError` failures from `com.clickhouse.data` when applications run on the module path. + (https://github.com/ClickHouse/clickhouse-java/issues/2669) + +- **[client-v2]** Fixed `Client.cancelTransportRequest(queryId)` being silently dropped when it landed between two + attempts of a retried operation (query, POJO insert and stream insert): the operation issued the next attempt anyway + and could complete successfully. The request of an attempt now stays registered until the whole operation is over, + and the cancellation is checked before every attempt, so a cancelled operation stops instead of sending another + request. (https://github.com/ClickHouse/clickhouse-java/issues/2989) +- **[data]** Fixed `BlockingPipedOutputStream.close()` not being idempotent under concurrency: the check of the + `closed` flag and the closing handshake were not atomic, so two threads closing the same stream (e.g. a writer + thread and a try-with-resources block) could both put the end-of-stream marker into the queue, and the second one + failed with `Close stream timed out after ms` once the reader had stopped consuming. Exactly one caller now + performs the handshake and runs the post-close action; a concurrent or repeated `close()` returns immediately. A + `close()` which fails while flushing the remaining data also marks the stream closed and runs the post-close + action, so the stream cannot stay half-closed. (https://github.com/ClickHouse/clickhouse-java/issues/3055) +- **[data]** Fixed `NonBlockingPipedOutputStream.close()` not being idempotent under concurrency. Two threads could + both flush and mutate the same pending buffer before the reader consumed it, silently replacing the payload with + an empty buffer and running the post-close action twice. Exactly one caller now flushes the pending data, enqueues + the end-of-stream marker, and runs the post-close action; concurrent or repeated `close()` calls return immediately. + (https://github.com/ClickHouse/clickhouse-java/issues/3057) +- **[jdbc-v2]** Fixed JDBC escape processing rewriting text inside string literals and quoted identifiers. Because + `PreparedStatement` inlines bound parameters into the statement text, a bound value containing `{fn ` (or `{d '...'}` + / `{ts '...'}`) was re-read as SQL syntax: the `{fn ` was removed together with the next `}` found anywhere in the + statement — usually the closing brace of an unrelated `Map`/`Tuple` literal in another value or row — corrupting the + inserted data or failing with a server-side `SYNTAX_ERROR`. Escape sequences are now recognized only outside of quoted + text, and a `{fn ...}` escape is unwrapped at its matching closing brace, so nested braces (e.g. a `{name:Type}` query + parameter or a nested escape) stay balanced. (https://github.com/ClickHouse/clickhouse-java/issues/2995) +- **[jdbc-v2]** Fixed prepared statements losing parameter markers after an empty `--` comment line or after + `SELECT * EXCEPT (...)`, which caused parameter binding to fail with `ArrayIndexOutOfBoundsException` for the + affected SQL parser backends. (https://github.com/ClickHouse/clickhouse-java/issues/3052) +- **[client-v2]** Fixed LZ4 input streams not closing their underlying HTTP response stream. Closing an LZ4 stream + returned by `QueryResponse.getInputStream()` now releases the wrapped transport stream, including after a partial + read. (https://github.com/ClickHouse/clickhouse-java/issues/2985) +- **[jdbc-v2]** Fixed the default JavaCC SQL parser aborting on a heredoc string (`$$body$$`, `$tag$body$tag$`) + whose body contains a character that is not a valid SQL token on its own, such as `!`, `&`, `|` or `~`. The + lexer had no heredoc token, so such a body raised a lexer error that left the statement classified as + `UNKNOWN` — an INSERT was reported as a result-set-bearing statement with no table name and no values-list + positions, which disables the batch values template and the table-name based paths. A heredoc is now lexed + as a single string literal. (https://github.com/ClickHouse/clickhouse-java/issues/3029) +- **[client-v2, jdbc-v2]** Reduced noisy and potentially sensitive logging; SQL that fails to parse is no + longer logged at `WARN` (it could contain credentials/PII). (https://github.com/ClickHouse/clickhouse-java/issues/2970) +- **[client-v2]** Fixed `BigDecimal` values written into a `Dynamic` column being silently truncated when the + value's scale exceeded the inferred width's maximum scale, and throwing an overflow error when the value + carried an integer part (e.g. `19.99`). The `Dynamic` type inference now sizes the `Decimal` width to hold + both the integer digits and the value's scale, keeps the scale as wide as the width allows without stealing + room from the integer part, and writes the actual column scale into the `Dynamic` type tag. Values that + already round-tripped losslessly are unchanged. (https://github.com/ClickHouse/clickhouse-java/issues/2966) +- **[client-v2]** Fixed the `RowBinary` writer throwing + `UnsupportedOperationException: Unsupported data type: SimpleAggregateFunction` when inserting into a + `SimpleAggregateFunction(func, T)` column (the reader already supported these columns). The value is now + serialized identically to its underlying type `T`, writing the `Nullable` null-marker byte when the + underlying type is nullable (e.g. `SimpleAggregateFunction(anyLast, Nullable(String))`), mirroring the + read path. (https://github.com/ClickHouse/clickhouse-java/issues/2477) +- **[client-v2]** Fixed the `Dynamic` type tag for a `SimpleAggregateFunction` type being written as a bare + `0x2E` byte. The binary type encoding also carries the function name, its parameters and its argument + types, so the server read the function name out of the value bytes that followed and failed with + `ATTEMPT_TO_READ_AFTER_EOF`. Since the client never infers a `SimpleAggregateFunction` from a Java value + and the reader cannot read one back out of a `Dynamic` column, this now fails fast with a clear + `ClientException` instead of producing a corrupt `RowBinary` stream (the same treatment `QBit` already + gets). (https://github.com/ClickHouse/clickhouse-java/issues/3007) +- **[client-v2, jdbc-v2]** Fixed several logging-layer defects. In `client-v2`, `HttpAPIClientHelper.shouldRetry` + threw a `ClassCastException` when a retryable `ServerException` was wrapped as the *cause* of another exception + (the branch matched on the cause but the cast used the outer exception); the retry decision is now taken from + whichever exception is the `ServerException`. Also in `client-v2`, a failure to build the HTTP client version + string is now logged at `WARN` with the throwable attached instead of a bare `INFO` message that discarded the + cause. In `jdbc-v2`, a failure to close the response after a query error now logs the close failure itself + instead of the already-propagated outer exception. (https://github.com/ClickHouse/clickhouse-java/issues/2968) + +- **[client-v2]** Fixed scalar `String` query parameters containing a tab (`0x09`), newline + (`0x0a`) or backslash being mishandled through the server's `param_` interface. A `{name:String}` + parameter value is parsed by the server with `deserializeTextEscaped`, which treated a raw tab or + newline as a field delimiter (failing the query with `BAD_QUERY_PARAMETER: ... isn't parsed completely`) + and a raw backslash as the start of an escape sequence (silently corrupting the value, e.g. `C:\temp` + became `C:emp`). `Client.query(sql, params, ...)` now escapes the backslash, tab and newline in a + scalar `String` parameter so any value round-trips; every other character the server reads verbatim — + including the single quote and carriage return — is left unchanged, so `Identifier` values and + pre-formatted `Array`/`Map` literals passed as a `String` still round-trip. The JDBC driver (`jdbc-v2`), + which inlines parameters as SQL literals and already escaped the backslash and single quote, is + unchanged and covered by a new regression test. (https://github.com/ClickHouse/clickhouse-java/issues/2781) + +- **[client-v2]** Fixed a `null` query-parameter value being sent as the literal string `"null"`, so + `Client.query(sql, params, ...)` binding a Java `null` to a scalar placeholder such as + `{x:Nullable(Decimal128(8))}` was rejected by the server with `BAD_QUERY_PARAMETER` + (`Value null cannot be parsed as Nullable(...)`). A top-level scalar `null` is now sent as the ClickHouse + `\N` NULL sentinel so it binds SQL `NULL`; a `null` nested inside an `Array`/`Map` parameter value + continues to render as the SQL `NULL` keyword. (https://github.com/ClickHouse/clickhouse-java/issues/2977) + +- **[client-v2]** Fixed binary array decoding for nullable element types so `Array(Nullable(Float64))` and similar columns now return boxed arrays such as `Double[]` instead of `Object[]`. This keeps null-supporting arrays aligned with their element type while preserving the existing `Object[]` fallback for Variant/Dynamic/Geometry arrays. (https://github.com/ClickHouse/clickhouse-java/issues/2846) + +- **[client-v2]** Fixed `Float32`/`Float64` columns throwing `ClassCastException` when a value of a + non-matching boxed numeric type was supplied through the `Object`-typed insert surface — for example a + `Double` (the natural type of a Java literal like `1.5`) for a `Float32` column, or a `Float` for a + `Float64` column. The `RowBinary` serializer now narrows any `Number` (and, like the `Int*` columns, + a `String`/`Boolean`) through `Number#floatValue()`/`Number#doubleValue()`, so the float columns accept + the same value types the integer columns already did. (https://github.com/ClickHouse/clickhouse-java/issues/2930) + +- **[client-v2]** Fixed a `NullPointerException` when serializing a `null` value into a non-nullable + `Enum8`/`Enum16` column. `SerializerUtils.serializeEnumData` had no `null` guard, so a `null` in a + non-nullable enum column reached `value.getClass()` and failed the RowBinary insert path with a confusing + NPE instead of a clear error. It now throws `IllegalArgumentException` naming the column, consistent with + the existing `IllegalArgumentException` for other unsupported enum values. Nullable enum columns are + unaffected. (https://github.com/ClickHouse/clickhouse-java/issues/2931) +- **[client-v2]** Fixed silent data corruption when serializing a Java `null` into a non-nullable + `Array(...)` column via `RowBinaryFormatWriter`. `RowBinaryFormatSerializer.writeValuePreamble` + special-cased `Array`, emitting a stray marker byte on top of the array length; the server read the + extra byte as a phantom extra row (single-column inserts) or as a column shift that failed the insert + with `CANNOT_READ_ALL_DATA` (multi-column inserts). A non-nullable `Array` cannot represent a `null`, + so it now throws `IllegalArgumentException` naming the column — consistent with every other non-nullable + type — in both the `RowBinary` and `RowBinaryWithDefaults` paths. Empty arrays (`[]`) still serialize + correctly, and `Dynamic` columns, which can hold a `null` as the implicit `Nothing` type, are + unaffected. (https://github.com/ClickHouse/clickhouse-java/issues/2938) +- **[client-v2]** Fixed POJO insert error classification so transport write failures such as java.net.SocketException: + Broken pipe (Write failed) are now surfaced as transfer/network errors instead of being wrapped as + DataSerializationException. This only changes the exception type reported for request-body transport failures during + Client.insert(...); actual POJO reflection/serialization failures are still reported as DataSerializationException. + (https://github.com/ClickHouse/clickhouse-java/issues/2729) +- **[client-v2]** Fixed binary varint decoding for length and count fields so overflowing or overlong values fail with an `IOException` instead of being decoded into corrupted or negative `int` values. (https://github.com/ClickHouse/clickhouse-java/issues/2902) + +- **[client-v2]** Fixed container query parameters being sent unquoted, so `Client.query(sql, params, settings)` binding + a `List` (or an array/`Map`) to a placeholder like `{ids:Array(Date)}` was rejected by the server with + `CANNOT_PARSE_INPUT_ASSERTION_FAILED`. Parameter values are now formatted by + `DataTypeConverter#convertParameterToString(Object)` before being sent: pass the raw Java value and the client renders + it into the text the server's `param_` interface expects — a `Collection`, array (object or primitive), or `Map` + becomes ClickHouse `Array` (`['2026-05-13']`) / `Map` (`{'k':'v'}`) text with `String`/temporal leaves single-quoted + and numeric/boolean leaves left unquoted, while a scalar is passed through unquoted as before. No manual + pre-formatting of container parameters is needed. (https://github.com/ClickHouse/clickhouse-java/issues/2897) + +- **[client-v2]** Fixed `DateTime`/`DateTime64` columns declared with a synthetic fixed-offset timezone name + (`Fixed/UTC±HH:MM:SS`, e.g. `Fixed/UTC+05:30:00`) being silently read in UTC instead of the declared offset. The + `RowBinary` reader now recovers the offset from the column's declared type. (https://github.com/ClickHouse/clickhouse-java/issues/2876) + +- **[jdbc-v2]** Fixed the ANTLR4 SQL parser backends (`jdbc_sql_parser=ANTLR4` and `ANTLR4_PARAMS_PARSER`) lexing + the body of a heredoc string (`$$body$$`, `$tag$body$tag$`) as ordinary SQL. The lexer had no heredoc token, so + every `$` was dropped as an unrecognized character and the body was parsed as identifiers, operators and + statement separators: a body that still looked like valid SQL was silently mis-parsed (wrong table name and + VALUES-list positions), and a body containing `;` — as well as the empty heredoc `$$$$` — was reported as a + parse error, which classifies an INSERT as a result-set-bearing statement with no values-list positions. A + heredoc is now lexed as a single string literal and accepted as a literal value (`INSERT ... VALUES` lists, + column expressions, settings), and a `$` inside an identifier (`a$b`, or an unterminated tag such as + `$foo$bar`) is part of the identifier, as the server reads it. + (https://github.com/ClickHouse/clickhouse-java/issues/3031) + +- **[jdbc-v2]** Fixed the beta RowBinary writer (`DriverProperties.BETA_ROW_BINARY_WRITER`) throwing + `NoSuchColumnException` for `INSERT` statements whose column names are backtick-quoted, in particular the + canonical `Nested` sub-column wire form `` `directory`.`id` ``. The SQL parser now unescapes each + backtick-quoted `INSERT` column-name component before the by-name server-schema lookup, matching how the + table and database identifiers are already handled. (https://github.com/ClickHouse/clickhouse-java/issues/2896) + +### Updated Dependencies + +- **[repo]** Upgraded `org.apache.httpcomponents.client5:httpclient5` from `5.4.4` to `5.6.4` in `client-v2` and + `clickhouse-http-client` to pick up the fixes of the newer 5.x releases, including known vulnerabilities. + (https://github.com/ClickHouse/clickhouse-java/issues/3078) + +### Docs & Examples + +- **[examples]** Converted the remaining Gradle-based example projects (`client-v2-apache-arrow`, `demo-service`, + `demo-kotlin-service`) to Maven so that every project under `examples/` builds with a single, consistent toolchain. + The Gradle wrapper and build scripts were removed and each project now has a standalone `pom.xml`. + (https://github.com/ClickHouse/clickhouse-java/issues/2915) + +## 0.10.0, + +[Release Migration Guide](docs/releases/0_10_0.md) + +### Breaking Changes + +- **[jdbc-v2]** The driver no longer hardcodes the server settings `async_insert=0` and `wait_end_of_query=0` on every JDBC + connection. This unblocks two scenarios that previously did not work: overriding these settings per connection or per + statement, and using the driver against read-only profiles that disallow `SETTINGS` overrides. There are two consequences: + - The driver now follows the server-side defaults for these settings (note: starting with ClickHouse 26.3, `async_insert` + defaults to `1`). Removing the explicit `wait_end_of_query=0` is a no-op against server defaults but lets users opt in to + `wait_end_of_query=1`. + - The row count returned by `java.sql.Statement.executeUpdate(java.lang.String)` (and the matching `PreparedStatement` + method) is no longer guaranteed to be accurate for INSERT statements when the server runs them asynchronously, and + parsing/data errors in the INSERT body may not surface synchronously as a `SQLException`. Previously these were + accurate because inserts were forced to be synchronous (see also https://github.com/ClickHouse/ClickHouse/issues/57768). + To restore the previous behavior, set `async_insert=0` (or `wait_for_async_insert=1`) per connection as server setting. + Read more about asynchronous insert: https://clickhouse.com/docs/optimize/asynchronous-inserts. + + (https://github.com/ClickHouse/clickhouse-java/issues/2652, https://github.com/ClickHouse/clickhouse-java/issues/2825) + +- **[client-v2]** `Client.Builder#build()` now throws `ClientMisconfigurationException` instead of + `IllegalArgumentException` for authentication and SSL misconfiguration (missing credentials, conflicting + authentication methods, missing client certificate when SSL authentication is enabled, and trust store used together + with a client certificate). Callers that relied on catching `IllegalArgumentException` from `build()` for these cases + must catch `ClientMisconfigurationException` (which extends `RuntimeException` via `ClientException`). (https://github.com/ClickHouse/clickhouse-java/pull/2812) + +- **[client-v2]** Combining `setUsername(...)` + `setPassword(...)` with a custom `Authorization` HTTP header ( + `httpHeader(HttpHeaders.AUTHORIZATION, ...)`) now fails at `Client.Builder#build()` with + `ClientMisconfigurationException` unless HTTP Basic authentication is explicitly disabled via + `useHTTPBasicAuth(false)`. Previously this combination was accepted and the custom `Authorization` header overrode + the user/password at request time. (https://github.com/ClickHouse/clickhouse-java/pull/2812) + +- **[client-v2]** The `access_token` configuration property (set via `Client.Builder#setAccessToken(String)` or directly + through `setOption`) is now actually applied to outgoing requests as the `Authorization` HTTP header value verbatim. + Previously the value was stored under `access_token` but never sent on the wire, so providing it alone had no effect + on authentication. Callers must include the scheme prefix themselves (e.g. `setAccessToken("Bearer ")`), or use + `useBearerTokenAuth(String)` which prepends `Bearer ` automatically. (https://github.com/ClickHouse/clickhouse-java/pull/2812) + +- **[client-v2]** `Client.Builder#useBearerTokenAuth(String)` now stores the bearer token under the `access_token` + configuration key (with the `Bearer ` prefix) instead of writing it directly into `http_header_authorization`. The + HTTP wire format is unchanged, but the token is no longer observable through `Client#getReadOnlyConfig()` under the + `http_header_authorization` key. (https://github.com/ClickHouse/clickhouse-java/pull/2812) + +- **[client-v2]** Fixed inconsistent use of `executionTimeout` parameter in `Client` component. The timeout was + previously set in milliseconds but mistakenly retrieved and used in seconds in some places. Now it correctly uses + milliseconds consistently. (https://github.com/ClickHouse/clickhouse-java/issues/2358) + +- **[client-v2]** The public `ClickHouseBinaryFormatWriter` interface gained two methods, `setString(String, byte[])` + and `setString(int, byte[])`, for writing raw `String`/`FixedString` bytes. Code that only *uses* the interface is + unaffected, but any third party that *implements* `ClickHouseBinaryFormatWriter` directly is source- and + binary-incompatible until it adds these methods (recompiling against the new version is required; otherwise an + `AbstractMethodError` can occur at runtime). + +- **[client-v2]** HTTP `503 Service Unavailable` responses are now surfaced as a connection-style failure ( + `java.net.ConnectException`) and are retried by default. Previously a `503` was treated as a server error ( + `ServerException`) and fell under the `ServerRetryable` fault cause. It has been moved to the `ConnectTimeout` fault + cause category so that connectivity/availability failures are handled uniformly with other connection errors. Callers + that specifically excluded `ServerRetryable` to avoid retrying `503` should now adjust their + `client_retry_on_failures` configuration to exclude `ConnectTimeout` instead. + +- **[client-v2]** Unexpected/unknown HTTP status codes (those the client cannot interpret as a ClickHouse response) now + throw a `ClientException` instead of a `ServerException`. Since the client cannot meaningfully handle these responses, + they are reported as a client-side error rather than being attributed to the server. + +### New Features + +- **[client-v2, jdbc-v2]** Added support for an application-supplied `javax.net.ssl.SSLContext`. In client-v2, + `Client.Builder.setSSLContext(SSLContext)` hands the client a fully pre-built context that is used as is; in + jdbc-v2 the same context may be passed as a live object in the connection `Properties` under the `ssl_context` + key (added with `Properties.put`, since it is not a string). Trust/key material options cannot be combined with + a custom context and are rejected; `ssl_mode` still applies but only to server hostname verification. This + supports in-memory TLS material that must never be written to disk, including behind connection pools that only + expose `java.util.Properties`. + - Examples for client-v2 https://github.com/ClickHouse/clickhouse-java/blob/main/examples/client-v2/src/main/java/com/clickhouse/examples/client_v2/SSLExamples.java + - Examples for jdbc-v2 https://github.com/ClickHouse/clickhouse-java/blob/main/examples/jdbc/src/main/java/com/clickhouse/examples/jdbc/SSLExamples.java + (https://github.com/ClickHouse/clickhouse-java/pull/2918, https://github.com/ClickHouse/clickhouse-java/issues/2909) + +- **[jdbc-v2, client-v2]** Implemented SSL modes configuration. Now it is possible to set `ssl_mode` to `DISABLED`, + `TRUST`, `VERIFY_CA` and `STRICT`. Note for V1 users: `NONE` is supported only by JDBC driver and mapped to `TRUST`. + Please migrate to the new naming. + - Examples for client-v2 https://github.com/ClickHouse/clickhouse-java/blob/main/examples/client-v2/src/main/java/com/clickhouse/examples/client_v2/SSLExamples.java + - Examples for jdbc-v2 https://github.com/ClickHouse/clickhouse-java/blob/main/examples/jdbc/src/main/java/com/clickhouse/examples/jdbc/SSLExamples.java + (https://github.com/ClickHouse/clickhouse-java/pull/2874, https://github.com/ClickHouse/clickhouse-java/issues/2389, + https://github.com/ClickHouse/clickhouse-java/issues/2309, https://github.com/ClickHouse/clickhouse-java/issues/2819) + +- **[jdbc-v2, client-v2] (beta)** Implemented standalone readers for `JSONEachRow` to provide scaffold for + reading this format. Additionally, it gives a way to map `Json` columns to custom types using JDBC driver. See examples + in https://github.com/ClickHouse/clickhouse-java/tree/main/examples/jdbc-v2-json-processors and https://github.com/ClickHouse/clickhouse-java/tree/main/examples/client-v2-json-processors. + (https://github.com/ClickHouse/clickhouse-java/pull/2871) + +- **[client-v2]** Added `Session` API to encapsulate and manage ClickHouse session settings (`session_id`, + `session_check`, `session_timeout`, `session_timezone`) as a reusable object. The `Session` instance can be applied to + any request settings using `applyTo()`, and session state can be cleared via `clearSession()`. Additionally, added + `resetOption(String)` to `InsertSettings`, `QuerySettings`, and `CommonSettings` to allow removing specific settings. + Settings explicitly set to `null` will not be sent to the server, which is useful for overriding global settings. + (https://github.com/ClickHouse/clickhouse-java/pull/2810) + +- **[client-v2]** Added runtime credential update APIs on `Client`: `updateUserAndPassword(String, String)`, + `updateAccessToken(String)`, and `updateBearerToken(String)`. Subsequent requests on the same `Client` instance use + the new credentials without rebuilding the client. The authentication method is fixed at construction time; calling a + runtime updater that does not match the configured method throws `ClientMisconfigurationException`. See + `docs/authentication.md` for details and migration guidance. (https://github.com/ClickHouse/clickhouse-java/pull/2812) + +- **[jdbc-v2]** Added `cluster_name` configuration property to specify a target cluster for statements like `KILL QUERY` + that require an `ON CLUSTER` clause to execute across all + nodes. (https://github.com/ClickHouse/clickhouse-java/issues/2837) + +- **[client-v2, jdbc-v2]** Added support for ClickHouse `Geometry` type for ClickHouse `25.11+`, where `Geometry` + changed from a `String` alias to `Variant(Point, Ring, LineString, MultiLineString, Polygon, MultiPolygon)` (client + still compatible with older versions). Includes client read/write handling and JDBC type mapping for retrieving and + inserting geometry values. Current writes infer the target geometry variant from array nesting depth, so `Ring` vs + `LineString` and `Polygon` vs `MultiLineString` are not yet distinguishable through the generic `Geometry` write + path. (https://github.com/ClickHouse/clickhouse-java/pull/2815) + +- **[jdbc-v2]** `ResultSet#getObject(int|String, Map>)` now accepts ClickHouse type names as map keys + in addition to the JDBC `SQLType` names it has always accepted. Only unwrapped type names are used for the lookup — + `Nullable(...)` and `LowCardinality(...)` wrappers are stripped and do not affect resolution, so a key like `"Int32"` + matches both `Int32` and `Nullable(Int32)` columns; keys like `"Nullable(Int32)"` are not recognized. Lookup order is + the `ClickHouseDataType` enum name (e.g. `"Int32"`, `"String"`, `"DateTime"`) then the JDBC `SQLType` name (e.g. + `"INTEGER"`, `"VARCHAR"`, `"TIMESTAMP"`); a missing entry leaves the value uncoerced. The feature is supported for + primitive ClickHouse types only — `Array`, `Tuple`, `Map`, `Nested`, and geometry types are not supported and continue + to be returned in their native form regardless of the user-supplied map. Existing maps keyed only by JDBC `SQLType` + names continue to work unchanged. (https://github.com/ClickHouse/clickhouse-java/pull/2865) + +- **[jdbc-v2]** Added support of custom mapping for JDBC types. Mainly used in cases when big integers should be + presented as string. Use `DriverProperties.JDBC_TYPE_MAPPINGS` (`jdbc_type_mappings`) and set needed type mapping + as `key=value[,]` list (For example, `Int32=Long,UInt64=String`). Deprecation notice: V1 property `typeMappings` is + supported but will be removed. Please migrate to the new property. + (https://github.com/ClickHouse/clickhouse-java/issues/2858) + +- **[client-v2, jdbc-v2]** Added opt-in binary string support through the `binary_string_support` configuration property + (or `Client.Builder#binaryStringSupport(boolean)`), disabled by default. The setting is resolved per operation from + the merged client and query settings, so it can be overridden for a single request via the `binary_string_support` + operation option (e.g. `QuerySettings#setOption(ClientConfigProperties.BINARY_STRING_SUPPORT.getKey(), true)`) + independently of the client-level default. When enabled, top-level `String` and `FixedString` columns are read + into a `StringValue` that preserves the raw bytes instead of decoding them into a `String`, allowing non-UTF-8/binary + content to round-trip byte-for-byte. `StringValue` exposes the bytes via `toByteArray()`/`asByteBuffer()` and + lazily decodes a `String` via `asString()` (UTF-8 by default, or a caller-supplied `Charset`). Values nested inside + containers (`Array`, `Map`, `Tuple`, `Nested`, `Variant`) continue to be read as `String`, since those types are not + expected to carry large/binary strings. On the JDBC side, `ResultSet#getBinaryStream(int)` and + `ResultSet#getBinaryStream(String)` are now implemented (previously unsupported) and, together with `getBytes(...)`, + return the raw column bytes. + +- **[client-v2]** Added `Client#cancelTransportRequest(String queryId)` to cancel an in-flight request that has not yet + received a response from the server, identified by the query id supplied in the operation settings. This aborts the + request on the client side (cancels the underlying IO operation) but does **not** issue a `KILL QUERY` on the server, + so a query that already started executing may continue to run server-side. It is recommended to use operation timeout + settings where possible; this API is intended for explicitly aborting a request from the client. + +### Improvements + +- **[jdbc-v2, client-v2]** Added support of hostnames with underscore (`_`) in them. Now it is possible to specify endpoint +like `ch_db_01`. This is mostly used in k8s environment. (https://github.com/ClickHouse/clickhouse-java/issues/2792, + https://github.com/ClickHouse/clickhouse-java/issues/2753) + +### Updated Dependencies + +- **[tests]** Bump org.postgresql:postgresql from 42.6.1 to 42.7.11 + +### Docs & Examples + +- **[client-v2]** Added example of working with `Apache Arrow` library using client. (https://github.com/ClickHouse/clickhouse-java/pull/2820) + +- **[repo]** Added a contribution guide. Please review and send us your feedback. (https://github.com/ClickHouse/clickhouse-java/pull/2859) + +- **[client-v2]** Added endpoint failover support: when multiple endpoints are configured and a request fails with a + retryable error (connect timeout, connection refused, HTTP 503, etc.), the client now automatically retries against + the next available endpoint instead of always targeting the first one. Failed endpoints are quarantined for 30 seconds + before being retried. (https://github.com/ClickHouse/clickhouse-java/issues/2855) + +### Bug Fixes + +- **[jdbc-v2, client-v2]** Fixed error handling for responses that not a ClickHouse error, like `404` response. (https://github.com/ClickHouse/clickhouse-java/issues/2803) + +- **[jdbc-v2, client-v2]** Fixed setting `null` as password. Previously it was converted to `null` literal. Still we recommend +passing empty string. (https://github.com/ClickHouse/clickhouse-java/pull/2809) + +- **[jdbc-v2, client-v2]** Fixed `ClickHouseBinaryFormatReader::getBigDecimal` silently truncating big integer. +(https://github.com/ClickHouse/clickhouse-java/issues/2748) + +- **[jdbc-v2, client-v2]** Fixed handling `NULL` values to `Variant` and `Dynamic` columns. Previously indicator +of `NULL` was not set and read. (https://github.com/ClickHouse/clickhouse-java/issues/2789, https://github.com/ClickHouse/clickhouse-java/issues/2791) + +- **[jdbc-v2]** Fixed `Statement.cancel()` throwing `SESSION_IS_LOCKED` when the statement was running inside a + ClickHouse session. The driver now accepts `session_id`, `session_check`, and `session_timeout` as first-class + connection properties and correctly suppresses them when issuing a `KILL QUERY` during cancellation. This ensures the + cancellation request runs outside the session and no longer contends with the running query for the session + lock. (https://github.com/ClickHouse/clickhouse-java/issues/2690, https://github.com/ClickHouse/clickhouse-java/issues/2881) + +- **[jdbc-v2]** Added option to specify cluster name for operations on cluster. One of them is `KILL QUERY .. ON CLUSTER `. + Use `DriverProperties.CLUSTER_NAME` (`jdbc_cluster_name`) to define name of the cluster to be used in such queries. + (https://github.com/ClickHouse/clickhouse-java/issues/2837) + +- **[jdbc-v2, client-v2]** Fixed writing nullable marker for nested `Tuple` and `Map values. (https://github.com/ClickHouse/clickhouse-java/issues/2721) + +- **[jdbc-v2]** Fixed `ResultSet.getObject` leaking the internal `StringValue` holder for `String`/`FixedString` + columns when `binary_string_support` is enabled. `getObject(column, byte[].class)` now returns the exact raw bytes, + and `getObject(column, Object.class)` and the no-type `getObject(column)` overloads now return a decoded `String` + instead of the internal holder. + +## 0.9.8 + +### Improvements + +- **[client-v2]** Added `Records#getSchema()` to expose table schema metadata even when query result is empty. (https://github.com/ClickHouse/clickhouse-java/pull/2777) + +### Bug Fixes + +- **[jdbc-v1, jdbc-v2]** Fixed type mapping/conversion behavior for large unsigned integer values (issue #2779), including related test coverage updates. (https://github.com/ClickHouse/clickhouse-java/issues/2779) + +- **[jdbc-v2]** Fixed off-by-one bug in `ArrayResultSet#next()` that could return `true` one extra time and then fail with `SQLException: No current row`. (https://github.com/ClickHouse/clickhouse-java/pull/2790) + +- **[jdbc-v2]** Fixed handling of malformed `SELECT`-like queries (for example, `SELECT a, FROM table`): previously, the driver could fail to read returned data when query type was not inferred before execution; now it detects a returned result set at runtime from the server response and reads it correctly. (https://github.com/ClickHouse/clickhouse-java/issues/2784) + +### Updated Dependencies + +- **[repo]** Upgraded `at.yawk.lz4:lz4-java` from `1.10.2` to `1.10.4` to pick up native performance fixes. (https://github.com/ClickHouse/clickhouse-java/pull/2778) + +- **[repo]** Upgraded `com.fasterxml.jackson.core:jackson-core` from `2.17.2` to `2.18.6` in `jdbc-v2`. (https://github.com/ClickHouse/clickhouse-java/pull/2767) + +- **[repo]** Upgraded `com.fasterxml.jackson.core:jackson-core` from `2.17.2` to `2.18.6` in `examples/client-v2`. (https://github.com/ClickHouse/clickhouse-java/pull/2766) + +## 0.9.7 + +### Breaking Changes + +- **[client-v2]** `Client.Builder#build()` now throws `ClientMisconfigurationException` when an unknown configuration property is passed. Previously, unknown properties were silently ignored what caused unexpected behaviour and problems. To restore the old behavior, add `ignore_unknown_config_key=true` to the client properties. JDBC driver-only properties (e.g. `ssl`) are no longer forwarded to the underlying client. (https://github.com/ClickHouse/clickhouse-java/issues/2658) + +- **[client-v2]** `Date`/`Date32` columns are now decoded as `LocalDate` without any timezone adjustment. Previously, `Date` values could be returned as `ZonedDateTime`. Applications that need a `ZonedDateTime` must construct it from the returned `LocalDate` using their own timezone context. More read about it - https://clickhouse.com/docs/integrations/language-clients/java/jdbc_date_time_guide (https://github.com/ClickHouse/clickhouse-java/pull/2727) + +### Improvements + +- **[client-v2]** Added option to send query parameters in the HTTP request body using multipart form encoding. Enabled via the new `client.http.use_form_request_for_query` configuration property (or `Client.Builder#useHttpFormDataForQuery()`). Useful when query parameters would exceed URL length limits. (https://github.com/ClickHouse/clickhouse-java/issues/2324) + +- **[client-v2]** `QueryResponse`, `InsertResponse`, and `CommandResponse` now expose `getServerDisplayName()` to retrieve the value of the `X-ClickHouse-Server-Display-Name` HTTP response header. A full map of whitelisted response headers is accessible via `getResponseHeaders()`. (https://github.com/ClickHouse/clickhouse-java/issues/2347) + +- **[client-v2]** Added `getObjectArray()` to `ClickHouseBinaryFormatReader` and `GenericRecord` to read `Array(...)` columns as `Object[]`, including recursive support for nested arrays. `getStringArray()` now also handles `Array(Enum*)` by returning enum names. (https://github.com/ClickHouse/clickhouse-java/issues/2738) + +- **[jdbc-v2]** Added support for reading and writing `byte[]` for `String`/`FixedString` columns. `PreparedStatement#setBytes()` encodes the value as a ClickHouse `unhex()` expression to preserve raw bytes; `ResultSet#getBytes()` returns the UTF-8 bytes of the string value. (https://github.com/ClickHouse/clickhouse-java/pull/2734) + +#### Date and Time Handling + +- **[docs]** Added a [document](https://clickhouse.com/docs/integrations/language-clients/java/jdbc_date_time_guide) that explains how driver handles different date/time values. + +- **[client-v2, jdbc-v2]** Fixed writing `Date`/`Time`/`Timestamp` values so that timezone conversions are performed explicitly. Previously, timezone-naïve encoding caused day-shift and precision bugs, especially when a `Calendar` or non-default JVM timezone was in use. (https://github.com/ClickHouse/clickhouse-java/issues/2701, https://github.com/ClickHouse/clickhouse-java/issues/2065, https://github.com/ClickHouse/clickhouse-java/issues/2496, https://github.com/ClickHouse/clickhouse-java/issues/1220, https://github.com/ClickHouse/clickhouse-java/issues/1117, https://github.com/ClickHouse/clickhouse-java/issues/1048, https://github.com/ClickHouse/clickhouse-java/issues/2381, https://github.com/ClickHouse/clickhouse-java/issues/1735, https://github.com/ClickHouse/clickhouse-java/issues/2542, https://github.com/ClickHouse/clickhouse-java/issues/2557) + +- **[client-v2, jdbc-v2]** Fixed reading `Date`/`Time`/`DateTime` values to be consistent and correct. `Date`/`Date32` are now decoded as `LocalDate` (no timezone), while `Time`/`Time64` are decoded as UTC-based `LocalDateTime`. New `getLocalTime()` API added to readers for `LocalTime` access. Reading date/time from `Dynamic` and `Variant` columns is also fixed. (https://github.com/ClickHouse/clickhouse-java/pull/2727) + +- **[jdbc-v2]** `DatabaseMetaDataImpl#getTypeInfo()` now returns type information in deterministic alphabetical order, matching v1 driver behavior. (https://github.com/ClickHouse/clickhouse-java/pull/2733) + +### SQL Parser (jdbc-v2) + +- **[jdbc-v2]** Fixed SQL parser failing to handle ClickHouse keywords used as table names or column aliases (e.g. `AS value`, `FROM date`). Both the ANTLR4 and JavaCC parsers now maintain a list of non-reserved keywords allowed in identifier positions. Also fixed parsing of `DESCRIBE (SELECT ...)`, `IN` operator in `SELECT`, and bracketed array access in expressions. (https://github.com/ClickHouse/clickhouse-java/issues/2718) + +- **[jdbc-v2]** Updated SQL parsers to recognise new ClickHouse keywords (`ABI`, `ARGUMENTS`, `DRY`, `LANGUAGE`, `RETURNS`, `RUN`) introduced in recent ClickHouse HEAD builds. Previously, queries using these keywords as identifiers would fail to parse. (https://github.com/ClickHouse/clickhouse-java/pull/2761) + +- **[jdbc-v2]** Fixed SQL parser failing on statements containing the `^` (caret) operator, such as `toJSONString(data.^header_index)`. A new `CARET` token is now recognised and allowed in dotted identifier chains. (https://github.com/ClickHouse/clickhouse-java/issues/2768) + +### Array and Collection Handling + +- **[jdbc-v2]** Fixed `DataTypeConverter` causing NPE when converting nested arrays to strings (e.g. `ResultSet#getString()` on an `Array(Array(...))` column). The converter is now re-entrancy-safe. Previously was throwing `NullPointerException`. (https://github.com/ClickHouse/clickhouse-java/issues/2723) + +- **[client-v2, jdbc-v2]** Fixed conversion of multi-dimensional `List`/array values into Java arrays. Child array dimensions are now calculated correctly from the parent depth level. Previously, converting `List>` or similar structures produced incorrect results. (https://github.com/ClickHouse/clickhouse-java/issues/2741) + +- **[client-v2]** Fixed binary array decoding when the first element of an array is an empty `Map` or `List`. Previously, this caused a type mismatch error during array population. (https://github.com/ClickHouse/clickhouse-java/issues/2499, https://github.com/ClickHouse/clickhouse-java/issues/2657, https://github.com/ClickHouse/clickhouse-java/issues/2703) + +### Bug Fixes + +- **[jdbc-v2]** Fixed `DatabaseMetaDataImpl#getJDBCMajorVersion()` and `getJDBCMinorVersion()` returning the ClickHouse driver version instead of the supported JDBC specification version. Now correctly returns `4` and `2` (JDBC 4.2). (https://github.com/ClickHouse/clickhouse-java/pull/2736) + +- **[jdbc-v2]** Added mapping for all known ClickHouse table engines to JDBC table types. Added `MATERIALIZED VIEW` as a new table type. `getTables()` now correctly reports remote/external engine tables that were previously invisible. (https://github.com/ClickHouse/clickhouse-java/issues/2664) + +- **[jdbc-v2]** Fixed `NullPointerException` in JDBC type metadata for columns of type `Map`, `IPv4`, `IPv6`, `UUID`, `BFloat16`, `Decimal256`, `Geometry`, all `Interval*` types, `JSON`, `LowCardinality`, `Nullable`, `Variant`, `Dynamic`, and others. Previously these types had no Java class mapping. (https://github.com/ClickHouse/clickhouse-java/issues/2711) + +- **[jdbc-v2]** ClickHouse server error codes are now propagated to `SQLException#getErrorCode()`. Previously, `SQLException` was created without the vendor error code, making it impossible to distinguish error types programmatically. (https://github.com/ClickHouse/clickhouse-java/issues/2717) + +- **[client-v2]** Fixed `max_execution_time` not being sent correctly to the server. Previously, it was treated as a client option and ignored by ClickHouse; it is now transmitted as a server setting. (https://github.com/ClickHouse/clickhouse-java/issues/2750) + +- **[client-v2]** Fixed `hasValue(columnName)` throwing an exception when the column name does not exist. Previously, referencing an unknown column or an out-of-range index via `hasValue()` would throw; it now returns `false`. (https://github.com/ClickHouse/clickhouse-java/issues/2755) + +- **[jdbc-v2]** Fixed `PreparedStatement` reporting missing parameters using 0-based indices in the error message. JDBC parameter positions are 1-based; the error message now reflects this correctly. (https://github.com/ClickHouse/clickhouse-java/pull/2749) + +- **[jdbc-v2]** Ported legacy JDBC v1 connection properties (`CUSTOM_HTTP_PARAMS`, `CUSTOM_SETTINGS`, `HTTP_CONNECTION_PROVIDER`, `REMEMBER_LAST_SET_ROLES`, `USE_SERVER_TIME_ZONE_FOR_DATES`) to JDBC v2. Required for integrations that share JDBC v1/v2 configuration code. `custom_http_params` and `custom_settings` values are converted to server settings automatically. (https://github.com/ClickHouse/clickhouse-java/pull/2757) + +- **[client-v2, jdbc-v2]** Fixed `custom_`-prefixed configuration properties not being forwarded to the server as custom settings when using the `custom_` prefix. Also fixed driver-only properties (e.g. `ssl`) being incorrectly forwarded to the HTTP client. (https://github.com/ClickHouse/clickhouse-java/issues/2658) + +## 0.9.6 +Release is aimed to address potential security risk in one of the dependencies (see below). We strongly recommend to upgrade. + +### Dependencies +- [repo] - upgraded `lz4-java` to `1.10.2`. Previously used version had a vulnerability https://www.cve.org/CVERecord?id=CVE-2025-66566. (https://github.com/ClickHouse/clickhouse-java/pull/2706) +- [repo] - upgraded `commons-lang3` to `3.20.0`. Previously used version had a vulnerability https://www.cve.org/CVERecord?id=CVE-2025-48924. (https://github.com/ClickHouse/clickhouse-java/pull/2696) + +### New Features +- [jdbc-v2] - support of dot notation for table names without quotes. (https://github.com/ClickHouse/clickhouse-java/issues/2650) + +## 0.9.5 + +### New Features +- [client-v2] Log durations in ISO-8601 duration format (https://github.com/ClickHouse/clickhouse-java/pull/2660) +- [jdbc-v2] Added support for `getResultSet()` method in Array data types. (https://github.com/ClickHouse/clickhouse-java/issues/1545, https://github.com/ClickHouse/clickhouse-java/issues/2683) +- [jdbc-v2] Added handling for `Time` and `Time64` data types in JDBC. (https://github.com/ClickHouse/clickhouse-java/issues/2682) + +### Bug Fixes +- [client-v2] Fixed reading columns of `Array` with component type `Variant`. For example, `Array(Variant(String, Int32))`. (https://github.com/ClickHouse/clickhouse-java/issues/2602) +- [jdbc-v2] Fixed backward compatibility for `getPrimaryKeys()` method. (https://github.com/ClickHouse/clickhouse-java/issues/2654) +- [jdbc-v2] Fixed array conversion issues with multidimensional arrays and nested arrays. (https://github.com/ClickHouse/clickhouse-java/issues/2457) +- [jdbc-v2] Fixed backward compatibility issues with older ClickHouse versions by replacing `::` cast operator with `CAST` function in prepared statements. Fixed issue with reading database version on older versions. (https://github.com/ClickHouse/clickhouse-java/issues/2654, https://github.com/ClickHouse/clickhouse-java/issues/2227) +- [jdbc-v2] Reviewed and fixed database metadata support flags and return values. (https://github.com/ClickHouse/clickhouse-java/issues/2520, https://github.com/ClickHouse/clickhouse-java/issues/778, https://github.com/ClickHouse/clickhouse-java/issues/2519) +- [jdbc-v2] Added verification tests for issues #1999, #1015, #2155 (https://github.com/ClickHouse/clickhouse-java/pull/2659) + + +## 0.9.4 + +### New Features +- [client-v2] Added support for different compression algorithms when HTTP compression is enabled. (https://github.com/ClickHouse/clickhouse-java/pull/2645) + +### Bug Fixes +- [client-v1] Fixes issue linked to a enabled by default HTTP compression in ClickHouse 25.10. (https://github.com/ClickHouse/clickhouse-java/issues/2636) + +## 0.9.3 + +### Important Changes +- [jdbc-v2] SQL parser from v1 is ported to v2 to address multiple issues with SQL parsing. The ANTLR4-based parser is +still an option and will be developed further. The main difference between parses is completeness of their grammar: +JavaCC grabs only needed information and skips parsing of the rest (what makes it work for most cases) while ANTLR4 +has more complete grammar and can detect type of some complex statements more accurate than JavaCC. +To use it set `com.clickhouse.jdbc.DriverProperties#SQL_PARSER` to `ANTLR4`. + (https://github.com/ClickHouse/clickhouse-java/pull/2579). This fixes issue: + - https://github.com/ClickHouse/clickhouse-java/issues/2574 + - https://github.com/ClickHouse/clickhouse-java/issues/2568 + - https://github.com/ClickHouse/clickhouse-java/issues/2537 + - https://github.com/ClickHouse/clickhouse-java/issues/2595 + - https://github.com/ClickHouse/clickhouse-java/issues/2617 + - https://github.com/ClickHouse/clickhouse-java/issues/2569 + - https://github.com/ClickHouse/clickhouse-java/issues/2570 + - https://github.com/ClickHouse/clickhouse-java/issues/2571 + - https://github.com/ClickHouse/clickhouse-java/issues/2572 + - https://github.com/ClickHouse/clickhouse-java/issues/2573 + - https://github.com/ClickHouse/clickhouse-java/issues/2609 + - https://github.com/ClickHouse/clickhouse-java/issues/2527 + +- [repo] New artifact `clickhouse-jdbc-all` added to address issue when maven package qualifiers may not be used. +This artifact should is a copy of `clickhouse-jdbc:all` but should be used only when required. (https://github.com/ClickHouse/clickhouse-java/issues/2625) + +### Improvements +- [client-v2] Added `getShortArray()` and `getStringArray()` to `ClickHouseBinaryFormatReader`. (https://github.com/ClickHouse/clickhouse-java/pull/2604) +- [client-v2] Added `result_rows` and `elapsed_time` to summary object. (https://github.com/ClickHouse/clickhouse-java/pull/1633/files) + +### Bug Fixes +- [jdbc-v2] Fixed issue with `maxRows` in `Statement` when additional settings were used to limit result set size. +It caused problems with read-only users because such users may not change settings in most cases. Now when `maxRows` is +set then `ResultSet` will skip extra rows. (https://github.com/ClickHouse/clickhouse-java/issues/2582) +- [jdbc-v2] Fixed issue with driver version. Previously version of a library was converted to minor and major version. +But this approach doesn't work well with `0.9.x` versions. Now major and minor versions are combined by shifting major. +Patch version becomes a minor version. (https://github.com/ClickHouse/clickhouse-java/issues/2410) +- [jdbc-v2, client-v2] Fixed converting different data types to a string. For example, there was an issue with IP +address when `toString()` was used and returned `\0.0.0.0` instead of `0.0.0.0`. (https://github.com/ClickHouse/clickhouse-java/issues/2575) +- [jdbc-v2] Fixed issues around spatial data (GEO types). (https://github.com/ClickHouse/clickhouse-java/issues/2577) +- [client-v2] Fixed issue with current user name. If user name is set then it will be used event after reading +server context. (https://github.com/ClickHouse/clickhouse-java/issues/2247) +- [client-v2] Fixed issue with network timeout settings when default value failed to be cast to Long. (https://github.com/ClickHouse/clickhouse-java/issues/2597) +- [jdbc-v2] Fixed getting metadata for nullable columns. (https://github.com/ClickHouse/clickhouse-java/issues/2586) +- [jdbc-v2, client-v2] Fixed issues related to reading JSON data type. Fixed reading JSON columns with arrays. Previously was causing exceptions like +`com.clickhouse.client.api.ClientException: Unsupported data type with tag 101 at ...` +(https://github.com/ClickHouse/clickhouse-java/issues/2598, https://github.com/ClickHouse/clickhouse-java/issues/2593, + https://github.com/ClickHouse/clickhouse-java/issues/2613, https://github.com/ClickHouse/clickhouse-java/issues/2102) +- [client-v2] Fixed configuration parameter type for `socket_linger` to match documentation. (https://github.com/ClickHouse/clickhouse-java/issues/2524) +- [client-v2] Fixed handling exceptions in http client code. Now response object is always closed to prevent connection leaking. (https://github.com/ClickHouse/clickhouse-java/pull/2615) +- [jdbc-v2, client-v2] Fixed issue with duplicate column names in a result set. (https://github.com/ClickHouse/clickhouse-java/issues/2459, https://github.com/ClickHouse/clickhouse-java/issues/2336) +- [jdbc-v2] Fixed ANTLR4 parse issue with `filter` clause along with aggregate function. (https://github.com/ClickHouse/clickhouse-java/pull/2631) + +## 0.9.2 + +### Improvements +- [jdbc-v2] `ResultSetImpl.getObject()` handles `java.time.Instant` + +### Improvements +- [jdbc-v2] Classes `com.clickhouse.jdbc.ClientInfoProperties` and `com.clickhouse.jdbc.DriverProperties` moved to public +API. (https://github.com/ClickHouse/clickhouse-java/pull/2521) +- [jdbc-v2] Implemented `isBeforeFirst`, `isAfterLast`, `isFirst`, `isLast` methods for `ResultSet` and `ResultSetMetaData`. +Improved test coverage for `ResultSetImpl`. (https://github.com/ClickHouse/clickhouse-java/pull/2530) +- [jdbc-v2] Implemented `createArray` and `createStruct` methods for `Connection` interface. Method `createStruct` +should be used to create `Tuple` values and `createArray` to create various arrays. (https://github.com/ClickHouse/clickhouse-java/pull/2523) +- [jdbc-v2] Implemented `setNetworkTimeout` of `Connection` interface. Used to fail fast when network operation fails. Related to +stale connection problem. (https://github.com/ClickHouse/clickhouse-java/pull/2522) +- [client-v2] Added support for JSON with predefined paths. Previously columns with definition like `JSON(a string, b.c Int32)` +were not supported. (https://github.com/ClickHouse/clickhouse-java/pull/2531) + +### Bug Fixes +- [jdbc-v2] Fixed issue creating array of tuples with `createArray` method of `Connection` interface. +(https://github.com/ClickHouse/clickhouse-java/issues/2360) +- [jdbc-v2] Fixed issue with reading nested arrays. (https://github.com/ClickHouse/clickhouse-java/issues/2539) +- [jdbc-v2] Fixed issue with not shaded antlr4-runtime dependency. Potential problem for Apache Spark users. (https://github.com/ClickHouse/clickhouse-java/issues/2553) +- [jdbc-v2] Fixed issue with parsing CTE for prepared statement. (https://github.com/ClickHouse/clickhouse-java/issues/2551) +- [jdbc-v2] Fixed issue with parsing SQL containing view parameters. (https://github.com/ClickHouse/clickhouse-java/issues/2547) +- [jdbc-v2] Fixed issue with `InsertSettings` when two concurrent insert operations sharing same settings object may be insert +wrong columns or to a wrong table. (https://github.com/ClickHouse/clickhouse-java/pull/2550) +- [jdbc-v2] Fixed issue with batch insert when it is not cleared after execution. Now batch is cleared after execution even on failure. (https://github.com/ClickHouse/clickhouse-java/issues/2548) +- [jdbc-v2] Fixed `DatabaseMetadataImpl` to return result set with defined by spec structure. Resolves issue for many database tools +relying on metadata. (https://github.com/ClickHouse/clickhouse-java/issues/2396) +- [jdbc-v2] Fixed `DatabaseMetadataImpl` to return empty result set where appropriate. (https://github.com/ClickHouse/clickhouse-java/issues/2517) +- [jdbc-v2] Fixed issue with verbose logging. (https://github.com/ClickHouse/clickhouse-java/issues/2148) +- [client-v2] Fixed issue with reading JSON with predefined paths. (https://github.com/ClickHouse/clickhouse-java/issues/2462) + +## 0.9.1 + +### New Features +- [client-v2] Added option to set SNI for SSL connections. See `com.clickhouse.client.api.Client.Builder.sslSocketSNI` (https://github.com/ClickHouse/clickhouse-java/pull/2467) +- [client-v2, jdbc-v2] Added support for `Time` and `Time64` data types. (https://github.com/ClickHouse/clickhouse-java/pull/2487) +- [client-v2] Added utility methods to format `Instant` to a DB DateTime format. See `com.clickhouse.client.api.DataTypeUtils.formatInstant(java.time.Instant, com.clickhouse.data.ClickHouseDataType)` (https://github.com/ClickHouse/clickhouse-java/pull/2501, https://github.com/ClickHouse/clickhouse-java/issues/2456) + + +### Improvements +- [client-v2] Improved exceptions and error handling. The `com.clickhouse.client.api.ClickHouseException` became a root for +`ServerException` and `ClientException`. `com.clickhouse.client.api.ClickHouseException.isRetryable` is added to determine if +exception caused in a retriable state. It means that catching such exception application may repeat operation to fix a +problem. (https://github.com/ClickHouse/clickhouse-java/pull/2453) +- [client-v2] Improved configuration handling. (https://github.com/ClickHouse/clickhouse-java/pull/2470) +- [client-v2] Custom type hint for the client to make `Reader` and `GenericRecord` return specific types. For example, +collection instead of `Array`. (https://github.com/ClickHouse/clickhouse-java/pull/2476) +- [client-v2] Added default port for HTTP and improved endpoint string validation - only one allowed. (https://github.com/ClickHouse/clickhouse-java/pull/2489) +- [client-v2] Improved handling error response. (https://github.com/ClickHouse/clickhouse-java/pull/2511) + +### Bug Fixes +- [client-v2] Fixed metrics to be parsed as Long to avoid integer overflow. Especially for elapsed time in nanos. (https://github.com/ClickHouse/clickhouse-java/pull/2465) +- [client-v2] Fixed a bug with passing some server settings from JDBC to the Client. (https://github.com/ClickHouse/clickhouse-java/issues/2359) +- [jdbc-v2] Fixed `DatabaseMetadata` to return correct column type. (https://github.com/ClickHouse/clickhouse-java/issues/2475) +- [jdbc-v2] Fixed problem with JDBC URL when Database names that include dash `-` were parsed wrong. (https://github.com/ClickHouse/clickhouse-java/issues/2463) +- [jdbc-v2] Fixed problem with nested arrays. Previously `ArrayValue` object was returned as element of a nested array. +After the fix array values contain primitive types. (https://github.com/ClickHouse/clickhouse-java/issues/2464, https://github.com/ClickHouse/clickhouse-java/issues/2340) +- [jdbc-v2] Fixed JDBC Connection's `isValid` method returning `true` all the time. (https://github.com/ClickHouse/clickhouse-java/issues/2472) +- [client-v2] Fixed problem with not closed response object when exception happens. That fixes connections leaking. (https://github.com/ClickHouse/clickhouse-java/issues/1741) +- [jdbc-v2] Fixed a set of issue in `StatementImpl`. (https://github.com/ClickHouse/clickhouse-java/issues/2414) +- [jdbc-v2] Fixed a set of issues in `PreparedStatementImpl`. (https://github.com/ClickHouse/clickhouse-java/issues/2418) +- [jdbc-v2] Fixed multiple issues linked to parsing statements in JDBC. (https://github.com/ClickHouse/clickhouse-java/issues/2450, https://github.com/ClickHouse/clickhouse-java/issues/2451, https://github.com/ClickHouse/clickhouse-java/issues/2461, https://github.com/ClickHouse/clickhouse-java/issues/2500, https://github.com/ClickHouse/clickhouse-java/issues/2493, https://github.com/ClickHouse/clickhouse-java/issues/2478) +- [jdbc-v2] Fixed result set returning not a DB compliant date/time format. (https://github.com/ClickHouse/clickhouse-java/issues/2408, https://github.com/ClickHouse/clickhouse-java/issues/2448) + +## 0.9.0 + +### Important Changes +- [repo] Artifacts `clickhouse-jdbc:http`, `clickhouse-jdbc:shaded-all`, `clickhouse-jdbc:shaded` are not published anymore. Use `clickhouse-jdbc:all` instead. It contains all required shaded libraries except `org.slf4j:slf4j-api` because it may not be shaded. +- [repo] Artifacts `jdbc-v2:all`, `jdbc-v2:http`, `jdbc-v2:shaded` are not published anymore because `jdbc-v2` is part of `clickhouse-jdbc:all`. + +### Bug Fixes +- [jdbc-v2] Fixed parsing SQL with alias in `FROM` clause when alias name is a keyword (https://github.com/ClickHouse/clickhouse-java/issues/2427) +- [client-v2] Fixed the issue when parameters are not passed if timeout is set. (https://github.com/ClickHouse/clickhouse-java/issues/2436, https://github.com/ClickHouse/clickhouse-java/issues/2096) +- [jdbc-v2] Fixed parser logging issue. Now parser uses same logger as the driver. (https://github.com/ClickHouse/clickhouse-java/issues/2428) +- [jdbc-v2] Fixed parsing `CREATE USER` statements. Now parser recognises it correctly. (https://github.com/ClickHouse/clickhouse-java/issues/2398) +- [jdbc-v2] Fixed parsing different CTEs statements. (https://github.com/ClickHouse/clickhouse-java/issues/2431, https://github.com/ClickHouse/clickhouse-java/issues/2391, https://github.com/ClickHouse/clickhouse-java/pull/2443) +- [jdbc-v2] Fixed parsing interval values. (https://github.com/ClickHouse/clickhouse-java/issues/2429) +- [repo] Fixed running unit/integration tests in the CI. (https://github.com/ClickHouse/clickhouse-java/issues/2430) +- [jdbc-v2] Fixed parsing parameters when function is in back quotes. (https://github.com/ClickHouse/clickhouse-java/issues/2422) +- [jdbc-v2] Fixed `ResultSet#findColumn` always returning `0`. (https://github.com/ClickHouse/clickhouse-java/issues/2375) + +### Misc +- [client-v2] Move all default values to `ClientConfigurationProperties` enum. (https://github.com/ClickHouse/clickhouse-java/issues/2269) + +### Docs +- [client-v2] Updated documentation of the configuration parameters and how to set server settings and custom HTTP headers. + +## 0.8.6 + +### Improvements +- [jdbc-v2] Now using Antlr4 to parse SQL statements to get meta information about it. +Invalid SQL still possible to execute if logic able to get all required information. (https://github.com/ClickHouse/clickhouse-java/pull/2351) +- [jdbc-v2] Now possible to use `RowBinaryWriter` if enabled by property `com.clickhouse.jdbc.internal.DriverProperties.BETA_ROW_BINARY_WRITER`. +This works well for batches. Single statements may get no benefits. (https://github.com/ClickHouse/clickhouse-java/pull/2316) + +### Bug Fixes + +- [jdbc-v2] Fixed parsing prepared statement arguments. (https://github.com/ClickHouse/clickhouse-java/issues/2348) +- [jdbc-v2] Fixed parsing role name when it contains `-`. (https://github.com/ClickHouse/clickhouse-java/issues/2325) +- [jdbc-v2] Fixed failure when `INSERT` doesn't contain `VALUES`. (https://github.com/ClickHouse/clickhouse-java/issues/2283) +- [jdbc-v2] Fixed parsing `INSERT` statement when `VALUES` is in lower case. (https://github.com/ClickHouse/clickhouse-java/issues/2354) +- [client-v2] Fixed NPE when async operation is request while client was built without an executor. +In this case default JVM executor will be used. (https://github.com/ClickHouse/clickhouse-java/issues/2355) +- [client-v2, jdbc-v2] Fixed conversion of IP addresses. When IPv4 stored as IPv6 it is correctly converted +to `Inet6Address`. Similar problem fixed for JDBC. (https://github.com/ClickHouse/clickhouse-java/issues/2342) +- [jdbc-v2] Fixed changing DB schema (DB name) on connection when `USE` statement executed. (https://github.com/ClickHouse/clickhouse-java/issues/2137) +- [client-v2] Fixed serializing POJO with primitive `boolean` fields. (https://github.com/ClickHouse/clickhouse-java/issues/2248) + +## 0.8.5 + +### Improvements +- [jdbc-v2] Added debug output for final SQL. (https://github.com/ClickHouse/clickhouse-java/issues/2249) + +### Bug Fixes +- [client-v2] Fixed creating TableSchema for materialized views. It resolves issues with POJO serde. (https://github.com/ClickHouse/clickhouse-java/issues/2118, +https://github.com/ClickHouse/clickhouse-java/issues/2025) +- [client-v2, jdbc-v2] Fixed handling `Nullable` inside `SimpleAggregateFunction` columns. (https://github.com/ClickHouse/clickhouse-java/issues/2110) +- [jdbc-v2] Fixed problem with server info request. It is fetched now when timezone of the server is set. (https://github.com/ClickHouse/clickhouse-java/issues/2191) +- [jdbc-v2] Fixed null response for `getIndexInfo()`. Empty Result Set is returned. (https://github.com/ClickHouse/clickhouse-java/issues/2286) +- [jdbc-v2] Fixed wrong `false` response in `DataBaseMetadata.supportsBatchUpdates()`. Returns `true` now. Please note that +no update is supported for result sets. +- [jdbc-v2] Fixed handling UUID data type in PreparedStatement. (https://github.com/ClickHouse/clickhouse-java/issues/2327) +- [jdbc-v2] Fixed unsigned integer type matching. `UInt8`, `UInt16`, `UInt32`, `UInt64`, `UInt128`, `UInt256` are presented as +`short`, `int`, `long`, `BigInteger`, `BigInteger`, `BigInteger` correspondingly. SQLType for them is `OTHER` because +JDBC (as well as Java) doesn't provide good mapping for unsigned integers. (https://github.com/ClickHouse/clickhouse-java/issues/2333) +- [jdbc-v2] Disallowed to call method from `Statement` interface on `PreparedStatement` instance according to the JDBC spec. (https://github.com/ClickHouse/clickhouse-java/issues/2339) + +## 0.8.4 + +### Examples +- [jdbc-v2] - JPA example added. (https://github.com/ClickHouse/clickhouse-java/pull/2301) + +### Bug Fixes +- [jdbc-v2] Added implementation of `ResultSetMetaData.getColumnClassName()` to return information +about class name of the value stored in a result. (https://github.com/ClickHouse/clickhouse-java/issues/2112) +- [client-v2] Fixed NPE when `Client.queryAll` used with `INSERT` statement because internally `columns` variable +was accessed and was null. (https://github.com/ClickHouse/clickhouse-java/issues/2150) +- [jdbc-v2] Fixed NPE when `ssl=true` was passed via an connection URL. (https://github.com/ClickHouse/clickhouse-java/issues/2206) +- [jdbc-v2] Fixed sending correct driver version. Problem occurs because context classloader had no access to +a resource file with versions. (https://github.com/ClickHouse/clickhouse-java/issues/2245) +- [jdbc-v2] Fixed incorrect flag for Array values when reading result set. (https://github.com/ClickHouse/clickhouse-java/issues/2266) +- [jdbc-v2] Fixed parsing parameters in PreparedStatement for cases when '?' is within quotes or similar. (https://github.com/ClickHouse/clickhouse-java/issues/2290) +- [jdbc-v2] Added implementation for `com.clickhouse.jdbc.PreparedStatementImpl#getMetaData`. +Complete metadata is returned only after statement execution. Partial metadata is returned before execution +of the statement. (https://github.com/ClickHouse/clickhouse-java/issues/2292) +- [jdbc-v2] Fixed `clearParameters` in `PreparedStatementImpl` to correctly reset parameters array. (https://github.com/ClickHouse/clickhouse-java/issues/2299) +- [jdbc-v2] Fixed logging. (https://github.com/ClickHouse/clickhouse-java/pull/2303) +- [jdbc-v2] Fixed metadata field `DATA_TYPE` being `String` (https://github.com/ClickHouse/clickhouse-java/issues/2240) + +## 0.8.3 + +### Improvements +- [client-v2] Support for native LZ4 compression (https://github.com/ClickHouse/clickhouse-java/issues/2274) + +### Bug Fixes +- [jdbc-v2] Fixed several issues with reading database metadata in JDBC driver. (https://github.com/ClickHouse/clickhouse-java/issues/2282) +- [jdbc-v2] Fixed settings client name in JDBC. (https://github.com/ClickHouse/clickhouse-java/issues/2233) +- [client-v2] Fixed reading data from columns defined as `Nullable(FixedString(N))`. (https://github.com/ClickHouse/clickhouse-java/issues/2218) +- [jdbc-v2] Fixed SQL parser failure to parse SQL statement with comments (https://github.com/ClickHouse/clickhouse-java/issues/2217) +- [client-v2] Fixed issue with excessive logging (https://github.com/ClickHouse/clickhouse-java/issues/2201) +- [jdbc-v2] Fixed handling IP addresses (https://github.com/ClickHouse/clickhouse-java/issues/2140) +- [jdbc] - Fixed missing LZ4 dependency in shaded package (https://github.com/ClickHouse/clickhouse-java/issues/2275) + +## 0.8.2 + +### Bug Fixes +- [jdbc-v2] - Significantly improved performance of JDBC inserts. (https://github.com/ClickHouse/clickhouse-java/pull/2165) +- [client-v1] - Removed unnecessary object creation and added executor pool configuration (https://github.com/ClickHouse/clickhouse-java/pull/2164) + +### Miscellaneous +- [client-v1] - Deprecated the old client, though it is still available. We've not yet set a date for its removal, this more serves as a head's up. + +## 0.8.1 + +### New Features +- [client-v2] - Added support for Variant Data Type for RowBinary format. Can be read into a POJO or using a reader. +Writing is supported only for POJO. (https://github.com/ClickHouse/clickhouse-java/pull/2130) +- [client-v2] - Added support for Dynamic Data Type for RowBinary format. Can be read into a POJO or using a reader. + Writing is supported only for POJO. (https://github.com/ClickHouse/clickhouse-java/pull/2130) +- [client-v2] - Added support for JSON Data Type for RowBinary format. Can be read into a POJO or using a reader. + Writing is supported only for POJO. (https://github.com/ClickHouse/clickhouse-java/pull/2130) +- [client-v2] - Added support for `ZonedDateTime` in POJO serde. (https://github.com/ClickHouse/clickhouse-java/issues/2117) +- [client-v2] - Added suport for micrometer metrics for Apache HTTP client connection pool. It is now possible to get metrics registered +in micrometer registry by calling `com.clickhouse.client.api.Client.Builder.registerClientMetrics(registry, groupName)`. (https://github.com/ClickHouse/clickhouse-java/issues/1901) + +### Bug Fixes +- [client-v2] - Fixed `getTableSchema(tableName, databaseName)` now it doesn't ignore database name. (https://github.com/ClickHouse/clickhouse-java/issues/2139) +- [client-v2] - Fixed `returnGeneratedValues` to not throw an exception. Keep in mind that ClickHouse doesn't support +returning anything in response for `INSERT` statements. This change was done just to make client usable with certain frameworks. (https://github.com/ClickHouse/clickhouse-java/issues/2121) +- [jdbc-v2] - Fixed changing database by `USE` statement. (https://github.com/ClickHouse/clickhouse-java/issues/2137) +- [jbdc-v2] - Fixed `ResultSetMetadata.getColumnClassName()` to return null instead of throwing exception. This change is done only +to allow certain frameworks work with the client. (https://github.com/ClickHouse/clickhouse-java/issues/2112) +- [jdbc-v2] - Fixed using statements with `WITH` in JDBC v2. Previously was causing NPE. (https://github.com/ClickHouse/clickhouse-java/issues/2132) +- [repo] - Fixed failing Test in Windows when attempting to locate files with reserved characters in the file path. (https://github.com/ClickHouse/clickhouse-java/issues/2114) + +## 0.8.0 + +### Highlights +- We've updated `ClickHouseDriver` and `ClickHouseDataSource` to default to using the new (`jdbc-v2`) implementation of the JDBC driver. Setting `clickhouse.jdbc.v1=true` will revert this change. + +#### JDBC Changes +- `jdbc-v2` - Removed support for Transaction Support. Early versions of the driver only simulated transaction support, which could have unexpected results. +- `jdbc-v2` - Removed support for Response Column Renaming. `ResultSet` was mutable - for efficiency sake they're now read-only +- `jdbc-v2` - Removed support for Multi-Statement SQL. Multi-statement support was only simulated, now it strictly follows 1:1 +- `jdbc-v2` - Removed support for Named Parameters. Not part of the JDBC spec +- `jdbc-v2` - Removed support for Stream-based `PreparedStatement`. Early version of the driver allowed for non-jdbc usage of `PreparedStatement` - if you desire such options, we recommend looking at client-v2. + +### New Features +- [client-v2, jdbc-v2] - Added support for Bearer token authentication like JWT. Now it is possible to specify encoded token while +creating a client and change while runtime using `com.clickhouse.client.api.Client.updateBearerToken`. (https://github.com/ClickHouse/clickhouse-java/issues/1834, https://github.com/ClickHouse/clickhouse-java/issues/1988) +- [client-v2] - Exposed connection pool metrics through Micrometer. It allows to monitor internal connection pool for number of active and leased connections. (https://github.com/ClickHouse/clickhouse-java/issues/1901) + +### Bug Fixes +- [client-v2] - Fixed construction of `User-Agent` header. Prev. implementation uses `class.getPackage().getImplementationVersion()` what returns +incorrect title and version when library is shaded. New implementation uses build time information from resource files generated while build. (https://github.com/ClickHouse/clickhouse-java/issues/2007) +- [client-v2] - Fixed multiple issues with handling connectivity disruption. Socket timeout is unlimited by default. Added retry on timeout. Added more information to exception message. +Please read the issue for more details. (https://github.com/ClickHouse/clickhouse-java/issues/1994) +- [client-v2] - Client doesn't close provided executor anymore letting application close it instead. (https://github.com/ClickHouse/clickhouse-java/issues/1956) +- [client-v2] - Removed unnecessary initialization to make startup time shorter. (https://github.com/ClickHouse/clickhouse-java/issues/2032) + +## 0.7.2 + +### New Components +- `jdbc-v2` - pre-release version of upcoming `clickhouse-jdbc` replacement. Supports basic functionality, works only with `client-v2`. +More information will be available after its release + +### Highlights +- [repo] Added shaded packages with requires a set of dependencies. Use `all` classified for `client-v2` full package. +Use `shaded-all` for old jdbc driver. +- [client-v2] New Data Writer API was added. It allows to gain a control over writing data to low-level output stream. +This API makes it possible to write compressed data directly to server. See `com.clickhouse.client.api.Client#insert(java.lang.String, com.clickhouse.client.api.DataStreamWriter, com.clickhouse.data.ClickHouseFormat, com.clickhouse.client.api.insert.InsertSettings)` +(https://github.com/ClickHouse/clickhouse-java/pull/2034) + + +### New Features +- [client-v2] Added ability to specify client name. It means `User-Agent` will be filled with proper information (https://github.com/ClickHouse/clickhouse-java/pull/1948) +- [client-v2] Implemented statement parameters for `queryAll` and `queryRecords` API methods. (https://github.com/ClickHouse/clickhouse-java/pull/1979) +- [client-v2] Implemented string to number conversion. (https://github.com/ClickHouse/clickhouse-java/pull/2014) +- [client-v1] Added basic auth support for proxies. Now you can specify username/password when connecting via a proxy that requires it with HttpURLConnection and Apache HttpClient. +- [client-v2] Enum columns can be read as string and number. Previously only as number. Now number matching string constant is returned when get as string. (https://github.com/ClickHouse/clickhouse-java/pull/2028) +- [client-v2] Client will load some server context (timezone, user) right after build. (https://github.com/ClickHouse/clickhouse-java/pull/2029) + +### Bug Fixes +- [jdbc] Fixed default value for `result_overflow_mode` setting. (https://github.com/ClickHouse/clickhouse-java/issues/1932) +- [client-v2] Fixed reading float/double values. Previously was prevented by incorrect overflow check. (https://github.com/ClickHouse/clickhouse-java/issues/1954) +- [client-v2] Fixed issue with enabling client compression. Previously flag was ignore in some cases. (https://github.com/ClickHouse/clickhouse-java/issues/1958) +- [client-v2] Fixed issue with reading `Array(UInt64)` because of incorrect class used to create internal array. (https://github.com/ClickHouse/clickhouse-java/issues/1990) +- [client-v2] Fixed ClickHouseLZ4OutputStream issue of sending empty frame when no data left in uncompressed buffer. (https://github.com/ClickHouse/clickhouse-java/issues/1993) +- [client-v2] Fix handling `ConnectTimeoutException` in retry and wrapping logic. (https://github.com/ClickHouse/clickhouse-java/pull/2015) + +## 0.7.1-patch1 + +### Bug Fixes +- [JDBC] Fixed `java.lang.NoClassDefFoundError: com/clickhouse/client/internal/apache/hc/core5/http2/HttpVersionPolicy` (https://github.com/ClickHouse/clickhouse-java/issues/1912) +- [client-v2] Fixed multiple issues with error message handling. (https://github.com/ClickHouse/clickhouse-java/issues/1906) +- [client-v2] Fixed primitive types conversion. Now client correctly handles numbers to boolean and vice-versa. (https://github.com/ClickHouse/clickhouse-java/issues/1908) + +## 0.7.1 + +### New Features +- [client-v2] Implemented more friendly number conversion. Now it is possible to convert smaller type to bigger one. +It is also possible to convert bigger into smaller if value fits into the range. (https://github.com/ClickHouse/clickhouse-java/issues/1852) +- [client-v2] Ported a feature that allows to remember DB roles for a client instance. See `com.clickhouse.client.api.Client#setDBRoles` +for details. (https://github.com/ClickHouse/clickhouse-java/issues/1832) +- [client-v2] Ported a feature that allows adding comments to a query. +See `com.clickhouse.client.api.insert.InsertSettings#logComment` and `com.clickhouse.client.api.query.QuerySettings#logComment` +for details. (https://github.com/ClickHouse/clickhouse-java/issues/1836) +- [client-v2] Added support for SSL Authentication with client certificates. (https://github.com/ClickHouse/clickhouse-java/issues/1837) +- [client-v2] Implemented a way to define a custom matching between a column name and a field in a POJO in `Client#register` method. (https://github.com/ClickHouse/clickhouse-java/issues/1866) +- [client-v1, client-v2] Implemented HTTP Basic authentication and made it a default auth method for HTTP interface. It +was done to address problem with passwords contianing special and UTF8 characters. New configuration option +`com.clickhouse.client.http.config.ClickHouseHttpOption.USE_BASIC_AUTHENTICATION` for client v1 is added. For client v2 +use `com.clickhouse.client.api.Client.Builder#useHTTPBasicAuth` method. (https://github.com/ClickHouse/clickhouse-java/issues/1305) + + +### Dependency Updates +- [client] Bumped org.apache.avro:avro version to 1.11.4 (https://github.com/ClickHouse/clickhouse-java/pull/1855) + +### Documentation +- [client] Added links to javadoc for all classes in the README.md (https://github.com/ClickHouse/clickhouse-java/pull/1878) + +### Bug Fixes +- [client-v2] Fixed deserializing nullable columns of `Nested` type (https://github.com/ClickHouse/clickhouse-java/issues/1858) +- [client-v2] Fixed dependencies needed for compression to work out of the box (https://github.com/ClickHouse/clickhouse-java/issues/1805) +- [client-v2] Fixed dependency on SNAPSHOT component (https://github.com/ClickHouse/clickhouse-java/issues/1853) +- [client-v2] Fixed using `scale` from a column definition when deserializing DateTime64 values (https://github.com/ClickHouse/clickhouse-java/issues/1851) +- [client-v2] Fixed applying database from insert settings (https://github.com/ClickHouse/clickhouse-java/issues/1868) +- [client-v2] Fixed error handling from server (https://github.com/ClickHouse/clickhouse-java/issues/1874) +- [client-v2] Fixed SerDe for SimpleAggregateFunction columns (https://github.com/ClickHouse/clickhouse-java/pull/1876) +- [client] Fixed handling error from server in response with `200 OK` status. Happens when +`send_progress_in_http_headers` is requested and query runs for a long time. (https://github.com/ClickHouse/clickhouse-java/issues/1821) +- [jdbc] Fixed incorrect error logging (https://github.com/ClickHouse/clickhouse-java/issues/1827) +- [client-v2] Fixed handling tuples in arrays (https://github.com/ClickHouse/clickhouse-java/issues/1882) +- [client-v2] Fixed passing `insert_duplication_token` through `InsertSettings`. (https://github.com/ClickHouse/clickhouse-java/issues/1877) + +## 0.7.0 + +### Deprecations +- Following deprecated components are removed: + - clickhouse-cli-client + - clickhouse-grpc-client + +### Important Changes +- [client-v2] New transport layer implementation is used by default. It is still possible to switch back +using old implementation by setting `com.clickhouse.client.api.Client.Builder#useNewImplementation` to `false`. (https://github.com/ClickHouse/clickhouse-java/pull/1847) + +### New Features +- [client-v2] Now there is an easy way to set custom HTTP headers globally for client and per operation. +See `com.clickhouse.client.api.Client.Builder.httpHeader(java.lang.String, java.lang.String)` for details. (https://github.com/ClickHouse/clickhouse-java/issues/1782) +- [client-v2] Now there is a way to set any server settings globally for client and per operation. +See `com.clickhouse.client.api.Client.Builder.serverSetting(java.lang.String, java.lang.String)` for details. (https://github.com/ClickHouse/clickhouse-java/issues/1782) +- [client-v2] Added support for writing AggregateFunction values (bitmap serialization). !! Reading is not +supported but will be added in the next release. (https://github.com/ClickHouse/clickhouse-java/pull/1814) +- [r2dbc] Defer connection creation. This allows pool to create a new instance on every subscription, +instead of always returning the same one. (https://github.com/ClickHouse/clickhouse-java/pull/1810) + +### Performance Improvements +- [client-v2] Improved reading fixed length data like numbers. It is possible to configure readers to +use pre-allocated buffers to avoid memory allocation for each data row/block. Significantly reduces GC pressure. +See `com.clickhouse.client.api.Client.Builder.allowBinaryReaderToReuseBuffers` for details. (https://github.com/ClickHouse/clickhouse-java/pull/1816) +- [client-v2] New API method introduced to read data directly to a POJO. Deserializers for POJO classes are compiled into +bytecode (with help of https://asm.ow2.io/ library) and optimized for each schema. It is great performance boost +because data is read without copying it into temporary structures. Code can be optimized by JVM while runtime as SerDe +code is implemented without reflection using JVM bytecode. Using bytecode makes handling primitive types without values boxing. (https://github.com/ClickHouse/clickhouse-java/pull/1794, +https://github.com/ClickHouse/clickhouse-java/pull/1826) +- [client-v2] Optimized reading columns - internally data is read into map of column-values. It is done +to allow reading same column more than once. Previously map was cleared each row what caused a lot +internal objects creation. Now values are overridden because schema doesn't change between rows. (https://github.com/ClickHouse/clickhouse-java/pull/1795) + +### Documentation +- [client-v2] Added example for Kotlin (https://github.com/ClickHouse/clickhouse-java/pull/1793) +- [doc] Main documentation on official ClickHouse website is updated. Each client has its own page with detailed information now. +Added documentation for the Client V2. See https://clickhouse.com/docs/en/integrations/java. + +### Bug Fixes +- [client-v2] Fix for cases when missing operation metrics were causing NPE. (https://github.com/ClickHouse/clickhouse-java/pull/1846) +- [client-v2] Fix for handling empty result by BinaryFormat readers. (https://github.com/ClickHouse/clickhouse-java/pull/1845) +- [jdbc] Content of an artifact 'clickhouse-jdbc-{version}-all.jar' is fixed and contains all required classes from `clickhouse-client` +and `clickhouse-data`. (https://github.com/ClickHouse/clickhouse-java/pull/1842) +- [client-v1, jdbc] Endpoints definition parsing fixed to grub properties correctly. Now even properties with key-value +pairs are supported. (https://github.com/ClickHouse/clickhouse-java/pull/1841, https://github.com/ClickHouse/clickhouse-java/issues/1665) + +## 0.6.5 + +### Deprecations +- Following components will be deprecated and removed in 0.7.0 release: + - clickhouse-cli-client + - clickhouse-grpc-client +- Projects cli-client and grpc-client are excluded from release and build. +- No more builds for non-lts Java versions - no more Java 9 release builds. + +### Performance Improvements +- [client-v2] `queryAll()` optimized to use less memory (https://github.com/ClickHouse/clickhouse-java/pull/1779) +- [client-v2] `Client.Builder#setClientNetworkBufferSize` introduced to allow increasing a buffer that is used +to transfer data from socket buffer to application memory. When set to >= of send/receive socket buffer size it +significantly reduces number of system calls and improves performance. (https://github.com/ClickHouse/clickhouse-java/pull/1784) + +### New Features +- [client-v2] Client will retry on `NoHttpResponseException` when using Apache HTTP client. + It is useful when close/stale connection is leased from connection pool. No client will +retry one more time instead of failing. `Client.Builder#retryOnFailures` and `Client.Builder#setMaxRetries` were +introduced to configure client behavior. (https://github.com/ClickHouse/clickhouse-java/pull/1768) + +### Bug Fixes +- [client-v2] Correct timezone used when reading DateTime values. Affects how date/datetime values + are read when `session_timezone` is used (https://github.com/ClickHouse/clickhouse-java/issues/1780) +- [client-v2] Fix reading big integers. Previously was causing incorrect values + (https://github.com/ClickHouse/clickhouse-java/issues/1786) + (https://github.com/ClickHouse/clickhouse-java/issues/1776) +- [client-v2] Fix server compressions when using a client instance concurrently +(https://github.com/ClickHouse/clickhouse-java/pull/1791) +- [client-v2] Fix reading arrays as list. Also affected reading nested arrays (https://github.com/ClickHouse/clickhouse-java/pull/1800) +- [client-v1] Fix handling summary metadata for write operations. Previously was causing empty metadata + +## 0.6.4 + +### Deprecations +- Following components will be deprecated and archived in next release: + - clickhouse-cli-client + - clickhouse-grpc-client +- No more builds for non-lts Java versions - no more Java 9 release builds. +- Lowest supported Java version will be 11. + - Java 11 support will be ended before the end of 2023. + - It is recommended to use Java 21. + +### Important Changes +- [Client-V1] Fix for handling DateTime without timezone when `session_timezone` is set. Now server timezone +is parsed from server response when present (https://github.com/ClickHouse/clickhouse-java/issues/1464) + +### New Features +- [Client-V1/Apache HTTP] More configuration parameters for connection management. Useful for tuning performance. +(https://github.com/ClickHouse/clickhouse-java/pull/1771) + - com.clickhouse.client.config.ClickHouseClientOption#CONNECTION_TTL - to configure connection time-to-live + - com.clickhouse.client.http.config.ClickHouseHttpOption#KEEP_ALIVE_TIMEOUT - to configure keep-alive timeout + - com.clickhouse.client.http.config.ClickHouseHttpOption#CONNECTION_REUSE_STRATEGY - defines how connection pool behaves. +If `FIFO` is selected then connections are reused in the order they were created. It results in even distribution of connections. +If `LIFO` is selected then connections are reused as soon they are returned to the pool. +Note: only for `APACHE_HTTP_CLIENT` connection provider. + - Additionally switched to using LAX connection pool for Apache Connection Manager to improve performance +for concurrent requests. +- [Client-V2] Connection pool configuration https://github.com/ClickHouse/clickhouse-java/pull/1766 + - com.clickhouse.client.api.Client.Builder.setConnectionRequestTimeout - to configure connection request timeout. +Important when there are no connections available in the pool to fail fast. + - com.clickhouse.client.api.Client.Builder.setMaxConnections - configures how soft limit of connections per host. +Note: Total number of connections is unlimited because in most cases there is one host. + - com.clickhouse.client.api.Client.Builder.setConnectionTTL - to limit connection live ignoring keep-alive from server. + - com.clickhouse.client.api.Client.Builder.setConnectionReuseStrategy - to configure how connections are used. +Select FIFO to reuse connections evenly or LIFO (default) to reuse the most recently active connections. +- [Client-V2] All operations are now executed in calling thread to avoid extra threads creation. +Async operations can be enabled by `com.clickhouse.client.api.Client.Builder.useAsyncRequests` (https://github.com/ClickHouse/clickhouse-java/pull/1767) +- [Client-V2] Content and HTTP native compression is supported now Currently only LZ4 is available. (https://github.com/ClickHouse/clickhouse-java/pull/1761) +- [Client-V2] HTTPS support added. Required to communicate with ClickHouse Cloud Services. +Client certificates are supported, too. (https://github.com/ClickHouse/clickhouse-java/pull/1753) +- [Client-V2] Added support for HTTP proxy (https://github.com/ClickHouse/clickhouse-java/pull/1748) + +### Documentation +- [Client-V2] Spring Demo Service as usage example (https://github.com/ClickHouse/clickhouse-java/pull/1765) +- [Client-V2] Examples for using text based formats (https://github.com/ClickHouse/clickhouse-java/pull/1752) + + +### Bug Fixes +- [Client-V2] Data is read fully from a stream. Important for Cloud instances (https://github.com/ClickHouse/clickhouse-java/pull/1759) +- [Client-V2] Timezone from a server response is now used to parse DateTime values (https://github.com/ClickHouse/clickhouse-java/pull/1763) +- [Client-V1] Timezone from a server response is now used to parse DateTime values (https://github.com/ClickHouse/clickhouse-java/issues/1464) + +## 0.6.3 + +### Important Changes +- [Client-V1] Changed how `User-Agent` string is generated. Now `ClickHouse-JavaClient` portion is appended in all cases. +It is still possible to set custom product name that will be the first part in `User-Agent` value. +(https://github.com/ClickHouse/clickhouse-java/issues/1698) + +### New Features +- [Client-V1/Apache HTTP] Retry on NoHttpResponseException in Apache HTTP client. +Should be used with causes because it is not always possible to resend request body. +Behaviour is controlled by `com.clickhouse.client.http.config.ClickHouseHttpOption#AHC_RETRY_ON_FAILURE`. +Works only for Apache HTTP client because based on its specific behavior(https://github.com/ClickHouse/clickhouse-java/pull/1721) +- [Client-V1/Apache HTTP] Connection validation before sending request. +Behaviour is controlled by `com.clickhouse.client.http.config.ClickHouseHttpOption#AHC_VALIDATE_AFTER_INACTIVITY`. +By default, connection is validated after being in the pool for 5 seconds. (https://github.com/ClickHouse/clickhouse-java/pull/1722) + +### Bug Fixes +- [Client-V2] Fix parsing endpoint URL to detect HTTPs (https://github.com/ClickHouse/clickhouse-java/issues/1718) +- [Client-V2] Fix handling asynchronous operations. Less extra threads created now. (https://github.com/ClickHouse/clickhouse-java/issues/1691) +- [Client-V2] Fix way of how settings are validated to let unsupported options to be added (https://github.com/ClickHouse/clickhouse-java/issues/1691) +- [Client-V1] Fix getting `localhost` effective IP address (https://github.com/ClickHouse/clickhouse-java/issues/1729) +- [Client-V2] Make client instance closeable to free underlying resource (https://github.com/ClickHouse/clickhouse-java/pull/1733) + +## 0.6.2 + +### New Features +- Describe non-executed SELECT queries in prepared statements to provide metadata (https://github.com/ClickHouse/clickhouse-java/issues/1430) +- Command execution in the client API (https://github.com/ClickHouse/clickhouse-java/pull/1693) +- Added `com.clickhouse.client.ClickHouseResponseSummary#getQueryId()` (https://github.com/ClickHouse/clickhouse-java/issues/1636) +- Added support for SSL for the Client V2 +- Added proxy support for Client V2 (https://github.com/ClickHouse/clickhouse-java/pull/1694) +- Added more examples for Client V2 (https://github.com/ClickHouse/clickhouse-java/pull/1709) + +## 0.6.1 + +### New Features +- Alpha version of the new client API. See example https://github.com/ClickHouse/clickhouse-java/tree/main/examples/client-v2. + +### Bug Fixes +- Fix proper handling of ClickHouseResult to release DB connection (https://github.com/ClickHouse/clickhouse-java/issues/1538 +- Fix getting response input stream for binary formats. + - https://github.com/ClickHouse/clickhouse-java/issues/1494 + - https://github.com/ClickHouse/clickhouse-java/issues/1567 + - https://github.com/ClickHouse/clickhouse-java/issues/1475 + - https://github.com/ClickHouse/clickhouse-java/issues/1376 +- Exception context remove from a message. Server information not present anymore but available thru `com.clickhouse.client.ClickHouseException#getServer` (https://github.com/ClickHouse/clickhouse-java/issues/1677) + +## 0.6.0-patch5 + +### Bug Fixes +- Roles (memorized by client) can be reset with 'SET ROLE NONE' query (https://github.com/ClickHouse/clickhouse-java/issues/1647) +- MaxBufferSize can be greater than internal MAX value now +- Updated example project to use the latest version of the client + +## 0.6.0-patch4 + +### New Features +- Added possibility to set client ID in `Referer` HTTP Header (https://github.com/ClickHouse/clickhouse-java/issues/1572) +- [HTTP] Persistence of a role after it is set by `SET ROLE ` + +### Bug Fixes +- Change RowBinaryWithDefaults settings. Output is changed from true to false +- Fix fail over for Apache HTTP client. Connect timeout error is not detected correctly +- Fix password logging in DEBUG (https://github.com/ClickHouse/clickhouse-java/issues/1571) +- Fix handling "NoHttpResponseException" in Apache HTTP client +- Fix param error in ByteUtils#equals in java9 + +## 0.6.0-patch3 + +### Bug Fixes +- Not detecting correctly ClickHouse error code +- Fix JDBC read error - Multidimensional empty errors raise a java.lang.ArrayStoreException + +## 0.6.0-patch2 + +### Dependency Updates +- org.apache.commons:commons-compress from 1.23.0 to 1.26.1 +- org.postgresql:postgresql from 42.6.0 to 42.6.1 + +## 0.6.0-patch1 + +### Bug Fixes +- Fix buffering issue caused by decompress flag not to work when working with HTTP Client. + +## 0.6.0 + +### WARNING -- ClickHouse CLI Client deprecation + +`clickhouse-cli-client` package is deprecated from version 0.6.0 and it's going to be removed in `0.7.0`. We recommend using [clickhouse-client](https://clickhouse.com/docs/en/interfaces/cli) instead. + +### WARNING -- ClickHouse GRPC Client deprecation + +`clickhouse-grpc-client` package is deprecated from version 0.6.0 and it's going to be removed in `0.7.0`. We recommend using [HTTP](https://github.com/ClickHouse/clickhouse-java/blob/main/examples/client/src/main/java/com/clickhouse/examples/jdbc/Main.java) protocol instead. + +### Breaking Changes + +- Remove WEB_CONTEXT support - [#1512](https://github.com/ClickHouse/clickhouse-java/issues/1512) + +### New Features + +- Add support in RowBinaryWithDefaults [#1508](https://github.com/ClickHouse/clickhouse-java/pull/1508) + +### Bug Fixes + +- Fix faulty node detection in ClickHouseNodes - [#1595](https://github.com/ClickHouse/clickhouse-java/pull/1511) +- Fix while getting addBatch with an exception drop the all batch [#1373](https://github.com/ClickHouse/clickhouse-java/issues/1373) +- Fix buffering issue caused by decompress flag not to work [#1500](https://github.com/ClickHouse/clickhouse-java/pull/1500) + +## 0.5.0 + +### Breaking Changes + +- ClickHouseByteBuffer can no longer be extended +- rename ClickHouseByteUtils methods by removing LE suffix +- change default databaseTerm from schema to catalog +- remove deprecated API load, dump and connect +- remove use_no_proxy settings + +### New Features + +- Adding new proxy support [#1338](https://github.com/ClickHouse/clickhouse-java/issues/1338) +- Add support for customer socket factory [#1391](https://github.com/ClickHouse/clickhouse-java/issues/1391) +- use VarHandle in JDK 9+ to read/write numbers +- Establish secured connection with custom Trust Store file +- Change default HTTP Client to Apache HTTP client [#1421](https://github.com/ClickHouse/clickhouse-java/issues/1421) + +### Bug Fixes + +- Java client threw confusing error when query is invalid. +- JDBC Driver correctly processes `AggregateFunction(Nested(...))` columns +- Incorrect parameter position +- Fix testing framework to support secured clickhouse server + +## 0.4.6, 2023-05-02 + +### New Features + +- ClickHouseStatement.setMirroredOutput() for dumping ResultSet. +- ClickHouseResponse.records(Class) for object mapping. +- Two new options(use_compilation & max_mapper_cache) reserved for future usage. + +### Bug Fixes + +- Too many socket fds generated by Apache HttpClient. +- NoClassDefFoundError with clickhouse-apache-http-client-jdbc. [#1319](https://github.com/ClickHouse/clickhouse-java/issues/1319) +- Nested array in tuple array is incorrectly deserialized. [#1324](https://github.com/ClickHouse/clickhouse-java/issues/1324) +- client certificate password exposure in exception. [#1331](https://github.com/ClickHouse/clickhouse-java/issues/1331) + +## 0.4.5, 2023-04-25 + +### Breaking Changes + +- refactor data processors and response classes to ensure input stream remain intact before first read: + - move ClickHouseSimpleRecord to com.clickhouse.data + - stop reading input stream when instantiating ClickHouseDataProcessor + - remove createRecord() method in ClickHouseDataProcessor along with some duplicated code + +### New Features + +- disable SQL rewrite for DELETE statement in ClickHouse 23.3+ + +### Bug Fixes + +- Slow when using Apache Http Client. [#1320](https://github.com/ClickHouse/clickhouse-java/issues/1320) +- ClickHouseResponse.getInputStream may return closed input stream. +- ConcurrentModificationException may occure during deserialization. [#1327](https://github.com/ClickHouse/clickhouse-java/pull/1327) +- ClickHouseSslContextProvider is not customizable. [#1329](https://github.com/ClickHouse/clickhouse-java/issues/1329) + +## 0.4.4, 2023-04-17 + +### Bug Fixes + +- flatten plugin 1.4.1 generated non-sense dependencies. + +## 0.4.3, 2023-04-17 + +### New Features + +- replace JavaCC21 with CongoCC + +### Bug Fixes + +- unable to convert empty string to default value when using text-based data format. +- r2dbc driver does not support most client options. [#1299](https://github.com/ClickHouse/clickhouse-java/issues/1299) +- incorrect content from Lz4InputStream when using text-based data format [#48446](https://github.com/ClickHouse/ClickHouse/issues/48446) + +## 0.4.2, 2023-03-21 + +### Breaking Changes + +- ClickHouseSqlStatement and \*ParserHandler in JDBC driver were refactored to support `compression` and `infile` in insert statement. + +### New Features + +- centralized configuration for JDBC driver using custom server setting `custom_jdbc_config`. +- support `BEGIN TRANSACTION`, `COMMIT`, and `ROLLBACK` in JDBC driver. [#975](https://github.com/ClickHouse/clickhouse-java/issues/975) +- new options for JDBC driver + - databaseTerm(catalog or schema, defaults to schema) [#1273](https://github.com/ClickHouse/clickhouse-java/issues/1273) + - externalDatabase(true or false, defaults to true) [#1245](https://github.com/ClickHouse/clickhouse-java/issues/1245) + - localFile(true of false, defaults to false) - whether to use local file for INFILE and OUTFILE or not + +### Bug Fixes + +- error while converting Nested values to Java maps. +- incorrect algorithm extracted from PEM. [#1274](https://github.com/ClickHouse/clickhouse-java/issues/1274) +- transaction failure introduced 0.4.0. +- respect node-specific credentials. [#1114](https://github.com/ClickHouse/clickhouse-java/issues/1114) +- USE statement does nothing. [#1160](https://github.com/ClickHouse/clickhouse-java/issues/1160) +- executeBatch does not support on cluster anymore. [#1261](https://github.com/ClickHouse/clickhouse-java/issues/1261) + +## 0.4.1, 2023-02-19 + +### Breaking Changes + +- changed option names - [#1203](https://github.com/ClickHouse/clickhouse-java/pull/1203) + - compress_alogrithm -> compress_algorithm + - decompress_alogrithm -> decompress_algorithm + +### New Features + +- added source in shaded jar for IDE friendly. [#1217](https://github.com/ClickHouse/clickhouse-java/pull/1217) +- iterable ClickHouseInputStream, which slightly improved performance of piping. [#1238](https://github.com/ClickHouse/clickhouse-java/pull/1238) +- ClickHouseOutputStream.transferBytes() is deprecated and it will be removed in 0.5. +- read() and write() methods in ClickHouseClient - connect() is deprecated and it will be removed in 0.5. +- make all dependencies of JDBC driver optional, along with configuration for building native binary. [#1247](https://github.com/ClickHouse/clickhouse-java/pull/1247) + +### Bug Fixes + +- incorrect nested array value. [#1221](https://github.com/ClickHouse/clickhouse-java/issues/1221) +- incorrect nested array values. [#1223](https://github.com/ClickHouse/clickhouse-java/issues/1223) +- potential endless loop when handling batch update error. [#1233](https://github.com/ClickHouse/clickhouse-java/issues/1233) +- exception when deserializing Array(FixedString(2)) from RowBinary. [#1235](https://github.com/ClickHouse/clickhouse-java/issues/1235) +- deserialization failure of Nested data type. [#1240](https://github.com/ClickHouse/clickhouse-java/issues/1240) + broken serde for Nested, which also impacted JSON. [#1242](https://github.com/ClickHouse/clickhouse-java/issues/1242) +- fix 64bit bitmap serialization issue. [#641](https://github.com/ClickHouse/clickhouse-java/issues/641), [#874](https://github.com/ClickHouse/clickhouse-java/issues/874), [#1141](https://github.com/ClickHouse/clickhouse-java/issues/1141) +- gRPC client may complete execution before closing response stream +- throw StreamCorruptedException instead of generic IOException when deserialization failed (mostly due to server error) + +## 0.4.0, 2023-01-19 + +### Breaking Changes + +- refactored `JdbcTypeMapping` to make it extensible. [#1075](https://github.com/ClickHouse/clickhouse-java/pull/1075) +- removed legacy driver `ru.yandex.*`.[#1089](https://github.com/ClickHouse/clickhouse-java/pull/1089) +- removed most deprecated methods and class members +- refactored data processor(for serialization and deserialization) and added new classes for unsigned types. [#1124](https://github.com/ClickHouse/clickhouse-java/pull/1124) +- refactored ClickHouseRequest/ClickHouseInputStream/ClickHouseOutputStream to better support compression. [#1174](https://github.com/ClickHouse/clickhouse-java/pull/1174), [#1189](https://github.com/ClickHouse/clickhouse-java/pull/1189) +- extracted `clickhouse-data` from `clickhouse-client` along with new packages. [#1197](https://github.com/ClickHouse/clickhouse-java/pull/1197) + ``` + com.clickhouse.config // generic configuration + com.clickhouse.data // data processing utilities + com.clickhouse.logging // generic logging utility + ``` +- added jdk17 in pom and changed build instructions + ```bash + mvn -Dj8 clean install # for jdk8, it was 'mvn clean install' + mvn clean install # for jdk17, it was 'mvn -Drelease clean install' + ``` + +### New Features + +- added R2DBC driver. [#914](https://github.com/ClickHouse/clickhouse-java/pull/914) +- enhanced ClickHouseClient for importing and exporting compressed file. [#1004](https://github.com/ClickHouse/clickhouse-java/pull/1004) +- added new option `custom_settings`. [#1059](https://github.com/ClickHouse/clickhouse-java/pull/1059) +- enhanced `ClickHouseRequestManager` to support query/session ID customization. [#1074](https://github.com/ClickHouse/clickhouse-java/pull/1074) +- added Apache HTTP Client 5 to support socket options. [#1146](https://github.com/ClickHouse/clickhouse-java/pull/1146) +- enhanced `clickhouse-grpc-client` to support request chunking and compression + - `decompress_alogrithm`(request compression): `BROTLI`[-1,11], `BZ2`, `DEFLATE`[0,9], `GZIP`[-1,9], `LZ4`[0,18], `XZ`[0,9], `ZSTD`[0,22] + - `compress_alogrithm`(response decompression): `DEFLATE`, `LZ4`, `ZSTD` + Note: typo will be fixed in v0.4.1. +- enhanced `clickhouse-http-client` to support zstd compression for both request and response + - `decompress_alogrithm`(request compression): `LZ4`[0,18], `ZSTD`[0,22] + - `compress_alogrithm`(response decompression): `BROTLI`, `BZ2`, `DEFLATE`, `GZIP`, `LZ4`, `XZ`, `ZSTD` + Note: typo will be fixed in v0.4.1. +- added stream-based prepared statement. [#1163](https://github.com/ClickHouse/clickhouse-java/pull/1163) +- browser-like client name (`select distinct http_user_agent from system.query_log`). [#1182](https://github.com/ClickHouse/clickhouse-java/pull/1182) +- enhanced `ClickHouseRequest` by treating input stream and `ClickHouseWriter` equally. [#1200](https://github.com/ClickHouse/clickhouse-java/pull/1200) + +### Bug Fixes + +- not able to cancel query when there's session_id. [#1035](https://github.com/ClickHouse/clickhouse-java/pull/1035) +- overflow error when handling BigInteger and BigDecimal in ClickHouseLongValue. [#1040](https://github.com/ClickHouse/clickhouse-java/pull/1040) +- not handling SimpleAggregateFunction properly. [#1054](https://github.com/ClickHouse/clickhouse-java/pull/1054) +- DELETE FROM was rewritten to ALTER TABLE DELETE even when lightweight deletion was enabled. [#1063](https://github.com/ClickHouse/clickhouse-java/pull/1063) +- forced headless format for inserting. [#1073](https://github.com/ClickHouse/clickhouse-java/pull/1073) +- missing the verb "be" in error messages. [#1137](https://github.com/ClickHouse/clickhouse-java/pull/1137) +- write time field data loss precision. [#1127](https://github.com/ClickHouse/clickhouse-java/pull/1127) +- fixed a few copy-paste error causing problem handling Geo types \ No newline at end of file diff --git a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReader.java b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReader.java index 018b0aeb6..303dbb9f3 100644 --- a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReader.java +++ b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReader.java @@ -1525,6 +1525,30 @@ public StringValue readStringValue() throws IOException { return new StringValue(readStringBytes(input, readVarInt(input))); } + /** + * Reads a {@code FixedString} value as raw bytes. + * + * @param length fixed length in bytes + * @return string value holding those bytes + * @throws IOException when IO error occurs + */ + public StringValue readFixedStringValue(int length) throws IOException { + if (length < 0) { + throw new IllegalArgumentException("FixedString length is negative"); + } + return new StringValue(readNBytes(input, length)); + } + + /** + * Reads a Dynamic type tag and returns the concrete column it describes. + * + * @return runtime column encoded by the tag + * @throws IOException when IO error occurs + */ + public ClickHouseColumn readDynamicType() throws IOException { + return readDynamicData(); + } + /** * Reads the raw bytes of a string from the input stream given its length. * diff --git a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/SerializerUtils.java b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/SerializerUtils.java index 50f9136e2..bf168398d 100644 --- a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/SerializerUtils.java +++ b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/SerializerUtils.java @@ -372,6 +372,18 @@ private static ClickHouseColumn enumValue2Column(Enum enumValue) { clickHouseEnum); } + private static int jsonLimit(ClickHouseColumn column, String name, int fallback) { + List parameters = column.getParameters(); + for (int i = 0; i < parameters.size(); i++) { + String parameter = parameters.get(i); + int eq = parameter.indexOf('='); + if (eq > 0 && parameter.substring(0, eq).trim().equals(name)) { + return Integer.parseInt(parameter.substring(eq + 1).trim()); + } + } + return fallback; + } + public static void writeDynamicTypeTag(OutputStream stream, ClickHouseColumn typeColumn) throws IOException { @@ -485,7 +497,23 @@ public static void writeDynamicTypeTag(OutputStream stream, ClickHouseColumn typ stream.write(binTag); break; case JSON: + // 0x30 + // (path, type)* + // Version 0 has no per-path flags. SKIP paths are not retained on the column, so a + // tag rebuilt from a parsed column does not repeat them. stream.write(binTag); + stream.write(0); + BinaryStreamUtils.writeVarInt(stream, jsonLimit(typeColumn, "max_dynamic_paths", 1024)); + BinaryStreamUtils.writeUnsignedInt8(stream, jsonLimit(typeColumn, "max_dynamic_types", 32)); + List typedPaths = typeColumn.getNestedColumns(); + BinaryStreamUtils.writeVarInt(stream, typedPaths.size()); + for (int i = 0; i < typedPaths.size(); i++) { + ClickHouseColumn typedPath = typedPaths.get(i); + BinaryStreamUtils.writeString(stream, typedPath.getColumnName()); + writeDynamicTypeTag(stream, typedPath); + } + BinaryStreamUtils.writeVarInt(stream, 0); + BinaryStreamUtils.writeVarInt(stream, 0); break; case SimpleAggregateFunction: case AggregateFunction: diff --git a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/stream/RowBinaryStreamReader.java b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/stream/RowBinaryStreamReader.java new file mode 100644 index 000000000..18830f142 --- /dev/null +++ b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/stream/RowBinaryStreamReader.java @@ -0,0 +1,207 @@ +package com.clickhouse.client.api.data_formats.stream; + +import com.clickhouse.client.api.ClientException; +import com.clickhouse.client.api.data_formats.internal.BinaryStreamReader; +import com.clickhouse.client.api.metadata.TableSchema; +import com.clickhouse.data.ClickHouseColumn; +import com.clickhouse.data.ClickHouseFormat; + +import java.io.EOFException; +import java.io.IOException; +import java.io.InputStream; +import java.io.PushbackInputStream; +import java.time.ZoneOffset; +import java.util.ArrayList; +import java.util.List; +import java.util.TimeZone; + +/** + * Reads {@code RowBinary}, {@code RowBinaryWithNames}, or {@code RowBinaryWithNamesAndTypes} + * without materializing nested containers. + *

+ * Experimental API. Not thread-safe. The caller owns the input stream. A failure desynchronizes + * the stream; the reader must not be reused. + */ +public final class RowBinaryStreamReader { + + private final PushbackInputStream stream; + + private final BinaryStreamReader binary; + + private final ClickHouseFormat format; + + private final TimeZone timeZone; + + private final byte[] scratch = new byte[8]; + + private final byte[] skipBuffer = new byte[256]; + + private TableSchema schema; + + private List columns; + + private boolean failed; + + private StreamReaderScope openRow; + + public RowBinaryStreamReader(InputStream in, TableSchema schema, ClickHouseFormat format) throws IOException { + this(in, schema, format, TimeZone.getTimeZone(ZoneOffset.UTC)); + } + + public RowBinaryStreamReader(InputStream in, TableSchema schema, ClickHouseFormat format, TimeZone timeZone) + throws IOException { + if (in == null) { + throw new IllegalArgumentException("Input stream is required"); + } + if (format != ClickHouseFormat.RowBinary && format != ClickHouseFormat.RowBinaryWithNames + && format != ClickHouseFormat.RowBinaryWithNamesAndTypes) { + throw new IllegalArgumentException( + "Only RowBinary, RowBinaryWithNames, and RowBinaryWithNamesAndTypes are supported"); + } + if (format != ClickHouseFormat.RowBinaryWithNamesAndTypes && schema == null) { + throw new IllegalArgumentException("Schema is required"); + } + this.stream = new PushbackInputStream(in, 1); + this.timeZone = timeZone == null ? TimeZone.getTimeZone(ZoneOffset.UTC) : timeZone; + this.binary = new BinaryStreamReader(stream, this.timeZone, null, + new BinaryStreamReader.CachingByteBufferAllocator(), true, null, true); + this.format = format; + if (schema != null) { + this.schema = schema; + this.columns = new ArrayList(schema.getColumns()); + } + if (format == ClickHouseFormat.RowBinaryWithNames) { + readNameHeader(); + } else if (format == ClickHouseFormat.RowBinaryWithNamesAndTypes) { + readNamesAndTypesHeader(); + } + } + + public boolean hasNext() throws IOException { + if (failed) { + throw new IllegalStateException("Reader is unusable after a previous failure"); + } + if (openRow != null && !openRow.isClosed()) { + throw new IllegalStateException("A row is still open"); + } + int next = stream.read(); + if (next < 0) { + return false; + } + stream.unread(next); + return true; + } + + public ValueReader beginRow() throws IOException { + if (failed) { + throw new IllegalStateException("Reader is unusable after a previous failure"); + } + if (openRow != null && !openRow.isClosed()) { + throw new IllegalStateException("A row is still open"); + } + if (!hasNext()) { + throw new EOFException("No more rows"); + } + openRow = StreamReaderScope.row(this, columns); + return openRow; + } + + public TableSchema getSchema() { + return schema; + } + + public ClickHouseFormat getFormat() { + return format; + } + + InputStream stream() { + return stream; + } + + BinaryStreamReader binary() { + return binary; + } + + TimeZone timeZone() { + return timeZone; + } + + byte[] scratch() { + return scratch; + } + + boolean isFailed() { + return failed; + } + + void fail() { + failed = true; + } + + void onRowClosed() { + openRow = null; + } + + void unread(int value) throws IOException { + stream.unread(value); + } + + void skipFully(int length) throws IOException { + int left = length; + while (left > 0) { + int n = stream.read(skipBuffer, 0, Math.min(skipBuffer.length, left)); + if (n < 0) { + throw new EOFException("End of stream reached before skipping " + length + " bytes"); + } + left -= n; + } + } + + private void readNameHeader() throws IOException { + List names = readNames(); + if (names.size() != columns.size()) { + throw new ClientException("RowBinaryWithNames column count " + names.size() + + " does not match schema " + columns.size()); + } + for (int i = 0; i < names.size(); i++) { + if (!names.get(i).equals(columns.get(i).getColumnName())) { + throw new ClientException("RowBinaryWithNames column '" + names.get(i) + + "' does not match schema column '" + columns.get(i).getColumnName() + "'"); + } + } + } + + private void readNamesAndTypesHeader() throws IOException { + List names = readNames(); + List header = new ArrayList(names.size()); + try { + for (int i = 0; i < names.size(); i++) { + header.add(ClickHouseColumn.of(names.get(i), binary.readString())); + } + } catch (IOException e) { + throw new ClientException("Failed to read header", e); + } + this.columns = header; + this.schema = new TableSchema(header); + } + + private List readNames() throws IOException { + int count; + try { + count = BinaryStreamReader.readVarInt(stream); + } catch (EOFException e) { + throw new ClientException("Failed to read header", e); + } catch (IOException e) { + throw new ClientException("Failed to read header", e); + } + List names = new ArrayList(count); + try { + for (int i = 0; i < count; i++) { + names.add(binary.readString()); + } + } catch (IOException e) { + throw new ClientException("Failed to read header", e); + } + return names; + } +} diff --git a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/stream/RowBinaryStreamWriter.java b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/stream/RowBinaryStreamWriter.java new file mode 100644 index 000000000..0867b5b06 --- /dev/null +++ b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/stream/RowBinaryStreamWriter.java @@ -0,0 +1,116 @@ +package com.clickhouse.client.api.data_formats.stream; + +import com.clickhouse.client.api.metadata.TableSchema; +import com.clickhouse.data.ClickHouseColumn; +import com.clickhouse.data.ClickHouseFormat; + +import java.io.IOException; +import java.io.OutputStream; +import java.time.ZoneOffset; +import java.util.ArrayList; +import java.util.List; +import java.util.TimeZone; + +/** + * Writes {@code RowBinary} or {@code RowBinaryWithDefaults} directly to an output stream. + * Nested values are scopes: a known container size is written through, an unknown size is buffered + * until the scope is closed. + *

+ * Experimental API. Not thread-safe. The caller owns {@code out}. A failure desynchronizes the + * stream; the writer must not be reused. Retry by writing the rows again to a new stream. + */ +public final class RowBinaryStreamWriter { + + private final OutputStream out; + + private final List columns; + + private final boolean defaultsSupport; + + private final ClickHouseFormat format; + + private final TimeZone timeZone; + + private int rowCount; + + private boolean failed; + + private StreamWriterScope openRow; + + public RowBinaryStreamWriter(OutputStream out, TableSchema schema, ClickHouseFormat format) { + this(out, schema, format, TimeZone.getTimeZone(ZoneOffset.UTC)); + } + + public RowBinaryStreamWriter(OutputStream out, TableSchema schema, ClickHouseFormat format, TimeZone timeZone) { + if (out == null) { + throw new IllegalArgumentException("Output stream is required"); + } + if (schema == null) { + throw new IllegalArgumentException("Schema is required"); + } + if (format != ClickHouseFormat.RowBinary && format != ClickHouseFormat.RowBinaryWithDefaults) { + throw new IllegalArgumentException("Only RowBinary and RowBinaryWithDefaults are supported"); + } + this.out = out; + this.columns = writableColumns(schema); + this.defaultsSupport = format == ClickHouseFormat.RowBinaryWithDefaults; + this.format = format; + this.timeZone = timeZone == null ? TimeZone.getTimeZone(ZoneOffset.UTC) : timeZone; + } + + public ValueWriter beginRow() throws IOException { + if (failed) { + throw new IllegalStateException("Writer is unusable after a previous failure"); + } + if (openRow != null && !openRow.isClosed()) { + throw new IllegalStateException("A row is still open"); + } + openRow = StreamWriterScope.row(this, columns); + return openRow; + } + + public int getRowCount() { + return rowCount; + } + + public ClickHouseFormat getFormat() { + return format; + } + + OutputStream out() { + return out; + } + + boolean defaultsSupport() { + return defaultsSupport; + } + + TimeZone timeZone() { + return timeZone; + } + + boolean isFailed() { + return failed; + } + + void fail() { + failed = true; + } + + void onRowClosed() { + openRow = null; + rowCount++; + } + + private static List writableColumns(TableSchema schema) { + List all = schema.getColumns(); + List kept = new ArrayList(all.size()); + for (int i = 0; i < all.size(); i++) { + ClickHouseColumn column = all.get(i); + if (!StreamTypes.omitted(column)) { + kept.add(column); + } + } + return kept; + } +} diff --git a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/stream/SlotCursor.java b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/stream/SlotCursor.java new file mode 100644 index 000000000..f506fdc22 --- /dev/null +++ b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/stream/SlotCursor.java @@ -0,0 +1,182 @@ +package com.clickhouse.client.api.data_formats.stream; + +import com.clickhouse.data.ClickHouseColumn; + +import java.util.List; + +/** + * Position inside one scope: a row, array, tuple, map, or a single variant/dynamic value. + */ +final class SlotCursor { + + static final int UNKNOWN = -1; + + enum Kind { + ROW, + ARRAY, + TUPLE, + MAP, + SINGLE, + JSON + } + + final Kind kind; + + private final List slots; + + private final ClickHouseColumn repeated; + + private final ClickHouseColumn mapKey; + + private final ClickHouseColumn mapValue; + + /** Expected number of completed values, or {@link #UNKNOWN}. */ + final int declared; + + private int index; + + private boolean expectValue; + + private SlotCursor(Kind kind, List slots, ClickHouseColumn repeated, + ClickHouseColumn mapKey, ClickHouseColumn mapValue, int declared) { + this.kind = kind; + this.slots = slots; + this.repeated = repeated; + this.mapKey = mapKey; + this.mapValue = mapValue; + this.declared = declared; + } + + static SlotCursor row(List columns) { + return new SlotCursor(Kind.ROW, columns, null, null, null, columns.size()); + } + + static SlotCursor array(ClickHouseColumn element, int size) { + return new SlotCursor(Kind.ARRAY, null, element, null, null, size); + } + + static SlotCursor tuple(List fields) { + return new SlotCursor(Kind.TUPLE, fields, null, null, null, fields.size()); + } + + static SlotCursor map(ClickHouseColumn key, ClickHouseColumn value, int pairs) { + return new SlotCursor(Kind.MAP, null, null, key, value, pairs); + } + + static SlotCursor single(ClickHouseColumn value) { + return new SlotCursor(Kind.SINGLE, null, value, null, null, 1); + } + + /** A JSON object. {@code paths} is the number of leaf paths, or {@link #UNKNOWN}. */ + static SlotCursor json(int paths) { + return new SlotCursor(Kind.JSON, null, null, null, null, paths); + } + + ClickHouseColumn current() { + switch (kind) { + case ROW: + case TUPLE: + if (index >= slots.size()) { + throw new IllegalStateException("No more values in this " + kind.name().toLowerCase()); + } + return slots.get(index); + case ARRAY: + if (declared != UNKNOWN && index >= declared) { + throw new IllegalStateException("No more array elements"); + } + return repeated; + case MAP: + if (expectValue) { + return mapValue; + } + if (declared != UNKNOWN && index >= declared) { + throw new IllegalStateException("No more map entries"); + } + return mapKey; + case SINGLE: + if (index >= 1) { + throw new IllegalStateException("The value has already been written"); + } + return repeated; + case JSON: + throw new IllegalStateException("A JSON value is written by path"); + default: + throw new IllegalStateException("Unknown scope"); + } + } + + ClickHouseColumn repeated() { + return repeated; + } + + boolean expectingValue() { + return expectValue; + } + + int completed() { + return index; + } + + int size() { + if (kind == Kind.ARRAY || kind == Kind.MAP) { + return declared; + } + if (kind == Kind.SINGLE) { + return 1; + } + if (kind == Kind.JSON) { + return declared; + } + return slots.size(); + } + + boolean hasNext() { + if (kind == Kind.MAP) { + return !expectValue && (declared == UNKNOWN || index < declared); + } + if (kind == Kind.JSON) { + return declared == UNKNOWN || index < declared; + } + int limit = kind == Kind.ARRAY ? declared : (kind == Kind.SINGLE ? 1 : slots.size()); + return index < limit; + } + + int remaining() { + if (kind == Kind.ARRAY) { + return declared - index; + } + throw new IllegalStateException("remaining() is only valid inside an array"); + } + + void advance() { + if (kind == Kind.MAP) { + if (!expectValue) { + expectValue = true; + } else { + expectValue = false; + index++; + } + return; + } + index++; + } + + /** Adds leaf paths contributed by a nested object that was flattened into this one. */ + void credit(int paths) { + index += paths; + } + + void forceComplete() { + index = declared == UNKNOWN ? 1 : declared; + expectValue = false; + } + + void checkComplete() { + if (kind == Kind.MAP && expectValue) { + throw new IllegalStateException("Map entry is missing a value"); + } + if (declared != UNKNOWN && index != declared) { + throw new IllegalStateException("Expected " + declared + " values but completed " + index); + } + } +} diff --git a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/stream/StreamReaderScope.java b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/stream/StreamReaderScope.java new file mode 100644 index 000000000..4afceaf38 --- /dev/null +++ b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/stream/StreamReaderScope.java @@ -0,0 +1,829 @@ +package com.clickhouse.client.api.data_formats.stream; + +import com.clickhouse.client.api.ClientException; +import com.clickhouse.client.api.data_formats.internal.BinaryStreamReader; +import com.clickhouse.client.api.data_formats.internal.StringValue; +import com.clickhouse.client.api.query.NullValueException; +import com.clickhouse.data.ClickHouseAggregateFunction; +import com.clickhouse.data.ClickHouseColumn; +import com.clickhouse.data.ClickHouseDataType; +import com.clickhouse.data.value.ClickHouseBitmap; + +import java.io.EOFException; +import java.io.IOException; +import java.math.BigDecimal; +import java.math.BigInteger; +import java.net.Inet4Address; +import java.net.Inet6Address; +import java.net.InetAddress; +import java.time.LocalDate; +import java.time.ZonedDateTime; +import java.util.List; +import java.util.TimeZone; +import java.util.UUID; + +/** + * Scope implementation shared by a row and every nested container on the read path. + */ +final class StreamReaderScope implements ValueReader { + + private final RowBinaryStreamReader owner; + + private final StreamReaderScope parent; + + private final SlotCursor cursor; + + private final int variantOrdinal; + + private StreamReaderScope child; + + private boolean markerConsumed; + + private boolean closed; + + private StreamReaderScope(RowBinaryStreamReader owner, StreamReaderScope parent, SlotCursor cursor, int variantOrdinal) { + this.owner = owner; + this.parent = parent; + this.cursor = cursor; + this.variantOrdinal = variantOrdinal; + } + + static StreamReaderScope row(RowBinaryStreamReader owner, List columns) { + return new StreamReaderScope(owner, null, SlotCursor.row(columns), -1); + } + + boolean isClosed() { + return closed; + } + + @Override + public byte readInt8() throws IOException { + prepare(ClickHouseDataType.Int8); + return finishByte(owner.binary().readByte()); + } + + @Override + public short readInt16() throws IOException { + prepare(ClickHouseDataType.Int16); + return finishShort(owner.binary().readShortLE()); + } + + @Override + public int readInt32() throws IOException { + prepare(ClickHouseDataType.Int32); + return finishInt(owner.binary().readIntLE()); + } + + @Override + public long readInt64() throws IOException { + prepare(ClickHouseDataType.Int64); + return finishLong(owner.binary().readLongLE()); + } + + @Override + public BigInteger readInt128() throws IOException { + prepare(ClickHouseDataType.Int128); + return finish(owner.binary().readBigIntegerLE(BinaryStreamReader.INT128_SIZE, false)); + } + + @Override + public BigInteger readInt256() throws IOException { + prepare(ClickHouseDataType.Int256); + return finish(owner.binary().readBigIntegerLE(BinaryStreamReader.INT256_SIZE, false)); + } + + @Override + public int readUInt8() throws IOException { + prepare(ClickHouseDataType.UInt8); + return finishInt(owner.binary().readUnsignedByte()); + } + + @Override + public int readUInt16() throws IOException { + prepare(ClickHouseDataType.UInt16); + return finishInt(owner.binary().readUnsignedShortLE()); + } + + @Override + public long readUInt32() throws IOException { + prepare(ClickHouseDataType.UInt32); + return finishLong(owner.binary().readUnsignedIntLE()); + } + + @Override + public BigInteger readUInt64() throws IOException { + prepare(ClickHouseDataType.UInt64); + return finish(owner.binary().readBigIntegerLE(BinaryStreamReader.INT64_SIZE, true)); + } + + @Override + public BigInteger readUInt128() throws IOException { + prepare(ClickHouseDataType.UInt128); + return finish(owner.binary().readBigIntegerLE(BinaryStreamReader.INT128_SIZE, true)); + } + + @Override + public BigInteger readUInt256() throws IOException { + prepare(ClickHouseDataType.UInt256); + return finish(owner.binary().readBigIntegerLE(BinaryStreamReader.INT256_SIZE, true)); + } + + @Override + public float readFloat32() throws IOException { + prepare(ClickHouseDataType.Float32); + return finishFloat(owner.binary().readFloatLE()); + } + + @Override + public double readFloat64() throws IOException { + prepare(ClickHouseDataType.Float64); + return finishDouble(owner.binary().readDoubleLE()); + } + + @Override + public float readBFloat16() throws IOException { + prepare(ClickHouseDataType.BFloat16); + return finishFloat(owner.binary().readBFloat16LE()); + } + + @Override + public boolean readBool() throws IOException { + prepare(ClickHouseDataType.Bool); + int value = owner.binary().readByte() & 0xFF; + cursor.advance(); + return value == 1; + } + + @Override + public String readString() throws IOException { + return readStringValue().asString(); + } + + @Override + public StringValue readStringValue() throws IOException { + ensureReadable(); + ClickHouseColumn effective = StreamTypes.unwrap(slot()); + ClickHouseDataType type = effective.getDataType(); + if (type != ClickHouseDataType.String && type != ClickHouseDataType.FixedString + && type != ClickHouseDataType.JSON) { + owner.fail(); + throw new IllegalStateException("Cannot read a string from " + StreamTypes.describe(effective)); + } + consumePresent(effective); + try { + StringValue value = type == ClickHouseDataType.FixedString + ? owner.binary().readFixedStringValue(effective.getPrecision()) + : owner.binary().readStringValue(); + cursor.advance(); + return value; + } catch (IOException e) { + owner.fail(); + throw e; + } + } + + @Override + public String readFixedString() throws IOException { + ensureReadable(); + ClickHouseColumn effective = StreamTypes.unwrap(slot()); + if (effective.getDataType() != ClickHouseDataType.FixedString) { + owner.fail(); + throw new IllegalStateException("Cannot read FixedString from " + StreamTypes.describe(effective)); + } + return readStringValue().asString(); + } + + @Override + public LocalDate readDate() throws IOException { + prepare(ClickHouseDataType.Date); + return finish(owner.binary().readDateAsLocalDate()); + } + + @Override + public LocalDate readDate32() throws IOException { + prepare(ClickHouseDataType.Date32); + return finish(owner.binary().readDate32AsLocalDate()); + } + + @Override + public ZonedDateTime readDateTime() throws IOException { + ClickHouseColumn column = prepare(ClickHouseDataType.DateTime, ClickHouseDataType.DateTime32); + TimeZone zone = column.getTimeZone() != null ? column.getTimeZone() : owner.timeZone(); + return finish(BinaryStreamReader.readDateTime32(owner.stream(), owner.scratch(), zone)); + } + + @Override + public ZonedDateTime readDateTime64() throws IOException { + ClickHouseColumn column = prepare(ClickHouseDataType.DateTime64); + TimeZone zone = column.getTimeZone() != null ? column.getTimeZone() : owner.timeZone(); + return finish(BinaryStreamReader.readDateTime64(owner.stream(), owner.scratch(), column.getScale(), zone)); + } + + @Override + public UUID readUUID() throws IOException { + prepare(ClickHouseDataType.UUID); + return finish(new UUID(owner.binary().readLongLE(), owner.binary().readLongLE())); + } + + @Override + public Inet4Address readIPv4() throws IOException { + prepare(ClickHouseDataType.IPv4); + byte[] raw = new byte[4]; + BinaryStreamReader.readNBytesLE(owner.stream(), raw, 0, 4); + return finish((Inet4Address) InetAddress.getByAddress(raw)); + } + + @Override + public Inet6Address readIPv6() throws IOException { + prepare(ClickHouseDataType.IPv6); + byte[] raw = BinaryStreamReader.readNBytes(owner.stream(), 16); + return finish((Inet6Address) InetAddress.getByAddress(raw)); + } + + @Override + public BigDecimal readDecimal() throws IOException { + ClickHouseColumn column = prepare(ClickHouseDataType.Decimal, ClickHouseDataType.Decimal32, + ClickHouseDataType.Decimal64, ClickHouseDataType.Decimal128, ClickHouseDataType.Decimal256); + return finish(owner.binary().readDecimal(column.getPrecision(), column.getScale())); + } + + @Override + public int readEnum8() throws IOException { + prepare(ClickHouseDataType.Enum8); + return finishInt(owner.binary().readByte()); + } + + @Override + public int readEnum16() throws IOException { + prepare(ClickHouseDataType.Enum16); + return finishInt(owner.binary().readShortLE()); + } + + @Override + public ClickHouseBitmap readBitmap() throws IOException { + ensureReadable(); + ClickHouseColumn effective = StreamTypes.unwrap(slot()); + if (effective.getDataType() != ClickHouseDataType.AggregateFunction + || effective.getAggregateFunction() != ClickHouseAggregateFunction.groupBitmap) { + owner.fail(); + throw new IllegalStateException("readBitmap is only valid for AggregateFunction(groupBitmap, ...) but was " + + StreamTypes.describe(effective)); + } + consumePresent(effective); + try { + ClickHouseBitmap value = ClickHouseBitmap.deserialize(owner.stream(), + effective.getNestedColumns().get(0).getDataType()); + cursor.advance(); + return value; + } catch (IOException e) { + owner.fail(); + throw e; + } + } + + @Override + public boolean isNull() throws IOException { + ensureReadable(); + if (markerConsumed) { + return false; + } + if (cursor.kind == SlotCursor.Kind.MAP && !cursor.expectingValue()) { + return false; + } + ClickHouseColumn effective = StreamTypes.unwrap(slot()); + if (effective.isNullable()) { + int marker = readMarker(); + if (marker == 1) { + cursor.advance(); + return true; + } + markerConsumed = true; + return false; + } + if (effective.getDataType() == ClickHouseDataType.Variant + || effective.getDataType() == ClickHouseDataType.Geometry) { + int ordinal = owner.binary().readByte() & 0xFF; + if (ordinal == 0xFF) { + cursor.advance(); + return true; + } + owner.unread(ordinal); + return false; + } + if (effective.getDataType() == ClickHouseDataType.Dynamic) { + return false; + } + return false; + } + + @Override + public boolean hasNext() { + return cursor.hasNext(); + } + + @Override + public int size() { + return cursor.size(); + } + + @Override + public int variantOrdinal() { + if (variantOrdinal < 0) { + throw new IllegalStateException("variantOrdinal is only valid on a variant scope"); + } + return variantOrdinal; + } + + @Override + public ClickHouseColumn runtimeType() { + if (cursor.kind != SlotCursor.Kind.SINGLE) { + throw new IllegalStateException("runtimeType is only valid on a variant or dynamic scope"); + } + return cursor.repeated(); + } + + @Override + public ValueReader beginArray() throws IOException { + return openCounted(false, false); + } + + @Override + public ValueReader beginTuple() throws IOException { + ensureReadable(); + ClickHouseColumn slot = slot(); + ClickHouseColumn effective = StreamTypes.unwrap(slot); + List fields = StreamTypes.tupleFields(effective); + if (fields == null) { + owner.fail(); + throw new IllegalStateException("beginTuple is not valid for " + StreamTypes.describe(effective)); + } + consumePresent(effective); + return nest(new StreamReaderScope(owner, this, SlotCursor.tuple(fields), -1)); + } + + @Override + public ValueReader beginMap() throws IOException { + ensureReadable(); + ClickHouseColumn slot = slot(); + ClickHouseColumn effective = StreamTypes.unwrap(slot); + if (effective.getDataType() != ClickHouseDataType.Map) { + owner.fail(); + throw new IllegalStateException("beginMap is not valid for " + StreamTypes.describe(effective)); + } + consumePresent(effective); + int pairs = readCount(); + return nest(new StreamReaderScope(owner, this, + SlotCursor.map(effective.getKeyInfo(), effective.getValueInfo(), pairs), -1)); + } + + @Override + public ValueReader beginNested() throws IOException { + return openCounted(true, false); + } + + @Override + public ValueReader beginVariant() throws IOException { + ensureReadable(); + ClickHouseColumn slot = slot(); + ClickHouseColumn effective = StreamTypes.unwrap(slot); + if (effective.getDataType() != ClickHouseDataType.Variant + && effective.getDataType() != ClickHouseDataType.Geometry) { + owner.fail(); + throw new IllegalStateException("beginVariant is not valid for " + StreamTypes.describe(effective)); + } + consumePresent(effective); + int ordinal = owner.binary().readByte() & 0xFF; + if (ordinal == 0xFF) { + cursor.advance(); + throw new NullValueException("Variant value is null"); + } + List alternatives = effective.getNestedColumns(); + if (ordinal >= alternatives.size()) { + owner.fail(); + throw new ClientException("Variant ordinal " + ordinal + " is out of range"); + } + return nest(new StreamReaderScope(owner, this, SlotCursor.single(alternatives.get(ordinal)), ordinal)); + } + + @Override + public ValueReader beginDynamic() throws IOException { + ensureReadable(); + ClickHouseColumn slot = slot(); + ClickHouseColumn effective = StreamTypes.unwrap(slot); + if (effective.getDataType() != ClickHouseDataType.Dynamic) { + owner.fail(); + throw new IllegalStateException("beginDynamic is not valid for " + StreamTypes.describe(effective)); + } + consumePresent(effective); + ClickHouseColumn runtime = owner.binary().readDynamicType(); + StreamReaderScope childScope = new StreamReaderScope(owner, this, SlotCursor.single(runtime), -1); + if (runtime.getDataType() == ClickHouseDataType.Nothing) { + childScope.cursor.forceComplete(); + } + return nest(childScope); + } + + @Override + public ValueReader beginQBit() throws IOException { + return openCounted(false, true); + } + + @Override + public int readInto(int[] dst, int off, int len) throws IOException { + ensureReadable(); + if (dst == null) { + throw new IllegalArgumentException("Destination array is required"); + } + if (off < 0 || len < 0 || off > dst.length - len) { + throw new IllegalArgumentException("readInto range is outside the destination array"); + } + if (cursor.kind != SlotCursor.Kind.ARRAY) { + owner.fail(); + throw new IllegalStateException("readInto is only valid inside an array"); + } + ClickHouseColumn element = StreamTypes.unwrap(cursor.repeated()); + if (element.getDataType() != ClickHouseDataType.Int32) { + owner.fail(); + throw new IllegalStateException("readInto expects Array(Int32) but was " + StreamTypes.describe(element)); + } + int count = Math.min(len, cursor.remaining()); + try { + for (int i = 0; i < count; i++) { + if (element.isNullable()) { + int marker = readMarker(); + if (marker == 1) { + owner.fail(); + throw new NullValueException("Array element is null"); + } + } + dst[off + i] = owner.binary().readIntLE(); + cursor.advance(); + } + } catch (IOException e) { + owner.fail(); + throw e; + } + return count; + } + + @Override + public void skip() throws IOException { + ensureReadable(); + ClickHouseColumn effective = StreamTypes.unwrap(slot()); + if (markerConsumed) { + markerConsumed = false; + } else if (effective.isNullable()) { + int marker = readMarker(); + if (marker == 1) { + cursor.advance(); + return; + } + } + try { + skipBody(effective); + } catch (IOException e) { + owner.fail(); + throw e; + } catch (RuntimeException e) { + owner.fail(); + throw e; + } + cursor.advance(); + } + + @Override + public void close() throws IOException { + if (closed) { + return; + } + if (owner.isFailed()) { + closed = true; + return; + } + if (child != null) { + owner.fail(); + closed = true; + throw new IllegalStateException("Unclosed nested value"); + } + try { + cursor.checkComplete(); + } catch (RuntimeException e) { + owner.fail(); + closed = true; + throw e; + } + closed = true; + if (parent != null) { + parent.child = null; + parent.cursor.advance(); + } else { + owner.onRowClosed(); + } + } + + private ValueReader openCounted(boolean nested, boolean qbit) throws IOException { + ensureReadable(); + ClickHouseColumn slot = slot(); + ClickHouseColumn effective = StreamTypes.unwrap(slot); + ClickHouseColumn element; + if (nested) { + if (effective.getDataType() != ClickHouseDataType.Nested) { + owner.fail(); + throw new IllegalStateException("beginNested is not valid for " + StreamTypes.describe(effective)); + } + element = ClickHouseColumn.tuple(effective.getColumnName(), false, + effective.getNestedColumns().toArray(new ClickHouseColumn[0])); + } else if (qbit) { + if (effective.getDataType() != ClickHouseDataType.QBit) { + owner.fail(); + throw new IllegalStateException("beginQBit is not valid for " + StreamTypes.describe(effective)); + } + element = StreamTypes.arrayElement(effective); + } else { + element = StreamTypes.arrayElement(effective); + if (element == null || effective.getDataType() == ClickHouseDataType.QBit) { + owner.fail(); + throw new IllegalStateException("beginArray is not valid for " + StreamTypes.describe(effective)); + } + } + consumePresent(effective); + int count = readCount(); + if (qbit && count != effective.getPrecision()) { + owner.fail(); + throw new ClientException("QBit column '" + effective.getColumnName() + "' expected exactly " + + effective.getPrecision() + " elements but received " + count); + } + return nest(new StreamReaderScope(owner, this, SlotCursor.array(element, count), -1)); + } + + private ClickHouseColumn prepare(ClickHouseDataType... allowed) throws IOException { + ensureReadable(); + ClickHouseColumn effective = StreamTypes.unwrap(slot()); + if (!StreamTypes.isOneOf(effective.getDataType(), allowed)) { + owner.fail(); + throw new IllegalStateException("Cannot read " + allowed[0] + " from " + StreamTypes.describe(effective)); + } + consumePresent(effective); + return effective; + } + + private void consumePresent(ClickHouseColumn effective) throws IOException { + if (markerConsumed) { + markerConsumed = false; + return; + } + if (effective.isNullable()) { + int marker = readMarker(); + if (marker == 1) { + cursor.advance(); + throw new NullValueException("Column '" + effective.getColumnName() + "' is null"); + } + } + } + + private int readMarker() throws IOException { + int marker = owner.binary().readByte() & 0xFF; + if (marker != 0 && marker != 1) { + owner.fail(); + throw new ClientException("Invalid null marker: " + marker); + } + return marker; + } + + private int readCount() throws IOException { + try { + return BinaryStreamReader.readVarInt(owner.stream()); + } catch (IOException e) { + owner.fail(); + throw e; + } + } + + private ClickHouseColumn slot() { + try { + return cursor.current(); + } catch (RuntimeException e) { + owner.fail(); + throw e; + } + } + + private void ensureReadable() { + if (owner.isFailed()) { + throw new IllegalStateException("Reader is unusable after a previous failure"); + } + if (closed) { + throw new IllegalStateException("Scope is closed"); + } + if (child != null) { + owner.fail(); + throw new IllegalStateException("A nested value is still open"); + } + } + + private byte finishByte(byte value) { + cursor.advance(); + return value; + } + + private short finishShort(short value) { + cursor.advance(); + return value; + } + + private int finishInt(int value) { + cursor.advance(); + return value; + } + + private long finishLong(long value) { + cursor.advance(); + return value; + } + + private float finishFloat(float value) { + cursor.advance(); + return value; + } + + private double finishDouble(double value) { + cursor.advance(); + return value; + } + + private T finish(T value) { + cursor.advance(); + return value; + } + + private StreamReaderScope nest(StreamReaderScope childScope) { + this.child = childScope; + return childScope; + } + + private void skipBody(ClickHouseColumn column) throws IOException { + ClickHouseDataType type = column.getDataType(); + switch (type) { + case Int8: + case UInt8: + case Bool: + case Enum8: + owner.binary().readByte(); + return; + case Int16: + case UInt16: + case Date: + case Enum16: + case BFloat16: + owner.skipFully(2); + return; + case Int32: + case UInt32: + case Float32: + case Date32: + case DateTime: + case DateTime32: + case Decimal32: + case IPv4: + case Time: + owner.skipFully(4); + return; + case Int64: + case UInt64: + case Float64: + case DateTime64: + case Decimal64: + case Time64: + case IntervalYear: + case IntervalQuarter: + case IntervalMonth: + case IntervalWeek: + case IntervalDay: + case IntervalHour: + case IntervalMinute: + case IntervalSecond: + case IntervalMicrosecond: + case IntervalMillisecond: + case IntervalNanosecond: + owner.skipFully(8); + return; + case Int128: + case UInt128: + case Decimal128: + case UUID: + case IPv6: + owner.skipFully(16); + return; + case Int256: + case UInt256: + case Decimal256: + owner.skipFully(32); + return; + case Decimal: + owner.skipFully(StreamTypes.decimalBytes(column.getPrecision())); + return; + case String: + case JSON: + owner.skipFully(BinaryStreamReader.readVarInt(owner.stream())); + return; + case FixedString: + owner.skipFully(column.getPrecision()); + return; + case Nothing: + return; + case Array: + case QBit: + skipRepeated(column.getNestedColumns().get(0), BinaryStreamReader.readVarInt(owner.stream())); + return; + case Nested: + int rows = BinaryStreamReader.readVarInt(owner.stream()); + List fields = column.getNestedColumns(); + for (int r = 0; r < rows; r++) { + for (int f = 0; f < fields.size(); f++) { + skipValue(fields.get(f)); + } + } + return; + case Map: + int pairs = BinaryStreamReader.readVarInt(owner.stream()); + for (int i = 0; i < pairs; i++) { + skipValue(column.getKeyInfo()); + skipValue(column.getValueInfo()); + } + return; + case Tuple: + List tupleFields = column.getNestedColumns(); + for (int i = 0; i < tupleFields.size(); i++) { + skipValue(tupleFields.get(i)); + } + return; + case Variant: + case Geometry: + int ordinal = owner.binary().readByte() & 0xFF; + if (ordinal == 0xFF) { + return; + } + skipValue(column.getNestedColumns().get(ordinal)); + return; + case Point: + owner.skipFully(16); + return; + case Ring: + case LineString: + case MultiPoint: + owner.skipFully(BinaryStreamReader.readVarInt(owner.stream()) * 16); + return; + case Polygon: + case MultiLineString: + skipRings(BinaryStreamReader.readVarInt(owner.stream())); + return; + case MultiPolygon: + int polygons = BinaryStreamReader.readVarInt(owner.stream()); + for (int i = 0; i < polygons; i++) { + skipRings(BinaryStreamReader.readVarInt(owner.stream())); + } + return; + case SimpleAggregateFunction: + case LowCardinality: + skipValue(column.getNestedColumns().get(0)); + return; + case Dynamic: + skipValue(owner.binary().readDynamicType()); + return; + case AggregateFunction: + if (column.getAggregateFunction() != ClickHouseAggregateFunction.groupBitmap) { + throw new ClientException("Cannot skip aggregate function " + column.getAggregateFunction()); + } + ClickHouseBitmap.deserialize(owner.stream(), column.getNestedColumns().get(0).getDataType()); + return; + default: + throw new ClientException("Cannot skip " + StreamTypes.describe(column)); + } + } + + private void skipRepeated(ClickHouseColumn element, int count) throws IOException { + for (int i = 0; i < count; i++) { + skipValue(element); + } + } + + private void skipRings(int count) throws IOException { + for (int i = 0; i < count; i++) { + owner.skipFully(BinaryStreamReader.readVarInt(owner.stream()) * 16); + } + } + + private void skipValue(ClickHouseColumn column) throws IOException { + ClickHouseColumn effective = StreamTypes.unwrap(column); + if (effective.isNullable()) { + int marker = owner.binary().readByte() & 0xFF; + if (marker == 1) { + return; + } + if (marker != 0) { + throw new ClientException("Invalid null marker: " + marker); + } + } + skipBody(effective); + } +} diff --git a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/stream/StreamTypes.java b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/stream/StreamTypes.java new file mode 100644 index 000000000..603ce6d20 --- /dev/null +++ b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/stream/StreamTypes.java @@ -0,0 +1,137 @@ +package com.clickhouse.client.api.data_formats.stream; + +import com.clickhouse.client.api.ClientException; +import com.clickhouse.data.ClickHouseColumn; +import com.clickhouse.data.ClickHouseDataType; + +import java.util.ArrayDeque; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; + +/** + * Shared column shapes for the streaming reader and writer. + */ +final class StreamTypes { + + static final List POINT_FIELDS = Collections.unmodifiableList(Arrays.asList( + ClickHouseColumn.of("x", "Float64"), + ClickHouseColumn.of("y", "Float64"))); + + static final ClickHouseColumn POINT = ClickHouseColumn.of("p", "Point"); + + static final ClickHouseColumn RING = ClickHouseColumn.of("r", "Ring"); + + static final ClickHouseColumn POLYGON = ClickHouseColumn.of("g", "Polygon"); + + static final ClickHouseColumn NOTHING = ClickHouseColumn.of("v", "Nothing"); + + private StreamTypes() { + } + + static boolean omitted(ClickHouseColumn column) { + return column.hasDefault() && column.getDefaultValue() != ClickHouseColumn.DefaultValue.DEFAULT; + } + + static ClickHouseColumn unwrap(ClickHouseColumn column) { + if (column.getDataType() == ClickHouseDataType.SimpleAggregateFunction) { + return column.getNestedColumns().get(0); + } + return column; + } + + static boolean isOneOf(ClickHouseDataType actual, ClickHouseDataType... allowed) { + for (int i = 0; i < allowed.length; i++) { + if (actual == allowed[i]) { + return true; + } + } + return false; + } + + static String describe(ClickHouseColumn column) { + String type = column.getOriginalTypeName(); + if (type == null || type.isEmpty()) { + type = column.getDataType().name(); + } + String name = column.getColumnName(); + if (name == null || name.isEmpty()) { + return type; + } + return name + " " + type; + } + + /** + * Element column of a length-prefixed array-like value, or null when {@code column} is not one. + */ + static ClickHouseColumn arrayElement(ClickHouseColumn column) { + ClickHouseDataType type = column.getDataType(); + if (type == ClickHouseDataType.Array || type == ClickHouseDataType.QBit) { + return column.getNestedColumns().get(0); + } + if (type == ClickHouseDataType.Ring || type == ClickHouseDataType.LineString + || type == ClickHouseDataType.MultiPoint) { + return POINT; + } + if (type == ClickHouseDataType.Polygon || type == ClickHouseDataType.MultiLineString) { + return RING; + } + if (type == ClickHouseDataType.MultiPolygon) { + return POLYGON; + } + return null; + } + + /** + * Tuple fields, or the two coordinates of a {@code Point}. Null when {@code column} is neither. + */ + static List tupleFields(ClickHouseColumn column) { + if (column.getDataType() == ClickHouseDataType.Tuple) { + return column.getNestedColumns(); + } + if (column.getDataType() == ClickHouseDataType.Point) { + return POINT_FIELDS; + } + return null; + } + + static int decimalBytes(int precision) { + if (precision <= ClickHouseDataType.Decimal32.getMaxScale()) { + return 4; + } + if (precision <= ClickHouseDataType.Decimal64.getMaxScale()) { + return 8; + } + if (precision <= ClickHouseDataType.Decimal128.getMaxScale()) { + return 16; + } + return 32; + } + + /** + * Rejects a runtime type whose dynamic tag encoding is incomplete and would desynchronize the stream. + */ + static void rejectDynamicPayload(ClickHouseColumn column) { + ArrayDeque pending = new ArrayDeque(); + pending.add(column); + while (!pending.isEmpty()) { + ClickHouseColumn current = pending.removeFirst(); + switch (current.getDataType()) { + case Tuple: + case Variant: + case Dynamic: + case JSON: + case Nested: + case QBit: + case AggregateFunction: + case SimpleAggregateFunction: + case Geometry: + throw new ClientException("Serializing a " + current.getDataType().name() + + " value inside a Dynamic column is not supported"); + default: + break; + } + pending.addAll(current.getNestedColumns()); + } + } +} diff --git a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/stream/StreamWriterScope.java b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/stream/StreamWriterScope.java new file mode 100644 index 000000000..44fa8d736 --- /dev/null +++ b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/stream/StreamWriterScope.java @@ -0,0 +1,1228 @@ +package com.clickhouse.client.api.data_formats.stream; + +import com.clickhouse.client.api.data_formats.internal.SerializerUtils; +import com.clickhouse.data.ClickHouseAggregateFunction; +import com.clickhouse.data.ClickHouseColumn; +import com.clickhouse.data.ClickHouseDataType; +import com.clickhouse.data.format.BinaryStreamUtils; +import com.clickhouse.data.value.ClickHouseBitmap; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.io.OutputStream; +import java.math.BigDecimal; +import java.math.BigInteger; +import java.net.Inet4Address; +import java.net.Inet6Address; +import java.nio.ByteBuffer; +import java.time.Instant; +import java.time.LocalDate; +import java.time.LocalDateTime; +import java.time.ZoneId; +import java.time.ZoneOffset; +import java.time.ZonedDateTime; +import java.util.List; +import java.util.UUID; + +/** + * Scope implementation shared by a row and every nested container. + */ +final class StreamWriterScope implements ValueWriter { + + private final RowBinaryStreamWriter owner; + + private final StreamWriterScope parent; + + private final SlotCursor cursor; + + private final OutputStream out; + + private final ByteArrayOutputStream buffer; + + /** JSON column whose typed paths apply, when this scope is a JSON object. */ + private ClickHouseColumn jsonType; + + /** Dotted prefix contributed by enclosing {@code beginObject} calls. */ + private String pathPrefix = ""; + + /** Nested object whose leaf paths belong to the parent JSON value. */ + private boolean flattensIntoParent; + + private StreamWriterScope child; + + private boolean closed; + + private static final ClickHouseColumn DYN_STRING = ClickHouseColumn.of("v", "String"); + + private static final ClickHouseColumn DYN_INT32 = ClickHouseColumn.of("v", "Int32"); + + private static final ClickHouseColumn DYN_INT64 = ClickHouseColumn.of("v", "Int64"); + + private static final ClickHouseColumn DYN_FLOAT64 = ClickHouseColumn.of("v", "Float64"); + + private static final ClickHouseColumn DYN_BOOL = ClickHouseColumn.of("v", "Bool"); + + private static final ClickHouseColumn DYN_JSON = ClickHouseColumn.of("v", "JSON"); + + private StreamWriterScope(RowBinaryStreamWriter owner, StreamWriterScope parent, SlotCursor cursor, + OutputStream out, ByteArrayOutputStream buffer) { + this.owner = owner; + this.parent = parent; + this.cursor = cursor; + this.out = out; + this.buffer = buffer; + } + + static StreamWriterScope row(RowBinaryStreamWriter owner, List columns) { + return new StreamWriterScope(owner, null, SlotCursor.row(columns), owner.out(), null); + } + + boolean isClosed() { + return closed; + } + + @Override + public void writeInt8(byte value) throws IOException { + prepare(ClickHouseDataType.Int8); + body(() -> BinaryStreamUtils.writeInt8(out, value)); + } + + @Override + public void writeInt8(int value) throws IOException { + checkRange(value, Byte.MIN_VALUE, Byte.MAX_VALUE, "Int8"); + writeInt8((byte) value); + } + + @Override + public void writeInt16(short value) throws IOException { + prepare(ClickHouseDataType.Int16); + body(() -> BinaryStreamUtils.writeInt16(out, value)); + } + + @Override + public void writeInt16(int value) throws IOException { + checkRange(value, Short.MIN_VALUE, Short.MAX_VALUE, "Int16"); + writeInt16((short) value); + } + + @Override + public void writeInt32(int value) throws IOException { + prepare(ClickHouseDataType.Int32); + body(() -> BinaryStreamUtils.writeInt32(out, value)); + } + + @Override + public void writeInt32(long value) throws IOException { + if (value != (int) value) { + owner.fail(); + throw new IllegalArgumentException("Int32 overflow: " + value); + } + writeInt32((int) value); + } + + @Override + public void writeInt64(long value) throws IOException { + prepare(ClickHouseDataType.Int64); + body(() -> BinaryStreamUtils.writeInt64(out, value)); + } + + @Override + public void writeInt128(BigInteger value) throws IOException { + requireValue(value); + prepare(ClickHouseDataType.Int128); + body(() -> BinaryStreamUtils.writeInt128(out, value)); + } + + @Override + public void writeInt256(BigInteger value) throws IOException { + requireValue(value); + prepare(ClickHouseDataType.Int256); + body(() -> BinaryStreamUtils.writeInt256(out, value)); + } + + @Override + public void writeUInt8(int value) throws IOException { + checkRange(value, 0, 255, "UInt8"); + prepare(ClickHouseDataType.UInt8); + body(() -> BinaryStreamUtils.writeUnsignedInt8(out, value)); + } + + @Override + public void writeUInt16(int value) throws IOException { + checkRange(value, 0, 65535, "UInt16"); + prepare(ClickHouseDataType.UInt16); + body(() -> BinaryStreamUtils.writeUnsignedInt16(out, value)); + } + + @Override + public void writeUInt32(long value) throws IOException { + if (value < 0 || value > 0xFFFFFFFFL) { + owner.fail(); + throw new IllegalArgumentException("UInt32 overflow: " + value); + } + prepare(ClickHouseDataType.UInt32); + body(() -> BinaryStreamUtils.writeUnsignedInt32(out, value)); + } + + @Override + public void writeUInt64(long value) throws IOException { + prepare(ClickHouseDataType.UInt64); + body(() -> BinaryStreamUtils.writeUnsignedInt64(out, value)); + } + + @Override + public void writeUInt64(BigInteger value) throws IOException { + requireValue(value); + prepare(ClickHouseDataType.UInt64); + body(() -> BinaryStreamUtils.writeUnsignedInt64(out, value)); + } + + @Override + public void writeUInt128(BigInteger value) throws IOException { + requireValue(value); + prepare(ClickHouseDataType.UInt128); + body(() -> BinaryStreamUtils.writeUnsignedInt128(out, value)); + } + + @Override + public void writeUInt256(BigInteger value) throws IOException { + requireValue(value); + prepare(ClickHouseDataType.UInt256); + body(() -> BinaryStreamUtils.writeUnsignedInt256(out, value)); + } + + @Override + public void writeFloat32(float value) throws IOException { + prepare(ClickHouseDataType.Float32); + body(() -> BinaryStreamUtils.writeFloat32(out, value)); + } + + @Override + public void writeFloat64(double value) throws IOException { + prepare(ClickHouseDataType.Float64); + body(() -> BinaryStreamUtils.writeFloat64(out, value)); + } + + @Override + public void writeBFloat16(float value) throws IOException { + prepare(ClickHouseDataType.BFloat16); + body(() -> BinaryStreamUtils.writeBFloat16(out, value)); + } + + @Override + public void writeBool(boolean value) throws IOException { + prepare(ClickHouseDataType.Bool); + body(() -> BinaryStreamUtils.writeBoolean(out, value)); + } + + @Override + public void writeString(String value) throws IOException { + if (value == null) { + writeNullFor(ClickHouseDataType.String, ClickHouseDataType.JSON); + return; + } + prepare(ClickHouseDataType.String, ClickHouseDataType.JSON); + body(() -> BinaryStreamUtils.writeString(out, value)); + } + + @Override + public void writeString(byte[] value) throws IOException { + if (value == null) { + writeNullFor(ClickHouseDataType.String, ClickHouseDataType.JSON); + return; + } + prepare(ClickHouseDataType.String, ClickHouseDataType.JSON); + body(() -> BinaryStreamUtils.writeString(out, value)); + } + + @Override + public void writeString(ByteBuffer value) throws IOException { + if (value == null) { + writeString((byte[]) null); + return; + } + ByteBuffer view = value.duplicate(); + int length = view.remaining(); + prepare(ClickHouseDataType.String, ClickHouseDataType.JSON); + body(() -> { + SerializerUtils.writeVarInt(out, length); + if (view.hasArray()) { + out.write(view.array(), view.arrayOffset() + view.position(), length); + } else { + byte[] copy = new byte[length]; + view.get(copy); + out.write(copy); + } + }); + } + + @Override + public void writeFixedString(String value) throws IOException { + if (value == null) { + writeNullFor(ClickHouseDataType.FixedString); + return; + } + ClickHouseColumn column = prepare(ClickHouseDataType.FixedString); + int length = column.getPrecision(); + body(() -> BinaryStreamUtils.writeFixedString(out, value, length)); + } + + @Override + public void writeFixedString(byte[] value) throws IOException { + if (value == null) { + writeNullFor(ClickHouseDataType.FixedString); + return; + } + ClickHouseColumn column = prepare(ClickHouseDataType.FixedString); + int length = column.getPrecision(); + body(() -> SerializerUtils.writeFixedStringBytes(out, value, length)); + } + + @Override + public void writeDate(LocalDate value) throws IOException { + requireValue(value); + prepare(ClickHouseDataType.Date); + body(() -> SerializerUtils.writeDate(out, value, ZoneOffset.UTC)); + } + + @Override + public void writeDate32(LocalDate value) throws IOException { + requireValue(value); + prepare(ClickHouseDataType.Date32); + body(() -> SerializerUtils.writeDate32(out, value, ZoneOffset.UTC)); + } + + @Override + public void writeDateTime(LocalDateTime value) throws IOException { + requireValue(value); + ClickHouseColumn column = prepare(ClickHouseDataType.DateTime, ClickHouseDataType.DateTime32); + ZoneId zone = zone(column); + body(() -> SerializerUtils.writeDateTime(out, value, zone)); + } + + @Override + public void writeDateTime(ZonedDateTime value) throws IOException { + requireValue(value); + ClickHouseColumn column = prepare(ClickHouseDataType.DateTime, ClickHouseDataType.DateTime32); + ZoneId zone = zone(column); + body(() -> SerializerUtils.writeDateTime(out, value, zone)); + } + + @Override + public void writeDateTime(Instant value) throws IOException { + requireValue(value); + ClickHouseColumn column = prepare(ClickHouseDataType.DateTime, ClickHouseDataType.DateTime32); + ZoneId zone = zone(column); + body(() -> SerializerUtils.writeDateTime(out, value, zone)); + } + + @Override + public void writeDateTime64(LocalDateTime value) throws IOException { + requireValue(value); + ClickHouseColumn column = prepare(ClickHouseDataType.DateTime64); + int scale = column.getScale(); + ZoneId zone = zone(column); + body(() -> SerializerUtils.writeDateTime64(out, value, scale, zone)); + } + + @Override + public void writeDateTime64(ZonedDateTime value) throws IOException { + requireValue(value); + ClickHouseColumn column = prepare(ClickHouseDataType.DateTime64); + int scale = column.getScale(); + ZoneId zone = zone(column); + body(() -> SerializerUtils.writeDateTime64(out, value, scale, zone)); + } + + @Override + public void writeUUID(UUID value) throws IOException { + requireValue(value); + prepare(ClickHouseDataType.UUID); + body(() -> BinaryStreamUtils.writeUuid(out, value)); + } + + @Override + public void writeIPv4(Inet4Address value) throws IOException { + requireValue(value); + prepare(ClickHouseDataType.IPv4); + body(() -> BinaryStreamUtils.writeInet4Address(out, value)); + } + + @Override + public void writeIPv6(Inet6Address value) throws IOException { + requireValue(value); + prepare(ClickHouseDataType.IPv6); + body(() -> BinaryStreamUtils.writeInet6Address(out, value)); + } + + @Override + public void writeDecimal(BigDecimal value) throws IOException { + requireValue(value); + ClickHouseColumn column = prepare(ClickHouseDataType.Decimal, ClickHouseDataType.Decimal32, + ClickHouseDataType.Decimal64, ClickHouseDataType.Decimal128, ClickHouseDataType.Decimal256); + int precision = column.getPrecision(); + int scale = column.getScale(); + body(() -> BinaryStreamUtils.writeDecimal(out, value, precision, scale)); + } + + @Override + public void writeEnum8(int value) throws IOException { + checkRange(value, Byte.MIN_VALUE, Byte.MAX_VALUE, "Enum8"); + prepare(ClickHouseDataType.Enum8); + body(() -> BinaryStreamUtils.writeInt8(out, value)); + } + + @Override + public void writeEnum8(String name) throws IOException { + writeEnum(name, ClickHouseDataType.Enum8); + } + + @Override + public void writeEnum16(int value) throws IOException { + checkRange(value, Short.MIN_VALUE, Short.MAX_VALUE, "Enum16"); + prepare(ClickHouseDataType.Enum16); + body(() -> BinaryStreamUtils.writeInt16(out, value)); + } + + @Override + public void writeEnum16(String name) throws IOException { + writeEnum(name, ClickHouseDataType.Enum16); + } + + @Override + public void writeNull() throws IOException { + ensureWritable(); + if (cursor.kind == SlotCursor.Kind.MAP && !cursor.expectingValue()) { + owner.fail(); + throw new IllegalArgumentException("Map keys are not nullable"); + } + ClickHouseColumn slot = slot(); + ClickHouseColumn effective = StreamTypes.unwrap(slot); + if (effective.isNullable()) { + try { + if (cursor.kind == SlotCursor.Kind.ROW && owner.defaultsSupport()) { + SerializerUtils.writeNonNull(out); + } + SerializerUtils.writeNull(out); + } catch (IOException e) { + owner.fail(); + throw e; + } + cursor.advance(); + return; + } + if (effective.getDataType() == ClickHouseDataType.Dynamic) { + writePresent(slot); + body(() -> SerializerUtils.writeDynamicTypeTag(out, StreamTypes.NOTHING)); + return; + } + if (effective.getDataType() == ClickHouseDataType.Variant + || effective.getDataType() == ClickHouseDataType.Geometry) { + writePresent(slot); + body(() -> out.write(0xFF)); + return; + } + owner.fail(); + throw new IllegalArgumentException("Cannot write null into non-nullable " + StreamTypes.describe(effective)); + } + + @Override + public void writeDefault() throws IOException { + ensureWritable(); + if (cursor.kind != SlotCursor.Kind.ROW || !owner.defaultsSupport()) { + owner.fail(); + throw new IllegalStateException("writeDefault is only valid for a top-level column of RowBinaryWithDefaults"); + } + ClickHouseColumn slot = slot(); + if (!slot.hasDefault() || slot.getDefaultValue() != ClickHouseColumn.DefaultValue.DEFAULT) { + owner.fail(); + throw new IllegalArgumentException("Column '" + slot.getColumnName() + "' has no DEFAULT"); + } + try { + SerializerUtils.writeNull(out); + } catch (IOException e) { + owner.fail(); + throw e; + } + cursor.advance(); + } + + @Override + public void writeBitmap(ClickHouseBitmap value) throws IOException { + if (value == null) { + owner.fail(); + throw new IllegalArgumentException("groupBitmap value is null"); + } + ensureWritable(); + ClickHouseColumn slot = slot(); + ClickHouseColumn effective = StreamTypes.unwrap(slot); + if (effective.getDataType() != ClickHouseDataType.AggregateFunction + || effective.getAggregateFunction() != ClickHouseAggregateFunction.groupBitmap) { + owner.fail(); + throw new IllegalStateException("writeBitmap is only valid for AggregateFunction(groupBitmap, ...) but was " + + StreamTypes.describe(effective)); + } + writePresent(slot); + body(() -> out.write(value.toBytes())); + } + + @Override + public ValueWriter beginArray() throws IOException { + return openArray(SlotCursor.UNKNOWN); + } + + @Override + public ValueWriter beginArray(int size) throws IOException { + if (size < 0) { + owner.fail(); + throw new IllegalArgumentException("Array size is negative"); + } + return openArray(size); + } + + @Override + public ValueWriter beginTuple() throws IOException { + ensureWritable(); + ClickHouseColumn slot = slot(); + ClickHouseColumn effective = StreamTypes.unwrap(slot); + List fields = StreamTypes.tupleFields(effective); + if (fields == null) { + owner.fail(); + throw new IllegalStateException("beginTuple is not valid for " + StreamTypes.describe(effective)); + } + writePresent(slot); + return nest(new StreamWriterScope(owner, this, SlotCursor.tuple(fields), out, null)); + } + + @Override + public ValueWriter beginMap() throws IOException { + return openMap(SlotCursor.UNKNOWN); + } + + @Override + public ValueWriter beginMap(int pairs) throws IOException { + if (pairs < 0) { + owner.fail(); + throw new IllegalArgumentException("Map size is negative"); + } + return openMap(pairs); + } + + @Override + public ValueWriter beginNested() throws IOException { + return openNested(SlotCursor.UNKNOWN); + } + + @Override + public ValueWriter beginNested(int rows) throws IOException { + if (rows < 0) { + owner.fail(); + throw new IllegalArgumentException("Nested row count is negative"); + } + return openNested(rows); + } + + @Override + public ValueWriter beginVariant(int ordinal) throws IOException { + if (ordinal < 0) { + owner.fail(); + throw new IllegalArgumentException("Variant ordinal is negative"); + } + ensureWritable(); + ClickHouseColumn slot = slot(); + ClickHouseColumn effective = StreamTypes.unwrap(slot); + if (effective.getDataType() != ClickHouseDataType.Variant + && effective.getDataType() != ClickHouseDataType.Geometry) { + owner.fail(); + throw new IllegalStateException("beginVariant is not valid for " + StreamTypes.describe(effective)); + } + List alternatives = effective.getNestedColumns(); + if (ordinal >= alternatives.size()) { + owner.fail(); + throw new IllegalArgumentException("Variant ordinal " + ordinal + " is out of range for " + + StreamTypes.describe(effective)); + } + writePresent(slot); + try { + BinaryStreamUtils.writeUnsignedInt8(out, ordinal); + } catch (IOException e) { + owner.fail(); + throw e; + } + return nest(new StreamWriterScope(owner, this, SlotCursor.single(alternatives.get(ordinal)), out, null)); + } + + @Override + public ValueWriter beginDynamic(ClickHouseColumn runtimeType) throws IOException { + if (runtimeType == null) { + owner.fail(); + throw new IllegalArgumentException("Dynamic runtime type is required"); + } + ensureWritable(); + ClickHouseColumn slot = slot(); + ClickHouseColumn effective = StreamTypes.unwrap(slot); + if (effective.getDataType() != ClickHouseDataType.Dynamic) { + owner.fail(); + throw new IllegalStateException("beginDynamic is not valid for " + StreamTypes.describe(effective)); + } + try { + StreamTypes.rejectDynamicPayload(runtimeType); + } catch (RuntimeException e) { + owner.fail(); + throw e; + } + writePresent(slot); + try { + SerializerUtils.writeDynamicTypeTag(out, runtimeType); + } catch (IOException e) { + owner.fail(); + throw e; + } catch (RuntimeException e) { + owner.fail(); + throw e; + } + StreamWriterScope childScope = new StreamWriterScope(owner, this, SlotCursor.single(runtimeType), out, null); + if (StreamTypes.unwrap(runtimeType).getDataType() == ClickHouseDataType.Nothing) { + childScope.cursor.forceComplete(); + } + return nest(childScope); + } + + @Override + public ValueWriter beginJson() throws IOException { + return openDocument(SlotCursor.UNKNOWN); + } + + @Override + public ValueWriter beginJson(int paths) throws IOException { + if (paths < 0) { + owner.fail(); + throw new IllegalArgumentException("JSON path count is negative"); + } + return openDocument(paths); + } + + private ValueWriter openDocument(int paths) throws IOException { + if (cursor.kind == SlotCursor.Kind.JSON) { + owner.fail(); + throw new IllegalStateException("Inside a JSON object use beginJson(path, paths)"); + } + ensureWritable(); + ClickHouseColumn slot = slot(); + ClickHouseColumn effective = StreamTypes.unwrap(slot); + if (effective.getDataType() != ClickHouseDataType.JSON) { + owner.fail(); + throw new IllegalStateException("beginJson is not valid for " + StreamTypes.describe(effective)); + } + writePresent(slot); + return openJsonValue(effective, "", paths); + } + + @Override + public void writeString(String path, String value) throws IOException { + if (value == null) { + writeNull(path); + return; + } + writeJsonScalar(path, ClickHouseDataType.String, DYN_STRING, () -> BinaryStreamUtils.writeString(out, value)); + } + + @Override + public void writeInt32(String path, int value) throws IOException { + writeJsonScalar(path, ClickHouseDataType.Int32, DYN_INT32, () -> BinaryStreamUtils.writeInt32(out, value)); + } + + @Override + public void writeInt64(String path, long value) throws IOException { + writeJsonScalar(path, ClickHouseDataType.Int64, DYN_INT64, () -> BinaryStreamUtils.writeInt64(out, value)); + } + + @Override + public void writeFloat64(String path, double value) throws IOException { + writeJsonScalar(path, ClickHouseDataType.Float64, DYN_FLOAT64, () -> BinaryStreamUtils.writeFloat64(out, value)); + } + + @Override + public void writeBool(String path, boolean value) throws IOException { + writeJsonScalar(path, ClickHouseDataType.Bool, DYN_BOOL, () -> BinaryStreamUtils.writeBoolean(out, value)); + } + + @Override + public void writeNull(String path) throws IOException { + ensureJson(); + String full = fullPath(path); + ClickHouseColumn typed = typedPath(full); + writePathName(full); + try { + if (typed == null) { + SerializerUtils.writeDynamicTypeTag(out, StreamTypes.NOTHING); + } else if (typed.isNullable()) { + SerializerUtils.writeNull(out); + } else { + throw new IllegalArgumentException("Cannot write null into non-nullable JSON path '" + full + "'"); + } + } catch (IOException e) { + owner.fail(); + throw e; + } catch (RuntimeException e) { + owner.fail(); + throw e; + } + cursor.advance(); + } + + @Override + public ValueWriter beginObject(String path) throws IOException { + return openObject(path, SlotCursor.UNKNOWN); + } + + @Override + public ValueWriter beginObject(String path, int paths) throws IOException { + if (paths < 0) { + owner.fail(); + throw new IllegalArgumentException("JSON path count is negative"); + } + return openObject(path, paths); + } + + private ValueWriter openObject(String path, int paths) throws IOException { + ensureJson(); + StreamWriterScope nested = new StreamWriterScope(owner, this, SlotCursor.json(paths), out, null); + nested.jsonType = jsonType; + nested.pathPrefix = fullPath(path) + "."; + nested.flattensIntoParent = true; + return nest(nested); + } + + @Override + public ValueWriter beginArray(String path, ClickHouseColumn elementType) throws IOException { + return beginJsonArray(path, elementType, SlotCursor.UNKNOWN); + } + + @Override + public ValueWriter beginArray(String path, ClickHouseColumn elementType, int size) throws IOException { + if (size < 0) { + owner.fail(); + throw new IllegalArgumentException("Array size is negative"); + } + return beginJsonArray(path, elementType, size); + } + + @Override + public ValueWriter beginJson(String path) throws IOException { + return beginNestedJson(path, SlotCursor.UNKNOWN); + } + + @Override + public ValueWriter beginJson(String path, int paths) throws IOException { + if (paths < 0) { + owner.fail(); + throw new IllegalArgumentException("JSON path count is negative"); + } + return beginNestedJson(path, paths); + } + + @Override + public ValueWriter beginQBit() throws IOException { + ensureWritable(); + ClickHouseColumn effective = StreamTypes.unwrap(slot()); + if (effective.getDataType() != ClickHouseDataType.QBit) { + owner.fail(); + throw new IllegalStateException("beginQBit is not valid for " + StreamTypes.describe(effective)); + } + return beginArray(effective.getPrecision()); + } + + @Override + public void writeInt8Array(byte[] values) throws IOException { + writePrimitiveArray(values, ClickHouseDataType.Int8, (index) -> BinaryStreamUtils.writeInt8(out, values[index])); + } + + @Override + public void writeInt16Array(short[] values) throws IOException { + writePrimitiveArray(values, values == null ? 0 : values.length, ClickHouseDataType.Int16, + (index) -> BinaryStreamUtils.writeInt16(out, values[index])); + } + + @Override + public void writeInt32Array(int[] values) throws IOException { + writePrimitiveArray(values, values == null ? 0 : values.length, ClickHouseDataType.Int32, + (index) -> BinaryStreamUtils.writeInt32(out, values[index])); + } + + @Override + public void writeInt64Array(long[] values) throws IOException { + writePrimitiveArray(values, values == null ? 0 : values.length, ClickHouseDataType.Int64, + (index) -> BinaryStreamUtils.writeInt64(out, values[index])); + } + + @Override + public void writeFloat32Array(float[] values) throws IOException { + writePrimitiveArray(values, values == null ? 0 : values.length, ClickHouseDataType.Float32, + (index) -> BinaryStreamUtils.writeFloat32(out, values[index])); + } + + @Override + public void writeFloat64Array(double[] values) throws IOException { + writePrimitiveArray(values, values == null ? 0 : values.length, ClickHouseDataType.Float64, + (index) -> BinaryStreamUtils.writeFloat64(out, values[index])); + } + + @Override + public void writeBFloat16Array(float[] values) throws IOException { + writePrimitiveArray(values, values == null ? 0 : values.length, ClickHouseDataType.BFloat16, + (index) -> BinaryStreamUtils.writeBFloat16(out, values[index])); + } + + @Override + public void writeBoolArray(boolean[] values) throws IOException { + writePrimitiveArray(values, values == null ? 0 : values.length, ClickHouseDataType.Bool, + (index) -> BinaryStreamUtils.writeBoolean(out, values[index])); + } + + @Override + public void close() throws IOException { + if (closed) { + return; + } + if (owner.isFailed()) { + closed = true; + return; + } + if (child != null) { + owner.fail(); + closed = true; + throw new IllegalStateException("Unclosed nested value"); + } + if (flattensIntoParent) { + try { + cursor.checkComplete(); + } catch (RuntimeException e) { + owner.fail(); + closed = true; + throw e; + } + closed = true; + parent.child = null; + parent.cursor.credit(cursor.completed()); + return; + } + try { + cursor.checkComplete(); + if (buffer != null) { + SerializerUtils.writeVarInt(parent.out, cursor.completed()); + buffer.writeTo(parent.out); + } + } catch (RuntimeException e) { + owner.fail(); + closed = true; + throw e; + } catch (IOException e) { + owner.fail(); + closed = true; + throw e; + } + closed = true; + if (parent != null) { + parent.child = null; + parent.cursor.advance(); + } else { + owner.onRowClosed(); + } + } + + private void writeEnum(String name, ClickHouseDataType type) throws IOException { + if (name == null) { + writeNullFor(type); + return; + } + ensureWritable(); + ClickHouseColumn slot = slot(); + ClickHouseColumn effective = StreamTypes.unwrap(slot); + if (effective.getDataType() != type) { + owner.fail(); + throw new IllegalStateException("Cannot write " + type + " into " + StreamTypes.describe(effective)); + } + int wire; + try { + wire = effective.getEnumConstants().value(name); + } catch (IllegalArgumentException e) { + owner.fail(); + throw e; + } + writePresent(slot); + if (type == ClickHouseDataType.Enum8) { + body(() -> BinaryStreamUtils.writeInt8(out, wire)); + } else { + body(() -> BinaryStreamUtils.writeInt16(out, wire)); + } + } + + private void writeNullFor(ClickHouseDataType... allowed) throws IOException { + ensureWritable(); + ClickHouseColumn effective = StreamTypes.unwrap(slot()); + if (!StreamTypes.isOneOf(effective.getDataType(), allowed)) { + owner.fail(); + throw new IllegalStateException("Cannot write " + allowed[0] + " into " + StreamTypes.describe(effective)); + } + writeNull(); + } + + private ValueWriter openJsonValue(ClickHouseColumn type, String prefix, int paths) throws IOException { + OutputStream destination = out; + ByteArrayOutputStream childBuffer = null; + if (paths == SlotCursor.UNKNOWN) { + childBuffer = new ByteArrayOutputStream(); + destination = childBuffer; + } else { + try { + SerializerUtils.writeVarInt(out, paths); + } catch (IOException e) { + owner.fail(); + throw e; + } + } + StreamWriterScope nested = new StreamWriterScope(owner, this, SlotCursor.json(paths), destination, childBuffer); + nested.jsonType = type; + nested.pathPrefix = prefix; + return nest(nested); + } + + private void writeJsonScalar(String path, ClickHouseDataType type, ClickHouseColumn dynamicType, IOAction action) + throws IOException { + ensureJson(); + String full = fullPath(path); + ClickHouseColumn typed = typedPath(full); + writePathName(full); + try { + if (typed == null) { + SerializerUtils.writeDynamicTypeTag(out, dynamicType); + action.run(); + } else { + ClickHouseColumn effective = StreamTypes.unwrap(typed); + if (effective.getDataType() != type) { + throw new IllegalStateException("Cannot write " + type + " into JSON path '" + full + + "' of type " + typed.getOriginalTypeName()); + } + if (typed.isNullable()) { + SerializerUtils.writeNonNull(out); + } + action.run(); + } + } catch (IOException e) { + owner.fail(); + throw e; + } catch (RuntimeException e) { + owner.fail(); + throw e; + } + cursor.advance(); + } + + private ValueWriter beginJsonArray(String path, ClickHouseColumn elementType, int size) throws IOException { + ensureJson(); + if (elementType == null) { + owner.fail(); + throw new IllegalArgumentException("JSON array element type is required"); + } + String full = fullPath(path); + ClickHouseColumn typed = typedPath(full); + writePathName(full); + ClickHouseColumn element = elementType; + try { + if (typed == null) { + ClickHouseColumn arrayType = ClickHouseColumn.of("v", "Array(" + elementType.getOriginalTypeName() + ")"); + SerializerUtils.writeDynamicTypeTag(out, arrayType); + element = StreamTypes.arrayElement(arrayType); + } else { + ClickHouseColumn effective = StreamTypes.unwrap(typed); + if (effective.getDataType() != ClickHouseDataType.Array) { + throw new IllegalStateException("Cannot write an array into JSON path '" + full + + "' of type " + typed.getOriginalTypeName()); + } + element = StreamTypes.arrayElement(effective); + if (typed.isNullable()) { + SerializerUtils.writeNonNull(out); + } + } + } catch (IOException e) { + owner.fail(); + throw e; + } catch (RuntimeException e) { + owner.fail(); + throw e; + } + return openLengthPrefixed(SlotCursor.array(element, size), size); + } + + private ValueWriter beginNestedJson(String path, int paths) throws IOException { + ensureJson(); + String full = fullPath(path); + ClickHouseColumn typed = typedPath(full); + writePathName(full); + ClickHouseColumn nestedType = DYN_JSON; + try { + if (typed == null) { + SerializerUtils.writeDynamicTypeTag(out, DYN_JSON); + } else { + nestedType = StreamTypes.unwrap(typed); + if (nestedType.getDataType() != ClickHouseDataType.JSON) { + throw new IllegalStateException("Cannot write JSON into path '" + full + + "' of type " + typed.getOriginalTypeName()); + } + if (typed.isNullable()) { + SerializerUtils.writeNonNull(out); + } + } + } catch (IOException e) { + owner.fail(); + throw e; + } catch (RuntimeException e) { + owner.fail(); + throw e; + } + return openJsonValue(nestedType, "", paths); + } + + private void ensureJson() { + ensureWritable(); + if (cursor.kind != SlotCursor.Kind.JSON) { + owner.fail(); + throw new IllegalStateException("A JSON path is written inside beginJson or beginObject"); + } + } + + private String fullPath(String path) { + if (path == null || path.isEmpty()) { + owner.fail(); + throw new IllegalArgumentException("JSON path is empty"); + } + return pathPrefix + path; + } + + private ClickHouseColumn typedPath(String full) { + if (jsonType == null || jsonType.getJsonPredefinedPaths() == null) { + return null; + } + return jsonType.getJsonPredefinedPaths().get(full); + } + + private void writePathName(String full) throws IOException { + try { + BinaryStreamUtils.writeString(out, full); + } catch (IOException e) { + owner.fail(); + throw e; + } + } + + private ValueWriter openArray(int size) throws IOException { + ensureWritable(); + ClickHouseColumn slot = slot(); + ClickHouseColumn effective = StreamTypes.unwrap(slot); + ClickHouseColumn element = StreamTypes.arrayElement(effective); + if (element == null) { + owner.fail(); + throw new IllegalStateException("beginArray is not valid for " + StreamTypes.describe(effective)); + } + if (effective.getDataType() == ClickHouseDataType.QBit && size != effective.getPrecision()) { + owner.fail(); + throw new IllegalArgumentException("QBit column '" + effective.getColumnName() + + "' expects exactly " + effective.getPrecision() + " elements but got " + + (size == SlotCursor.UNKNOWN ? "an unknown size" : Integer.toString(size))); + } + writePresent(slot); + return openLengthPrefixed(SlotCursor.array(element, size), size); + } + + private ValueWriter openNested(int rows) throws IOException { + ensureWritable(); + ClickHouseColumn slot = slot(); + ClickHouseColumn effective = StreamTypes.unwrap(slot); + if (effective.getDataType() != ClickHouseDataType.Nested) { + owner.fail(); + throw new IllegalStateException("beginNested is not valid for " + StreamTypes.describe(effective)); + } + ClickHouseColumn tuple = ClickHouseColumn.tuple(effective.getColumnName(), false, + effective.getNestedColumns().toArray(new ClickHouseColumn[0])); + writePresent(slot); + return openLengthPrefixed(SlotCursor.array(tuple, rows), rows); + } + + private ValueWriter openMap(int pairs) throws IOException { + ensureWritable(); + ClickHouseColumn slot = slot(); + ClickHouseColumn effective = StreamTypes.unwrap(slot); + if (effective.getDataType() != ClickHouseDataType.Map) { + owner.fail(); + throw new IllegalStateException("beginMap is not valid for " + StreamTypes.describe(effective)); + } + writePresent(slot); + return openLengthPrefixed(SlotCursor.map(effective.getKeyInfo(), effective.getValueInfo(), pairs), pairs); + } + + private ValueWriter openLengthPrefixed(SlotCursor childCursor, int size) throws IOException { + OutputStream destination = out; + ByteArrayOutputStream childBuffer = null; + if (size == SlotCursor.UNKNOWN) { + childBuffer = new ByteArrayOutputStream(); + destination = childBuffer; + } else { + try { + SerializerUtils.writeVarInt(out, size); + } catch (IOException e) { + owner.fail(); + throw e; + } + } + return nest(new StreamWriterScope(owner, this, childCursor, destination, childBuffer)); + } + + private void writePrimitiveArray(byte[] values, ClickHouseDataType elementType, ElementWriter writer) throws IOException { + if (values == null) { + writeNullFor(ClickHouseDataType.Array, ClickHouseDataType.QBit); + return; + } + writePrimitiveArray(values, values.length, elementType, writer); + } + + private void writePrimitiveArray(Object values, int length, ClickHouseDataType elementType, ElementWriter writer) + throws IOException { + if (values == null) { + writeNullFor(ClickHouseDataType.Array, ClickHouseDataType.QBit); + return; + } + ensureWritable(); + ClickHouseColumn slot = slot(); + ClickHouseColumn effective = StreamTypes.unwrap(slot); + ClickHouseDataType dataType = effective.getDataType(); + if (dataType != ClickHouseDataType.Array && dataType != ClickHouseDataType.QBit) { + owner.fail(); + throw new IllegalStateException("Cannot write a primitive array into " + StreamTypes.describe(effective)); + } + if (dataType == ClickHouseDataType.QBit && length != effective.getPrecision()) { + owner.fail(); + throw new IllegalArgumentException("QBit column '" + effective.getColumnName() + + "' expects exactly " + effective.getPrecision() + " elements but got " + length); + } + ClickHouseColumn element = StreamTypes.unwrap(StreamTypes.arrayElement(effective)); + if (element.getDataType() != elementType) { + owner.fail(); + throw new IllegalStateException("Cannot write " + elementType + " array into " + StreamTypes.describe(effective)); + } + boolean nullable = element.isNullable(); + writePresent(slot); + try { + SerializerUtils.writeVarInt(out, length); + for (int i = 0; i < length; i++) { + if (nullable) { + SerializerUtils.writeNonNull(out); + } + writer.write(i); + } + } catch (IOException e) { + owner.fail(); + throw e; + } catch (RuntimeException e) { + owner.fail(); + throw e; + } + cursor.advance(); + } + + private ClickHouseColumn prepare(ClickHouseDataType... allowed) throws IOException { + ensureWritable(); + ClickHouseColumn slot = slot(); + ClickHouseColumn effective = StreamTypes.unwrap(slot); + if (!StreamTypes.isOneOf(effective.getDataType(), allowed)) { + owner.fail(); + throw new IllegalStateException("Cannot write " + allowed[0] + " into " + StreamTypes.describe(effective)); + } + writePresent(slot); + return effective; + } + + private void writePresent(ClickHouseColumn slot) throws IOException { + ClickHouseColumn effective = StreamTypes.unwrap(slot); + try { + if (cursor.kind == SlotCursor.Kind.ROW && owner.defaultsSupport()) { + SerializerUtils.writeNonNull(out); + } + if (effective.isNullable()) { + SerializerUtils.writeNonNull(out); + } + } catch (IOException e) { + owner.fail(); + throw e; + } + } + + private void body(IOAction action) throws IOException { + try { + action.run(); + } catch (IOException e) { + owner.fail(); + throw e; + } catch (RuntimeException e) { + owner.fail(); + throw e; + } + cursor.advance(); + } + + private ClickHouseColumn slot() { + try { + return cursor.current(); + } catch (RuntimeException e) { + owner.fail(); + throw e; + } + } + + private void ensureWritable() { + if (owner.isFailed()) { + throw new IllegalStateException("Writer is unusable after a previous failure"); + } + if (closed) { + throw new IllegalStateException("Scope is closed"); + } + if (child != null) { + owner.fail(); + throw new IllegalStateException("A nested value is still open"); + } + } + + private void requireValue(Object value) { + if (value == null) { + owner.fail(); + throw new IllegalArgumentException("Value is null"); + } + } + + private void checkRange(long value, long min, long max, String type) { + if (value < min || value > max) { + owner.fail(); + throw new IllegalArgumentException(type + " overflow: " + value); + } + } + + private ZoneId zone(ClickHouseColumn column) { + if (column.getTimeZone() != null) { + return column.getTimeZone().toZoneId(); + } + return owner.timeZone().toZoneId(); + } + + private StreamWriterScope nest(StreamWriterScope childScope) { + this.child = childScope; + return childScope; + } + + private interface IOAction { + void run() throws IOException; + } + + private interface ElementWriter { + void write(int index) throws IOException; + } +} diff --git a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/stream/ValueReader.java b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/stream/ValueReader.java new file mode 100644 index 000000000..d2b2e13f9 --- /dev/null +++ b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/stream/ValueReader.java @@ -0,0 +1,126 @@ +package com.clickhouse.client.api.data_formats.stream; + +import com.clickhouse.client.api.data_formats.internal.StringValue; +import com.clickhouse.data.ClickHouseColumn; +import com.clickhouse.data.value.ClickHouseBitmap; + +import java.io.IOException; +import java.math.BigDecimal; +import java.math.BigInteger; +import java.net.Inet4Address; +import java.net.Inet6Address; +import java.time.LocalDate; +import java.time.LocalDateTime; +import java.time.ZonedDateTime; +import java.util.UUID; + +/** + * One scope of a streaming RowBinary read. The verbs match {@link ValueWriter}. + *

+ * Experimental API. Not thread-safe. The caller owns the input stream. + */ +public interface ValueReader extends AutoCloseable { + + byte readInt8() throws IOException; + + short readInt16() throws IOException; + + int readInt32() throws IOException; + + long readInt64() throws IOException; + + BigInteger readInt128() throws IOException; + + BigInteger readInt256() throws IOException; + + int readUInt8() throws IOException; + + int readUInt16() throws IOException; + + long readUInt32() throws IOException; + + BigInteger readUInt64() throws IOException; + + BigInteger readUInt128() throws IOException; + + BigInteger readUInt256() throws IOException; + + float readFloat32() throws IOException; + + double readFloat64() throws IOException; + + float readBFloat16() throws IOException; + + boolean readBool() throws IOException; + + String readString() throws IOException; + + StringValue readStringValue() throws IOException; + + String readFixedString() throws IOException; + + LocalDate readDate() throws IOException; + + LocalDate readDate32() throws IOException; + + ZonedDateTime readDateTime() throws IOException; + + ZonedDateTime readDateTime64() throws IOException; + + UUID readUUID() throws IOException; + + Inet4Address readIPv4() throws IOException; + + Inet6Address readIPv6() throws IOException; + + BigDecimal readDecimal() throws IOException; + + int readEnum8() throws IOException; + + int readEnum16() throws IOException; + + ClickHouseBitmap readBitmap() throws IOException; + + /** + * Consumes a SQL NULL and returns true, or returns false and leaves the body to be read. + * On a non-nullable {@code Variant} or {@code Geometry}, discriminator {@code 0xFF} is null. + */ + boolean isNull() throws IOException; + + boolean hasNext(); + + int size(); + + int variantOrdinal(); + + ClickHouseColumn runtimeType(); + + ValueReader beginArray() throws IOException; + + ValueReader beginTuple() throws IOException; + + ValueReader beginMap() throws IOException; + + ValueReader beginNested() throws IOException; + + ValueReader beginVariant() throws IOException; + + ValueReader beginDynamic() throws IOException; + + ValueReader beginQBit() throws IOException; + + /** + * Fills {@code dst} with up to {@code len} {@code Int32} elements of the current array. + * + * @return number of elements written into {@code dst} + */ + int readInto(int[] dst, int off, int len) throws IOException; + + /** + * Skips the current value without building a list, map, or array. + */ + void skip() throws IOException; + + @Override + void close() throws IOException; +} diff --git a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/stream/ValueWriter.java b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/stream/ValueWriter.java new file mode 100644 index 000000000..228669a59 --- /dev/null +++ b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/stream/ValueWriter.java @@ -0,0 +1,202 @@ +package com.clickhouse.client.api.data_formats.stream; + +import com.clickhouse.data.ClickHouseColumn; +import com.clickhouse.data.value.ClickHouseBitmap; + +import java.io.IOException; +import java.math.BigDecimal; +import java.math.BigInteger; +import java.net.Inet4Address; +import java.net.Inet6Address; +import java.nio.ByteBuffer; +import java.time.Instant; +import java.time.LocalDate; +import java.time.LocalDateTime; +import java.time.ZonedDateTime; +import java.util.UUID; + +/** + * One scope of a streaming RowBinary write: a row, an array, a tuple, a map, a nested row, + * a variant arm, or a dynamic value. + *

+ * Experimental API. Not thread-safe. The caller owns the output stream. + */ +public interface ValueWriter extends AutoCloseable { + + void writeInt8(byte value) throws IOException; + + void writeInt8(int value) throws IOException; + + void writeInt16(short value) throws IOException; + + void writeInt16(int value) throws IOException; + + void writeInt32(int value) throws IOException; + + void writeInt32(long value) throws IOException; + + void writeInt64(long value) throws IOException; + + void writeInt128(BigInteger value) throws IOException; + + void writeInt256(BigInteger value) throws IOException; + + void writeUInt8(int value) throws IOException; + + void writeUInt16(int value) throws IOException; + + void writeUInt32(long value) throws IOException; + + void writeUInt64(long value) throws IOException; + + void writeUInt64(BigInteger value) throws IOException; + + void writeUInt128(BigInteger value) throws IOException; + + void writeUInt256(BigInteger value) throws IOException; + + void writeFloat32(float value) throws IOException; + + void writeFloat64(double value) throws IOException; + + void writeBFloat16(float value) throws IOException; + + void writeBool(boolean value) throws IOException; + + void writeString(String value) throws IOException; + + void writeString(byte[] value) throws IOException; + + void writeString(ByteBuffer value) throws IOException; + + void writeFixedString(String value) throws IOException; + + void writeFixedString(byte[] value) throws IOException; + + void writeDate(LocalDate value) throws IOException; + + void writeDate32(LocalDate value) throws IOException; + + void writeDateTime(LocalDateTime value) throws IOException; + + void writeDateTime(ZonedDateTime value) throws IOException; + + void writeDateTime(Instant value) throws IOException; + + void writeDateTime64(LocalDateTime value) throws IOException; + + void writeDateTime64(ZonedDateTime value) throws IOException; + + void writeUUID(UUID value) throws IOException; + + void writeIPv4(Inet4Address value) throws IOException; + + void writeIPv6(Inet6Address value) throws IOException; + + void writeDecimal(BigDecimal value) throws IOException; + + void writeEnum8(int value) throws IOException; + + void writeEnum8(String name) throws IOException; + + void writeEnum16(int value) throws IOException; + + void writeEnum16(String name) throws IOException; + + void writeNull() throws IOException; + + void writeDefault() throws IOException; + + void writeBitmap(ClickHouseBitmap value) throws IOException; + + ValueWriter beginArray() throws IOException; + + ValueWriter beginArray(int size) throws IOException; + + ValueWriter beginTuple() throws IOException; + + ValueWriter beginMap() throws IOException; + + ValueWriter beginMap(int pairs) throws IOException; + + ValueWriter beginNested() throws IOException; + + ValueWriter beginNested(int rows) throws IOException; + + ValueWriter beginVariant(int ordinal) throws IOException; + + ValueWriter beginDynamic(ClickHouseColumn runtimeType) throws IOException; + + ValueWriter beginQBit() throws IOException; + + /** + * Opens a binary {@code JSON} object. The number of leaf paths is written when the scope closes. + * A nested object opened with {@link #beginObject(String)} contributes its leaf paths to this count. + */ + ValueWriter beginJson() throws IOException; + + /** + * Opens a binary {@code JSON} object with a known number of leaf paths. The count is written immediately. + */ + ValueWriter beginJson(int paths) throws IOException; + + /** Dynamic {@code String} path. A null value is a JSON null. */ + void writeString(String path, String value) throws IOException; + + /** Dynamic {@code Int32} path. */ + void writeInt32(String path, int value) throws IOException; + + /** Dynamic {@code Int64} path. */ + void writeInt64(String path, long value) throws IOException; + + /** Dynamic {@code Float64} path. */ + void writeFloat64(String path, double value) throws IOException; + + /** Dynamic {@code Bool} path. */ + void writeBool(String path, boolean value) throws IOException; + + /** JSON null at {@code path}: the path is present and its dynamic type is {@code Nothing}. */ + void writeNull(String path) throws IOException; + + /** + * Nested object stored as dotted paths ({@code address.city}) on the enclosing JSON value. + * The leaf count is not known yet. + */ + ValueWriter beginObject(String path) throws IOException; + + /** + * Nested object stored as dotted paths. {@code paths} is the number of leaf paths it contributes. + */ + ValueWriter beginObject(String path, int paths) throws IOException; + + /** Dynamic {@code Array} path. The element type selects the array type tag. */ + ValueWriter beginArray(String path, ClickHouseColumn elementType) throws IOException; + + /** Dynamic {@code Array} path with a known element count. */ + ValueWriter beginArray(String path, ClickHouseColumn elementType, int size) throws IOException; + + /** Dynamic path whose value is itself a JSON document, for example one element of {@code Array(JSON)}. */ + ValueWriter beginJson(String path) throws IOException; + + /** Dynamic JSON path with a known number of leaf paths inside that document. */ + ValueWriter beginJson(String path, int paths) throws IOException; + + void writeInt8Array(byte[] values) throws IOException; + + void writeInt16Array(short[] values) throws IOException; + + void writeInt32Array(int[] values) throws IOException; + + void writeInt64Array(long[] values) throws IOException; + + void writeFloat32Array(float[] values) throws IOException; + + void writeFloat64Array(double[] values) throws IOException; + + void writeBFloat16Array(float[] values) throws IOException; + + void writeBoolArray(boolean[] values) throws IOException; + + @Override + void close() throws IOException; +} diff --git a/client-v2/src/test/java/com/clickhouse/client/api/data_formats/stream/RowBinaryStreamRoundTripTest.java b/client-v2/src/test/java/com/clickhouse/client/api/data_formats/stream/RowBinaryStreamRoundTripTest.java new file mode 100644 index 000000000..863450098 --- /dev/null +++ b/client-v2/src/test/java/com/clickhouse/client/api/data_formats/stream/RowBinaryStreamRoundTripTest.java @@ -0,0 +1,672 @@ +package com.clickhouse.client.api.data_formats.stream; + +import com.clickhouse.client.api.ClientException; +import com.clickhouse.client.api.data_formats.RowBinaryFormatWriter; +import com.clickhouse.client.api.data_formats.internal.BinaryStreamReader; +import com.clickhouse.client.api.data_formats.internal.StringValue; +import com.clickhouse.client.api.metadata.TableSchema; +import com.clickhouse.client.api.query.NullValueException; +import com.clickhouse.data.ClickHouseColumn; +import com.clickhouse.data.ClickHouseDataType; +import com.clickhouse.data.ClickHouseFormat; +import com.clickhouse.data.format.BinaryStreamUtils; +import com.clickhouse.data.value.ClickHouseBitmap; +import org.testng.Assert; +import org.testng.annotations.Test; + +import java.io.ByteArrayInputStream; +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.TimeZone; + +public class RowBinaryStreamRoundTripTest { + + @Test + public void scalarsRoundTripInTheMiddleOfTheRow() throws IOException { + TableSchema schema = schema("id Int32", "name String", "tail Float64"); + byte[] streamed = streamed(schema, ClickHouseFormat.RowBinary, row -> { + row.writeInt32(7); + row.writeString("abc"); + row.writeFloat64(1.5d); + }); + byte[] buffered = buffered(schema, ClickHouseFormat.RowBinary, writer -> { + writer.setValue("id", 7); + writer.setValue("name", "abc"); + writer.setValue("tail", 1.5d); + }); + Assert.assertEquals(streamed, buffered); + + BinaryStreamReader reader = binary(streamed); + Assert.assertEquals(reader.readValue(column(schema, 0)), Integer.valueOf(7)); + Assert.assertEquals(reader.readValue(column(schema, 1)), "abc"); + Assert.assertEquals(reader.readValue(column(schema, 2)), 1.5d); + + RowBinaryStreamReader streamReader = new RowBinaryStreamReader( + new ByteArrayInputStream(streamed), schema, ClickHouseFormat.RowBinary); + Assert.assertTrue(streamReader.hasNext()); + try (ValueReader row = streamReader.beginRow()) { + Assert.assertEquals(row.readInt32(), 7); + Assert.assertEquals(row.readString(), "abc"); + Assert.assertEquals(row.readFloat64(), 1.5d); + } + Assert.assertFalse(streamReader.hasNext()); + } + + @Test + public void nullableStringAndNull() throws IOException { + TableSchema schema = schema("id Int32", "name Nullable(String)", "tail Float64"); + ByteArrayOutputStream out = new ByteArrayOutputStream(); + RowBinaryStreamWriter writer = new RowBinaryStreamWriter(out, schema, ClickHouseFormat.RowBinary); + try (ValueWriter row = writer.beginRow()) { + row.writeInt32(1); + row.writeString((String) null); + row.writeFloat64(2.5d); + } + try (ValueWriter row = writer.beginRow()) { + row.writeInt32(2); + row.writeString("ok"); + row.writeFloat64(3.5d); + } + byte[] streamed = out.toByteArray(); + BinaryStreamReader reader = binary(streamed); + Assert.assertEquals(reader.readValue(column(schema, 0)), Integer.valueOf(1)); + Assert.assertNull(reader.readValue(column(schema, 1))); + Assert.assertEquals(reader.readValue(column(schema, 2)), 2.5d); + Assert.assertEquals(reader.readValue(column(schema, 0)), Integer.valueOf(2)); + Assert.assertEquals(reader.readValue(column(schema, 1)), "ok"); + Assert.assertEquals(reader.readValue(column(schema, 2)), 3.5d); + + RowBinaryStreamReader streamReader = new RowBinaryStreamReader( + new ByteArrayInputStream(streamed), schema, ClickHouseFormat.RowBinary); + try (ValueReader row = streamReader.beginRow()) { + Assert.assertEquals(row.readInt32(), 1); + Assert.assertTrue(row.isNull()); + Assert.assertEquals(row.readFloat64(), 2.5d); + } + try (ValueReader row = streamReader.beginRow()) { + Assert.assertEquals(row.readInt32(), 2); + Assert.assertFalse(row.isNull()); + Assert.assertEquals(row.readString(), "ok"); + Assert.assertEquals(row.readFloat64(), 3.5d); + } + } + + @Test + public void arraysKnownAndUnknownSizeMatchBufferedWriter() throws IOException { + TableSchema schema = schema("id Int32", "items Array(Nullable(String))", "tail Float64"); + List items = Arrays.asList("a", null, "b"); + byte[] unknown = streamed(schema, ClickHouseFormat.RowBinary, row -> { + row.writeInt32(1); + try (ValueWriter array = row.beginArray()) { + array.writeString("a"); + array.writeString((String) null); + array.writeString("b"); + } + row.writeFloat64(1.5d); + }); + byte[] known = streamed(schema, ClickHouseFormat.RowBinary, row -> { + row.writeInt32(1); + try (ValueWriter array = row.beginArray(items.size())) { + for (String item : items) { + array.writeString(item); + } + } + row.writeFloat64(1.5d); + }); + byte[] fromBuffer = buffered(schema, ClickHouseFormat.RowBinary, writer -> { + writer.setValue("id", 1); + writer.setValue("items", items); + writer.setValue("tail", 1.5d); + }); + Assert.assertEquals(unknown, known); + Assert.assertEquals(known, fromBuffer); + + BinaryStreamReader reader = binary(known); + Assert.assertEquals(reader.readValue(column(schema, 0)), Integer.valueOf(1)); + Assert.assertEquals(asList(reader.readValue(column(schema, 1))), items); + Assert.assertEquals(reader.readValue(column(schema, 2)), 1.5d); + } + + @Test + public void nestedArraysMatchBufferedWriter() throws IOException { + TableSchema schema = schema("id Int32", "matrix Array(Array(Nullable(String)))", "tail Float64"); + List> matrix = Arrays.asList( + Arrays.asList("a", null), + Collections.emptyList()); + byte[] unknown = streamed(schema, ClickHouseFormat.RowBinary, row -> { + row.writeInt32(4); + try (ValueWriter outer = row.beginArray()) { + try (ValueWriter inner = outer.beginArray()) { + inner.writeString("a"); + inner.writeString((String) null); + } + try (ValueWriter inner = outer.beginArray(0)) { + } + } + row.writeFloat64(1.5d); + }); + byte[] known = streamed(schema, ClickHouseFormat.RowBinary, row -> { + row.writeInt32(4); + try (ValueWriter outer = row.beginArray(2)) { + try (ValueWriter inner = outer.beginArray(2)) { + inner.writeString("a"); + inner.writeString((String) null); + } + try (ValueWriter inner = outer.beginArray()) { + } + } + row.writeFloat64(1.5d); + }); + byte[] fromBuffer = buffered(schema, ClickHouseFormat.RowBinary, writer -> { + writer.setValue("id", 4); + writer.setValue("matrix", matrix); + writer.setValue("tail", 1.5d); + }); + Assert.assertEquals(unknown, known); + Assert.assertEquals(known, fromBuffer); + BinaryStreamReader reader = binary(known); + Assert.assertEquals(reader.readValue(column(schema, 0)), Integer.valueOf(4)); + Assert.assertEquals(asList(reader.readValue(column(schema, 1))), matrix); + Assert.assertEquals(reader.readValue(column(schema, 2)), 1.5d); + } + + @Test + public void tupleWithArrayInTheMiddle() throws IOException { + TableSchema schema = schema("id Int32", "t Tuple(Int32, Array(Nullable(String)), Float64)", "tail Float64"); + byte[] streamed = streamed(schema, ClickHouseFormat.RowBinary, row -> { + row.writeInt32(1); + try (ValueWriter tuple = row.beginTuple()) { + tuple.writeInt32(9); + try (ValueWriter array = tuple.beginArray(2)) { + array.writeString("x"); + array.writeString((String) null); + } + tuple.writeFloat64(0.5d); + } + row.writeFloat64(1.5d); + }); + byte[] fromBuffer = buffered(schema, ClickHouseFormat.RowBinary, writer -> { + writer.setValue("id", 1); + writer.setValue("t", Arrays.asList(9, Arrays.asList("x", null), 0.5d)); + writer.setValue("tail", 1.5d); + }); + Assert.assertEquals(streamed, fromBuffer); + BinaryStreamReader reader = binary(streamed); + Assert.assertEquals(reader.readValue(column(schema, 0)), Integer.valueOf(1)); + Object[] tuple = (Object[]) reader.readValue(column(schema, 1)); + Assert.assertEquals(tuple[0], 9); + Assert.assertEquals(asList(tuple[1]), Arrays.asList("x", null)); + Assert.assertEquals(tuple[2], 0.5d); + Assert.assertEquals(reader.readValue(column(schema, 2)), 1.5d); + } + + @Test + public void mapWithNestedArray() throws IOException { + TableSchema schema = schema("id Int32", "m Map(String, Array(Int32))", "tail Float64"); + Map> map = new LinkedHashMap>(); + map.put("a", Arrays.asList(1, 2)); + map.put("b", Collections.emptyList()); + byte[] streamed = streamed(schema, ClickHouseFormat.RowBinary, row -> { + row.writeInt32(1); + try (ValueWriter values = row.beginMap(2)) { + values.writeString("a"); + try (ValueWriter array = values.beginArray(2)) { + array.writeInt32(1); + array.writeInt32(2); + } + values.writeString("b"); + try (ValueWriter array = values.beginArray()) { + } + } + row.writeFloat64(1.5d); + }); + byte[] fromBuffer = buffered(schema, ClickHouseFormat.RowBinary, writer -> { + writer.setValue("id", 1); + writer.setValue("m", map); + writer.setValue("tail", 1.5d); + }); + Assert.assertEquals(streamed, fromBuffer); + } + + @Test + public void unflattenedNested() throws IOException { + TableSchema schema = schema("id Int32", "n Nested(n1 Int32, n2 Nullable(String))", "tail Float64"); + byte[] streamed = streamed(schema, ClickHouseFormat.RowBinary, row -> { + row.writeInt32(1); + try (ValueWriter nested = row.beginNested(2)) { + try (ValueWriter tuple = nested.beginTuple()) { + tuple.writeInt32(5); + tuple.writeString("a"); + } + try (ValueWriter tuple = nested.beginTuple()) { + tuple.writeInt32(6); + tuple.writeString((String) null); + } + } + row.writeFloat64(1.5d); + }); + byte[] fromBuffer = buffered(schema, ClickHouseFormat.RowBinary, writer -> { + writer.setValue("id", 1); + writer.setValue("n", Arrays.asList(Arrays.asList(5, "a"), Arrays.asList(6, null))); + writer.setValue("tail", 1.5d); + }); + Assert.assertEquals(streamed, fromBuffer); + BinaryStreamReader reader = binary(streamed); + Assert.assertEquals(reader.readValue(column(schema, 0)), Integer.valueOf(1)); + List rows = asList(reader.readValue(column(schema, 1))); + Assert.assertEquals(((Object[]) rows.get(0))[0], 5); + Assert.assertEquals(((Object[]) rows.get(0))[1], "a"); + Assert.assertEquals(((Object[]) rows.get(1))[0], 6); + Assert.assertNull(((Object[]) rows.get(1))[1]); + Assert.assertEquals(reader.readValue(column(schema, 2)), 1.5d); + } + + @Test + public void qbitAndBulkArrayUseTheKnownSizePath() throws IOException { + TableSchema qbit = schema("id Int32", "v QBit(Float64, 4)", "tail Float64"); + double[] vector = new double[] {1.0d, 2.0d, 3.0d, 4.0d}; + byte[] bulk = streamed(qbit, ClickHouseFormat.RowBinary, row -> { + row.writeInt32(1); + row.writeFloat64Array(vector); + row.writeFloat64(1.5d); + }); + byte[] cursor = streamed(qbit, ClickHouseFormat.RowBinary, row -> { + row.writeInt32(1); + try (ValueWriter values = row.beginQBit()) { + for (double value : vector) { + values.writeFloat64(value); + } + } + row.writeFloat64(1.5d); + }); + byte[] fromBuffer = buffered(qbit, ClickHouseFormat.RowBinary, writer -> { + writer.setValue("id", 1); + writer.setValue("v", vector); + writer.setValue("tail", 1.5d); + }); + Assert.assertEquals(bulk, cursor); + Assert.assertEquals(cursor, fromBuffer); + + TableSchema ints = schema("id Int32", "v Array(Int32)", "tail Float64"); + int[] numbers = new int[] {4, 5, 6}; + byte[] bulkInts = streamed(ints, ClickHouseFormat.RowBinary, row -> { + row.writeInt32(1); + row.writeInt32Array(numbers); + row.writeFloat64(1.5d); + }); + byte[] loopInts = streamed(ints, ClickHouseFormat.RowBinary, row -> { + row.writeInt32(1); + try (ValueWriter array = row.beginArray(numbers.length)) { + for (int number : numbers) { + array.writeInt32(number); + } + } + row.writeFloat64(1.5d); + }); + Assert.assertEquals(bulkInts, loopInts); + } + + @Test + public void defaultsPreamble() throws IOException { + ClickHouseColumn withDefault = ClickHouseColumn.of("d", "Nullable(Int32)"); + withDefault.setHasDefault(true); + withDefault.setDefaultValue(ClickHouseColumn.DefaultValue.DEFAULT); + ClickHouseColumn nullable = ClickHouseColumn.of("n", "Nullable(Int32)"); + ClickHouseColumn plain = ClickHouseColumn.of("v", "Int32"); + TableSchema schema = new TableSchema(Arrays.asList(withDefault, nullable, plain)); + + byte[] streamed = streamed(schema, ClickHouseFormat.RowBinaryWithDefaults, row -> { + row.writeDefault(); + row.writeNull(); + row.writeInt32(3); + }); + byte[] fromBuffer = buffered(schema, ClickHouseFormat.RowBinaryWithDefaults, writer -> { + writer.setValue("d", null); + writer.setValue("n", null); + writer.setValue("v", 3); + }); + Assert.assertEquals(streamed, fromBuffer); + Assert.assertEquals(streamed, new byte[] {1, 0, 1, 0, 3, 0, 0, 0}); + + byte[] value = streamed(schema, ClickHouseFormat.RowBinaryWithDefaults, row -> { + row.writeInt32(4); + row.writeInt32(5); + row.writeInt32(6); + }); + Assert.assertEquals(value[0], 0); + Assert.assertEquals(value[1], 0); + } + + @Test + public void rawStringBytesRoundTrip() throws IOException { + TableSchema schema = schema("id Int32", "raw String", "tail Float64"); + byte[] raw = new byte[] {(byte) 0xFF, 0x00, (byte) 0xFE}; + byte[] streamed = streamed(schema, ClickHouseFormat.RowBinary, row -> { + row.writeInt32(1); + row.writeString(raw); + row.writeFloat64(1.5d); + }); + BinaryStreamReader reader = new BinaryStreamReader(new ByteArrayInputStream(streamed), + TimeZone.getTimeZone("UTC"), null, new BinaryStreamReader.CachingByteBufferAllocator(), + true, null, true); + Assert.assertEquals(reader.readValue(column(schema, 0)), Integer.valueOf(1)); + Object value = reader.readValue(column(schema, 1)); + Assert.assertTrue(value instanceof StringValue); + Assert.assertEquals(((StringValue) value).toByteArray(), raw); + Assert.assertEquals(reader.readValue(column(schema, 2)), 1.5d); + + RowBinaryStreamReader streamReader = new RowBinaryStreamReader( + new ByteArrayInputStream(streamed), schema, ClickHouseFormat.RowBinary); + try (ValueReader row = streamReader.beginRow()) { + Assert.assertEquals(row.readInt32(), 1); + Assert.assertEquals(row.readStringValue().toByteArray(), raw); + Assert.assertEquals(row.readFloat64(), 1.5d); + } + } + + @Test + public void variantAndDynamic() throws IOException { + TableSchema variantSchema = schema("id Int32", "v Variant(String, Int32)", "tail Float64"); + ClickHouseColumn variant = column(variantSchema, 1); + int stringOrdinal = -1; + for (int i = 0; i < variant.getNestedColumns().size(); i++) { + if (variant.getNestedColumns().get(i).getDataType() == ClickHouseDataType.String) { + stringOrdinal = i; + } + } + Assert.assertTrue(stringOrdinal >= 0); + int ordinal = stringOrdinal; + byte[] streamed = streamed(variantSchema, ClickHouseFormat.RowBinary, row -> { + row.writeInt32(1); + try (ValueWriter value = row.beginVariant(ordinal)) { + value.writeString("hi"); + } + row.writeFloat64(1.5d); + }); + byte[] fromBuffer = buffered(variantSchema, ClickHouseFormat.RowBinary, writer -> { + writer.setValue("id", 1); + writer.setValue("v", "hi"); + writer.setValue("tail", 1.5d); + }); + Assert.assertEquals(streamed, fromBuffer); + + TableSchema dynamicSchema = schema("id Int32", "d Dynamic", "tail Float64"); + byte[] dynamicInt = streamed(dynamicSchema, ClickHouseFormat.RowBinary, row -> { + row.writeInt32(1); + try (ValueWriter value = row.beginDynamic(ClickHouseColumn.of("v", "Int32"))) { + value.writeInt32(7); + } + row.writeFloat64(1.5d); + }); + byte[] dynamicFromBuffer = buffered(dynamicSchema, ClickHouseFormat.RowBinary, writer -> { + writer.setValue("id", 1); + writer.setValue("d", 7); + writer.setValue("tail", 1.5d); + }); + Assert.assertEquals(dynamicInt, dynamicFromBuffer); + + byte[] dynamicArray = streamed(dynamicSchema, ClickHouseFormat.RowBinary, row -> { + row.writeInt32(1); + try (ValueWriter value = row.beginDynamic(ClickHouseColumn.of("v", "Array(Nullable(String))"))) { + try (ValueWriter array = value.beginArray(2)) { + array.writeString("a"); + array.writeString((String) null); + } + } + row.writeFloat64(1.5d); + }); + byte[] dynamicArrayFromBuffer = buffered(dynamicSchema, ClickHouseFormat.RowBinary, writer -> { + writer.setValue("id", 1); + writer.setValue("d", Arrays.asList("a", null)); + writer.setValue("tail", 1.5d); + }); + Assert.assertEquals(dynamicArray, dynamicArrayFromBuffer); + + RowBinaryStreamReader reader = new RowBinaryStreamReader( + new ByteArrayInputStream(dynamicInt), dynamicSchema, ClickHouseFormat.RowBinary); + try (ValueReader row = reader.beginRow()) { + Assert.assertEquals(row.readInt32(), 1); + try (ValueReader value = row.beginDynamic()) { + Assert.assertEquals(value.runtimeType().getDataType(), ClickHouseDataType.Int32); + Assert.assertEquals(value.readInt32(), 7); + } + Assert.assertEquals(row.readFloat64(), 1.5d); + } + } + + @Test + public void dynamicNullIsNothingAndIncompleteTypesAreRejected() throws IOException { + TableSchema schema = schema("d Dynamic"); + byte[] streamed = streamed(schema, ClickHouseFormat.RowBinary, ValueWriter::writeNull); + byte[] fromBuffer = buffered(schema, ClickHouseFormat.RowBinary, writer -> writer.setValue("d", null)); + Assert.assertEquals(streamed, fromBuffer); + + RowBinaryStreamReader reader = new RowBinaryStreamReader( + new ByteArrayInputStream(streamed), schema, ClickHouseFormat.RowBinary); + try (ValueReader row = reader.beginRow()) { + try (ValueReader value = row.beginDynamic()) { + Assert.assertEquals(value.runtimeType().getDataType(), ClickHouseDataType.Nothing); + } + } + + Assert.expectThrows(ClientException.class, () -> streamed(schema, ClickHouseFormat.RowBinary, row -> + row.beginDynamic(ClickHouseColumn.of("v", "Tuple(Int32, String)")))); + Assert.expectThrows(ClientException.class, () -> streamed(schema, ClickHouseFormat.RowBinary, row -> + row.beginDynamic(ClickHouseColumn.of("v", "Array(Tuple(Int32))")))); + Assert.expectThrows(ClientException.class, () -> streamed(schema, ClickHouseFormat.RowBinary, row -> + row.beginDynamic(ClickHouseColumn.of("v", "QBit(Float64, 4)")))); + } + + @Test + public void jsonIsAStringAndStructuredWriteIsRejected() throws IOException { + TableSchema schema = schema("id Int32", "j JSON", "tail Float64"); + byte[] streamed = streamed(schema, ClickHouseFormat.RowBinary, row -> { + row.writeInt32(1); + row.writeString("{\"a\":1}"); + row.writeFloat64(1.5d); + }); + byte[] fromBuffer = buffered(schema, ClickHouseFormat.RowBinary, writer -> { + writer.setValue("id", 1); + writer.setValue("j", "{\"a\":1}"); + writer.setValue("tail", 1.5d); + }); + Assert.assertEquals(streamed, fromBuffer); + Assert.expectThrows(IllegalStateException.class, () -> streamed(schema, ClickHouseFormat.RowBinary, row -> { + row.writeInt32(1); + row.beginArray(); + })); + } + + @Test + public void groupBitmapRoundTrip() throws IOException { + TableSchema schema = schema("id Int32", "b AggregateFunction(groupBitmap, UInt32)", "tail Float64"); + ClickHouseBitmap bitmap = ClickHouseBitmap.wrap(1, 2, 3); + byte[] streamed = streamed(schema, ClickHouseFormat.RowBinary, row -> { + row.writeInt32(1); + row.writeBitmap(bitmap); + row.writeFloat64(1.5d); + }); + byte[] fromBuffer = buffered(schema, ClickHouseFormat.RowBinary, writer -> { + writer.setValue("id", 1); + writer.setValue("b", bitmap); + writer.setValue("tail", 1.5d); + }); + Assert.assertEquals(streamed, fromBuffer); + BinaryStreamReader reader = binary(streamed); + Assert.assertEquals(reader.readValue(column(schema, 0)), Integer.valueOf(1)); + Assert.assertEquals(((ClickHouseBitmap) reader.readValue(column(schema, 1))).toBytes(), bitmap.toBytes()); + Assert.assertEquals(reader.readValue(column(schema, 2)), 1.5d); + } + + @Test + public void readerSkipsArrayWithoutLosingTheNextColumn() throws IOException { + TableSchema schema = schema("id Int32", "items Array(Int32)", "tail Float64"); + int[] numbers = new int[32]; + for (int i = 0; i < numbers.length; i++) { + numbers[i] = i; + } + byte[] streamed = streamed(schema, ClickHouseFormat.RowBinary, row -> { + row.writeInt32(1); + row.writeInt32Array(numbers); + row.writeFloat64(1.5d); + }); + RowBinaryStreamReader reader = new RowBinaryStreamReader( + new ByteArrayInputStream(streamed), schema, ClickHouseFormat.RowBinary); + try (ValueReader row = reader.beginRow()) { + Assert.assertEquals(row.readInt32(), 1); + row.skip(); + Assert.assertEquals(row.readFloat64(), 1.5d); + } + } + + @Test + public void readIntoFillsCallerBuffer() throws IOException { + TableSchema schema = schema("v Array(Int32)", "tail Float64"); + byte[] streamed = streamed(schema, ClickHouseFormat.RowBinary, row -> { + row.writeInt32Array(new int[] {8, 9, 10}); + row.writeFloat64(1.5d); + }); + RowBinaryStreamReader reader = new RowBinaryStreamReader( + new ByteArrayInputStream(streamed), schema, ClickHouseFormat.RowBinary); + try (ValueReader row = reader.beginRow()) { + try (ValueReader array = row.beginArray()) { + Assert.assertEquals(array.size(), 3); + int[] dst = new int[3]; + Assert.assertEquals(array.readInto(dst, 0, dst.length), 3); + Assert.assertEquals(dst, new int[] {8, 9, 10}); + Assert.assertFalse(array.hasNext()); + } + Assert.assertEquals(row.readFloat64(), 1.5d); + } + } + + @Test + public void namesAndTypesHeaderBuildsTheSchema() throws IOException { + TableSchema schema = schema("id Int32", "name String"); + ByteArrayOutputStream out = new ByteArrayOutputStream(); + BinaryStreamUtils.writeVarInt(out, 2); + BinaryStreamUtils.writeString(out, "id"); + BinaryStreamUtils.writeString(out, "name"); + BinaryStreamUtils.writeString(out, column(schema, 0).getOriginalTypeName()); + BinaryStreamUtils.writeString(out, column(schema, 1).getOriginalTypeName()); + try (ValueWriter row = new RowBinaryStreamWriter(out, schema, ClickHouseFormat.RowBinary).beginRow()) { + row.writeInt32(3); + row.writeString("z"); + } + RowBinaryStreamReader reader = new RowBinaryStreamReader( + new ByteArrayInputStream(out.toByteArray()), null, ClickHouseFormat.RowBinaryWithNamesAndTypes); + Assert.assertEquals(reader.getSchema().getColumns().size(), 2); + Assert.assertEquals(reader.getSchema().getColumns().get(0).getColumnName(), "id"); + Assert.assertEquals(reader.getSchema().getColumns().get(1).getDataType(), ClickHouseDataType.String); + try (ValueReader row = reader.beginRow()) { + Assert.assertEquals(row.readInt32(), 3); + Assert.assertEquals(row.readString(), "z"); + } + } + + @Test + public void negativePaths() throws IOException { + TableSchema arraySchema = schema("v Array(Int32)"); + Assert.expectThrows(IllegalStateException.class, () -> { + RowBinaryStreamWriter writer = new RowBinaryStreamWriter( + new ByteArrayOutputStream(), arraySchema, ClickHouseFormat.RowBinary); + ValueWriter row = writer.beginRow(); + ValueWriter array = row.beginArray(); + array.writeInt32(1); + row.close(); + }); + Assert.expectThrows(IllegalStateException.class, () -> streamed(arraySchema, ClickHouseFormat.RowBinary, row -> { + try (ValueWriter array = row.beginArray(2)) { + array.writeInt32(1); + } + })); + Assert.expectThrows(IllegalArgumentException.class, () -> + streamed(schema("v Int32"), ClickHouseFormat.RowBinary, row -> row.writeInt32(1L << 40))); + Assert.expectThrows(IllegalArgumentException.class, () -> + streamed(schema("v Int32"), ClickHouseFormat.RowBinary, row -> row.writeNull())); + Assert.expectThrows(IllegalStateException.class, () -> + streamed(schema("v Array(Int32)"), ClickHouseFormat.RowBinaryWithDefaults, row -> { + try (ValueWriter array = row.beginArray()) { + array.writeDefault(); + } + })); + Assert.expectThrows(IllegalArgumentException.class, () -> + streamed(schema("v Int32"), ClickHouseFormat.RowBinaryWithDefaults, ValueWriter::writeDefault)); + Assert.expectThrows(IllegalStateException.class, () -> streamed(schema("m Map(String, Int32)"), ClickHouseFormat.RowBinary, row -> { + try (ValueWriter map = row.beginMap(1)) { + map.writeString("a"); + } + })); + Assert.expectThrows(IllegalArgumentException.class, () -> + streamed(schema("v QBit(Float64, 4)"), ClickHouseFormat.RowBinary, + row -> row.writeFloat64Array(new double[] {1.0d}))); + Assert.expectThrows(NullValueException.class, () -> { + byte[] bytes = streamed(schema("v Nullable(Int32)"), ClickHouseFormat.RowBinary, ValueWriter::writeNull); + RowBinaryStreamReader reader = new RowBinaryStreamReader( + new ByteArrayInputStream(bytes), schema("v Nullable(Int32)"), ClickHouseFormat.RowBinary); + try (ValueReader row = reader.beginRow()) { + row.readInt32(); + } + }); + Assert.expectThrows(IllegalStateException.class, () -> + streamed(schema("a Int32", "b Int32"), ClickHouseFormat.RowBinary, row -> row.writeInt32(1))); + } + + private static TableSchema schema(String... definitions) { + List columns = new ArrayList(); + for (String definition : definitions) { + int space = definition.indexOf(' '); + columns.add(ClickHouseColumn.of(definition.substring(0, space), definition.substring(space + 1).trim())); + } + return new TableSchema(columns); + } + + private static ClickHouseColumn column(TableSchema schema, int index) { + return schema.getColumns().get(index); + } + + private static byte[] streamed(TableSchema schema, ClickHouseFormat format, StreamingRow row) throws IOException { + ByteArrayOutputStream out = new ByteArrayOutputStream(); + RowBinaryStreamWriter writer = new RowBinaryStreamWriter(out, schema, format); + try (ValueWriter value = writer.beginRow()) { + row.accept(value); + } + return out.toByteArray(); + } + + private static byte[] buffered(TableSchema schema, ClickHouseFormat format, BufferedRow row) throws IOException { + ByteArrayOutputStream out = new ByteArrayOutputStream(); + RowBinaryFormatWriter writer = new RowBinaryFormatWriter(out, schema, format); + row.accept(writer); + writer.commitRow(); + return out.toByteArray(); + } + + private static BinaryStreamReader binary(byte[] bytes) { + return new BinaryStreamReader(new ByteArrayInputStream(bytes), TimeZone.getTimeZone("UTC"), null, + new BinaryStreamReader.CachingByteBufferAllocator(), true, null, false); + } + + @SuppressWarnings("unchecked") + private static List asList(Object value) { + if (value instanceof BinaryStreamReader.ArrayValue) { + return ((BinaryStreamReader.ArrayValue) value).asList(); + } + return (List) value; + } + + private interface StreamingRow { + void accept(ValueWriter row) throws IOException; + } + + private interface BufferedRow { + void accept(RowBinaryFormatWriter writer) throws IOException; + } +} diff --git a/client-v2/src/test/java/com/clickhouse/client/api/data_formats/stream/RowBinaryStreamServerTest.java b/client-v2/src/test/java/com/clickhouse/client/api/data_formats/stream/RowBinaryStreamServerTest.java new file mode 100644 index 000000000..9be3605f3 --- /dev/null +++ b/client-v2/src/test/java/com/clickhouse/client/api/data_formats/stream/RowBinaryStreamServerTest.java @@ -0,0 +1,297 @@ +package com.clickhouse.client.api.data_formats.stream; + +import com.clickhouse.client.BaseIntegrationTest; +import com.clickhouse.client.ClickHouseNode; +import com.clickhouse.client.ClickHouseProtocol; +import com.clickhouse.client.ClickHouseServerForTest; +import com.clickhouse.client.api.Client; +import com.clickhouse.client.api.command.CommandSettings; +import com.clickhouse.client.api.data_formats.internal.BinaryStreamReader; +import com.clickhouse.client.api.enums.Protocol; +import com.clickhouse.client.api.insert.InsertSettings; +import com.clickhouse.client.api.internal.ServerSettings; +import com.clickhouse.client.api.metadata.TableSchema; +import com.clickhouse.client.api.query.GenericRecord; +import com.clickhouse.data.ClickHouseColumn; +import com.clickhouse.data.ClickHouseFormat; +import com.clickhouse.data.ClickHouseVersion; +import org.testng.Assert; +import org.testng.annotations.AfterMethod; +import org.testng.annotations.BeforeMethod; +import org.testng.annotations.Test; + +import java.io.IOException; +import java.util.Arrays; +import java.util.List; +import java.util.Map; +import java.util.concurrent.TimeUnit; + +/** + * Inserts rows with {@link RowBinaryStreamWriter} and reads them back from a server. + */ +@Test(groups = {"integration"}) +public class RowBinaryStreamServerTest extends BaseIntegrationTest { + + private static final int TIMEOUT_SECONDS = 30; + + private Client client; + + @BeforeMethod(groups = {"integration"}) + public void setUp() { + ClickHouseNode node = getServer(ClickHouseProtocol.HTTP); + client = new Client.Builder() + .addEndpoint(Protocol.HTTP, node.getHost(), node.getPort(), isCloud()) + .setUsername("default") + .setPassword(ClickHouseServerForTest.getPassword()) + .compressServerResponse(false) + .compressClientRequest(false) + .useHttpCompression(false) + .setDefaultDatabase(ClickHouseServerForTest.getDatabase()) + .serverSetting(ServerSettings.ASYNC_INSERT, "0") + .serverSetting(ServerSettings.WAIT_END_OF_QUERY, "1") + .build(); + } + + @AfterMethod(groups = {"integration"}) + public void tearDown() { + if (client != null) { + client.close(); + } + } + + @Test(groups = {"integration"}) + public void scalarsAndNullableString() throws Exception { + List rows = insert("stream_scalars", + "id Int32, name Nullable(String), tail Float64", writer -> { + try (ValueWriter row = writer.beginRow()) { + row.writeInt32(1); + row.writeString((String) null); + row.writeFloat64(2.5d); + } + try (ValueWriter row = writer.beginRow()) { + row.writeInt32(2); + row.writeString("ok"); + row.writeFloat64(3.5d); + } + }); + Assert.assertEquals(rows.get(0).getInteger("id"), 1); + Assert.assertNull(rows.get(0).getObject("name")); + Assert.assertEquals(rows.get(0).getDouble("tail"), 2.5d); + Assert.assertEquals(rows.get(1).getInteger("id"), 2); + Assert.assertEquals(rows.get(1).getString("name"), "ok"); + Assert.assertEquals(rows.get(1).getDouble("tail"), 3.5d); + } + + @Test(groups = {"integration"}) + public void arrayTupleMapAndNested() throws Exception { + List arrays = insert("stream_array", + "id Int32, items Array(Nullable(String)), tail Float64", writer -> { + try (ValueWriter row = writer.beginRow()) { + row.writeInt32(1); + try (ValueWriter items = row.beginArray(3)) { + items.writeString("a"); + items.writeString((String) null); + items.writeString("b"); + } + row.writeFloat64(1.5d); + } + }); + Assert.assertEquals(asList(arrays.get(0).getObject("items")), Arrays.asList("a", null, "b")); + Assert.assertEquals(arrays.get(0).getDouble("tail"), 1.5d); + + List tuples = insert("stream_tuple", + "id Int32, t Tuple(Int32, Array(Nullable(String)), Float64), tail Float64", writer -> { + try (ValueWriter row = writer.beginRow()) { + row.writeInt32(1); + try (ValueWriter tuple = row.beginTuple()) { + tuple.writeInt32(9); + try (ValueWriter values = tuple.beginArray(2)) { + values.writeString("x"); + values.writeString((String) null); + } + tuple.writeFloat64(0.5d); + } + row.writeFloat64(1.5d); + } + }); + Object[] tuple = (Object[]) tuples.get(0).getObject("t"); + Assert.assertEquals(tuple[0], 9); + Assert.assertEquals(asList(tuple[1]), Arrays.asList("x", null)); + Assert.assertEquals(((Number) tuple[2]).doubleValue(), 0.5d); + + List maps = insert("stream_map", + "id Int32, m Map(String, Array(Int32)), tail Float64", writer -> { + try (ValueWriter row = writer.beginRow()) { + row.writeInt32(1); + try (ValueWriter map = row.beginMap(2)) { + map.writeString("a"); + try (ValueWriter values = map.beginArray(2)) { + values.writeInt32(1); + values.writeInt32(2); + } + map.writeString("b"); + try (ValueWriter values = map.beginArray(0)) { + } + } + row.writeFloat64(1.5d); + } + }); + @SuppressWarnings("unchecked") + Map map = (Map) maps.get(0).getObject("m"); + Assert.assertEquals(asList(map.get("a")), Arrays.asList(1, 2)); + Assert.assertEquals(asList(map.get("b")), Arrays.asList()); + + List nested = insert("stream_nested", + "id Int32, n Nested(n1 Int32, n2 Nullable(String)), tail Float64", writer -> { + try (ValueWriter row = writer.beginRow()) { + row.writeInt32(1); + try (ValueWriter rows = row.beginNested(2)) { + try (ValueWriter inner = rows.beginTuple()) { + inner.writeInt32(5); + inner.writeString("a"); + } + try (ValueWriter inner = rows.beginTuple()) { + inner.writeInt32(6); + inner.writeString((String) null); + } + } + row.writeFloat64(1.5d); + } + }); + List rows = asList(nested.get(0).getObject("n")); + Assert.assertEquals(((Object[]) rows.get(0))[0], 5); + Assert.assertEquals(((Object[]) rows.get(0))[1], "a"); + Assert.assertEquals(((Object[]) rows.get(1))[0], 6); + Assert.assertNull(((Object[]) rows.get(1))[1]); + } + + @Test(groups = {"integration"}) + public void binaryJsonWithNestedObjectsNullsAndArrays() throws Exception { + if (jsonUnsupported()) { + return; + } + List rows = insert("stream_json", "id Int32, doc JSON", writer -> { + try (ValueWriter row = writer.beginRow()) { + row.writeInt32(1); + try (ValueWriter doc = row.beginJson()) { + doc.writeString("name", "ada"); + try (ValueWriter address = doc.beginObject("address", 2)) { + address.writeString("city", "Berlin"); + address.writeNull("zip"); + } + try (ValueWriter tags = doc.beginArray("tags", ClickHouseColumn.of("e", "Nullable(String)"), 3)) { + tags.writeString("a"); + tags.writeNull(); + tags.writeString("b"); + } + try (ValueWriter items = doc.beginArray("items", ClickHouseColumn.of("e", "JSON"), 2)) { + try (ValueWriter first = items.beginJson(1)) { + first.writeInt64("a", 100L); + } + try (ValueWriter second = items.beginJson(1)) { + second.writeString("b", "name"); + } + } + } + } + try (ValueWriter row = writer.beginRow()) { + row.writeInt32(2); + try (ValueWriter doc = row.beginJson(0)) { + } + } + }); + + Map doc = jsonMap(rows.get(0).getObject("doc")); + Assert.assertEquals(doc.get("name"), "ada"); + Assert.assertEquals(doc.get("address.city"), "Berlin"); + Assert.assertEquals(asList(doc.get("tags")), Arrays.asList("a", null, "b")); + List items = asList(doc.get("items")); + Assert.assertEquals(items.size(), 2); + Assert.assertEquals(jsonMap(items.get(0)).get("a"), 100L); + Assert.assertEquals(jsonMap(items.get(1)).get("b"), "name"); + Assert.assertTrue(jsonMap(rows.get(1).getObject("doc")).isEmpty()); + + List fields = client.queryAll( + "SELECT doc.address.city AS city, doc.address.zip AS zip, toJSONString(doc) AS text " + + "FROM stream_json WHERE id = 1"); + Assert.assertEquals(fields.get(0).getString("city"), "Berlin"); + Assert.assertNull(fields.get(0).getObject("zip")); + String text = fields.get(0).getString("text"); + Assert.assertTrue(text.contains("\"city\":\"Berlin\""), text); + Assert.assertTrue(text.contains("\"tags\":[\"a\",null,\"b\"]"), text); + Assert.assertTrue(text.contains("\"items\":[{\"a\":100},{\"b\":\"name\"}]"), text); + } + + @Test(groups = {"integration"}) + public void typedJsonPathsSkipTheDynamicTag() throws Exception { + if (jsonUnsupported()) { + return; + } + List rows = insert("stream_json_typed", + "doc JSON(name String, count Int64)", writer -> { + try (ValueWriter row = writer.beginRow()) { + try (ValueWriter doc = row.beginJson(2)) { + doc.writeString("name", "ada"); + doc.writeInt64("count", 7L); + } + } + }); + Map doc = jsonMap(rows.get(0).getObject("doc")); + Assert.assertEquals(doc.get("name"), "ada"); + Assert.assertEquals(doc.get("count"), 7L); + } + + private List insert(String table, String columns, RowWriter write) throws Exception { + CommandSettings command = new CommandSettings(); + command.serverSetting("allow_experimental_json_type", "1"); + client.execute("DROP TABLE IF EXISTS " + table, command).get(TIMEOUT_SECONDS, TimeUnit.SECONDS); + String nested = columns.contains("Nested(") ? " SETTINGS flatten_nested = 0" : ""; + client.execute("CREATE TABLE " + table + " (" + columns + ") ORDER BY tuple()" + nested, + command).get(TIMEOUT_SECONDS, TimeUnit.SECONDS); + TableSchema schema = client.getTableSchema(table); + client.insert(table, out -> { + RowBinaryStreamWriter writer = new RowBinaryStreamWriter(out, schema, ClickHouseFormat.RowBinary); + write.accept(writer); + }, ClickHouseFormat.RowBinary, new InsertSettings()).get(TIMEOUT_SECONDS, TimeUnit.SECONDS); + return client.queryAll("SELECT * FROM " + table + " ORDER BY " + orderBy(columns)); + } + + private static String orderBy(String columns) { + int comma = columns.indexOf(','); + String first = comma < 0 ? columns : columns.substring(0, comma); + String name = first.trim().split(" ")[0]; + if ("id".equals(name)) { + return "id"; + } + return "tuple()"; + } + + private boolean jsonUnsupported() throws Exception { + if (isCloud()) { + return true; + } + List version = client.queryAll("SELECT version()"); + return ClickHouseVersion.of(version.get(0).getString(1)).check("(,24.8]"); + } + + @SuppressWarnings("unchecked") + private static Map jsonMap(Object value) { + Assert.assertTrue(value instanceof Map, String.valueOf(value)); + return (Map) value; + } + + @SuppressWarnings("unchecked") + private static List asList(Object value) { + if (value instanceof BinaryStreamReader.ArrayValue) { + return ((BinaryStreamReader.ArrayValue) value).asList(); + } + if (value instanceof List) { + return (List) value; + } + throw new AssertionError("Expected a list but was " + (value == null ? "null" : value.getClass().getName())); + } + + private interface RowWriter { + void accept(RowBinaryStreamWriter writer) throws IOException; + } +} diff --git a/docs/features.md b/docs/features.md index 1bd3588ca..2c5d37df7 100644 --- a/docs/features.md +++ b/docs/features.md @@ -27,6 +27,7 @@ This document lists stable, user-visible behavior in `client-v2` and `jdbc-v2` t - Geometry type support: For ClickHouse `25.11+`, where `Geometry` changed from a string alias to `Variant(Point, Ring, LineString, MultiLineString, Polygon, MultiPolygon)`, the client reads and writes `Geometry` values through generic records, binary readers, POJO binding, and SQL parameter formatting, using Java array dimensionality to represent the geometry shape. ClickHouse `26.8+` adds `MultiPoint` to that variant, appended after the existing six so their discriminators are unchanged; a `MultiPoint` value read from a `Geometry` column is returned as `double[][]`, the same shape as `Ring` and `LineString`. - MultiPoint type support: For ClickHouse `26.8+`, reads and writes the `MultiPoint` geo type, which is transmitted exactly like `Ring` and `LineString` (`Array(Point)`) and is represented as `double[][]` through generic records, binary readers, POJO binding, and SQL parameter formatting. A `MultiPoint` held in a `Dynamic` column is decoded on read; it cannot be written into a `Dynamic` column, because the client infers the type from the Java value and `double[][]` resolves to `Ring`. - Nested type support: Un-flattened `Nested(f1 T1, ..., fN TN)` columns (tables created with `flatten_nested = 0`) can be written through the insert path (`Client#insert`) using `RowBinaryFormatWriter#setValue`, and are read back through the binary readers and generic records. The column is serialized the same way it is read — identically to `Array(Tuple(T1, ..., TN))`, a var-uint row count followed by one tuple per nested row — so the value supplied for the column is a `List` (or array) of tuples, one tuple per nested row, each carrying the N field values in declaration order. +- Streaming RowBinary reader and writer: Experimental `RowBinaryStreamWriter` and `RowBinaryStreamReader` in `com.clickhouse.client.api.data_formats.stream` write and read RowBinary values as scopes, so an `Array`, `Map`, `Tuple`, unflattened `Nested`, `QBit`, `Variant`, `Dynamic`, or binary `JSON` value can be produced or consumed one element at a time without building the whole container first. A `JSON` document is a list of paths: `beginJson` opens it, `writeString(path, value)` / `writeInt64(path, value)` / `writeNull(path)` write a dynamic path, `beginObject(path)` flattens a nested object into dotted paths, and `beginArray(path, elementType)` writes an array, including an array of JSON objects. `writeString` without a path remains the string form of a JSON column. A length-prefixed container opened with a known size writes the varint immediately and sends element bytes straight to the stream; an unknown size buffers the encoded bytes and writes the length when the scope is closed. `writeString(byte[])` preserves bytes that are not valid UTF-8. The writer supports `RowBinary` and `RowBinaryWithDefaults` and writes no header — the `INSERT` statement carries names and types. The reader supports `RowBinary` when the caller supplies the schema, `RowBinaryWithNames` with the same schema, and `RowBinaryWithNamesAndTypes` with the schema taken from the header. A column left unwritten is an error. `RowBinaryFormatWriter`, POJO insert, and JDBC are unchanged. `Native` is not supported on this path. See [reader_writer_api.md](reader_writer_api.md). - Insert APIs: Supports inserting registered POJOs, raw streams, and callback-driven writers, with optional column lists and format selection. - Insert controls: Supports insert-specific settings such as deduplication token, query id, compression behavior, and request headers. - Command execution: Executes DDL or other non-result commands and exposes response summaries and operation metrics. diff --git a/docs/integration-client.md b/docs/integration-client.md index 4e1d291bb..95969a86a 100644 --- a/docs/integration-client.md +++ b/docs/integration-client.md @@ -859,6 +859,92 @@ public void writeEventsStream(Client client, TableSchema schema, List eve Populate each row with `setValue(column, value)` (by name or 1-based index) and finish it with `commitRow()`. Do not close the stream yourself — the client manages the lifecycle. +When a column is itself a container and you do not already hold the whole value, use the experimental streaming writer inside the same callback. `RowBinaryStreamWriter` writes each row and each nested value as a scope. Pass the element count to `beginArray` when you know it: the length goes out immediately and the elements follow with no extra copy. Close every scope, including the row. A column you do not write is an error. `RowBinaryFormatWriter` stays the API for named cells and for values you already have in memory. The matching reader is `RowBinaryStreamReader`. The full contract is in [reader_writer_api.md](reader_writer_api.md). + +The example below expects columns `id Int64`, `name String`, and `tags Array(String)`, in that order. + +```java +import com.clickhouse.client.api.Client; +import com.clickhouse.client.api.data_formats.stream.RowBinaryStreamWriter; +import com.clickhouse.client.api.data_formats.stream.ValueWriter; +import com.clickhouse.client.api.insert.InsertResponse; +import com.clickhouse.client.api.insert.InsertSettings; +import com.clickhouse.client.api.metadata.TableSchema; +import com.clickhouse.data.ClickHouseFormat; + +public static class TaggedEvent { + public long id; + public String name; + public List tags; +} + +public void writeTaggedEvents(Client client, TableSchema schema, List events) throws Exception { + ClickHouseFormat format = ClickHouseFormat.RowBinary; + + try (InsertResponse response = client.insert("events", out -> { + RowBinaryStreamWriter writer = new RowBinaryStreamWriter(out, schema, format); + for (TaggedEvent event : events) { + try (ValueWriter row = writer.beginRow()) { + row.writeInt64(event.id); + row.writeString(event.name); + List tags = event.tags == null ? Collections.emptyList() : event.tags; + try (ValueWriter tagValues = row.beginArray(tags.size())) { + for (String tag : tags) { + tagValues.writeString(tag); + } + } + } + } + }, format, new InsertSettings()).get()) { + // handle response metrics + } +} +``` + +A `JSON` column is written the same way. `beginJson` opens the document. A nested object becomes dotted paths, `writeNull` writes a JSON null, and an array of objects is an `Array(JSON)` whose elements are documents. The column in this example is `doc JSON`. + +```java +import com.clickhouse.client.api.Client; +import com.clickhouse.client.api.data_formats.stream.RowBinaryStreamWriter; +import com.clickhouse.client.api.data_formats.stream.ValueWriter; +import com.clickhouse.client.api.insert.InsertResponse; +import com.clickhouse.client.api.insert.InsertSettings; +import com.clickhouse.client.api.metadata.TableSchema; +import com.clickhouse.data.ClickHouseColumn; +import com.clickhouse.data.ClickHouseFormat; + +public void writeProfile(Client client, TableSchema schema) throws Exception { + ClickHouseFormat format = ClickHouseFormat.RowBinary; + ClickHouseColumn nullableString = ClickHouseColumn.of("e", "Nullable(String)"); + ClickHouseColumn json = ClickHouseColumn.of("e", "JSON"); + + try (InsertResponse response = client.insert("profiles", out -> { + RowBinaryStreamWriter writer = new RowBinaryStreamWriter(out, schema, format); + try (ValueWriter row = writer.beginRow()) { + try (ValueWriter doc = row.beginJson()) { + doc.writeString("name", "ada"); + try (ValueWriter address = doc.beginObject("address", 2)) { + address.writeString("city", "Berlin"); + address.writeNull("zip"); + } + try (ValueWriter tags = doc.beginArray("tags", nullableString, 3)) { + tags.writeString("a"); + tags.writeNull(); + tags.writeString("b"); + } + try (ValueWriter items = doc.beginArray("items", json, 1)) { + try (ValueWriter item = items.beginJson(1)) { + item.writeInt64("a", 100L); + } + } + } + } + }, format, new InsertSettings()).get()) { + // handle response metrics + } +} +``` + **Key classes:** | Class | Role | diff --git a/docs/reader_writer_api.md b/docs/reader_writer_api.md new file mode 100644 index 000000000..6dbfee5e8 --- /dev/null +++ b/docs/reader_writer_api.md @@ -0,0 +1,247 @@ +# Reader Writer API Spec + +## Abstract + +The client already has a reader and a writer, but the two are not the same shape, and both materialize nested values before the application can touch them. This document specifies a streaming API that is symmetrical for the types that can be both read and written. Some ClickHouse types remain read-only; those limits are listed below. + +## Data flow + +A value is encoded on one side and decoded on the other. That encoding depends on the ClickHouse type and the server version, not on the application's domain types. An `Int32` is always the same four little-endian bytes. `String`, `Array`, and `Dynamic` each have their own codec. + +The application does not have to use the same Java type as the column. It may write a Java `long` into an `Int32`, which is a conversion (and an overflow check), not a different codec. + +The format is the container around those codecs. It decides how rows are framed and how nulls and defaults are marked. `RowBinary` and `RowBinaryWithDefaults` are row-oriented: each row is column values in schema order, and each value uses the RowBinary codec. `RowBinaryWithNamesAndTypes` adds a header of names and then types. `Native` is different: values are grouped into column blocks. `Dynamic` adds a type tag in front of the codec payload. + +Three layers, in order: + +1. **Conversion.** Java value to the canonical form of the ClickHouse type (`long` to `Int32` with a range check, `LocalDate` to a date). This is the public read and write methods. +2. **Format framing.** Null markers and, for `RowBinaryWithDefaults`, the default marker. This is a property of the column (nullable, has a `DEFAULT`) and of the format. It is not part of the type codec. Nested `Nullable` elements use only the inner null byte. The default marker exists only on a top-level column. +3. **Codec.** Canonical bytes: little-endian numbers, varint lengths, dynamic type tags. The codec does not accept an `Object` and does not look at JDBC types. + +Today the public API is a buffered, JDBC-style row (`setValue` / `commitRow`, `next` / `getInteger`). That API stays. Connectors that already have a cursor need to push or pull nested values without building the whole container first. That is the streaming API in this document. + +## A scope is a sequence of values + +A row, an array, a tuple, a map, a nested row, a variant arm, and a dynamic value are the same thing: an ordered sequence of values. The type of the next value comes from the schema (or, inside `Dynamic`, from the runtime type). The same verbs work at every level. Nesting is opening a child scope. Closing the child commits it and returns to the parent. + +Types are not repeated on each call. The caller passes a `TableSchema`. For `RowBinaryWithNamesAndTypes` the reader takes the schema from the header. The current column or element type selects the codec, the null marker, and the default marker. That is how format knowledge moves through the layers. `Dynamic` is the exception: the schema only says `Dynamic`, so `beginDynamic` supplies the runtime type. + +The API is positional. Columns are written and read in schema order. Named setters stay on the buffered writer. On write, `MATERIALIZED` and `ALIAS` columns are omitted, matching `RowBinaryFormatWriter`. On read, the schema is the columns present in the stream; nothing is omitted. + +Closing a scope is explicit (`close()`, or try-with-resources). The next value does not implicitly end an open container. An open child when the parent is closed is an error. Implicit close makes a nested array ambiguous: the next call could be another element or the next column. + +```java +void writeData(MyDTO dto, OutputStream out, TableSchema schema) throws IOException { + RowBinaryStreamWriter writer = new RowBinaryStreamWriter( + out, schema, ClickHouseFormat.RowBinaryWithDefaults); + try (ValueWriter row = writer.beginRow()) { + row.writeInt32(dto.getIntValue1()); + row.writeString(dto.getStrVal()); // null is SQL NULL when the column is Nullable + + try (ValueWriter items = row.beginArray()) { // size unknown: buffer encoded bytes + for (String li : dto.getList()) { + items.writeString(li); + } + } + + try (ValueWriter matrix = row.beginArray(dto.tableData().size())) { // size known: straight to the stream + for (String[] line : dto.tableData()) { + try (ValueWriter cells = matrix.beginArray(line.length)) { + for (String col : line) { + cells.writeString(col); + } + } + } + } + row.writeDate(dto.getStartDate()); + } +} +``` + +The reader has the same shape. It does not take a destination collection. + +```java +void readData(MyDTO dto, InputStream in, TableSchema schema) throws IOException { + RowBinaryStreamReader reader = new RowBinaryStreamReader( + in, schema, ClickHouseFormat.RowBinary); + if (!reader.hasNext()) { + return; + } + try (ValueReader row = reader.beginRow()) { + dto.setIntValue1(row.readInt32()); + dto.setStrVal(row.readString()); + List list = new ArrayList(); + try (ValueReader items = row.beginArray()) { + while (items.hasNext()) { + list.add(items.readString()); + } + } + dto.setList(list); + } +} +``` + +`beginArray()` belongs outside the loop. Calling it inside the loop would open a new array for every element. + +## Streaming and zero-copy + +`Array`, `Map`, `Nested`, and `QBit` are length-prefixed: a varint count, then the elements. `Tuple` has no count; it is N fields in schema order. Because the count comes first, a container whose size is not known yet cannot be written straight to the caller stream. + +Two open modes: + +- `beginArray(int size)`, `beginMap(int pairs)`, `beginNested(int rows)` with `size >= 0`. The varint is written immediately. Element bytes go to the current stream. `close()` checks the count. This is the path that does not copy the payload. +- `beginArray()`, `beginMap()`, `beginNested()`. Element bytes go to a growable buffer. `close()` writes the varint and then the buffer. The encoded bytes are copied once. A nested unknown-size container is copied again into its parent buffer. Callers who know the size avoid that copy. + +Zero-copy rules: + +- There is no row `Object[]`. A scalar is encoded when its write method is called. +- `writeString(byte[])` and `writeString(ByteBuffer)` write a varint length and the existing bytes. There is no UTF-8 round-trip. Bytes that are not valid UTF-8 are preserved. A heap `ByteBuffer` is written from its backing array. A direct buffer is copied once into a temporary array. +- Those payload bytes are not copied again when every open ancestor was opened with a known size, or has no length prefix (`Tuple`, a variant arm, a dynamic value, a top-level column). If any ancestor is buffering, the bytes are copied into that buffer. The copy is required because the length sits in front of the bytes. +- `Tuple` never buffers on its own. Only a length-prefixed field inside it does. +- Readers do not allocate the collection. `beginArray()` reads the varint into a remaining count. `readInt32()` returns a primitive. `readStringValue()` returns a `StringValue` over the bytes just read (one `byte[]`, no `String` unless the caller asks). `readString()` decodes UTF-8 and is lossy for invalid UTF-8. `readInto(int[] dst, int off, int len)` fills a caller buffer for an `Array(Int32)`. +- `hasNext()` is the remaining count of the current array, nested rows, or map pairs. After a map key, the next read or write must be the value. `hasNext()` is true only at the start of a pair, so the loop is "key, then value". +- `size()` is the total count read from the varint, or the number of tuple fields or row columns. It is not the number of values still remaining. + +`QBit` is a fixed-length array. `beginQBit()` and `writeFloat64Array(double[])` (and the matching primitive helpers) require the length to equal the dimension. An unknown size is rejected. On the wire a `QBit` is still a varint plus elements; that layout is a RowBinary detail, not a reason to treat `QBit` as `Array`. + +Bulk helpers (`writeInt32Array(int[])`, and the same for `Int8`, `Int16`, `Int64`, `Float32`, `Float64`, `BFloat16`, and `Bool`) write a whole primitive array as the current value. They use the known-size path. The loop form stays for a cursor source. A nullable element type is written with a non-null marker per element, because a primitive array cannot hold null. A null element still uses the loop and `writeNull()`. + +On any API or IO failure the stream is desynchronized. The writer or reader is not reusable. One writer or reader, one thread, one stream. The caller owns the stream and closes it. + +`Date` and `Date32` are encoded in UTC, the same as the buffered writer. `DateTime` and `DateTime64` use the column time zone, or UTC when the column has none. A constructor `TimeZone` overrides that default. + +## Writer + +`com.clickhouse.client.api.data_formats.stream.RowBinaryStreamWriter` + +```text +RowBinaryStreamWriter(OutputStream out, TableSchema schema, ClickHouseFormat format) +RowBinaryStreamWriter(OutputStream out, TableSchema schema, ClickHouseFormat format, TimeZone timeZone) +ValueWriter beginRow() +int getRowCount() +ClickHouseFormat getFormat() +``` + +Formats: `RowBinary` and `RowBinaryWithDefaults` only. There is no header. The `INSERT` statement carries names and types. + +`ValueWriter` is implemented by every scope. `close()` commits the scope. A second `close()` does nothing. Production code is Java 8. + +Scalars, named for the ClickHouse type. The argument is the canonical Java type. A second overload exists where a wider Java type needs a range check (`writeInt32(long)`). There is no `Object` overload. + +- `writeInt8`, `writeInt16`, `writeInt32`, `writeInt64`, `writeInt128`, `writeInt256` +- `writeUInt8`, `writeUInt16`, `writeUInt32`, `writeUInt64`, `writeUInt128`, `writeUInt256` +- `writeFloat32`, `writeFloat64`, `writeBFloat16`, `writeBool` +- `writeString(String)`, `writeString(byte[])`, `writeString(ByteBuffer)` +- `writeFixedString(String)`, `writeFixedString(byte[])` — length is the column precision +- `writeDate(LocalDate)`, `writeDate32(LocalDate)` +- `writeDateTime` for `LocalDateTime`, `ZonedDateTime`, and `Instant` +- `writeDateTime64` for `LocalDateTime` and `ZonedDateTime` +- `writeUUID`, `writeIPv4`, `writeIPv6`, `writeDecimal` +- `writeEnum8` and `writeEnum16` for the numeric wire value or the enum name +- `writeBitmap` for `AggregateFunction(groupBitmap, ...)` only + +`writeString(null)` (and the other reference overloads) writes SQL NULL when the current value is a nullable string. A null passed to the wrong type is a type error. `writeNull()` writes SQL NULL for whatever nullable value is current. + +`writeDefault()` writes the `RowBinaryWithDefaults` default marker and no body. It is legal only for a top-level column whose default kind is `DEFAULT`. It is not legal inside an array, a tuple, or a map, and it is not legal for `RowBinary`. + +Containers: + +- `beginArray()`, `beginArray(int size)` +- `beginTuple()` +- `beginMap()`, `beginMap(int pairs)` — keys and values alternate; a map key is not nullable; an odd number of writes is an error +- `beginNested()`, `beginNested(int rows)` — unflattened `Nested(f1 T1, ..., fN TN)` only, wire-identical to `Array(Tuple(T1, ..., TN))`. Each row is a `beginTuple()`. Flattened Nested (`flatten_nested = 1`) is already separate `Array` columns in the schema. +- `beginVariant(int ordinal)` — one value of that alternative, which may itself be a container. Ordinal `0xFF` is `writeNull()` on a `Variant` or `Geometry` value. +- `beginDynamic(ClickHouseColumn runtimeType)` — writes the dynamic type tag, then one value of that type. +- `beginQBit()` — `beginArray(dimension)`. + +`LowCardinality(T)` and `SimpleAggregateFunction(func, T)` are written as `T`. A nullable argument of a simple aggregate function still gets the nested null marker. + +`row.close()` requires every non-omitted column to have been written. A missing column is an error. It is not silently NULL or DEFAULT. That differs from `RowBinaryFormatWriter`, which treats an unset cell as null. The buffered writer does not change. + +A negative size is an error. Unknown size is the no-argument method, not `-1`. + +## Reader + +`com.clickhouse.client.api.data_formats.stream.RowBinaryStreamReader` + +```text +RowBinaryStreamReader(InputStream in, TableSchema schema, ClickHouseFormat format) +RowBinaryStreamReader(InputStream in, TableSchema schema, ClickHouseFormat format, TimeZone timeZone) +boolean hasNext() +ValueReader beginRow() +TableSchema getSchema() +``` + +Formats: + +- `RowBinary`: schema required, no header. +- `RowBinaryWithNames`: schema required. The name header is read and checked against the schema. +- `RowBinaryWithNamesAndTypes`: schema is read from the header. A schema passed by the caller is not used. + +`Native` is out of scope. It is column blocks, and `QBit` uses a different bit-plane layout there. Streaming a Native row still means buffering the block. + +`hasNext()` peeks one byte. It is false at end of stream. It throws if a row is still open. `beginRow()` throws `EOFException` when there is no row. + +`ValueReader` mirrors the writer: `readInt32()`, `readString()`, `readStringValue()`, and the other scalar twins, plus `beginArray()`, `beginTuple()`, `beginMap()`, `beginNested()`, `beginVariant()`, `beginDynamic()`, `beginQBit()`. `beginVariant()` reads the discriminator. `variantOrdinal()` reports it. `beginDynamic()` reads the type tag. `runtimeType()` returns the concrete column. A `Dynamic` `Nothing` value has no body; the scope is already complete and `runtimeType()` is `Nothing`. + +`isNull()` consumes a SQL NULL and returns true, or returns false and leaves the body to be read. On a non-nullable `Variant` or `Geometry` it recognizes discriminator `0xFF`. Primitive reads of SQL NULL throw `NullValueException` and consume the null, same as the buffered getters. `skip()` walks the current value without building a `List`, `Map`, or `ArrayValue`. `readInto` fills a caller `int[]` from an `Array(Int32)`. + +`close()` requires every column, field, or element the caller is responsible for to have been consumed. Use `skip()` for the rest. `hasNext()` and `size()` behave as described above. + +Strings inside containers are returned as `StringValue` from `readStringValue()`, including nested strings. The buffered reader still returns nested strings as `String`. + +## Format framing + +Top-level column, `RowBinary`: + +- Nullable null: one byte `1`, no body. +- Nullable value: one byte `0`, then the body. +- Non-nullable value: body only. +- Non-nullable null: error, except `Dynamic` (encoded as `Nothing`) and `Variant` / `Geometry` (discriminator `0xFF`). + +Top-level column, `RowBinaryWithDefaults`. Every column gets a default marker, including columns that have no default expression: + +- `writeDefault()` on a `DEFAULT` column: one byte `1`, no body. +- Value: one byte `0` (not a default). If the column is nullable, another `0`, then the body. +- SQL NULL on a nullable column that is not `writeDefault()`: `0`, then `1`, no body. +- SQL NULL on a non-nullable column: error, with the same `Dynamic` and `Variant` exceptions. The "not a default" byte is still written. + +Nested `Nullable` (an array element, a tuple field, a map value): one byte, `1` for null and `0` for a value, then the body. No default marker. + +## Container encodings + +- `Array`: varint count, then elements. A nullable element is a null byte plus the body. +- `Tuple`: fields in order, no count. +- `Map`: varint pair count, then key, value, key, value. Key order is the order of the write calls. +- `Nested` (unflattened): same bytes as `Array(Tuple(...))`. +- `QBit`: same bytes as an array whose length is the dimension. +- `Variant` and `Geometry`: one discriminator byte, then one value. `0xFF` is null. `Geometry` alternatives are `Point`, `Ring`, `LineString`, `MultiLineString`, `Polygon`, `MultiPolygon`, and `MultiPoint`. `Point` is a tuple of two `Float64`. `Ring`, `LineString`, and `MultiPoint` are arrays of points. `Polygon` and `MultiLineString` are arrays of rings. `MultiPolygon` is an array of polygons. There is no separate geo API. +- `Dynamic`: type tag (the existing dynamic tag encoding), then one value. +- `JSON` (binary object, not the string form): varint path count, then each path as a string name followed by its value. A dynamic path prefixes the value with a dynamic type tag. A typed path does not. `Nothing` (`0x00`) is a JSON null. Nested objects written with `beginObject` do not have their own count; their leaves are paths of the enclosing document. A path whose value is a JSON document (an element of `Array(JSON)`) is a dynamic `JSON` type tag plus another path list. + +## Symmetry limits + +These types are not given a new structured write path: + +- `AggregateFunction` other than `groupBitmap`. `writeBitmap` / `readBitmap` cover `groupBitmap` only. +- Structured `JSON` is a map of paths, written with `beginJson`. A dynamic path is a name plus a dynamic type tag and a value. A typed path declared on the column is the value only. A nested object opened with `beginObject` is stored as dotted paths (`address.city`) on the same document. `writeNull(path)` writes a JSON null (`Nothing`), which is not the same as leaving the path out. An array of objects is `beginArray(path, JSON column)` and each element is `beginJson`. `writeString` without a path is still the string form of a `JSON` column, used when the server is reading JSON as a string. The streaming reader still reads that string form. Structured documents are read back through the existing binary reader, which returns a map of paths. +- Inside `Dynamic`, payloads whose type tag the codec does not finish: `Tuple`, `Variant`, `Dynamic`, `JSON`, `Nested`, `QBit`, `AggregateFunction`, `SimpleAggregateFunction`, and `Geometry`. `beginDynamic` rejects them, including when they are nested inside an array or a map. Calling the existing tag writer for `Tuple` would emit a partial tag and desynchronize the stream. + +`LowCardinality` is the inner type on the RowBinary value path. The low-cardinality flag is not a dictionary. + +## Errors + +- `IllegalStateException`: wrong verb for the current type, unclosed child, declared size mismatch, extra value, missing column, `writeDefault()` in the wrong place, use after a failure. +- `IllegalArgumentException`: overflow (`writeInt32(long)` and the other ranged overloads), null into a non-nullable value, negative size, `QBit` length, odd map entry, `writeDefault()` on a column without `DEFAULT`. +- `NullValueException`: primitive read of SQL NULL, or `beginVariant()` on discriminator `0xFF`. +- `ClientException`: unsupported `Dynamic` payload, malformed header. +- `IOException`: the stream failed. +- `EOFException`: `beginRow()` when `hasNext()` is false. + +After any of these the reader or writer throws `IllegalStateException` on further use. Do not retry the same stream. + +## What does not change + +`ClickHouseBinaryFormatWriter` and `RowBinaryFormatWriter` stay the buffered, name-addressable API. POJO insert and JDBC keep using them. The streaming types are a separate experimental API, used inside `DataStreamWriter.onOutput`. A retry still calls `onRetry()` and then `onOutput` again; the caller writes the rows again. + +The materialized container codecs (`serializeArrayData`, `serializeMapData`, `serializeTupleData`) stay the path that walks a finished `List`, array, or `Map`. The streaming writer does not call them.