Fix interrupted PostgreSQL PREPAREs, SQLite JSON/UUID read as numbers, Has/Is query names; MySQL :copyfrom takes any Enumerable

README.md

@@ -108,7 +108,9 @@ per table, one module of constants per enum), `connection.rb` (shared helpers) a
   one at a time.
 * **Style:** SQL lives in frozen heredoc constants (`GET_AUTHOR_SQL`), comments above a query become its doc comment,
   table / column / enum comments of the schema become doc comments of the models. The code is RuboCop-clean; where a
-  name from your schema breaks a RuboCop naming rule (`"UserId"`, `naïve`), that class or method gets a `rubocop:disable`.
+  name from your schema breaks a RuboCop naming rule (`"UserId"`, `naïve`), that class or method gets a `rubocop:disable`
+  (on SQLite keep the query files ASCII, see [Limitations](#limitations)). Query names such as `HasAuthors` / `IsActive`
+  keep their name (`has_authors`) with `Naming/PredicatePrefix` disabled.
   A result column without a usable name (`SELECT 1, now()`) is named by its position: `column_1`, `column_2`, ...
 
 ### Transactions
@@ -156,7 +158,7 @@ end
 | `:execlastid` | the new id | PostgreSQL / SQLite: needs `RETURNING id`; MySQL: the auto-increment id |
 | `:execresult` | `ExecResult` (`rows_affected`, `last_insert_id`) | `last_insert_id` is `nil` on PostgreSQL (use `RETURNING`) |
 | `:batchexec` / `:batchone` / `:batchmany` | `nil` / `Array` of rows or `nil` / `Array` of `Array`s of rows | take an `Array` of `<Query>Args` and run the query once per element, on one connection |
-| `:copyfrom` | `nil` | bulk insert of an `Array` of `<Query>Args`; all rows or none (see below) |
+| `:copyfrom` | `nil` | bulk insert of an `Array` (or any `Enumerable`, also lazy) of `<Query>Args`; all rows or none (see below) |
 
 `:copyfrom` uses `COPY ... FROM STDIN` on PostgreSQL and batched multi-row `INSERT`s on SQLite. On MySQL it uses
 `LOAD DATA LOCAL INFILE`, which needs `local_infile: true` in the `Mysql2::Client` params and `local_infile=1` on the
@@ -191,7 +193,8 @@ server; duplicate keys and invalid values raise `Mysql2::Error` instead of being
   are not valid constant names still get one: `'end'` → `END_`, `'all'` → `ALL_`, `'2x'` → `VALUE_2X`, `'a b'` → `A_B`;
   values that end up with the same name get `_2`, `_3`, ...
 * SQLite follows its [type affinity rules](https://www.sqlite.org/datatype3.html#determination_of_column_affinity)
-  (`VARCHAR(255)` → `String`, `DECIMAL(10,5)` → `Float`, `UNSIGNED BIG INT` → `Integer`); `JSON` and `UUID` columns are `String`s.
+  (`VARCHAR(255)` → `String`, `DECIMAL(10,5)` → `Float`, `UNSIGNED BIG INT` → `Integer`); `JSON` and `UUID` columns are `String`s,
+  also when SQLite stored numeric-looking text as a number (`'42'` → `"42"`, in SQLite's spelling: `'1.50'` → `"1.5"`).
   The generated SQLite code uses `Monitor`: if you run Steep, add `library "monitor"` to your `Steepfile`.
 
 ### Names
@@ -217,6 +220,9 @@ server; duplicate keys and invalid values raise `Mysql2::Error` instead of being
   and fibers stop too; `busy_handler_timeout=` waits without blocking them.
 * SQLite transactions belong to the thread that opened them; fibers sharing the same database take turns, and a
   transaction holds it until it ends. Only the yielded `tx` runs inside the transaction from other fibers / threads.
+* SQLite: sqlc (1.27 – 1.31 at least) misplaces the SQL of a query file that contains a non-ASCII character (in a
+  name, literal or comment): that query and the following ones lose characters at the end (`FROM user` instead of
+  `FROM users`). This is a sqlc bug, not a plugin one; keep SQLite query files ASCII and pass such values as parameters.
 * Not applicable to Ruby (they exist in the C# plugin): `withAsyncSuffix` / `withCancellationToken` (use the Fiber
   scheduler), `useDapper`, target framework / csproj options (`moduleName` replaces the namespace).
 

docs/Coverage.md

@@ -43,3 +43,7 @@ CI runs `test/coverage_map_test.rb`, which checks every linked test method exist
 | Interrupts at `BEGIN` / `COMMIT` (`Thread#raise`, raised by the statement) roll back and leave no open transaction | [`test_interrupts_around_begin_and_commit_never_leave_the_connection_in_a_transaction`](../tests/end2end_pg.rb#test_interrupts_around_begin_and_commit_never_leave_the_connection_in_a_transaction) | [`test_interrupts_around_begin_and_commit_never_leave_the_connection_in_a_transaction`](../tests/end2end_mysql2.rb#test_interrupts_around_begin_and_commit_never_leave_the_connection_in_a_transaction) | [`test_interrupts_around_begin_and_commit_never_leave_the_database_in_a_transaction`](../tests/end2end_sqlite3.rb#test_interrupts_around_begin_and_commit_never_leave_the_database_in_a_transaction) | |
 | PostgreSQL statements dropped by `DEALLOCATE ALL` / `DISCARD ALL` are prepared again | [`test_statements_are_prepared_again_after_deallocate_all`](../tests/end2end_pg.rb#test_statements_are_prepared_again_after_deallocate_all) | not MySQL (no server-side prepared statements) | not SQLite | [`test_statements_are_named_after_the_module_and_query_class`](../test/features_test.rb#test_statements_are_named_after_the_module_and_query_class) |
 | `:execresult` with `RETURNING` runs the statement | covered by `:execresult` semantics | covered by `:execresult` semantics | [`test_execresult_with_returning_runs_the_statement`](../tests/end2end_sqlite3.rb#test_execresult_with_returning_runs_the_statement) | |
+| `:copyfrom` takes any `Enumerable` (empty, lazy, `Enumerator`) | [`test_copyfrom_takes_any_enumerable`](../tests/end2end_pg.rb#test_copyfrom_takes_any_enumerable) | [`test_copyfrom_takes_any_enumerable`](../tests/end2end_mysql2.rb#test_copyfrom_takes_any_enumerable) | [`test_copyfrom_takes_any_enumerable`](../tests/end2end_sqlite3.rb#test_copyfrom_takes_any_enumerable) | |
+| Query names like `HasAuthors` / `IsActive` keep their name and stay RuboCop-clean | [`test_predicate_like_query_names_are_kept`](../tests/end2end_pg.rb#test_predicate_like_query_names_are_kept) | [`test_predicate_like_query_names_are_kept`](../tests/end2end_mysql2.rb#test_predicate_like_query_names_are_kept) | [`test_predicate_like_query_names_are_kept`](../tests/end2end_sqlite3.rb#test_predicate_like_query_names_are_kept) | [`test_predicate_like_query_names_are_rubocop_clean`](../test/features_test.rb#test_predicate_like_query_names_are_rubocop_clean) |
+| SQLite `JSON` / `UUID` columns are `String`s, also numeric-looking text | not PostgreSQL | not MySQL | [`test_json_and_uuid_columns_return_strings_that_look_numeric`](../tests/end2end_sqlite3.rb#test_json_and_uuid_columns_return_strings_that_look_numeric) | |
+| PostgreSQL `PREPARE` interrupted (Timeout / `Thread#raise`, an exception raised by it) after the server ran it leaves the statement usable | [`test_an_interrupt_during_prepare_leaves_the_statement_usable`](../tests/end2end_pg.rb#test_an_interrupt_during_prepare_leaves_the_statement_usable), [`test_a_statement_left_prepared_by_a_failed_prepare_is_used`](../tests/end2end_pg.rb#test_a_statement_left_prepared_by_a_failed_prepare_is_used) | not MySQL (no server-side prepared statements) | not SQLite | |

docs/Testing.md

@@ -40,7 +40,8 @@ They `require` the generated code from `examples/` and run it against a real dat
   isolation, `break` / `throw`, a rescued failed statement on PostgreSQL, SQLite threads sharing a database, the
   yielded `tx` used from other fibers / threads), models named like Ruby constants (`Exception`, `Data`, ...), pool
   injection, logging, bang finders, row members named like row methods (`initialize`), MySQL changed vs. matched
-  rows, SQLite busy waiting, `:copyfrom` with awkward and binary values and its all-or-nothing behaviour,
+  rows, SQLite busy waiting, `:copyfrom` with awkward and binary values, lazy enumerators and its all-or-nothing behaviour,
+  PostgreSQL `PREPARE`s interrupted after the server ran them, SQLite JSON / UUID text that looks numeric,
   Fiber-scheduler concurrency (`async` gem) and statement-leak checks.
 
 ## CI workflows (`.github/workflows/`)

examples/authors/mysql/query.sql

@@ -224,3 +224,6 @@ SELECT COUNT(*) AS n FROM exceptions;
 
 -- name: TruncateExceptions :exec
 TRUNCATE TABLE exceptions;
+
+-- name: HasAuthors :one
+SELECT EXISTS (SELECT 1 FROM authors) AS has_any;

examples/authors/postgresql/query.sql

@@ -263,3 +263,6 @@ INSERT INTO late.feelings (id, moods) VALUES ($1, $2);
 
 -- name: GetLateFeeling :one
 SELECT moods FROM late.feelings WHERE id = $1;
+
+-- name: HasAuthors :one
+SELECT EXISTS (SELECT 1 FROM authors) AS has_any;

examples/authors/sqlite/query.sql

@@ -220,3 +220,6 @@ INSERT INTO exceptions (id) VALUES (NULL);
 
 -- name: CountExceptions :one
 SELECT COUNT(*) AS n FROM exceptions;
+
+-- name: HasAuthors :one
+SELECT EXISTS (SELECT 1 FROM authors) AS has_any;

examples/mysql2/query_sql.rb

@@ -490,6 +490,13 @@ class CountExceptionsRow < ::Data.define(:n)
     TRUNCATE TABLE exceptions
   SQL
 
+  HAS_AUTHORS_SQL = <<~'SQL'
+    SELECT EXISTS (SELECT 1 FROM authors) AS has_any
+  SQL
+
+  class HasAuthorsRow < ::Data.define(:has_any)
+  end
+
   class QuerySql
     # Raised by the bang variants (get_author!) when the query returns no row.
     class NotFound < ::StandardError; end
@@ -757,14 +764,12 @@ def get_authors_by_ids_and_names(ids:, names:)
     # Bulk insert: pass an Array of CopyAuthorsArgs.
     def copy_authors(rows)
       with_client('copy_authors') do |client|
-        return if rows.empty?
-
         ::Tempfile.create(['copy_authors', '.csv']) do |file|
           rows.each do |row|
             file.write(csv_line([row.name, row.bio]))
           end
           file.flush
-          load_data(client, "LOAD DATA LOCAL INFILE '#{client.escape(file.path)}' INTO TABLE `authors` CHARACTER SET binary FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' ESCAPED BY '\\\\' LINES TERMINATED BY '\\n' (`name`, `bio`)")
+          load_data(client, "LOAD DATA LOCAL INFILE '#{client.escape(file.path)}' INTO TABLE `authors` CHARACTER SET binary FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' ESCAPED BY '\\\\' LINES TERMINATED BY '\\n' (`name`, `bio`)") if file.pos.positive?
         end
         nil
       end
@@ -912,29 +917,25 @@ def list_regressions_by_ids(id:, id_2:, id_3:)
 
     def copy_regressions(rows)
       with_client('copy_regressions') do |client|
-        return if rows.empty?
-
         ::Tempfile.create(['copy_regressions', '.csv']) do |file|
           rows.each do |row|
             file.write(csv_line([row.order, row.note]))
           end
           file.flush
-          load_data(client, "LOAD DATA LOCAL INFILE '#{client.escape(file.path)}' INTO TABLE `regressions` CHARACTER SET binary FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' ESCAPED BY '\\\\' LINES TERMINATED BY '\\n' (`order`, `note`)")
+          load_data(client, "LOAD DATA LOCAL INFILE '#{client.escape(file.path)}' INTO TABLE `regressions` CHARACTER SET binary FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' ESCAPED BY '\\\\' LINES TERMINATED BY '\\n' (`order`, `note`)") if file.pos.positive?
         end
         nil
       end
     end
 
     def copy_regression_rows(rows)
       with_client('copy_regression_rows') do |client|
-        return if rows.empty?
-
         ::Tempfile.create(['copy_regression_rows', '.csv']) do |file|
           rows.each do |row|
             file.write(csv_line([row.id, row.order, row.note, row.payload]))
           end
           file.flush
-          load_data(client, "LOAD DATA LOCAL INFILE '#{client.escape(file.path)}' INTO TABLE `regressions` CHARACTER SET binary FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' ESCAPED BY '\\\\' LINES TERMINATED BY '\\n' (`id`, `order`, `note`, `payload`)")
+          load_data(client, "LOAD DATA LOCAL INFILE '#{client.escape(file.path)}' INTO TABLE `regressions` CHARACTER SET binary FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' ESCAPED BY '\\\\' LINES TERMINATED BY '\\n' (`id`, `order`, `note`, `payload`)") if file.pos.positive?
         end
         nil
       end
@@ -1001,44 +1002,38 @@ def truncate_regressions
 
     def copy_copy_types(rows)
       with_client('copy_copy_types') do |client|
-        return if rows.empty?
-
         ::Tempfile.create(['copy_copy_types', '.csv']) do |file|
           rows.each do |row|
             file.write(csv_line([row.id, row.code, row.doc, row.flags, row.amount]))
           end
           file.flush
-          load_data(client, "LOAD DATA LOCAL INFILE '#{client.escape(file.path)}' INTO TABLE `copy_types` CHARACTER SET binary FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' ESCAPED BY '\\\\' LINES TERMINATED BY '\\n' (`id`, `code`, @sqlc_v2, `flags`, `amount`) SET `doc` = CONVERT(@sqlc_v2 USING utf8mb4)")
+          load_data(client, "LOAD DATA LOCAL INFILE '#{client.escape(file.path)}' INTO TABLE `copy_types` CHARACTER SET binary FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' ESCAPED BY '\\\\' LINES TERMINATED BY '\\n' (`id`, `code`, @sqlc_v2, `flags`, `amount`) SET `doc` = CONVERT(@sqlc_v2 USING utf8mb4)") if file.pos.positive?
         end
         nil
       end
     end
 
     def copy_copy_types_ignore(rows)
       with_client('copy_copy_types_ignore') do |client|
-        return if rows.empty?
-
         ::Tempfile.create(['copy_copy_types_ignore', '.csv']) do |file|
           rows.each do |row|
             file.write(csv_line([row.id, row.code]))
           end
           file.flush
-          load_data(client, "LOAD DATA LOCAL INFILE '#{client.escape(file.path)}' IGNORE INTO TABLE `copy_types` CHARACTER SET binary FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' ESCAPED BY '\\\\' LINES TERMINATED BY '\\n' (`id`, `code`)", strict: false)
+          load_data(client, "LOAD DATA LOCAL INFILE '#{client.escape(file.path)}' IGNORE INTO TABLE `copy_types` CHARACTER SET binary FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' ESCAPED BY '\\\\' LINES TERMINATED BY '\\n' (`id`, `code`)", strict: false) if file.pos.positive?
         end
         nil
       end
     end
 
     def copy_copy_types_replace(rows)
       with_client('copy_copy_types_replace') do |client|
-        return if rows.empty?
-
         ::Tempfile.create(['copy_copy_types_replace', '.csv']) do |file|
           rows.each do |row|
             file.write(csv_line([row.id, row.code]))
           end
           file.flush
-          load_data(client, "LOAD DATA LOCAL INFILE '#{client.escape(file.path)}' REPLACE INTO TABLE `copy_types` CHARACTER SET binary FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' ESCAPED BY '\\\\' LINES TERMINATED BY '\\n' (`id`, `code`)")
+          load_data(client, "LOAD DATA LOCAL INFILE '#{client.escape(file.path)}' REPLACE INTO TABLE `copy_types` CHARACTER SET binary FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' ESCAPED BY '\\\\' LINES TERMINATED BY '\\n' (`id`, `code`)") if file.pos.positive?
         end
         nil
       end
@@ -1314,6 +1309,23 @@ def truncate_exceptions
       end
     end
 
+    # rubocop:disable Naming/PredicatePrefix
+    def has_authors
+      with_client('has_authors') do |client|
+        result = client.query(bind_params(client, HAS_AUTHORS_SQL, []), as: :array)
+        row = result.first
+        row && HasAuthorsRow.new(row[0])
+      end
+    end
+    # rubocop:enable Naming/PredicatePrefix
+
+    # rubocop:disable Naming/PredicatePrefix
+    # Like #has_authors, but raises NotFound when there is no row.
+    def has_authors!(...)
+      has_authors(...) || raise(NotFound, 'HasAuthors returned no row')
+    end
+    # rubocop:enable Naming/PredicatePrefix
+
     private
 
     def instrument(name)

examples/mysql2/query_sql.rbs

@@ -531,6 +531,13 @@ module Mysql2Codegen
 
   TRUNCATE_EXCEPTIONS_SQL: ::String
 
+  HAS_AUTHORS_SQL: ::String
+
+  class HasAuthorsRow
+    attr_reader has_any: untyped
+    def initialize: (  untyped has_any) -> void
+  end
+
   class QuerySql
     class NotFound < ::StandardError
     end
@@ -687,6 +694,10 @@ module Mysql2Codegen
 
     def truncate_exceptions: -> void
 
+    def has_authors: -> HasAuthorsRow?
+
+    def has_authors!: -> HasAuthorsRow
+
     private
 
     def instrument: [T] (::String name) { () -> T } -> T

examples/pg/query_sql.rb

@@ -581,6 +581,13 @@ class CountExceptionsRow < ::Data.define(:n)
   class GetLateFeelingRow < ::Data.define(:moods)
   end
 
+  HAS_AUTHORS_SQL = <<~'SQL'
+    SELECT EXISTS (SELECT 1 FROM authors) AS has_any
+  SQL
+
+  class HasAuthorsRow < ::Data.define(:has_any)
+  end
+
   class QuerySql
     # Raised by the bang variants (get_author!) when the query returns no row.
     class NotFound < ::StandardError; end
@@ -1527,6 +1534,23 @@ def get_late_feeling!(...)
       get_late_feeling(...) || raise(NotFound, 'GetLateFeeling returned no row')
     end
 
+    # rubocop:disable Naming/PredicatePrefix
+    def has_authors
+      with_client('has_authors') do |client|
+        result = run_prepared(client, 'PgCodegen::QuerySql#has_authors', HAS_AUTHORS_SQL)
+        row = result.first
+        row && HasAuthorsRow.new(row['has_any'])
+      end
+    end
+    # rubocop:enable Naming/PredicatePrefix
+
+    # rubocop:disable Naming/PredicatePrefix
+    # Like #has_authors, but raises NotFound when there is no row.
+    def has_authors!(...)
+      has_authors(...) || raise(NotFound, 'HasAuthors returned no row')
+    end
+    # rubocop:enable Naming/PredicatePrefix
+
     private
 
     def instrument(name)
@@ -1589,22 +1613,32 @@ def string_arrays(client)
     end
 
     # Executes a statement prepared once per connection. A statement dropped by hand (DEALLOCATE ALL, DISCARD ALL) is
-    # prepared again; inside a transaction the error has already aborted it, so it is raised.
+    # prepared again, and one the server already has (its PREPARE was interrupted after it ran) is used; inside a
+    # transaction the error has already aborted it, so it is raised.
     def run_prepared(client, name, sql, params = [])
-      unless client.prepared_statements.include?(name)
-        client.prepare(name, sql)
-        client.prepared_statements.add(name)
-      end
+      prepare(client, name, sql) unless client.prepared_statements.include?(name)
       client.exec_prepared(name, params)
     rescue ::PG::InvalidSqlStatementName
       client.prepared_statements.delete(name)
       raise unless client.transaction_status == ::PG::PQTRANS_IDLE
 
-      client.prepare(name, sql)
-      client.prepared_statements.add(name)
+      prepare(client, name, sql)
       client.exec_prepared(name, params)
     end
 
+    # Timeout / Thread#raise wait until the statement is prepared and recorded. An exception raised by PREPARE itself
+    # (a stopped Async task) can still leave the statement prepared on the server: it is recorded when PREPARE fails
+    # because the statement exists.
+    def prepare(client, name, sql)
+      ::Thread.handle_interrupt(::Object => :never) do
+        client.prepare(name, sql)
+        client.prepared_statements.add(name)
+      rescue ::PG::DuplicatePstatement
+        client.prepared_statements.add(name)
+        raise unless client.transaction_status == ::PG::PQTRANS_IDLE
+      end
+    end
+
     # bytea parameters are sent escaped: binary Strings (NUL bytes, invalid UTF-8) are not valid text.
     def encode_bytea(value)
       value && ::PG::TextEncoder::Bytea.new.encode(value)

examples/pg/query_sql.rbs

@@ -603,6 +603,13 @@ module PgCodegen
     def initialize: (  ::Array[::String] moods) -> void
   end
 
+  HAS_AUTHORS_SQL: ::String
+
+  class HasAuthorsRow
+    attr_reader has_any: bool
+    def initialize: (  bool has_any) -> void
+  end
+
   class QuerySql
     class NotFound < ::StandardError
     end
@@ -793,6 +800,10 @@ module PgCodegen
 
     def get_late_feeling!: (id: ::Integer) -> GetLateFeelingRow
 
+    def has_authors: -> HasAuthorsRow?
+
+    def has_authors!: -> HasAuthorsRow
+
     private
 
     def instrument: [T] (::String name) { () -> T } -> T
@@ -806,6 +817,7 @@ module PgCodegen
     def string_arrays: (untyped client) -> untyped
 
     def run_prepared: (untyped client, ::String name, ::String sql, ?::Array[untyped] params) -> untyped
+    def prepare: (untyped client, ::String name, ::String sql) -> void
 
     def encode_bytea: (::String? value) -> ::String?