[kvlonge]

Files sources can now read CSV files in encodings other than UTF-8, and skip lines before the header.

What changes

• csv.encoding (default utf-8) accepts:
  • cp1252;
  • latin-1 and its other spellings (iso88591, iso-8859-1, cp819, …), read byte for byte as ISO-8859-1;
  • strict ascii;
  • utf-16le and utf-16be;
  • any other label of the WHATWG Encoding Standard (windows-1250, iso-8859-15, shift_jis, …).
• Errors:
  • An unknown label fails plan.
  • A byte the encoding doesn't define (cp1252's 0x81/0x8D/0x8F/0x90/0x9D, or anything over 127 in ascii) fails the table with the file, line and byte. A bad UTF-8 file fails the same way.
  • A UTF-16 file whose byte order mark names the other byte order fails, naming the right setting.
  • A file cut off inside a character fails at its last byte.
• csv.skip_rows skips lines (ending at \n or \r\n) before the header. Quotes in skipped lines don't matter, and error line numbers count the skipped lines. A file whose lines end in a bare \r fails rather than being skipped whole.
• Default path unchanged: UTF-8 is still read as it is, with no extra copy.
• Snowflake direct COPY: both settings now pass through as ENCODING and SKIP_HEADER = skip_rows + header. Snowflake reads latin-1, windows-1250–1256, iso-8859-2/5/6/7/15, koi8-r and utf-16le/be the way SQLLoad's reader does. ascii, East Asian encodings and other unmapped encodings fail plan for the direct route.

Snowflake live check

One X-Small run used copy_to_internal_stage with synthetic files. The schema was dropped and the warehouse suspended afterwards. The loaded values were byte-identical to SQLLoad's own reader (compared with HEX_ENCODE) for:

• cp1252 with two skipped lines (one holding an unclosed quote) and CRLF inside a quoted field;
• latin-1 with 0x80 (Snowflake's ISO88591 reads it as U+0080, byte for byte);
• UTF-16LE with a byte order mark.

Known difference: Snowflake loads a byte cp1252 leaves undefined (0x81) as the control character U+0081, where SQLLoad's reader fails. plan catches such a byte when it falls in the sampled rows. The docs say so.

Tests

• Reader: a table in files_source/tests/test_read_text_files.rs covers each encoding, the error cases, skip_rows and its line numbers.
• Snowflake: COPY SQL cases in test_direct_copies.rs, plus a test of the encoding mapping.
• Local runs: make check, make test-unit and make test-python (353 passed) all pass.
• Review: one independent review (five small findings, all fixed with tests) and a follow-up check of the fixes, which found nothing.

The docs mark both settings "From 0.35.0 (not released yet)".

[cloudflare-workers-and-pages]

Deploying sqlload with    Cloudflare Pages

Latest commit:	4b635aa
Status:	✅  Deploy successful!
Preview URL:	https://9dde67a3.sqlload.pages.dev
Branch Preview URL:	https://feat-csv-encoding.sqlload.pages.dev

View logs