Parallelize large-file CSV loading (13-20x), and fix dialect guessing on files with JSON columns

[quocson95]

Opening a large CSV pinned a single core and froze the window until it finished.
This makes the load saturate all cores, keeps the window painting, and adds a
working Cancel. It also fixes a pre-existing bug that made any file with a JSON
column open with the wrong number of columns, silently.

Numbers

M4 Pro (8P + 4E), Release -O2, best of three. "before" is a harness built from
the pre-change sources — it reproduces every golden, which is what makes the
comparison trustworthy rather than self-referential.

workload	before	after
117 MB, 6 cols, 15% quoted, UTF-8	1.338 s	0.099 s	13.5×
111 MB TSV, no quotes	1.313 s	0.114 s	11.5×
156 MB, 1000 cols × 20k rows	1.443 s	0.073 s	19.8×
arrangeColumns, 1000 cols	10 698 ms	33 ms	324×

That last row was the post-load freeze: the column-width scan walked each row
once per column, and ran after the progress window had already been hidden.

How it works

csvfsm.hh holds the one state machine every consumer runs — the serial parser,
the chunk prescan, and the parallel parse — so the transitions cannot drift
apart. The design turns on a single observation: at a physical line start the
parser's structural state is exactly one bit (enclosed). A one-bit transfer
function composes in O(k), so each chunk is prescanned under both carry-in
hypotheses and the true carry chain is resolved serially afterwards. No guessing,
no verify round, no probabilistic fallback.

The file is mapped once and every consumer reads the same bytes — dialect
guessing, encoding detection, and the parse — removing eight seekg(0) re-reads
and a full extra streaming pass. UTF-8 validation is multi-threaded and provably
equivalent to utf8::is_valid, which is what licenses skipping fixUtf8() on
the parallel path.

Workers never touch a widget or call Fl::check(), so FLTK stays effectively
single-threaded: no thread-safe build, no Fl::lock, no Fl::awake.

The JSON bug (second commit)

A row like 5226,101,...,"{""user"": {""id"": ""566674135""}}",... is 13
comma-separated fields. guessDefinition also probes :, and because JSON is
dense with colons every row split into the same 87 pieces — consistently, so
the variance metric read as a perfect zero. The tie-break was "more columns
wins", so : beat , 60 to 13. Zero variance also kept confidence at 1.00,
so the format dialog never appeared and the file silently opened with 87 columns.

tableStatistics() skips row 0, so nothing noticed that : left the header as
one field while exploding every data row. Two tie-breakers now sit after
variance and before the column count: header consistency, and the share of
fields left holding an orphaned quote. Both only ever break a tie, so no
candidate that uniquely explains the row lengths can be displaced.

Blast radius across the 61-file corpus: 3 files change dialect, all three to
the correct one — including bs_escaped_delim, which now finds the backslash
escape it actually uses.

Behaviour changes that need a changelog line

• Windows: a 0x1A (Ctrl-Z) byte no longer silently truncates the table.
  The mapping reads raw bytes; such files now open in full.
• UTF-8 files over 200 MB now open without the "choose your format" modal.

Bugs found and fixed on the way

• getNextCodeUnit() read uninitialised stack when EOF cut a UTF-16 code unit
  short — the garbage could equal the 0x000A line terminator, so UTF-16 line
  splitting depended on stack contents.
• ENC_NONE files fell into the fixed-width code-unit reader and parsed as an
  empty table.
• A UTF-16LE file starting with U+xx00 (一 and much of CJK) matched the
  UTF-32LE BOM test, which compared octet[2] twice, and opened empty.
• guessEncoding() read uninitialised octets for files under 4 bytes.
• Fl::check() inside a std::sort comparator (latent UB) — replaced with an
  injected progress callback.
• Moving the load onto a background thread exposed three more: a redraw racing
  the storage rebuild, a Cancel that could report success after truncating, and
  an unguarded bad_alloc that would have called std::terminate.

Testing

There were no tests before this. tests/run_all.sh runs a headless harness that
links without FLTK — itself the forcing function that keeps the data layer
decoupled from the UI:

check	what it proves
goldens	the serial path produces the same bytes it did before
istream vs mmap	the two readers agree byte-for-byte
serial vs parallel	agreement at 7 different chunk counts
boundary sweep	a forced chunk boundary at every byte offset of every corpus file, plus pairs and triples
differential fuzz	120k random-content / random-dialect / random-chunking comparisons
UTF-8 validation	the parallel validator equals utf8::is_valid across 17 thread counts
column scan	one-pass equals the per-cell reference
reuse check	parseCsvStream keeps no state between calls
guess check	the guessed dialect for every corpus file is unchanged

Clean under ASan+UBSan and TSan. The corpus is generated (tests/gen_corpus.sh)
and gitignored.

Where to focus review

• src/csvfsm.hh — the shared state machine. The transitions reproduce the
  historical parser exactly, quirks included; the comments say which and why.
• src/csvloader.cpp — the prescan/compose/parse/assemble pipeline.
• src/csvguess.cpp — the ranking change, the only behaviour change to how
  existing files are interpreted.

Risk and revert

The parallel path is gated in one place (CsvLoader::planLoad) and refuses
anything it cannot prove exact: non-UTF-8 encodings, invalid UTF-8, non-ASCII
structural characters, probing paths. TCRUNCHER_DISABLE_PARALLEL=1 forces the
serial path at runtime — that is the switch to tell a bug reporter to flip. Even
with it set, the file still loads ~3.4× faster than before.