Repository navigation
Native cell pull reads, shared Arrow column builder with dictionary strings, CLI preview, faster string and date reads - #177
Merged
Merged
Conversation
`excelreader preview <file>` prints the shape, column names and inferred types, then the first and last rows in a box-drawn table, like a polars DataFrame. --rows, --max-columns and --max-width cap the table; --fast stops after the first rows instead of scanning the whole sheet; --columns picks columns by name or 0-based index. CSV columns get types inferred from their text. Also for the other commands: - sheets and schema write UTF-8 when redirected, and UTF-16 through WriteConsoleW on a Windows console, so the shell's code page is never changed. - Error messages drop the " (Parameter 'x')" suffix. - The -s, -o, -f and -d aliases the README documents now work.
…and bump arrow version to 60.1.0
ISheetWriter.SetShowGridLines(bool) must be called before the first row, like SetColumnWidth. XLSX writes <sheetView showGridLines="0"> only when gridlines are hidden, so default output is unchanged; XLSB clears the fDspGrid bit of BrtBeginWsView; XLS clears it in WINDOW2 and carries it to continuation sheets; CSV ignores it. The interface member has an empty default implementation so sheet writers implemented outside the library keep compiling.
xl_rows_next_cells(rows, &cells, &count) returns the next row as cursor-owned xl_cell_view structs, valid until the next call on the cursor or xl_rows_close. Text is copied once; numbers stay doubles (has_number/number), so XLSB and XLS numbers are never formatted to text and the caller never parses them back. Append-only export, no ABI bump. On the 65K fixtures (NativeAOT, medians) it reads XLSB in 20.8 ms against 33.1 ms for xl_rows_next_view (-37%) and CSV in 10.2 ms against 12.3 ms (-17%); XLSX is parse-bound (~-1%). CellViewBuffer now does both shapes: Borrow for xl_rows_foreach and xl_aggregate, Copy for the cursor. A row left pending by xl_rows_next is handed out as text only. The PGO trainer now also runs xl_rows_next_cells and xl_rows_foreach, and the profile is regenerated: without it the new path gained only 5% on CSV and lost 3% on XLSX. xl_parse_typed is unchanged with the new profile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RowCursor::next_row now gets cell views with binary numbers instead of xl_rows_next_view's text-only row. Counting the cells of the 65K CSV drops from 11.8 ms to 10.3 ms against zsv. A cell that carries only a number (XLSB, XLS) now has an empty CellView::value and its number in CellView::number, as for_each_row already gave it; read it with as_number() or text(). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RowCursor::next_row borrows cursor-owned cell views instead of decoding an xl_rows_next blob from its own buffer, so it no longer keeps a buffer or retries on XL_BUFFER_TOO_SMALL. A cell that carries only a number (XLSB, XLS) now has empty as_str() and as_bytes(), as for_each_row already gave it; read it with number() or text(). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rings The typed column builder behind xl_parse_typed moves from ExcelReader.Native into ExcelReader.Core (Reader/Columnar), with the sequential, CSV-partitioned and XLSX/XLSB pipeline paths. TypedApi and ArrowApi now only map its columns onto xl_table and the Arrow C structs, and ExcelReader.Arrow's ToArrowRecordBatch uses it in place of its own ColumnAppender. On 65K x 14 columns (JIT) the managed conversion drops from 40.3 to 25.0 ms on XLSB, 62.6 to 44.4 ms on XLSX and 27.6 to 21.3 ms on CSV, the same as xl_parse_arrow; BDN allocation is 2.4-2.6x lower. ArrowReadOptions.DegreeOfParallelism (default 1) splits the read across threads. ExcelReader.Arrow reaches Core internals through InternalsVisibleTo, so its package now depends on the exact Core version. ExcelColumnType.StringDictionaryColumn / XL_T_STRING_DICT (7) reads a text column as an Arrow dictionary: int32 indices into the column's distinct strings. A shared string is looked up by its SST index, other text by hash; parallel parts are merged by remapping indices. Only the Arrow outputs take it; xl_parse_typed and the typed reader reject it. Parse time: XLSB -12%, XLSX -5%, CSV unchanged. Append-only, no ABI bump. A blank cell in a nullable string column is now null on every path; the C ABI used to return an empty string there. The PGO trainer also runs xl_parse_arrow with a dictionary column, and the profile is regenerated (the builder rename had dropped it). With it, xl_parse_typed is unchanged on XLSB, ~+1% on XLSX and +2.5% on CSV. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ColumnType.STRING_DICT reads a text column as an Arrow dictionary, so to_pandas() and to_polars() return a Categorical. Only the Arrow outputs take it; parse() and parse_batches() raise. A streamed read builds one dictionary per batch. A blank cell in a nullable text column is now None rather than "". The parse() docstring now says parallelism applies to XLSX, XLSB and CSV. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Set ColumnType::StringDict on a column from infer_schema and pass the
schema to to_arrow_with_schema / to_arrow_stream_with_schema to get a
DictionaryArray<Int32Type>. It has no derive field type; the typed reads
reject it.
A blank cell in an Option<String> field is now None rather than
Some("").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Cell.GetString and the shared-string cache built strings through Encoding.UTF8.GetString, which counts characters and then transcodes, two passes that dominate short cell values. Text that is all ASCII now widens straight into the new string; other text keeps the UTF-8 decoder. The number-formatting branch of GetString, with its stackalloc, moved out of line so ordinary calls get a lighter frame. Reading every field of the 65K sales CSV as strings drops from about 31 ms to 20.5 ms. Accented text gets slightly faster too, with no change in allocation.
An inline string holding an entity, an _xHHHH_ escape or a carriage return left the canonical cell path for the general one. A single <t> run holds no '<', so it can be decoded straight into the row values, with the same result as the general path. Inline text with escapes reads 20-25% faster; plain text is unchanged.
InternStrings used one content cache for the whole reader. On mixed data the mostly-unique columns evicted the categorical ones, so most lookups missed. A miss cost more than allocating the string, and interning the 65K sales CSV took 85% longer than not interning. Each column now gets its own cache, created the first time a cell of that column is read as a string. Every 1024 lookups the cache counts its hits; under one in sixteen, the column skips the cache for the next 64K cells and then samples again. The hash now mixes 8-byte words in place of HashCode.AddBytes. On the sales CSV, interning drops from 37.9 ms to 24.3 ms, against 20.7 ms without it, and allocates half as much (16.9 vs 35.8 MiB). On a generated 50K x 10 CSV it drops from 20.3 ms to 12.3 ms.
Text dates in CSV typed reads took the fast path only for ISO-8601. The invariant culture's own DateTime.ToString() output, MM/dd/yyyy HH:mm:ss, fell through to a UTF-16 decode and DateTime.TryParse, about 150 ns per value. With the default invariant culture, that exact shape (and the date-only MM/dd/yyyy) is now parsed directly. Any other spelling still goes to the culture parser, so results do not change. On a 50K x 10 CSV whose dates use that format, a DateTime-only parse drops from 11 ms to 3.4 ms and the full typed parse from 19 ms to 12 ms.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #177 +/- ##
==========================================
+ Coverage 92.37% 92.43% +0.05%
==========================================
Files 294 302 +8
Lines 24938 25381 +443
Branches 4042 4161 +119
==========================================
+ Hits 23037 23460 +423
- Misses 1294 1300 +6
- Partials 607 621 +14
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
develop is 15 commits ahead of master. The main changes:
xl_rows_next_cells(append-only export, no ABI bump). The C++ and RustRowCursornow read through it.ExcelReader.Nativeinto Core.xl_parse_typed,xl_parse_arrowandExcelReader.Arrow'sToArrowRecordBatchall use it.XL_T_STRING_DICT/ColumnType.STRING_DICT/ColumnType::StringDictread a text column as an Arrow dictionary.previewcommand. Prints shape, column types and head/tail rows, like a polars DataFrame.SetShowGridLines. Hides a sheet's gridlines in XLSX, XLSB and XLS. CSV ignores it.InternStringskeeps a cache per column.MM/dd/yyyy HH:mm:ssdates directly.Performance
65K × 14 fixtures (NativeAOT or JIT as noted), medians:
xl_rows_next_cellsvsxl_rows_next_view)ToArrowRecordBatch, XLSB / XLSX / CSV (JIT)stringInternStringsArrow allocation is 2.4–2.6x lower. Plain-text XLSX and number-only paths are unchanged.
Compared with OfficeIMO 3.6.0 using its own benchmark adapters on the same machine, ExcelReader is faster or tied on every read lane: XLSX/XLSB/XLS, CSV raw, typed, quoted, multiline and parallel. CSV writes are tied.
Behavior changes
null/None. The C ABI used to return"".RowCursor. In C++ and Rust, an XLSB/XLS cell that carries only a number now has an emptyvalue/as_str(). Read it withnumber()/as_number()ortext().ExcelReader.Arrownow depends on the exact Core version, because it uses Core internals. Apache.Arrow is bumped to 60.1.0, andArrowArrayStreamReaderis replaced byRecordBatchStreamReader.Testing
DateTime.Parse;xl_parse_typedwas measured on XLSX, XLSB and CSV before and after; it is neutral.