Skip to content

Bug-hunt fixes (culture-safe numbers, cancellation, model mapping) and four measured Core speedups - #187

Merged
GabrielMarquezMatte merged 53 commits into
masterfrom
develop
Nov 18, 2026
Merged

GabrielMarquezMatte merged 53 commits into
masterfrom
develop

Conversation

@GabrielMarquezMatte

Copy link
Copy Markdown
Owner

Summary

develop is 38 commits ahead of master: 33 fixes from a third bug hunt and 5 performance changes from a Core audit. Every fix has a regression test that failed before it. Every perf change was measured with an interleaved A/B against the pre-perf develop.

Reader

  • Workbook-stored numbers are parsed invariantly, whatever the culture. XLSX only stores a parsed double for <v> text of 15 digits or fewer. Longer text, such as 0.30000000000000004 or any computed double Excel writes, was parsed with ExcelParserOptions.Culture. Under pt-BR, 35.840000000000003 became 35840000000000004, and under fr-FR it failed to parse. Cell.TryParse and TryParseSpanParsable now use the invariant culture for Number/Date cells, and string/CSV cells keep the caller's culture.
  • Memory-backed XLSX/XLSB sheets honor the cancellation token. OpenSheetAsync dropped ct when the ZIP index was backed by memory, which covers FromXlsx(ReadOnlyMemory) and any writable MemoryStream.
  • Time-only t="d" cells now read in 1904 workbooks and with a negative offset (10:30:00-03:00). They used to lose their serial, or pick up today's date.
  • Prefix detection skips comments, PIs and DOCTYPE. A <Tag> inside a leading comment used to make a prefixed (x:) sheet read zero rows.
  • HasEntryWithPrefix is ASCII case-insensitive like TryGetEntry. Before, xl/Tables/ made workbook.Tables silently empty.
  • DetectFileFormat returns Unknown for encrypted ZIPs and ZIPs past the entry limit instead of throwing.
  • SheetDataReader.GetFieldType/GetSchemaTable report string when GetValue will return a string (e.g. an out-of-range date serial).

Parser and generator

  • Parse(IExcelSheet) on a CsvSheet uses the CSV map. Text dates read correctly, integers are no longer read as TimeOnly, and blank lines no longer produce phantom models.
  • Parallel CSV:
    • a partitioned parse honors the ParseAsync token;
    • WithCancellation links tokens instead of replacing them;
    • the sequential fallback can be enumerated twice;
    • a source shorter than HeaderRow yields nothing, like the sequential parser.
  • Reflection mapping (FromAttributes, record writer):
    • skips indexers;
    • binds only the most-derived property when one is hidden with new;
    • an [ExcelIgnore] on the most-derived declaration suppresses the name. It no longer binds through the base.
  • The generator honors attributes inherited through override (ExcelColumn, ExcelIgnore, ExcelRequired, ExcelConverter, ExcelStyle), matching reflection.

Writer

  • XLSB WriteRow finishes the row when a cell is rejected. Before, the sheet and the whole package could never be ended.
  • A rejected write keeps the pending cell reference and style (XLSX Skip then Write(NaN)). The next cell used to land in column A. XLSB and XLS also kept a pending style on rejected numbers and dates.
  • XLS:
    • a Workbook stream past 2 GiB throws ExcelLimitExceededException instead of wrapping its offsets;
    • continuation sheet names are cut on a code-point boundary.

Arrow, native ABI, CLI

  • ToArrowRecordBatch without a schema makes inferred columns nullable. A blank past the 100-row sample used to throw.
  • Native exports:
    • every export zeroes its out params before validating, so a C caller freeing a failed xl_buffer no longer frees stack garbage;
    • a NULL column-spec name is XL_INVALID_ARGUMENT;
    • xl_open_memory accepts (NULL, 0), and every rejection sets a last error.
  • CLI:
    • convert errors start on their own line;
    • preview --rows <huge> no longer allocates the whole tail buffer up front.

Bindings

  • .NET Framework:
    • skips indexers and hidden new properties;
    • Aggregate rethrows the failure the native run ended on, not the first one in wall-clock time;
    • GetText falls back to G17 when .NET Framework's "R" doesn't round-trip.
  • Rust:
    • the out_of_range check only reads I64 columns, bounded by the column length. A hand-written ExcelMapper could read past a native buffer from safe code;
    • each aggregation failure gets its own status, so a stale panic from a discarded chunk can't replace the real outcome.
  • Python:
    • column buffers are copied without ctypes.string_at, whose size is a C int (silent truncation past 2 GiB);
    • an Arrow table that stays multi-chunk raises ValueError instead of writing only its first chunk.

Performance

Change Scenario Min Median
Per-thread deflate hash table, no 256 KB clear per ZIP entry (output byte-identical) 500 × 10-row XLSX writes −21% −29%
Hand-written enumerator for the parallel CSV merge (no async-iterator resume per row) SheetParallel.ParseAsync struct model, 300k-row CSV, dop 4 −12% −15%
Blank cells skip failing parses (TryAppendNull, empty guards in Cell) ToArrowRecordBatch, 150k-row XLSX, 30% blanks −14% −9%
Non-seekable input spooled into one exact-size array 6.7 MB XLSX via a non-seekable stream allocations 16.7 → 6.7 MB time within noise

The controls (typed XLSX parse, 100k-row XLSX write) are neutral.

Evidence

Repro tests, before → after:

ParserFeatureTests.CultureDoesNotApplyToWorkbookStoredNumbers   (XLSX <v>35.840000000000003</v>, pt-BR)
  before: 35840000000000004          after: 35.84
MidStreamCancellationTests.XlsxMemoryBackedEnumeratorHonorsCancellation
  before: rows yielded after cancel  after: OperationCanceledException
CsvParserTests.ACsvSheetHeldAsIExcelSheetParsesLikeTheCsvOverload
  before: When == 0001-01-01         after: 2024-01-15 10:30
XlsbWriterTests.BulkXlsbCellRowRejectedCellKeepsSheetUsable
  before: End() throws "The active XlsbRowWriter must be disposed"   after: workbook ends
CellStyleWriteTests.RejectedWriteKeepsColumnAndPendingStyle   (xlsx/xlsb/xls × 5 rejected values)
  before: "x" in column A, unstyled  after: column C, bold
ExportsOutParamTests (7 exports)
  before: out param still 0x5A garbage   after: zeroed
DeflateTableReuseTests
  reused-table output == forced-clear output, byte for byte, including wrap-around near uint.MaxValue
  • dotnet test tests/ExcelReader.Tests on the merged develop: 4632 passed, 0 failed
  • On the branches: NetFramework tests 187 passed (1 x86 skip), cargo test -p excelreader green, Python 272 passed (1 xfail)

Merge Danger

Door: two-way. There are no public API, C header or XL_ABI_VERSION changes, and every commit reverts on its own.

Blast Radius: behavior

  • Culture: ExcelParserOptions.Culture / Cell.TryParse(provider) no longer apply to workbook Number/Date cells. Callers who relied on a culture to parse workbook numbers get invariant results now (the correct ones).
  • Model mapping: generated maps on models that override properties may now bind different headers (inherited [ExcelColumn]) or skip columns (inherited [ExcelIgnore]). This brings them in line with FromAttributes.
  • Arrow schema: ToArrowRecordBatch() without a schema reports every inferred field as nullable.
  • SheetDataReader: a column whose first row holds an unreadable value is typed string.
  • Writer output: compressed bytes are unchanged (verified against a forced-clear table). XLS workbooks past 2 GiB now throw instead of producing a corrupt file.
  • Memory: each thread that writes ZIP entries keeps one 256 KB hash table for its lifetime. Parallel writes keep about cores × 256 KB.
  • Known, not changed here:
    • parallel XLSX writes with shared strings are not byte-reproducible run to run (pre-existing);
    • PrefetchStream's documented chunk copy is still open.

XlsbSheetWriter.WriteRow had no try/finally around the cell loop, so a rejected
value (NaN, oversized text) left the row active and every later StartRow,
WriteRow or End threw, losing the whole package. The row is now finished in a
finally, keeping the cells written before the rejection, which matches what
disposing an XlsbRowWriter after a rejected Write does. XLSX and XLS have no
bulk WriteRow, so only XLSB was affected.
TypeMap bound every public instance property, so a POCO with an indexer
made Read<T> throw NotSupportedException for 'Item' even though no column
could ever match it. The writer already filtered indexers.
…ulture

XLSX numbers with more than 15 significant digits keep only their text, and
Cell.TryParse parsed that text with the caller's culture: under pt-BR
35.840000000000003 became 35840000000000004, under fr-FR it failed. Number and
Date cells now always parse invariantly (also in the ISpanParsable path);
string and CSV cells still honor the culture.
Parse(IExcelSheet) always used the Excel type map, so a CSV sheet reached
through IExcelWorkbook read text dates as default, accepted integer TimeOnly
cells as midnight, and turned blank lines into all-default models (or a
required-value failure). It now picks the CSV map for a CsvSheet and the
generic projector skips CSV blank records like the CSV-specialized one.
GetProperties returns both members when a derived class redeclares a
property with a different type, so typed reads filled both and typed
writes emitted two columns with the same header. Reader and writer now
share one property walk that keeps only the most-derived member per name,
as Core's generator does.
…d on

Fail kept whichever callback exception happened first in wall-clock time,
but the native pipeline reports the earliest failing chunk in sheet order
and may discard and re-read later chunks. With failures in several chunks
the rethrown exception varied run to run. Each failure now returns its own
positive code, which native passes back verbatim, so the exception is
looked up by the status the run returned.
…LSB sheets

OpenSheetAsync took a shortcut to the sync OpenSheet when the zip was
memory-backed (ReadOnlyMemory overloads, or a writable MemoryStream exposing
its buffer), which built the enumerator with a default token, so a cancelled
token never stopped the read.
…ted write

XlsxRowWriter consumed the pending explicit cell reference (set by Skip) and
the pending Style() as call arguments, before CellFormatter validated the value.
A rejected NaN, infinite or pre-epoch date therefore dropped both, so the next
cell lost its r= attribute and landed in the wrong column. Row state is now
committed only after the cell is written.

XLSB and XLS had the same pending-style loss: EffectiveStyle ran before the
non-finite and epoch checks. Those checks now run first.
ParallelCsvEnumerable only saw the token handed to GetAsyncEnumerator, so
the ct given to SheetParallel.ParseAsync was dropped on the partitioned
path and a cancelled parse read every row. It now links both tokens, as
the XLSX pipeline does.
GetText formatted with "R", which on .NET Framework x64 can drop the last
bit (0.84551240822557006 came back one ulp off from XLSB and XLS). The
"R" text is kept when it parses back to the value, otherwise G17 is used.
…again

When the data after the header fit in one chunk, the factory opened a
private CsvReader and handed it to the CsvEnumerable, whose first
enumeration disposed it; a second await foreach threw
ObjectDisposedException. The fallback now reads the caller's reader
sheet, which opens a fresh cursor per enumeration, so there is no owned
reader left to dispose.
…sets

BuildGlobals summed sheet lengths in int, so a workbook stream over 2 GiB
wrapped the BOUNDSHEET lbPlyPos, INDEX and DBCELL offsets negative and the
container came out corrupt. BIFF8 stores those as 32-bit offsets and a CFB v3
stream must stay under 2 GiB, so the sum is now checked and throws
ExcelLimitExceededException (WorkbookStreamBytes). The OLE writer's sector
rounding of the stream size is also done in long, since a size within 511 bytes
of int.MaxValue overflowed it.
…ed with another

GetAsyncEnumerator replaced the token captured at creation (the
SheetParallel.ParseAsync ct) with any cancelable one passed to it, so
.WithCancellation(other) dropped the original. Both are now linked when
both can cancel, and the async enumerator disposes the linked source.
DetectFileFormat only caught InvalidDataException, so a password-protected
ZIP (NotSupportedException) or one with more entries than MaxZipEntries
(ExcelLimitExceededException) escaped instead of returning the documented
Unknown. Opening such a file still throws NotSupportedException, which the
error guide documents for features the reader does not support.
CreateContinuation cut the base name by UTF-16 code unit to fit the 31-char
limit with its " (n)" suffix, so a name with an astral character at the cut
point produced a continuation sheet ending in a lone high surrogate, which the
AddSheet name validation would have rejected. The cut now drops the whole pair.
…e the header row

CsvHeaderBinder threw InvalidOperationException when the source had
fewer rows than HeaderRow, while the sequential parser yields an empty
sequence. The binder now reports no data start, so the factory returns
the empty result.
…prefix

A tag inside a leading comment or processing instruction was taken for the root element, so a prefixed sheet, workbook, styles or shared-strings part read as unprefixed and yielded nothing.
…egative offset

The time-only anchor was fixed at 1899-12-31, which is before the 1904 epoch and so had no serial. A trailing '-hh:mm' offset also made the value look like a date and picked up today's date. Anchor at the workbook epoch and detect time-only by the absence of a date part before the first ':'.
HasEntryWithPrefix compared bytes exactly while TryGetEntry ignores ASCII
case, so a package storing xl/Tables/table1.xml made table discovery bail out
early and Tables came back silently empty.
GetProperties includes indexers, and binding one threw ArgumentException
from Expression.Property inside the cached Lazy, so any model with a
public indexer could never be parsed. Indexers are now filtered out, as
the writer and the source generator already do.
…adable typed cells

GetValue falls back to the cell's text when a Number, Date or Boolean cell
cannot be read as that type (e.g. a date-styled serial past DateTime's range),
but GetFieldType and GetSchemaTable still reported double/DateTime/bool. The
type now follows the same check, so it always matches GetValue's value.
…ne with new

Reflection returns both the hiding and the hidden property, so
FromAttributes threw "Ambiguous header binding" and the record writer
emitted the column twice. Both now walk the type hierarchy most-derived
first and keep one property per name, as the source generator does.
xl_parse_typed, xl_parse_arrow, xl_write_typed_to_memory,
xl_encrypt_package_to_memory, xl_open_write_handle, xl_add_style and
xl_rows_next_cells returned InvalidArgument without touching a valid out
pointer, so a caller freeing its buffer afterwards freed stack garbage.
A NULL name pointer reached Encoding.UTF8.GetString and surfaced as
XL_ERROR 'Value cannot be null' instead of XL_INVALID_ARGUMENT.
ctypes.string_at takes a C int, so a typed column or buffer past 2 GiB was
silently truncated or raised SystemError. Route every computed-size copy
through _native.copy_bytes, which sizes a c_char array from the address.
…error

C fixed over an empty array yields NULL, which an empty in-memory CSV
should open from. Rejected open requests now also report why.
…writing only its first chunk

combine_chunks leaves binary columns split past 2 GiB, and taking batches[0]
silently dropped the rest. The int32-offset ABI cannot take that column anyway,
so raise a ValueError.
…eclaration

MappableProperties checked [ExcelIgnore] before claiming the name, so a
derived property marked [ExcelIgnore] let the base declaration of the
same name bind, and for an override that calls the very setter the
attribute was meant to skip.
…ngth

out_of_range read every column with a fits check as row_count i64 values, so a
hand-written binding with fits on a bool column read 8 bytes per row from a
1-byte-per-row buffer from safe code, and an empty names slice panicked on
names[0]. Check only I64 columns, stop at the column length, and tolerate no names.
ExcelColumn, ExcelIgnore, ExcelRequired, ExcelConverter and ExcelStyle are
Inherited, and reflection resolves them up the override chain; the generated
map only read the overriding symbol. An ignore on the most-derived
declaration now also suppresses the name instead of binding the base
declaration.
…rence sample

Inference samples 100 rows, so a column without a blank in the sample was
non-nullable and a later blank threw. Inferred columns are now all nullable;
an explicit schema keeps its own nullability.
…s tail lazily

convert: an error after progress text landed on the same line; end the
progress line first and drop the blank line printed on failure.
preview: --rows allocated a rows/2 tail array before reading anything,
which threw OutOfMemoryException for huge values; it now grows with the
rows actually read.
store_panic kept the first payload forever and finish resumed it whenever the
status was i32::MAX, so a panic from a discarded CSV chunk replaced a later
panic or a legitimate Err(i32::MAX). Each panic now gets a status below
i32::MAX, a caller code inside that window is recorded the same way, and finish
maps the status the run ended on back to that exact failure. A discarded panic
still cannot fail a run that returns XL_OK.
A non-seekable input was copied through a doubling MemoryStream (256 B start), allocating ~3x the input in zeroed LOH arrays plus a copy per doubling. Now pooled 64K..1M chunks are assembled into a single uninitialized exact-size array. Reading the 6.7 MB XLSX through a non-seekable stream was 8-9% slower than seekable.
… the array limit

The check ran only after the whole stream was read, so an oversized input was held in pooled chunks before failing; MemoryStream used to fail as it crossed the limit.
A blank cell in a nullable typed column ran the full failing parse (FastDate, double.TryParse,
DateTime.TryParseExact x4) before recording a null: 56 ns per blank date versus 11.5 ns for a valid one,
45 ns for a blank double. TypedTableBuilder now appends the null directly for a missing or empty cell
in a nullable column, writing the same default and validity bit as before. Cell.TryGetDateTime and
Cell.TryParse<T> return false at once for an empty text cell of a type whose parser never accepts empty.
Model parsing already skips CellType.Empty cells before calling the parser.
…er row

The merge thread of a partitioned SheetParallel.ParseAsync<T> handed out
every row through an async iterator, paying a full state-machine resume
per row (21-22 ns/row in isolation). A hand-written enumerator now
returns rows of the current chunk synchronously and only goes async at
chunk boundaries (5.5 ns/row). Chunk taking, failure row numbers,
cancellation per chunk, list pooling and teardown are unchanged;
DisposeAsync is idempotent and also runs on completion and on a fault.
…r entry

Every DeflateEncoder (one per ZIP entry and one per parallel segment)
rented and cleared a 64K-slot hash table, about 15 us of a 49 us
10-row XLSX write. Each thread now keeps one table, and each encoder
starts its positions past the previous encoder's, so stale slots fail
the distance check exactly as zero slots did. Output is byte-identical.
The table is still cleared once its base passes 2^30 or after a stream
that slid its window. Measured: writing a 10-row XLSX is 31-45% faster.
@GabrielMarquezMatte GabrielMarquezMatte changed the title Develop Bug-hunt fixes (culture-safe numbers, cancellation, model mapping) and four measured Core speedups Nov 18, 2026
@codecov

codecov Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.61017% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.42%. Comparing base (04b3c25) to head (ab711ac).

Files with missing lines Patch % Lines
src/ExcelReader.Core/Reader/Xlsx/XlsxXml.cs 85.00% 1 Missing and 2 partials ⚠️
rust/excelreader/src/aggregate.rs 96.29% 2 Missing ⚠️
...r.Core/Parser/ParallelCsv/ParallelCsvEnumerable.cs 97.01% 0 Missing and 2 partials ⚠️
.../ExcelReader.Core/Reader/Internal/StreamSpooler.cs 96.00% 1 Missing and 1 partial ⚠️
src/ExcelReader.Cli/CliCommands.cs 97.22% 1 Missing ⚠️
...lReader.Core/Reader/Columnar/TypedColumnBuilder.cs 90.00% 0 Missing and 1 partial ⚠️
...Reader.Core/Reader/Xlsx/XlsxWorkbook.Enumerator.cs 87.50% 0 Missing and 1 partial ⚠️
src/ExcelReader.Core/Writer/Xls/XlsSheetWriter.cs 85.71% 0 Missing and 1 partial ⚠️
src/ExcelReader.Core/Writer/Zip/DeflateEncoder.cs 94.44% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #187      +/-   ##
==========================================
+ Coverage   92.32%   92.42%   +0.10%     
==========================================
  Files         302      304       +2     
  Lines       25602    25856     +254     
  Branches     4219     4273      +54     
==========================================
+ Hits        23636    23898     +262     
+ Misses       1345     1334      -11     
- Partials      621      624       +3     
Flag Coverage Δ
cpp 88.77% <ø> (ø)
csharp 92.32% <96.50%> (+0.11%) ⬆️
python 96.27% <100.00%> (+<0.01%) ⬆️
rust 92.72% <96.61%> (+0.11%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@GabrielMarquezMatte
GabrielMarquezMatte merged commit 25e6768 into master Nov 18, 2026
47 checks passed
GabrielMarquezMatte added a commit that referenced this pull request Nov 18, 2026
Bug-hunt fixes (culture-safe numbers, cancellation, model mapping) and four measured Core speedups
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant