Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
9a43825
Bump the test group with 1 update
dependabot[bot] Oct 26, 2026
b91a3a4
Bump Microsoft.CodeAnalysis.CSharp from 5.6.0 to 5.9.0
dependabot[bot] Oct 26, 2026
b88917d
Merge pull request #84 from GabrielMarquezMatte/dependabot/nuget/src/…
GabrielMarquezMatte Oct 26, 2026
56d66e4
Merge pull request #83 from GabrielMarquezMatte/dependabot/nuget/test…
GabrielMarquezMatte Oct 26, 2026
038831b
chore: promote public API for v2.0.0
GabrielMarquezMatte Oct 26, 2026
c0d5d97
Merge pull request #85 from GabrielMarquezMatte/chore/promote-public-…
GabrielMarquezMatte Oct 26, 2026
b79ad8b
fix(native): keep the existing file when fw_write_file fails
Oct 26, 2026
fb2110c
fix(native): reject TIME and TIMESTAMP values outside their range
Oct 26, 2026
123c643
fix(python): scale decimals exactly whatever the decimal context
Oct 26, 2026
1471906
fix(cpp): report a std::ostream with failbit set as a failed write
Oct 26, 2026
3bf23c3
fix(cpp): keep the bytes read before the end of an exception-enabled …
Oct 26, 2026
daea5c7
fix: parse every name of an enum alias
Oct 26, 2026
5f595e4
fix: read a mix of spaces and the trim character as blank
Oct 26, 2026
8754887
fix(native): skip a DECIMAL whose scaling overflows instead of failin…
Oct 26, 2026
fa472eb
fix: reject a column that ends past int.MaxValue
Oct 26, 2026
35aa355
fix(generator): escape keyword names and reject generic models
Oct 26, 2026
b406221
chore: prepare 2.0.1
Oct 26, 2026
0238927
fix(native): check STRING offsets against the column's data length
Oct 26, 2026
c13ee6d
fix: write a record that formats as zero bytes
Oct 26, 2026
540bb89
fix(native): reject DECIMAL values the reader cannot read back
Oct 26, 2026
03ee6f7
fix: parse a negative enum value written with zero padding
Oct 26, 2026
bc23b78
fix: honor a format provider's positive sign in the fast number parsers
Oct 26, 2026
fac69e1
fix: decode non-UTF-8 text before parsing UTF-8-parsable columns
Oct 26, 2026
32766c9
fix(native): give fw_write_file's staging file a short name of its own
Oct 26, 2026
cba8d70
docs: list the second round of fixes in the 2.0.1 changelog
Oct 26, 2026
59781e5
refactor(native): report invalid arguments through NativeApi.InvalidA…
Oct 26, 2026
7b2647f
refactor: create attribute converters through one helper
Oct 26, 2026
3fec1ee
fix: stop a parallel stream aggregation at the failure that decides it
Oct 26, 2026
0fddedd
perf: look up zero-padded negative enum codes without allocating
Oct 26, 2026
027aabd
perf(native): trim a whole-source STRING column's block before publis…
Oct 26, 2026
1813cb9
perf(native): copy whole bytes when merging byte-aligned validity bit…
Oct 26, 2026
fcecfa9
docs: list the performance audit's changes in the 2.0.1 changelog
Oct 26, 2026
662d9dd
fix: add comment for tiered compilation quick JIT setting in project …
Oct 28, 2026
e7d6bc6
Bump the test group with 1 update
dependabot[bot] Oct 29, 2026
b32d157
Merge pull request #87 from GabrielMarquezMatte/dependabot/nuget/test…
GabrielMarquezMatte Oct 29, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -191,7 +191,7 @@ Studio can load it. It is an incremental generator: `Extract` turns each candida
change a model's columns does not regenerate it.

`Emit` validates the model — partial, top-level, a supported type for each column, layout, converter
type, overflow, ASCII padding — and reports diagnostics `FWP001`–`FWP012` (`FWP007` and `FWP009` are
type, overflow, ASCII padding — and reports diagnostics `FWP001`–`FWP014` (`FWP007` and `FWP009` are
retired) instead of generating code
for an invalid model. Otherwise it writes `TryParse` and `TryFormat` as straight-line code: one
`FixedWidthRuntime` call per column. Each column's settings become two `static readonly` fields, a
Expand Down
47 changes: 44 additions & 3 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,49 @@ All notable changes to this project are documented here.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [2.0.0]
## [2.0.1]

### Fixed
- `fw_write_file` replaced an existing file before formatting, so a failed write left it truncated. It
now writes a sibling file and moves it into place only on success.
- `fw_write_*` wrapped `TIME` and `TIMESTAMP` microseconds outside the supported range into a valid
value; they are now an error.
- A `DECIMAL` value whose scaling overflowed `decimal` failed the whole read instead of a parse error
that `skip_invalid` can skip.
- C++ wrapper: `write_stream` reported success to a `std::ostream` with only `failbit` set, and
`Reader::open_stream` failed at the end of a `std::istream` with exceptions enabled.
- Python: `DECIMAL` scaling, written and in `Table.to_arrow`, rounded to the caller's `decimal`
context precision.
- An enum alias (a second name for a value) did not parse.
- A nullable column of spaces and its trim character mixed, such as `" 0 0 "`, was not read as `null`.
- A layout whose column ends past `int.MaxValue` was accepted and threw while reading; it is now
rejected, and the generator reports `FWP014`.
- Generated models with C# keyword names (`@event`) produced code that did not compile, and a generic
model crashed the generator; it now reports `FWP013`.
- `fw_write_*` trusted a `STRING` column's offsets: offsets past its `data_len`, descending or negative,
and a negative `data_len`, are now `FW_INVALID_ARGUMENT` instead of reading past the declared data.
- `fw_write_*` wrote `DECIMAL` values of 19 digits, which reading the output rejects; they are now an
error, as on reading.
- Writing a record that formats as zero bytes with an empty `NewLine`, such as a generated model with
no columns, grew its buffer without end.
- A negative enum value written as a zero-padded number (`-0001`) did not parse back.
- A format provider whose positive sign is not `"+"` was ignored by the fast `double`, `float` and
`decimal` parsers, which accepted `"+1"`.
- A non-UTF-8 encoding was ignored by `decimal` and other UTF-8-parsed columns holding non-ASCII text,
such as a Latin-1 no-break space grouping digits.
- A parallel `AggregateAsync` over a stream kept reading to the end after a failure that decided the
result, such as a line that fails to parse, so a stream that stays open delayed the error until the
caller cancelled. It now stops at the chunk holding the failure.

Not released yet.
### Changed
- A whole-source native read gives back the unused part of a `STRING` column's block before handing it
over: the block was sized for the column's full width on every row, and a table of short text in a wide
column kept all of it (126 MiB instead of 5 MiB for 1,000,000 rows of 2 bytes in 128).
- Parsing a negative enum code written with zero padding (`-0001`) no longer allocates a string.
- Merging the null bitmaps of a parallel native read copies whole bytes where a partition starts on a byte
boundary.

## [2.0.0]

2.0 reads and writes **bytes only**, where 1.x had a `char` and a UTF-8 version of each reader,
parser and writer, and goes through **one entry point**: every `FixedWidth` method comes in an
Expand Down Expand Up @@ -249,7 +289,8 @@ Initial release.
reflection-free `TryParse` generated at compile time, with diagnostics (FWP001-FWP007) for invalid
layouts and unsupported column types.

[2.0.0]: https://github.com/GabrielMarquezMatte/FixedWidthParser/compare/v1.1.1...HEAD
[2.0.1]: https://github.com/GabrielMarquezMatte/FixedWidthParser/compare/v2.0.0...HEAD
[2.0.0]: https://github.com/GabrielMarquezMatte/FixedWidthParser/compare/v1.1.1...v2.0.0
[1.1.1]: https://github.com/GabrielMarquezMatte/FixedWidthParser/compare/v1.1.0...v1.1.1
[1.1.0]: https://github.com/GabrielMarquezMatte/FixedWidthParser/compare/v1.0.0...v1.1.0
[1.0.0]: https://github.com/GabrielMarquezMatte/FixedWidthParser/releases/tag/v1.0.0
44 changes: 30 additions & 14 deletions cpp/tests/read.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,24 @@ static void check_table(const fw::Table& table)
CHECK(table.is_valid(2, 1));
}

// The codes of every batch read from `in`, checking each read succeeds.
static std::vector<std::string> streamed_codes(std::istream& in, const fw::Layout& layout)
{
auto streamed = fw::Reader::open_stream(in, layout, {.read = skip_header_and_invalid(), .batch_rows = 2});
CHECK(streamed.has_value());
std::vector<std::string> codes;
while (true) {
auto batch = streamed->next();
CHECK(batch.has_value());
if (!batch->has_value()) {
return codes;
}
for (int64_t row = 0; row < (*batch)->row_count(); ++row) {
codes.emplace_back((*batch)->string(1, row));
}
}
}

int main()
{
auto layout = cotahist_layout();
Expand Down Expand Up @@ -63,20 +81,18 @@ int main()

// The same batches from a std::istream, read forward without seeking.
std::istringstream in{std::string(content)};
auto streamed = fw::Reader::open_stream(in, layout, {.read = skip_header_and_invalid(), .batch_rows = 2});
CHECK(streamed.has_value());
std::vector<std::string> codes;
while (true) {
auto batch = streamed->next();
CHECK(batch.has_value());
if (!batch->has_value()) {
break;
}
for (int64_t row = 0; row < (*batch)->row_count(); ++row) {
codes.emplace_back((*batch)->string(1, row));
}
}
CHECK((codes == std::vector<std::string>{"PETR4", "VALE3", "ITUB4"}));
CHECK((streamed_codes(in, layout) == std::vector<std::string>{"PETR4", "VALE3", "ITUB4"}));

// A stream with exceptions enabled throws at its end too; the bytes before it still count.
std::istringstream throwing{std::string(content)};
throwing.exceptions(std::ios::failbit | std::ios::badbit);
CHECK((streamed_codes(throwing, layout) == std::vector<std::string>{"PETR4", "VALE3", "ITUB4"}));

// A stream that fails before its end is an error, not an empty file.
std::istringstream failing{std::string(content)};
failing.setstate(std::ios::failbit);
auto refused = fw::Reader::open_stream(failing, layout, {.read = skip_header_and_invalid()});
CHECK(!refused.has_value() || !refused->next().has_value());

std::vector<fw::ColumnSpec> overlap{
{.name = "a", .start = 0, .length = 4, .type = fw::Type::String},
Expand Down
9 changes: 9 additions & 0 deletions cpp/tests/write.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,15 @@ int main()
CHECK(fw::write_stream(out, *table, layout, {.degree_of_parallelism = 4, .batch_rows = 1}).has_value());
CHECK(out.str() == std::string(serial->begin(), serial->end()));

// failbit alone stops a stream's writes too, so it is an error, as badbit is.
std::ostringstream failed;
failed.setstate(std::ios::failbit);
auto refused = fw::write_stream(failed, *table, layout);
CHECK(!refused.has_value() && refused.error().message.find("std::ostream") != std::string::npos);
CHECK(failed.str().empty());
std::ostream unbuffered{nullptr};
CHECK(!fw::write_stream(unbuffered, *table, layout).has_value());

const std::string path = "fw_cpp_write.txt";
CHECK(fw::write_file(path, *table, layout).has_value());
std::FILE* file = std::fopen(path.c_str(), "rb");
Expand Down
2 changes: 1 addition & 1 deletion docs/guide/models.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,4 +115,4 @@ On .NET, Windows-1252 and other code pages need `Encoding.RegisterProvider(CodeP

## Validation

Invalid layouts fail fast with an `InvalidOperationException` from `FixedWidthLayout.Build` or `FromAttributes`, or a generator diagnostic on generated models: negative `Start`, `Length < 1`, overlapping columns, a non-ASCII `Padding` or `TrimChar` (FWP012), an unsupported column type and a converter for the wrong type. Adjacent columns are valid. An attribute model that can be read but not written (or the reverse), such as one with a column type that only formats, gets a layout anyway; the side it cannot do throws on first use.
Invalid layouts fail fast with an `InvalidOperationException` from `FixedWidthLayout.Build` or `FromAttributes`, or a generator diagnostic on generated models: negative `Start`, `Length < 1`, a column ending past `int.MaxValue` (FWP014), overlapping columns, a non-ASCII `Padding` or `TrimChar` (FWP012), an unsupported column type, a converter for the wrong type and, for generated models, a generic type (FWP013). Adjacent columns are valid. An attribute model that can be read but not written (or the reverse), such as one with a column type that only formats, gets a layout anyway; the side it cannot do throws on first use.
3 changes: 2 additions & 1 deletion python/src/fixedwidth/table.py
Original file line number Diff line number Diff line change
Expand Up @@ -106,9 +106,10 @@ def _arrow_column(self, name: str, pa: Any) -> Any:
validity = pa.foreign_buffer(column.validity, (n + 7) // 8, base=self._owner) if column.validity else None
if kind is ColumnType.DECIMAL:
# ponytail: decimals go through Python objects; fw.read_arrow() hands them over natively.
# From a string, exactly: scaleb would round to the caller's decimal context.
mask = self.validity(name)
values = self[name]
decimals = [None if mask is not None and not mask[i] else Decimal(int(v)).scaleb(-column.scale) for i, v in enumerate(values)]
decimals = [None if mask is not None and not mask[i] else Decimal(f"{int(v)}E-{column.scale}") for i, v in enumerate(values)]
return pa.array(decimals, type=pa.decimal128(18, column.scale))
if kind is ColumnType.STRING:
offsets = pa.foreign_buffer(column.values, (n + 1) * 4, base=self._owner)
Expand Down
7 changes: 6 additions & 1 deletion python/src/fixedwidth/writer.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
from __future__ import annotations

import ctypes
import decimal
import io
import operator
import os
Expand Down Expand Up @@ -192,11 +193,15 @@ def _decimals(spec: Column, array: Any, pa: Any, np: Any) -> Any:
return scaled.astype(np.int64)


# Scales without rounding, whatever the caller's decimal context.
_EXACT = decimal.Context(prec=decimal.MAX_PREC)


def _unscaled(value: Any, spec: Column) -> int:
"""A decimal as the integer the column stores; more decimals than its scale would be lost, so they fail."""
if value is None:
return 0
scaled = value.scaleb(spec.scale)
scaled = value.scaleb(spec.scale, _EXACT)
if scaled != scaled.to_integral_value():
raise FixedWidthError(f"value {value} of column '{spec.name}' has more decimals than its scale {spec.scale}")
return _check_int64(int(scaled), value, spec)
Expand Down
13 changes: 13 additions & 0 deletions python/tests/test_arrow.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,19 @@ def test_read_arrow_decimal():
assert str(table.column("v")[0].as_py()) == "123.45"


def test_table_to_arrow_decimal_ignores_the_decimal_context():
import decimal

columns = fw.Layout([fw.Column("v", 0, 20, fw.Decimal, scale=2)])
table = fw.read(b"-0000000123456789012\n", columns)

with decimal.localcontext() as context:
context.prec = 6
arrow = table.to_arrow()

assert str(arrow.column("v")[0].as_py()) == "-1234567890.12"


def test_arrow_stream_to_pyarrow_and_polars(cotahist, options):
reader_options = fw.ReaderOptions(read=options, batch_rows=2)

Expand Down
30 changes: 30 additions & 0 deletions python/tests/test_decimal.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
from __future__ import annotations

import decimal
from decimal import Decimal

import pytest

import fixedwidth as fw
from fixedwidth.writer import _unscaled

MONEY = fw.Column("money", 0, 12, fw.Decimal, scale=2)


@pytest.mark.parametrize("precision", [3, 6, 28])
def test_scaling_ignores_the_decimal_context(precision):
with decimal.localcontext() as context:
context.prec = precision

assert _unscaled(Decimal("12345.67"), MONEY) == 1234567
assert _unscaled(Decimal("-92233720368547758.08"), MONEY) == -(2**63)
with pytest.raises(fw.FixedWidthError, match="12345.678"):
_unscaled(Decimal("12345.678"), MONEY)


def test_scaling_keeps_nulls_zero_and_int64_bounds():
assert _unscaled(None, MONEY) == 0
assert _unscaled(Decimal("0"), MONEY) == 0
assert _unscaled(Decimal("92233720368547758.07"), MONEY) == 2**63 - 1
with pytest.raises(fw.FixedWidthError, match="int64"):
_unscaled(Decimal("92233720368547758.08"), MONEY)
Loading
Loading