diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 5b620f5..adf3676 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -191,7 +191,7 @@ Studio can load it. It is an incremental generator: `Extract` turns each candida change a model's columns does not regenerate it. `Emit` validates the model — partial, top-level, a supported type for each column, layout, converter -type, overflow, ASCII padding — and reports diagnostics `FWP001`–`FWP012` (`FWP007` and `FWP009` are +type, overflow, ASCII padding — and reports diagnostics `FWP001`–`FWP014` (`FWP007` and `FWP009` are retired) instead of generating code for an invalid model. Otherwise it writes `TryParse` and `TryFormat` as straight-line code: one `FixedWidthRuntime` call per column. Each column's settings become two `static readonly` fields, a diff --git a/CHANGELOG.md b/CHANGELOG.md index 4175656..45ee293 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,9 +5,49 @@ All notable changes to this project are documented here. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## [2.0.0] +## [2.0.1] + +### Fixed +- `fw_write_file` replaced an existing file before formatting, so a failed write left it truncated. It + now writes a sibling file and moves it into place only on success. +- `fw_write_*` wrapped `TIME` and `TIMESTAMP` microseconds outside the supported range into a valid + value; they are now an error. +- A `DECIMAL` value whose scaling overflowed `decimal` failed the whole read instead of a parse error + that `skip_invalid` can skip. +- C++ wrapper: `write_stream` reported success to a `std::ostream` with only `failbit` set, and + `Reader::open_stream` failed at the end of a `std::istream` with exceptions enabled. +- Python: `DECIMAL` scaling, written and in `Table.to_arrow`, rounded to the caller's `decimal` + context precision. +- An enum alias (a second name for a value) did not parse. +- A nullable column of spaces and its trim character mixed, such as `" 0 0 "`, was not read as `null`. +- A layout whose column ends past `int.MaxValue` was accepted and threw while reading; it is now + rejected, and the generator reports `FWP014`. +- Generated models with C# keyword names (`@event`) produced code that did not compile, and a generic + model crashed the generator; it now reports `FWP013`. +- `fw_write_*` trusted a `STRING` column's offsets: offsets past its `data_len`, descending or negative, + and a negative `data_len`, are now `FW_INVALID_ARGUMENT` instead of reading past the declared data. +- `fw_write_*` wrote `DECIMAL` values of 19 digits, which reading the output rejects; they are now an + error, as on reading. +- Writing a record that formats as zero bytes with an empty `NewLine`, such as a generated model with + no columns, grew its buffer without end. +- A negative enum value written as a zero-padded number (`-0001`) did not parse back. +- A format provider whose positive sign is not `"+"` was ignored by the fast `double`, `float` and + `decimal` parsers, which accepted `"+1"`. +- A non-UTF-8 encoding was ignored by `decimal` and other UTF-8-parsed columns holding non-ASCII text, + such as a Latin-1 no-break space grouping digits. +- A parallel `AggregateAsync` over a stream kept reading to the end after a failure that decided the + result, such as a line that fails to parse, so a stream that stays open delayed the error until the + caller cancelled. It now stops at the chunk holding the failure. -Not released yet. +### Changed +- A whole-source native read gives back the unused part of a `STRING` column's block before handing it + over: the block was sized for the column's full width on every row, and a table of short text in a wide + column kept all of it (126 MiB instead of 5 MiB for 1,000,000 rows of 2 bytes in 128). +- Parsing a negative enum code written with zero padding (`-0001`) no longer allocates a string. +- Merging the null bitmaps of a parallel native read copies whole bytes where a partition starts on a byte + boundary. + +## [2.0.0] 2.0 reads and writes **bytes only**, where 1.x had a `char` and a UTF-8 version of each reader, parser and writer, and goes through **one entry point**: every `FixedWidth` method comes in an @@ -249,7 +289,8 @@ Initial release. reflection-free `TryParse` generated at compile time, with diagnostics (FWP001-FWP007) for invalid layouts and unsupported column types. -[2.0.0]: https://github.com/GabrielMarquezMatte/FixedWidthParser/compare/v1.1.1...HEAD +[2.0.1]: https://github.com/GabrielMarquezMatte/FixedWidthParser/compare/v2.0.0...HEAD +[2.0.0]: https://github.com/GabrielMarquezMatte/FixedWidthParser/compare/v1.1.1...v2.0.0 [1.1.1]: https://github.com/GabrielMarquezMatte/FixedWidthParser/compare/v1.1.0...v1.1.1 [1.1.0]: https://github.com/GabrielMarquezMatte/FixedWidthParser/compare/v1.0.0...v1.1.0 [1.0.0]: https://github.com/GabrielMarquezMatte/FixedWidthParser/releases/tag/v1.0.0 diff --git a/cpp/tests/read.cpp b/cpp/tests/read.cpp index e0848a6..e53cc16 100644 --- a/cpp/tests/read.cpp +++ b/cpp/tests/read.cpp @@ -19,6 +19,24 @@ static void check_table(const fw::Table& table) CHECK(table.is_valid(2, 1)); } +// The codes of every batch read from `in`, checking each read succeeds. +static std::vector streamed_codes(std::istream& in, const fw::Layout& layout) +{ + auto streamed = fw::Reader::open_stream(in, layout, {.read = skip_header_and_invalid(), .batch_rows = 2}); + CHECK(streamed.has_value()); + std::vector codes; + while (true) { + auto batch = streamed->next(); + CHECK(batch.has_value()); + if (!batch->has_value()) { + return codes; + } + for (int64_t row = 0; row < (*batch)->row_count(); ++row) { + codes.emplace_back((*batch)->string(1, row)); + } + } +} + int main() { auto layout = cotahist_layout(); @@ -63,20 +81,18 @@ int main() // The same batches from a std::istream, read forward without seeking. std::istringstream in{std::string(content)}; - auto streamed = fw::Reader::open_stream(in, layout, {.read = skip_header_and_invalid(), .batch_rows = 2}); - CHECK(streamed.has_value()); - std::vector codes; - while (true) { - auto batch = streamed->next(); - CHECK(batch.has_value()); - if (!batch->has_value()) { - break; - } - for (int64_t row = 0; row < (*batch)->row_count(); ++row) { - codes.emplace_back((*batch)->string(1, row)); - } - } - CHECK((codes == std::vector{"PETR4", "VALE3", "ITUB4"})); + CHECK((streamed_codes(in, layout) == std::vector{"PETR4", "VALE3", "ITUB4"})); + + // A stream with exceptions enabled throws at its end too; the bytes before it still count. + std::istringstream throwing{std::string(content)}; + throwing.exceptions(std::ios::failbit | std::ios::badbit); + CHECK((streamed_codes(throwing, layout) == std::vector{"PETR4", "VALE3", "ITUB4"})); + + // A stream that fails before its end is an error, not an empty file. + std::istringstream failing{std::string(content)}; + failing.setstate(std::ios::failbit); + auto refused = fw::Reader::open_stream(failing, layout, {.read = skip_header_and_invalid()}); + CHECK(!refused.has_value() || !refused->next().has_value()); std::vector overlap{ {.name = "a", .start = 0, .length = 4, .type = fw::Type::String}, diff --git a/cpp/tests/write.cpp b/cpp/tests/write.cpp index a776564..bcbe1de 100644 --- a/cpp/tests/write.cpp +++ b/cpp/tests/write.cpp @@ -27,6 +27,15 @@ int main() CHECK(fw::write_stream(out, *table, layout, {.degree_of_parallelism = 4, .batch_rows = 1}).has_value()); CHECK(out.str() == std::string(serial->begin(), serial->end())); + // failbit alone stops a stream's writes too, so it is an error, as badbit is. + std::ostringstream failed; + failed.setstate(std::ios::failbit); + auto refused = fw::write_stream(failed, *table, layout); + CHECK(!refused.has_value() && refused.error().message.find("std::ostream") != std::string::npos); + CHECK(failed.str().empty()); + std::ostream unbuffered{nullptr}; + CHECK(!fw::write_stream(unbuffered, *table, layout).has_value()); + const std::string path = "fw_cpp_write.txt"; CHECK(fw::write_file(path, *table, layout).has_value()); std::FILE* file = std::fopen(path.c_str(), "rb"); diff --git a/docs/guide/models.md b/docs/guide/models.md index ca322f3..e9c047e 100644 --- a/docs/guide/models.md +++ b/docs/guide/models.md @@ -115,4 +115,4 @@ On .NET, Windows-1252 and other code pages need `Encoding.RegisterProvider(CodeP ## Validation -Invalid layouts fail fast with an `InvalidOperationException` from `FixedWidthLayout.Build` or `FromAttributes`, or a generator diagnostic on generated models: negative `Start`, `Length < 1`, overlapping columns, a non-ASCII `Padding` or `TrimChar` (FWP012), an unsupported column type and a converter for the wrong type. Adjacent columns are valid. An attribute model that can be read but not written (or the reverse), such as one with a column type that only formats, gets a layout anyway; the side it cannot do throws on first use. +Invalid layouts fail fast with an `InvalidOperationException` from `FixedWidthLayout.Build` or `FromAttributes`, or a generator diagnostic on generated models: negative `Start`, `Length < 1`, a column ending past `int.MaxValue` (FWP014), overlapping columns, a non-ASCII `Padding` or `TrimChar` (FWP012), an unsupported column type, a converter for the wrong type and, for generated models, a generic type (FWP013). Adjacent columns are valid. An attribute model that can be read but not written (or the reverse), such as one with a column type that only formats, gets a layout anyway; the side it cannot do throws on first use. diff --git a/python/src/fixedwidth/table.py b/python/src/fixedwidth/table.py index fc36c65..9d56e81 100644 --- a/python/src/fixedwidth/table.py +++ b/python/src/fixedwidth/table.py @@ -106,9 +106,10 @@ def _arrow_column(self, name: str, pa: Any) -> Any: validity = pa.foreign_buffer(column.validity, (n + 7) // 8, base=self._owner) if column.validity else None if kind is ColumnType.DECIMAL: # ponytail: decimals go through Python objects; fw.read_arrow() hands them over natively. + # From a string, exactly: scaleb would round to the caller's decimal context. mask = self.validity(name) values = self[name] - decimals = [None if mask is not None and not mask[i] else Decimal(int(v)).scaleb(-column.scale) for i, v in enumerate(values)] + decimals = [None if mask is not None and not mask[i] else Decimal(f"{int(v)}E-{column.scale}") for i, v in enumerate(values)] return pa.array(decimals, type=pa.decimal128(18, column.scale)) if kind is ColumnType.STRING: offsets = pa.foreign_buffer(column.values, (n + 1) * 4, base=self._owner) diff --git a/python/src/fixedwidth/writer.py b/python/src/fixedwidth/writer.py index 31678a2..88fdb09 100644 --- a/python/src/fixedwidth/writer.py +++ b/python/src/fixedwidth/writer.py @@ -3,6 +3,7 @@ from __future__ import annotations import ctypes +import decimal import io import operator import os @@ -192,11 +193,15 @@ def _decimals(spec: Column, array: Any, pa: Any, np: Any) -> Any: return scaled.astype(np.int64) +# Scales without rounding, whatever the caller's decimal context. +_EXACT = decimal.Context(prec=decimal.MAX_PREC) + + def _unscaled(value: Any, spec: Column) -> int: """A decimal as the integer the column stores; more decimals than its scale would be lost, so they fail.""" if value is None: return 0 - scaled = value.scaleb(spec.scale) + scaled = value.scaleb(spec.scale, _EXACT) if scaled != scaled.to_integral_value(): raise FixedWidthError(f"value {value} of column '{spec.name}' has more decimals than its scale {spec.scale}") return _check_int64(int(scaled), value, spec) diff --git a/python/tests/test_arrow.py b/python/tests/test_arrow.py index 9e711e4..7ff92cf 100644 --- a/python/tests/test_arrow.py +++ b/python/tests/test_arrow.py @@ -31,6 +31,19 @@ def test_read_arrow_decimal(): assert str(table.column("v")[0].as_py()) == "123.45" +def test_table_to_arrow_decimal_ignores_the_decimal_context(): + import decimal + + columns = fw.Layout([fw.Column("v", 0, 20, fw.Decimal, scale=2)]) + table = fw.read(b"-0000000123456789012\n", columns) + + with decimal.localcontext() as context: + context.prec = 6 + arrow = table.to_arrow() + + assert str(arrow.column("v")[0].as_py()) == "-1234567890.12" + + def test_arrow_stream_to_pyarrow_and_polars(cotahist, options): reader_options = fw.ReaderOptions(read=options, batch_rows=2) diff --git a/python/tests/test_decimal.py b/python/tests/test_decimal.py new file mode 100644 index 0000000..25cef7f --- /dev/null +++ b/python/tests/test_decimal.py @@ -0,0 +1,30 @@ +from __future__ import annotations + +import decimal +from decimal import Decimal + +import pytest + +import fixedwidth as fw +from fixedwidth.writer import _unscaled + +MONEY = fw.Column("money", 0, 12, fw.Decimal, scale=2) + + +@pytest.mark.parametrize("precision", [3, 6, 28]) +def test_scaling_ignores_the_decimal_context(precision): + with decimal.localcontext() as context: + context.prec = precision + + assert _unscaled(Decimal("12345.67"), MONEY) == 1234567 + assert _unscaled(Decimal("-92233720368547758.08"), MONEY) == -(2**63) + with pytest.raises(fw.FixedWidthError, match="12345.678"): + _unscaled(Decimal("12345.678"), MONEY) + + +def test_scaling_keeps_nulls_zero_and_int64_bounds(): + assert _unscaled(None, MONEY) == 0 + assert _unscaled(Decimal("0"), MONEY) == 0 + assert _unscaled(Decimal("92233720368547758.07"), MONEY) == 2**63 - 1 + with pytest.raises(fw.FixedWidthError, match="int64"): + _unscaled(Decimal("92233720368547758.08"), MONEY) diff --git a/src/FixedWidthParser.Generator/FixedWidthModelGenerator.cs b/src/FixedWidthParser.Generator/FixedWidthModelGenerator.cs index 5876018..d2deaf0 100644 --- a/src/FixedWidthParser.Generator/FixedWidthModelGenerator.cs +++ b/src/FixedWidthParser.Generator/FixedWidthModelGenerator.cs @@ -74,6 +74,20 @@ public sealed class FixedWidthModelGenerator : IIncrementalGenerator "Column '{0}' in '{1}' sets {2} to '{3}', which is not an ASCII character; column positions are bytes, so it must be one byte in every supported encoding", "FixedWidthParser", DiagnosticSeverity.Error, isEnabledByDefault: true); + private static readonly DiagnosticDescriptor MustNotBeGeneric = new( + "FWP013", "Fixed-width model must not be generic", + "Type '{0}' is generic; generated fixed-width parsing currently supports only non-generic types", + "FixedWidthParser", DiagnosticSeverity.Error, isEnabledByDefault: true); + + private static readonly DiagnosticDescriptor ColumnEndOverflow = new( + "FWP014", "Fixed-width column ends past int.MaxValue", + "Column '{0}' in '{1}' starts at {2} with Length {3}, so it ends past int.MaxValue", + "FixedWidthParser", DiagnosticSeverity.Error, isEnabledByDefault: true); + + // The namespace as C# source: dotted, without 'global::', keywords escaped. + private static readonly SymbolDisplayFormat NamespaceFormat = + SymbolDisplayFormat.FullyQualifiedFormat.WithGlobalNamespaceStyle(SymbolDisplayGlobalNamespaceStyle.Omitted); + public void Initialize(IncrementalGeneratorInitializationContext context) { var models = context.SyntaxProvider.CreateSyntaxProvider( @@ -123,7 +137,7 @@ public void Initialize(IncrementalGeneratorInitializationContext context) string? ns = symbol.ContainingNamespace.IsGlobalNamespace ? null - : symbol.ContainingNamespace.ToDisplayString(); + : symbol.ContainingNamespace.ToDisplayString(NamespaceFormat); return new ModelInfo( ns, @@ -133,6 +147,7 @@ public void Initialize(IncrementalGeneratorInitializationContext context) new EquatableArray(columns.ToImmutable()), declaration.Modifiers.Any(m => m.IsKind(SyntaxKind.PartialKeyword)), symbol.ContainingType is not null, + symbol.TypeParameters.Length > 0, declaration.Identifier.GetLocation()); } @@ -366,6 +381,12 @@ private static void Emit(SourceProductionContext spc, ModelInfo model) spc.ReportDiagnostic(Diagnostic.Create(MustBePartial, model.Location, model.TypeName)); return; } + // ponytail: rejected rather than emitted; supporting them means carrying type parameters and constraints. + if (model.IsGeneric) + { + spc.ReportDiagnostic(Diagnostic.Create(MustNotBeGeneric, model.Location, model.TypeName)); + return; + } // The interface needs both methods, so any column error blocks the whole emission. bool error = false; @@ -380,7 +401,8 @@ private static void Emit(SourceProductionContext spc, ModelInfo model) return; } - spc.AddSource(model.FullyQualifiedName.Replace("global::", string.Empty) + ".FixedWidth.g.cs", BuildSource(model)); + // A hint name takes no '@', which escapes a keyword in the qualified name. + spc.AddSource(model.FullyQualifiedName.Replace("global::", string.Empty).Replace("@", string.Empty) + ".FixedWidth.g.cs", BuildSource(model)); } private static bool ReportColumnErrors(SourceProductionContext spc, ModelInfo model, ColumnInfo column) @@ -427,6 +449,10 @@ void Report(DiagnosticDescriptor descriptor, params object?[] args) { Report(InvalidLength, column.Name, model.TypeName, column.Length); } + else if (column.Start > int.MaxValue - column.Length) + { + Report(ColumnEndOverflow, column.Name, model.TypeName, column.Start, column.Length); + } return error; } @@ -481,7 +507,7 @@ private static string BuildSource(ModelInfo model) indent = string.Empty; } - sb.Append(indent).Append(model.TypeKeywords).Append(' ').Append(model.TypeName).AppendLine(); + sb.Append(indent).Append(model.TypeKeywords).Append(' ').Append(Identifier(model.TypeName)).AppendLine(); sb.Append(indent).AppendLine("{"); string body = indent + " "; @@ -577,7 +603,7 @@ private static void AppendParseMethod(StringBuilder sb, string body, ModelInfo m sb.Append(stmt).AppendLine("{"); for (int i = 0; i < columns.Length; i++) { - sb.Append(stmt).Append(" ").Append(columns[i].Name).Append(" = __v").Append(i).AppendLine(","); + sb.Append(stmt).Append(" ").Append(Identifier(columns[i].Name)).Append(" = __v").Append(i).AppendLine(","); } sb.Append(stmt).AppendLine("};"); sb.Append(stmt).AppendLine("return true;"); @@ -662,11 +688,11 @@ private static void AppendColumnFormat(StringBuilder sb, string stmt, ColumnInfo string slice = "__line.Slice(" + c.Start + ", " + c.Length + ")"; if (!c.IsNullable) { - AppendColumnKindFormat(sb, stmt, c, slice, i, "model." + c.Name); + AppendColumnKindFormat(sb, stmt, c, slice, i, "model." + Identifier(c.Name)); return; } - sb.Append(stmt).Append("if (model.").Append(c.Name).Append(" is { } __u").Append(i).AppendLine(")"); + sb.Append(stmt).Append("if (model.").Append(Identifier(c.Name)).Append(" is { } __u").Append(i).AppendLine(")"); sb.Append(stmt).AppendLine("{"); AppendColumnKindFormat(sb, stmt + " ", c, slice, i, "__u" + i); sb.Append(stmt).AppendLine("}"); @@ -700,6 +726,12 @@ private static void AppendColumnKindFormat(StringBuilder sb, string stmt, Column } } + // A symbol's name as C# source: a keyword, such as a member declared as @event, needs its '@' back. + private static string Identifier(string name) + { + return SyntaxFacts.GetKeywordKind(name) == SyntaxKind.None ? name : "@" + name; + } + private static string ByteLiteral(char value) { return "(byte)" + SymbolDisplay.FormatLiteral(value, true); @@ -742,8 +774,9 @@ private readonly record struct ColumnInfo( WriteKind WKind, int Alignment, char Padding, string? Format, int Overflow, char TrimChar, int TrimMode); private sealed class ModelInfo(string? ns, string typeKeywords, string typeName, string fullyQualifiedName, - EquatableArray columns, bool isPartial, bool isNested, Location location) : IEquatable + EquatableArray columns, bool isPartial, bool isNested, bool isGeneric, Location location) : IEquatable { + public bool IsGeneric { get; } = isGeneric; public string? Namespace { get; } = ns; public string TypeKeywords { get; } = typeKeywords; public string TypeName { get; } = typeName; @@ -761,6 +794,7 @@ public bool Equals(ModelInfo? other) return other is not null && IsPartial == other.IsPartial && IsNested == other.IsNested + && IsGeneric == other.IsGeneric && string.Equals(Namespace, other.Namespace, StringComparison.Ordinal) && string.Equals(TypeKeywords, other.TypeKeywords, StringComparison.Ordinal) && string.Equals(TypeName, other.TypeName, StringComparison.Ordinal) @@ -787,6 +821,7 @@ public override int GetHashCode() hash = hash * 31 + Columns.GetHashCode(); hash = hash * 31 + (IsPartial ? 1 : 0); hash = hash * 31 + (IsNested ? 1 : 0); + hash = hash * 31 + (IsGeneric ? 1 : 0); return hash; } } diff --git a/src/FixedWidthParser.Generator/FixedWidthParser.Generator.csproj b/src/FixedWidthParser.Generator/FixedWidthParser.Generator.csproj index c90cb18..31e9f6a 100644 --- a/src/FixedWidthParser.Generator/FixedWidthParser.Generator.csproj +++ b/src/FixedWidthParser.Generator/FixedWidthParser.Generator.csproj @@ -22,7 +22,7 @@ - + diff --git a/src/FixedWidthParser.Native/Exports.Arrow.cs b/src/FixedWidthParser.Native/Exports.Arrow.cs index 3a93d0b..5fcf650 100644 --- a/src/FixedWidthParser.Native/Exports.Arrow.cs +++ b/src/FixedWidthParser.Native/Exports.Arrow.cs @@ -11,7 +11,7 @@ public static int ReadArrowFile(byte* path, int pathLength, nint layout, NativeR { if (path is null || pathLength <= 0 || outArray is null || outSchema is null) { - return InvalidArgument("fw_read_arrow_file needs a path, out_array and out_schema."); + return NativeApi.InvalidArgument("fw_read_arrow_file needs a path, out_array and out_schema."); } *outArray = default; *outSchema = default; @@ -23,7 +23,7 @@ public static int ReadArrowMemory(byte* data, long dataLength, nint layout, Nati { if ((data is null && dataLength != 0) || dataLength < 0 || outArray is null || outSchema is null) { - return InvalidArgument("fw_read_arrow_memory needs data for a non-zero data_len, out_array and out_schema."); + return NativeApi.InvalidArgument("fw_read_arrow_memory needs data for a non-zero data_len, out_array and out_schema."); } *outArray = default; *outSchema = default; @@ -35,7 +35,7 @@ public static int ReadArrowStreamFile(byte* path, int pathLength, nint layout, N { if (path is null || pathLength <= 0 || outStream is null) { - return InvalidArgument("fw_read_arrow_stream_file needs a path and out_stream."); + return NativeApi.InvalidArgument("fw_read_arrow_stream_file needs a path and out_stream."); } *outStream = default; return WithLayout(layout, l => ArrowApi.OpenStreamFile(new ReadOnlySpan(path, pathLength), l, options, outStream)); @@ -46,7 +46,7 @@ public static int ReadArrowStreamMemory(byte* data, long dataLength, nint layout { if ((data is null && dataLength != 0) || dataLength < 0 || outStream is null) { - return InvalidArgument("fw_read_arrow_stream_memory needs data for a non-zero data_len, and out_stream."); + return NativeApi.InvalidArgument("fw_read_arrow_stream_memory needs data for a non-zero data_len, and out_stream."); } *outStream = default; return WithLayout(layout, l => ArrowApi.OpenStreamMemory(data, dataLength, l, options, outStream)); @@ -57,7 +57,7 @@ public static int ReadArrowStreamStream(NativeStreamRaw* stream, nint layout, Na { if (outStream is null) { - return InvalidArgument("fw_read_arrow_stream_stream needs out_stream."); + return NativeApi.InvalidArgument("fw_read_arrow_stream_stream needs out_stream."); } *outStream = default; #pragma warning disable CA2000 // The reader takes the stream over, or it is disposed below. diff --git a/src/FixedWidthParser.Native/Exports.Layout.cs b/src/FixedWidthParser.Native/Exports.Layout.cs index be29098..982d0fe 100644 --- a/src/FixedWidthParser.Native/Exports.Layout.cs +++ b/src/FixedWidthParser.Native/Exports.Layout.cs @@ -10,12 +10,12 @@ public static int LayoutCreate(NativeColumnSpecRaw* columns, int columnCount, Na { if (outLayout is null || columns is null || columnCount <= 0) { - return InvalidArgument("fw_layout_create needs columns, a positive column_count and out_layout."); + return NativeApi.InvalidArgument("fw_layout_create needs columns, a positive column_count and out_layout."); } *outLayout = 0; if (columns->StructSize != sizeof(NativeColumnSpecRaw)) { - return InvalidArgument($"fw_column_spec struct_size is {columns->StructSize}; expected {sizeof(NativeColumnSpecRaw)}."); + return NativeApi.InvalidArgument($"fw_column_spec struct_size is {columns->StructSize}; expected {sizeof(NativeColumnSpecRaw)}."); } try { @@ -38,11 +38,5 @@ public static void LayoutFree(nint layout) { NativeHandleTable.TryUnregister(layout, out _); } - - private static int InvalidArgument(string message) - { - NativeApi.SetLastError(message); - return NativeStatus.InvalidArgument; - } } } diff --git a/src/FixedWidthParser.Native/Exports.Reader.cs b/src/FixedWidthParser.Native/Exports.Reader.cs index 6a12dc3..7c2003d 100644 --- a/src/FixedWidthParser.Native/Exports.Reader.cs +++ b/src/FixedWidthParser.Native/Exports.Reader.cs @@ -10,7 +10,7 @@ public static int ReaderOpenFile(byte* path, int pathLength, nint layout, Native { if (path is null || pathLength <= 0 || outReader is null) { - return InvalidArgument("fw_reader_open_file needs a path and out_reader."); + return NativeApi.InvalidArgument("fw_reader_open_file needs a path and out_reader."); } *outReader = 0; return WithLayout(layout, l => @@ -26,7 +26,7 @@ public static int ReaderOpenMemory(byte* data, long dataLength, nint layout, Nat { if ((data is null && dataLength != 0) || dataLength < 0 || outReader is null) { - return InvalidArgument("fw_reader_open_memory needs data for a non-zero data_len, and out_reader."); + return NativeApi.InvalidArgument("fw_reader_open_memory needs data for a non-zero data_len, and out_reader."); } *outReader = 0; return WithLayout(layout, l => @@ -42,7 +42,7 @@ public static int ReaderOpenStream(NativeStreamRaw* stream, nint layout, NativeR { if (outReader is null) { - return InvalidArgument("fw_reader_open_stream needs out_reader."); + return NativeApi.InvalidArgument("fw_reader_open_stream needs out_reader."); } *outReader = 0; #pragma warning disable CA2000 // The reader takes the stream over, or it is disposed below. @@ -76,7 +76,7 @@ public static int ReaderNext(nint reader, NativeTableRaw* outTable) { if (outTable is null) { - return InvalidArgument("fw_reader_next needs out_table."); + return NativeApi.InvalidArgument("fw_reader_next needs out_table."); } return ReaderApi.Next(reader, outTable); } diff --git a/src/FixedWidthParser.Native/Exports.Reading.cs b/src/FixedWidthParser.Native/Exports.Reading.cs index ece4cc3..fd96ca6 100644 --- a/src/FixedWidthParser.Native/Exports.Reading.cs +++ b/src/FixedWidthParser.Native/Exports.Reading.cs @@ -11,7 +11,7 @@ public static int ReadFile(byte* path, int pathLength, nint layout, NativeReadOp { if (path is null || pathLength <= 0 || outTable is null) { - return InvalidArgument("fw_read_file needs a path and out_table."); + return NativeApi.InvalidArgument("fw_read_file needs a path and out_table."); } *outTable = default; return WithLayout(layout, l => ReadApi.ReadFile(new ReadOnlySpan(path, pathLength), l, options, outTable, outResult)); @@ -22,7 +22,7 @@ public static int ReadMemory(byte* data, long dataLength, nint layout, NativeRea { if ((data is null && dataLength != 0) || dataLength < 0 || outTable is null) { - return InvalidArgument("fw_read_memory needs data for a non-zero data_len, and out_table."); + return NativeApi.InvalidArgument("fw_read_memory needs data for a non-zero data_len, and out_table."); } *outTable = default; return WithLayout(layout, l => ReadApi.ReadMemory(data, dataLength, l, options, outTable, outResult)); diff --git a/src/FixedWidthParser.Native/Exports.Writer.cs b/src/FixedWidthParser.Native/Exports.Writer.cs index aa03a60..493c8a2 100644 --- a/src/FixedWidthParser.Native/Exports.Writer.cs +++ b/src/FixedWidthParser.Native/Exports.Writer.cs @@ -11,7 +11,7 @@ public static int WriteFile(byte* path, int pathLength, NativeTableRaw* table, n { if (path is null || pathLength <= 0 || table is null) { - return InvalidArgument("fw_write_file needs a path and a table."); + return NativeApi.InvalidArgument("fw_write_file needs a path and a table."); } return WithLayout(layout, l => WriteApi.WriteFile(new ReadOnlySpan(path, pathLength), table, l, options)); } @@ -21,7 +21,7 @@ public static int WriteMemory(NativeTableRaw* table, nint layout, NativeWriteOpt { if (table is null || outBuffer is null) { - return InvalidArgument("fw_write_memory needs a table and out_buffer."); + return NativeApi.InvalidArgument("fw_write_memory needs a table and out_buffer."); } *outBuffer = default; return WithLayout(layout, l => WriteApi.WriteMemory(table, l, options, outBuffer)); @@ -32,7 +32,7 @@ public static int WriteStream(NativeTableRaw* table, nint layout, NativeWriteOpt { if (table is null) { - return InvalidArgument("fw_write_stream needs a table."); + return NativeApi.InvalidArgument("fw_write_stream needs a table."); } #pragma warning disable CA2000 // Disposed by the using below; the sink has nothing to release. if (!CallbackWriteStream.TryCreate(sink, out var created)) diff --git a/src/FixedWidthParser.Native/GrowableNativeBuffer.cs b/src/FixedWidthParser.Native/GrowableNativeBuffer.cs index e6a6b2e..f2440b6 100644 --- a/src/FixedWidthParser.Native/GrowableNativeBuffer.cs +++ b/src/FixedWidthParser.Native/GrowableNativeBuffer.cs @@ -86,6 +86,33 @@ internal void AddRange(ReadOnlySpan values) Count += values.Length; } + /// Elements the buffer can hold before it grows. + internal int Capacity => _capacity; + + /// + /// Gives back the room past elements when it is at least half of an owned block, so a + /// block reserved for more than it got does not stay that large for as long as its owner keeps it. Memory lent + /// by is left alone, and so is the block if the allocator cannot shrink it. + /// + internal void TrimExcess(int length) + { + length = Math.Max(length, 1); + if (_block is null || length > _capacity / 2) + { + return; + } + try + { + _data = (T*)_block.Resize(checked((nint)length * sizeof(T))); + _capacity = length; + Count = Math.Min(Count, length); + } + catch (OutOfMemoryException) + { + // ReAllocHGlobal leaves the block as it was. + } + } + /// Hands the block to the caller, who frees it with . internal IntPtr Detach() { diff --git a/src/FixedWidthParser.Native/Layouts/NativeColumnSpec.cs b/src/FixedWidthParser.Native/Layouts/NativeColumnSpec.cs index 27e0d01..bd8a1b2 100644 --- a/src/FixedWidthParser.Native/Layouts/NativeColumnSpec.cs +++ b/src/FixedWidthParser.Native/Layouts/NativeColumnSpec.cs @@ -15,6 +15,9 @@ internal static class NativeType internal const int MaxScale = 18; + /// A DECIMAL's unscaled value has at most 18 digits, so it stays below this in magnitude. + internal const long DecimalLimit = 1_000_000_000_000_000_000; + /// The day DATE values count from, as Arrow's date32 does. internal static readonly int EpochDay = new DateOnly(1970, 1, 1).DayNumber; diff --git a/src/FixedWidthParser.Native/NativeOptions.cs b/src/FixedWidthParser.Native/NativeOptions.cs index 9610640..fb66f27 100644 --- a/src/FixedWidthParser.Native/NativeOptions.cs +++ b/src/FixedWidthParser.Native/NativeOptions.cs @@ -16,8 +16,7 @@ internal static int TryRead(T* options, string name, out T value) where T : u int size = *(int*)options; if (size != sizeof(T)) { - NativeApi.SetLastError($"{name} struct_size is {size}; expected {sizeof(T)}."); - return NativeStatus.InvalidArgument; + return NativeApi.InvalidArgument($"{name} struct_size is {size}; expected {sizeof(T)}."); } value = *options; return NativeStatus.Ok; diff --git a/src/FixedWidthParser.Native/Reading/ColumnBuilder.cs b/src/FixedWidthParser.Native/Reading/ColumnBuilder.cs index c0168a8..a903e59 100644 --- a/src/FixedWidthParser.Native/Reading/ColumnBuilder.cs +++ b/src/FixedWidthParser.Native/Reading/ColumnBuilder.cs @@ -16,7 +16,6 @@ namespace FixedWidthParser.Native.Reading internal sealed unsafe class ColumnBuilder : IDisposable { private const int StackChars = 256; - private const long DecimalLimit = 1_000_000_000_000_000_000; private const DateTimeStyles Utc = DateTimeStyles.AdjustToUniversal | DateTimeStyles.AssumeUniversal; private readonly NativeColumn _column; @@ -270,7 +269,14 @@ private static void ConcatStrings(ReadOnlySpan parts, long rows, offsets = (int*)lentOffsets.Detach(); } target->Values = offsets is null ? (void*)parts[0]._ints.Detach() : offsets; - target->Data = lent is null ? data : (byte*)lent.Detach(); + if (lent is not null) + { + // The arena lent every row the column's full width; trimmed text is often far shorter, and the table + // would keep the difference for its lifetime. Every partition has moved in by now. + lent.TrimExcess((int)total); + data = (byte*)lent.Detach(); + } + target->Data = data; target->DataLen = total; } @@ -307,7 +313,15 @@ private static void MoveTo(ReadOnlySpan bytes, byte* to) foreach (ref readonly var part in parts) { var partBits = part._validity is null ? default : part._validity.Span; - for (int r = 0; r < part.Rows && !partBits.IsEmpty; r++) + int r = 0; + // A partition starting on a byte boundary copies its whole bytes; its last few rows, and a partition + // starting mid-byte, go bit by bit, so no padding bit lands on another partition's rows. + if (!partBits.IsEmpty && (row & 7) == 0) + { + r = part.Rows & ~7; + partBits[..(r >> 3)].CopyTo(new Span(bits + (row >> 3), r >> 3)); + } + for (; r < part.Rows && !partBits.IsEmpty; r++) { if ((partBits[r >> 3] & (1 << (r & 7))) == 0) { @@ -365,14 +379,20 @@ private static bool TryDecimal(ReadOnlySpan text, int scale, out long unsc { if (TryInteger(text, out unscaled)) { - return unscaled is > -DecimalLimit and < DecimalLimit; + return unscaled is > -NativeType.DecimalLimit and < NativeType.DecimalLimit; } if (!decimal.TryParse(text, NumberStyles.Number, CultureInfo.InvariantCulture, out decimal value)) { return false; } - decimal scaled = value * (decimal)NativeType.Pow10[scale]; - if (scaled != decimal.Truncate(scaled) || scaled <= -DecimalLimit || scaled >= DecimalLimit) + // Bounded before scaling: a value near decimal's range would overflow the product and throw, not fail the parse. + decimal factor = (decimal)NativeType.Pow10[scale]; + if (Math.Abs(value) >= NativeType.DecimalLimit / factor) + { + return false; + } + decimal scaled = value * factor; + if (scaled != decimal.Truncate(scaled)) { return false; } diff --git a/src/FixedWidthParser.Native/Reading/ReadApi.cs b/src/FixedWidthParser.Native/Reading/ReadApi.cs index 8595308..8d4d57d 100644 --- a/src/FixedWidthParser.Native/Reading/ReadApi.cs +++ b/src/FixedWidthParser.Native/Reading/ReadApi.cs @@ -75,8 +75,7 @@ private static int CheckResult(NativeReadResultRaw* result) { if (result is not null && result->StructSize != sizeof(NativeReadResultRaw)) { - NativeApi.SetLastError($"fw_read_result struct_size is {result->StructSize}; expected {sizeof(NativeReadResultRaw)}."); - return NativeStatus.InvalidArgument; + return NativeApi.InvalidArgument($"fw_read_result struct_size is {result->StructSize}; expected {sizeof(NativeReadResultRaw)}."); } return NativeStatus.Ok; } diff --git a/src/FixedWidthParser.Native/Reading/ReadSettings.cs b/src/FixedWidthParser.Native/Reading/ReadSettings.cs index 090f448..842fe55 100644 --- a/src/FixedWidthParser.Native/Reading/ReadSettings.cs +++ b/src/FixedWidthParser.Native/Reading/ReadSettings.cs @@ -37,13 +37,11 @@ internal static unsafe int TryCreate(NativeReadOptionsRaw* options, out ReadSett } if (raw.HeaderLines < 0 || raw.DegreeOfParallelism < 0 || raw.MaxLineLength < 0 || raw.SkipByteOrderMark is < 0 or > OptTrue) { - NativeApi.SetLastError("fw_read_options: header_lines, degree_of_parallelism and max_line_length must not be negative, and skip_byte_order_mark must be an FW_OPT_* value."); - return NativeStatus.InvalidArgument; + return NativeApi.InvalidArgument("fw_read_options: header_lines, degree_of_parallelism and max_line_length must not be negative, and skip_byte_order_mark must be an FW_OPT_* value."); } if (!TryEncoding(raw.Encoding, out var encoding)) { - NativeApi.SetLastError($"fw_read_options: encoding {raw.Encoding} is not a code page this library knows."); - return NativeStatus.InvalidArgument; + return NativeApi.InvalidArgument($"fw_read_options: encoding {raw.Encoding} is not a code page this library knows."); } settings = new ReadSettings(raw, encoding); return NativeStatus.Ok; diff --git a/src/FixedWidthParser.Native/Reading/ReaderApi.cs b/src/FixedWidthParser.Native/Reading/ReaderApi.cs index c954ede..8e95179 100644 --- a/src/FixedWidthParser.Native/Reading/ReaderApi.cs +++ b/src/FixedWidthParser.Native/Reading/ReaderApi.cs @@ -177,8 +177,7 @@ private static int TrySettings(NativeReaderOptionsRaw* options, out ReadSettings } if (raw.BatchRows is < 0 or > int.MaxValue || (raw.Read.StructSize != 0 && raw.Read.StructSize != sizeof(NativeReadOptionsRaw))) { - NativeApi.SetLastError("fw_reader_options: batch_rows must be 0..2147483647, and read.struct_size must be 0 or sizeof(fw_read_options)."); - return NativeStatus.InvalidArgument; + return NativeApi.InvalidArgument("fw_reader_options: batch_rows must be 0..2147483647, and read.struct_size must be 0 or sizeof(fw_read_options)."); } batchRows = raw.BatchRows; var read = raw.Read with { StructSize = sizeof(NativeReadOptionsRaw) }; diff --git a/src/FixedWidthParser.Native/Writing/TableFormatter.cs b/src/FixedWidthParser.Native/Writing/TableFormatter.cs index 297664f..02ec457 100644 --- a/src/FixedWidthParser.Native/Writing/TableFormatter.cs +++ b/src/FixedWidthParser.Native/Writing/TableFormatter.cs @@ -34,8 +34,7 @@ internal static int TryCreate(NativeTableRaw* table, NativeLayout layout, Encodi string? error = Check(table, layout); if (error is not null) { - NativeApi.SetLastError(error); - return NativeStatus.InvalidArgument; + return NativeApi.InvalidArgument(error); } formatter = new TableFormatter(layout, table, encoding); return NativeStatus.Ok; @@ -56,14 +55,40 @@ internal static int TryCreate(NativeTableRaw* table, NativeLayout layout, Encodi { return $"Column '{column.Name}' has type {raw.Type} (scale {raw.Scale}) in the table and {column.Type} (scale {column.Scale}) in the layout."; } - if (raw.Length != table->RowCount || (raw.Values is null && raw.Length > 0) || (raw.Data is null && raw.DataLen > 0)) + if (raw.Length != table->RowCount || (raw.Values is null && raw.Length > 0) || (raw.Data is null && raw.DataLen > 0) || raw.DataLen < 0) { return $"Column '{column.Name}' has {raw.Length} values for {table->RowCount} rows, or a buffer that does not match its length."; } + if (column.Type == NativeType.String && !OffsetsWithinData(raw)) + { + return $"Column '{column.Name}' has string offsets that descend or fall outside its {raw.DataLen} data bytes."; + } } return null; } + /// True when a string column's offsets ascend within 0..DataLen, so no row reads past its data. + private static bool OffsetsWithinData(NativeColumnRaw raw) + { + if (raw.Length == 0) + { + return true; + } + var offsets = (int*)raw.Values; + if (offsets[0] < 0) + { + return false; + } + for (long row = 0; row < raw.Length; row++) + { + if (offsets[row + 1] < offsets[row]) + { + return false; + } + } + return offsets[raw.Length] <= raw.DataLen; + } + /// Writes row into , whose length is the layout's line length. internal void FormatRow(long row, Span line) { @@ -98,18 +123,39 @@ private void FormatValue(NativeColumn column, NativeColumnRaw* raw, long row, Sp FixedWidthRuntime.FormatUtf8Value(DateOnly.FromDayNumber(NativeType.EpochDay + ((int*)raw->Values)[row]), slice, in _context, options, column.Name); return; case NativeType.Time: - FixedWidthRuntime.FormatUtf8Value(new TimeOnly(((long*)raw->Values)[row] * TimeSpan.TicksPerMicrosecond), slice, in _context, options, column.Name); + FixedWidthRuntime.FormatUtf8Value(new TimeOnly(Ticks(column, raw, row, 0, TimeSpan.TicksPerDay - 1)), slice, in _context, options, column.Name); return; case NativeType.Timestamp: - var ticks = DateTime.UnixEpoch.Ticks + (((long*)raw->Values)[row] * TimeSpan.TicksPerMicrosecond); + long epoch = DateTime.UnixEpoch.Ticks; + var ticks = epoch + Ticks(column, raw, row, DateTime.MinValue.Ticks - epoch, DateTime.MaxValue.Ticks - epoch); FixedWidthRuntime.FormatUtf8Value(new DateTime(ticks, DateTimeKind.Unspecified), slice, in _context, options, column.Name); return; + case NativeType.Decimal: + long unscaled = ((long*)raw->Values)[row]; + // The reader takes at most 18 digits; writing more would make a line it cannot read back. + if (unscaled is <= -NativeType.DecimalLimit or >= NativeType.DecimalLimit) + { + throw new InvalidOperationException($"Value {unscaled} of column \"{column.Name}\" has more than 18 digits, which a DECIMAL cannot read back."); + } + FixedWidthRuntime.FormatUtf8Value(unscaled, slice, in _context, options, column.Name); + return; default: FixedWidthRuntime.FormatUtf8Value(((long*)raw->Values)[row], slice, in _context, options, column.Name); return; } } + /// The row's microseconds as ticks, checked against .. before scaling so it cannot wrap. + private static long Ticks(NativeColumn column, NativeColumnRaw* raw, long row, long min, long max) + { + long micros = ((long*)raw->Values)[row]; + if (micros < min / TimeSpan.TicksPerMicrosecond || micros > max / TimeSpan.TicksPerMicrosecond) + { + throw new InvalidOperationException($"Value {micros} of column \"{column.Name}\" is out of range in microseconds."); + } + return micros * TimeSpan.TicksPerMicrosecond; + } + private void FormatFloat(NativeColumn column, double value, Span slice) { if (column.Scale == 0) diff --git a/src/FixedWidthParser.Native/Writing/WriteApi.cs b/src/FixedWidthParser.Native/Writing/WriteApi.cs index 351b13c..d481cb9 100644 --- a/src/FixedWidthParser.Native/Writing/WriteApi.cs +++ b/src/FixedWidthParser.Native/Writing/WriteApi.cs @@ -44,20 +44,38 @@ internal static int WriteFile(ReadOnlySpan path, NativeTableRaw* table, Na { return status; } + string target = Encoding.UTF8.GetString(path); + // Writes a sibling, then moves it over the target: a failed write leaves an existing file as it was. Its + // name is short and independent of the target's, which may already be as long as a name can be. + string temp = Path.Combine(Path.GetDirectoryName(target) ?? string.Empty, Path.GetRandomFileName() + ".tmp"); try { long total = formatter!.Rows * lineBytes; // Asynchronous, though every write waits: Windows runs one synchronous I/O at a time on a handle, // which would serialize the parallel batches. - using var handle = File.OpenHandle(Encoding.UTF8.GetString(path), FileMode.Create, FileAccess.Write, FileShare.None, FileOptions.Asynchronous, total); - RandomAccess.SetLength(handle, total); - return Run(formatter, settings!, lineBytes, (first, count) => WriteBatch(handle, formatter, settings!, layout, first, count, lineBytes)); + using (var handle = File.OpenHandle(temp, FileMode.CreateNew, FileAccess.Write, FileShare.None, FileOptions.Asynchronous, total)) + { + RandomAccess.SetLength(handle, total); + status = Run(formatter, settings!, lineBytes, (first, count) => WriteBatch(handle, formatter, settings!, layout, first, count, lineBytes)); + } + if (status == NativeStatus.Ok) + { + File.Move(temp, target, overwrite: true); + } + return status; } catch (Exception ex) when (ex is IOException or UnauthorizedAccessException or ArgumentException) { NativeApi.SetLastError(ex.Message); return NativeStatus.Error; } + finally + { + if (File.Exists(temp)) + { + File.Delete(temp); + } + } } /// diff --git a/src/FixedWidthParser.Native/include/fixedwidth.hpp b/src/FixedWidthParser.Native/include/fixedwidth.hpp index 79d2cf6..62856b7 100644 --- a/src/FixedWidthParser.Native/include/fixedwidth.hpp +++ b/src/FixedWidthParser.Native/include/fixedwidth.hpp @@ -94,23 +94,24 @@ namespace fw return reinterpret_cast(text.data()); } - // fw_stream.read over a std::istream: a short count at the end, -1 once the stream is bad. An exception - // must not unwind into the library, so it fails the read too. + // fw_stream.read over a std::istream: a short count at the end, -1 once the stream is bad or fails before + // its end. An exception must not unwind into the library; one that the end itself throws, from a stream + // with exceptions enabled, still keeps the bytes read before it. inline int64_t read_istream(void* user_data, uint8_t* buffer, int64_t length) noexcept { constexpr std::string_view failed = "the std::istream failed"; + auto& in = *static_cast(user_data); try { - auto& in = *static_cast(user_data); in.read(reinterpret_cast(buffer), static_cast(length)); - if (!in.bad()) - { - return static_cast(in.gcount()); - } } catch (...) { } + if (!in.bad() && (in.eof() || !in.fail())) + { + return static_cast(in.gcount()); + } fw_set_source_error(bytes(failed), static_cast(failed.size())); return -1; } @@ -448,7 +449,7 @@ namespace fw namespace detail { - // fw_sink.write over a std::ostream: -1 once the stream is bad, or if it throws. + // fw_sink.write over a std::ostream: -1 once the stream fails, or if it throws. inline int32_t write_ostream(void* user_data, const uint8_t* buffer, int64_t length) noexcept { constexpr std::string_view failed = "the std::ostream failed"; @@ -456,7 +457,7 @@ namespace fw { auto& out = *static_cast(user_data); out.write(reinterpret_cast(buffer), static_cast(length)); - if (!out.bad()) + if (!out.fail()) { return 0; } diff --git a/src/FixedWidthParser/Attributes/ColumnLayoutValidator.cs b/src/FixedWidthParser/Attributes/ColumnLayoutValidator.cs index 9af3a65..7e489d3 100644 --- a/src/FixedWidthParser/Attributes/ColumnLayoutValidator.cs +++ b/src/FixedWidthParser/Attributes/ColumnLayoutValidator.cs @@ -21,6 +21,11 @@ public static void Validate(Span<(int Start, int Length, string Name)> columns, throw new InvalidOperationException( $"Column \"{Name}\" in {modelName} has an invalid Length ({Length}); it must be >= 1."); } + if (Start > int.MaxValue - Length) + { + throw new InvalidOperationException( + $"Column \"{Name}\" in {modelName} starts at {Start} with Length {Length}, so it ends past int.MaxValue."); + } } if (columns.Length < 2) diff --git a/src/FixedWidthParser/Attributes/ModelColumns.cs b/src/FixedWidthParser/Attributes/ModelColumns.cs index 9699fa4..8323ba0 100644 --- a/src/FixedWidthParser/Attributes/ModelColumns.cs +++ b/src/FixedWidthParser/Attributes/ModelColumns.cs @@ -46,6 +46,22 @@ void Visit(MemberInfo member) } } + /// + /// A new , after checking that it converts . + /// The parser and the writer each create their own, since a converter may carry state. + /// + public static object CreateConverter(Type converterType, Type valueType, string memberName) + { + var requiredInterface = typeof(IFixedWidthConverter<>).MakeGenericType(valueType); + if (!requiredInterface.IsAssignableFrom(converterType)) + { + throw new InvalidOperationException( + $"Converter '{converterType}' for column '{memberName}' must implement '{requiredInterface}'."); + } + return Activator.CreateInstance(converterType) + ?? throw new InvalidOperationException($"Could not instantiate converter '{converterType}' for column '{memberName}'."); + } + /// /// Builds the by-ref model parameter and the member access for , /// returning the member's type too. The caller turns this into a setter (an assignment) or a diff --git a/src/FixedWidthParser/FixedWidthContext.cs b/src/FixedWidthParser/FixedWidthContext.cs index 22d98bc..0c1dfd0 100644 --- a/src/FixedWidthParser/FixedWidthContext.cs +++ b/src/FixedWidthParser/FixedWidthContext.cs @@ -44,8 +44,8 @@ public IFormatProvider FormatProvider internal bool StandardDateSeparators { get => _formatProvider is null || field; private init; } /// - /// The provider's decimal separator as one ASCII byte when its negative sign is "-", so a plain - /// number can be written without going through the provider; 0 otherwise. + /// The provider's decimal separator as one ASCII byte when its signs are "-" and "+", so a plain + /// number can be read and written without going through the provider; 0 otherwise. /// // ponytail: read once, so a NumberFormatInfo mutated after this context was built is not seen. internal byte DecimalSeparator { get => _formatProvider is null ? (byte)'.' : field; private init; } @@ -57,7 +57,8 @@ private static byte PlainDecimalSeparator(IFormatProvider? provider) return (byte)'.'; } var info = NumberFormatInfo.GetInstance(provider); - if (!string.Equals(info.NegativeSign, "-", StringComparison.Ordinal) || info.NumberDecimalSeparator is not [var separator] || !char.IsAscii(separator)) + if (!string.Equals(info.NegativeSign, "-", StringComparison.Ordinal) || !string.Equals(info.PositiveSign, "+", StringComparison.Ordinal) + || info.NumberDecimalSeparator is not [var separator] || !char.IsAscii(separator)) { return 0; } diff --git a/src/FixedWidthParser/FixedWidthParser.csproj b/src/FixedWidthParser/FixedWidthParser.csproj index e5816e2..03b0f68 100644 --- a/src/FixedWidthParser/FixedWidthParser.csproj +++ b/src/FixedWidthParser/FixedWidthParser.csproj @@ -31,7 +31,7 @@ FixedWidthParser.NET - 2.0.0-dev + 2.0.1-dev Gabriel Matte FixedWidthParser High-performance, low-allocation parsing and writing of fixed-width (flat) files for .NET. Attribute-driven column mapping, sync/async batch reading (IEnumerable/IAsyncEnumerable), configurable formatting, culture awareness and ref struct support. diff --git a/src/FixedWidthParser/Formatters/FormatterBuilder.cs b/src/FixedWidthParser/Formatters/FormatterBuilder.cs index 8505b49..e08f314 100644 --- a/src/FixedWidthParser/Formatters/FormatterBuilder.cs +++ b/src/FixedWidthParser/Formatters/FormatterBuilder.cs @@ -70,14 +70,7 @@ private static IColumnFormatter BuildValueFormatter( { if (attribute.Converter is { } converterType) { - var requiredInterface = typeof(IFixedWidthConverter<>).MakeGenericType(valueType); - if (!requiredInterface.IsAssignableFrom(converterType)) - { - throw new InvalidOperationException( - $"Converter '{converterType}' for column '{member.Name}' must implement '{requiredInterface}'."); - } - var converter = Activator.CreateInstance(converterType) - ?? throw new InvalidOperationException($"Could not instantiate converter '{converterType}' for column '{member.Name}'."); + var converter = ModelColumns.CreateConverter(converterType, valueType, member.Name); return (IColumnFormatter)Activator.CreateInstance( typeof(ConverterColumnFormatter<,,>).MakeGenericType(typeof(TModel), valueType, converterType), attribute.Start, attribute.Length, options, member.Name, getter, converter)!; diff --git a/src/FixedWidthParser/Infrastructure/FixedWidthRuntime.cs b/src/FixedWidthParser/Infrastructure/FixedWidthRuntime.cs index 5150050..8ccdb46 100644 --- a/src/FixedWidthParser/Infrastructure/FixedWidthRuntime.cs +++ b/src/FixedWidthParser/Infrastructure/FixedWidthRuntime.cs @@ -37,7 +37,7 @@ internal static ReadOnlySpan TrimColumn(ReadOnlySpan column, in Colu [MethodImpl(MethodImplOptions.AggressiveInlining)] public static bool IsBlank(ReadOnlySpan column, in ColumnParseOptions options) { - return column.Trim(options.TrimByte).Trim((byte)' ').IsEmpty; + return column.IndexOfAnyExcept(options.TrimByte, (byte)' ') < 0; } /// Decodes a string column with the context's encoding, interning it through the context's pool when there is one. @@ -107,9 +107,40 @@ public static bool TryParse(ReadOnlySpan column, in FixedWidthCont value = Unsafe.As(ref fast); return true; } + // The UTF-8 parser reads ASCII the same in every supported encoding, but not other bytes, such as + // a Latin-1 no-break space grouping digits. + if (!context.IsUtf8 && !Ascii.IsValid(trimmed)) + { + return ParseTranscoded(trimmed, in context, out value); + } return TValue.TryParse(trimmed, context.FormatProvider, out value); } + // Decodes with the context's encoding and parses the text's UTF-8, on the stack when it fits. + [SkipLocalsInit] + private static bool ParseTranscoded(ReadOnlySpan bytes, in FixedWidthContext context, [MaybeNullWhen(false)] out TValue value) + where TValue : IUtf8SpanParsable + { + Span chars = stackalloc char[StackBufferSize]; + var text = Decode(bytes, context.Encoding, chars, out char[]? rentedChars); + int maxBytes = Encoding.UTF8.GetMaxByteCount(text.Length); + byte[]? rentedBytes = maxBytes <= StackBufferSize ? null : ArrayPool.Shared.Rent(maxBytes); + Span utf8 = rentedBytes is null ? stackalloc byte[StackBufferSize] : rentedBytes; + try + { + int written = Encoding.UTF8.GetBytes(text, utf8); + return TValue.TryParse(utf8[..written], context.FormatProvider, out value); + } + finally + { + Return(rentedChars); + if (rentedBytes is not null) + { + ArrayPool.Shared.Return(rentedBytes); + } + } + } + /// Parses an column that has no UTF-8 parser, decoding it with the context's encoding first. public static bool TryParseDecoded(ReadOnlySpan column, in FixedWidthContext context, in ColumnParseOptions options, [MaybeNullWhen(false)] out TValue value) where TValue : ISpanParsable @@ -146,9 +177,11 @@ private static class EnumNames private static FrozenDictionary.AlternateLookup> Build() { var map = new Dictionary(StringComparer.OrdinalIgnoreCase); - foreach (TEnum member in Enum.GetValues()) + // By name: an alias shares its value, so the value's ToString would name only one of them. + foreach (string name in Enum.GetNames()) { - map[member.ToString()] = member; + var member = Enum.Parse(name); + map[name] = member; map[member.ToString("D")] = member; } return map.ToFrozenDictionary(StringComparer.OrdinalIgnoreCase).GetAlternateLookup>(); @@ -161,14 +194,38 @@ public static bool TryLookup(ReadOnlySpan text, out TEnum value) { return true; } - // A zero-padded code is its number: 02 is 2, and 00 is 0. - int digits = text.IndexOfAnyExcept('0'); - if (digits == 0 || text.IsEmpty) + // A zero-padded code is its number: 02 is 2, 00 is 0, and -0001 (a negative value right-aligned + // with zero padding, as the writer makes it) is -1. + int sign = text.StartsWith('-') ? 1 : 0; + var unsigned = text[sign..]; + int digits = unsigned.IndexOfAnyExcept('0'); + if (digits == 0 || unsigned.IsEmpty) + { + return false; + } + var number = digits < 0 ? "0" : unsigned[digits..]; + if (number.ContainsAnyExceptInRange('0', '9')) + { + return false; + } + return sign == 0 ? Lookup.TryGetValue(number, out value) : TryLookupNegative(number, out value); + } + + // Its own method, so the stack buffer costs nothing to the lookups that do not need it. No enum value has more + // than 20 digits, so a longer number names no member, and the sign fits beside the rest. + [SkipLocalsInit] + [MethodImpl(MethodImplOptions.NoInlining)] + private static bool TryLookupNegative(ReadOnlySpan digits, out TEnum value) + { + Span negative = stackalloc char[21]; + if (digits.Length >= negative.Length) { + value = default; return false; } - var number = digits < 0 ? "0" : text[digits..]; - return !number.ContainsAnyExceptInRange('0', '9') && Lookup.TryGetValue(number, out value); + negative[0] = '-'; + digits.CopyTo(negative[1..]); + return Lookup.TryGetValue(negative[..(digits.Length + 1)], out value); } } diff --git a/src/FixedWidthParser/Processors/ColumnParserFactory.cs b/src/FixedWidthParser/Processors/ColumnParserFactory.cs index 617c948..ff750db 100644 --- a/src/FixedWidthParser/Processors/ColumnParserFactory.cs +++ b/src/FixedWidthParser/Processors/ColumnParserFactory.cs @@ -1,5 +1,6 @@ using System.Diagnostics.CodeAnalysis; using System.Reflection; +using FixedWidthParser.Attributes; using FixedWidthParser.Infrastructure; namespace FixedWidthParser.Processors @@ -35,14 +36,7 @@ private static ColumnParser CreateValue(Type valueType, Delegate { if (converterType is not null) { - var requiredInterface = typeof(IFixedWidthConverter<>).MakeGenericType(valueType); - if (!requiredInterface.IsAssignableFrom(converterType)) - { - throw new InvalidOperationException( - $"Converter '{converterType}' for column '{memberName}' must implement '{requiredInterface}'."); - } - var converter = Activator.CreateInstance(converterType) - ?? throw new InvalidOperationException($"Could not instantiate converter '{converterType}' for column '{memberName}'."); + var converter = ModelColumns.CreateConverter(converterType, valueType, memberName); return (ColumnParser)Method(nameof(ForConverter), typeof(TModel), valueType, converterType) .Invoke(null, [setter, converter, options])!; } diff --git a/src/FixedWidthParser/PublicAPI.Shipped.txt b/src/FixedWidthParser/PublicAPI.Shipped.txt index 7dc5c58..d15c8a7 100644 --- a/src/FixedWidthParser/PublicAPI.Shipped.txt +++ b/src/FixedWidthParser/PublicAPI.Shipped.txt @@ -1 +1,200 @@ #nullable enable +FixedWidthParser.Alignment +FixedWidthParser.Alignment.Left = 0 -> FixedWidthParser.Alignment +FixedWidthParser.Alignment.Right = 1 -> FixedWidthParser.Alignment +FixedWidthParser.ColumnGetter +FixedWidthParser.ColumnSetter +FixedWidthParser.FixedColumnAttribute +FixedWidthParser.FixedColumnAttribute.Alignment.get -> FixedWidthParser.Alignment +FixedWidthParser.FixedColumnAttribute.Alignment.set -> void +FixedWidthParser.FixedColumnAttribute.Converter.get -> System.Type? +FixedWidthParser.FixedColumnAttribute.Converter.set -> void +FixedWidthParser.FixedColumnAttribute.FixedColumnAttribute(int start, int length) -> void +FixedWidthParser.FixedColumnAttribute.Format.get -> string? +FixedWidthParser.FixedColumnAttribute.Format.set -> void +FixedWidthParser.FixedColumnAttribute.Length.get -> int +FixedWidthParser.FixedColumnAttribute.Overflow.get -> FixedWidthParser.OverflowBehavior +FixedWidthParser.FixedColumnAttribute.Overflow.set -> void +FixedWidthParser.FixedColumnAttribute.Padding.get -> char +FixedWidthParser.FixedColumnAttribute.Padding.set -> void +FixedWidthParser.FixedColumnAttribute.Start.get -> int +FixedWidthParser.FixedColumnAttribute.TrimChar.get -> char +FixedWidthParser.FixedColumnAttribute.TrimChar.set -> void +FixedWidthParser.FixedColumnAttribute.TrimMode.get -> FixedWidthParser.TrimMode +FixedWidthParser.FixedColumnAttribute.TrimMode.set -> void +FixedWidthParser.FixedWidth +FixedWidthParser.FixedWidthColumnOptions +FixedWidthParser.FixedWidthColumnOptions.Alignment.get -> FixedWidthParser.Alignment +FixedWidthParser.FixedWidthColumnOptions.Alignment.init -> void +FixedWidthParser.FixedWidthColumnOptions.FixedWidthColumnOptions() -> void +FixedWidthParser.FixedWidthColumnOptions.Format.get -> string? +FixedWidthParser.FixedWidthColumnOptions.Format.init -> void +FixedWidthParser.FixedWidthColumnOptions.Overflow.get -> FixedWidthParser.OverflowBehavior +FixedWidthParser.FixedWidthColumnOptions.Overflow.init -> void +FixedWidthParser.FixedWidthColumnOptions.Padding.get -> char +FixedWidthParser.FixedWidthColumnOptions.Padding.init -> void +FixedWidthParser.FixedWidthColumnOptions.TrimChar.get -> char +FixedWidthParser.FixedWidthColumnOptions.TrimChar.init -> void +FixedWidthParser.FixedWidthColumnOptions.TrimMode.get -> FixedWidthParser.TrimMode +FixedWidthParser.FixedWidthColumnOptions.TrimMode.init -> void +FixedWidthParser.FixedWidthContext +FixedWidthParser.FixedWidthContext.Encoding.get -> System.Text.Encoding! +FixedWidthParser.FixedWidthContext.Encoding.init -> void +FixedWidthParser.FixedWidthContext.Equals(FixedWidthParser.FixedWidthContext other) -> bool +FixedWidthParser.FixedWidthContext.FixedWidthContext() -> void +FixedWidthParser.FixedWidthContext.FormatProvider.get -> System.IFormatProvider! +FixedWidthParser.FixedWidthContext.FormatProvider.init -> void +FixedWidthParser.FixedWidthContext.ReportFailedColumn(string! columnName, int start, int length) -> void +FixedWidthParser.FixedWidthContext.ReportShortLine(int requiredLength) -> void +FixedWidthParser.FixedWidthContext.StringPool.get -> CommunityToolkit.HighPerformance.Buffers.StringPool? +FixedWidthParser.FixedWidthContext.StringPool.init -> void +FixedWidthParser.FixedWidthLayout +FixedWidthParser.FixedWidthLayout +FixedWidthParser.FixedWidthLayoutBuilder +FixedWidthParser.FixedWidthLayoutBuilder.Column(string! name, int start, int length, FixedWidthParser.ColumnGetter? get, FixedWidthParser.ColumnSetter? set, FixedWidthParser.FixedWidthColumnOptions? options = null) -> FixedWidthParser.FixedWidthLayoutBuilder! +FixedWidthParser.FixedWidthLayoutBuilder.Column(string! name, int start, int length, FixedWidthParser.ColumnGetter? get, FixedWidthParser.ColumnSetter? set, FixedWidthParser.FixedWidthColumnOptions? options = null) -> FixedWidthParser.FixedWidthLayoutBuilder! +FixedWidthParser.FixedWidthLayoutBuilder.ConvertedColumn(string! name, int start, int length, FixedWidthParser.ColumnGetter? get, FixedWidthParser.ColumnSetter? set, FixedWidthParser.IFixedWidthConverter! converter, FixedWidthParser.FixedWidthColumnOptions? options = null) -> FixedWidthParser.FixedWidthLayoutBuilder! +FixedWidthParser.FixedWidthLayoutBuilder.DecodedColumn(string! name, int start, int length, FixedWidthParser.ColumnGetter? get, FixedWidthParser.ColumnSetter? set, FixedWidthParser.FixedWidthColumnOptions? options = null) -> FixedWidthParser.FixedWidthLayoutBuilder! +FixedWidthParser.FixedWidthLayoutBuilder.EnumColumn(string! name, int start, int length, FixedWidthParser.ColumnGetter? get, FixedWidthParser.ColumnSetter? set, FixedWidthParser.FixedWidthColumnOptions? options = null) -> FixedWidthParser.FixedWidthLayoutBuilder! +FixedWidthParser.FixedWidthLayoutBuilder.Factory(System.Func! factory) -> FixedWidthParser.FixedWidthLayoutBuilder! +FixedWidthParser.FixedWidthLayoutBuilder.NullableColumn(string! name, int start, int length, FixedWidthParser.ColumnGetter? get, FixedWidthParser.ColumnSetter? set, FixedWidthParser.FixedWidthColumnOptions? options = null) -> FixedWidthParser.FixedWidthLayoutBuilder! +FixedWidthParser.FixedWidthLayoutBuilder.NullableConvertedColumn(string! name, int start, int length, FixedWidthParser.ColumnGetter? get, FixedWidthParser.ColumnSetter? set, FixedWidthParser.IFixedWidthConverter! converter, FixedWidthParser.FixedWidthColumnOptions? options = null) -> FixedWidthParser.FixedWidthLayoutBuilder! +FixedWidthParser.FixedWidthLayoutBuilder.NullableDecodedColumn(string! name, int start, int length, FixedWidthParser.ColumnGetter? get, FixedWidthParser.ColumnSetter? set, FixedWidthParser.FixedWidthColumnOptions? options = null) -> FixedWidthParser.FixedWidthLayoutBuilder! +FixedWidthParser.FixedWidthLayoutBuilder.NullableEnumColumn(string! name, int start, int length, FixedWidthParser.ColumnGetter? get, FixedWidthParser.ColumnSetter? set, FixedWidthParser.FixedWidthColumnOptions? options = null) -> FixedWidthParser.FixedWidthLayoutBuilder! +FixedWidthParser.FixedWidthParseException +FixedWidthParser.FixedWidthParseException.ColumnLength.get -> int +FixedWidthParser.FixedWidthParseException.ColumnName.get -> string? +FixedWidthParser.FixedWidthParseException.ColumnStart.get -> int +FixedWidthParser.FixedWidthParseException.FixedWidthParseException() -> void +FixedWidthParser.FixedWidthParseException.FixedWidthParseException(string! message) -> void +FixedWidthParser.FixedWidthParseException.FixedWidthParseException(string! message, System.Exception! innerException) -> void +FixedWidthParser.FixedWidthParseException.LineNumber.get -> long +FixedWidthParser.FixedWidthParseException.RawValue.get -> string? +FixedWidthParser.FixedWidthReaderOptions +FixedWidthParser.FixedWidthReaderOptions.BufferSize.get -> int +FixedWidthParser.FixedWidthReaderOptions.BufferSize.init -> void +FixedWidthParser.FixedWidthReaderOptions.DegreeOfParallelism.get -> int +FixedWidthParser.FixedWidthReaderOptions.DegreeOfParallelism.init -> void +FixedWidthParser.FixedWidthReaderOptions.Encoding.get -> System.Text.Encoding? +FixedWidthParser.FixedWidthReaderOptions.Encoding.init -> void +FixedWidthParser.FixedWidthReaderOptions.FixedWidthReaderOptions() -> void +FixedWidthParser.FixedWidthReaderOptions.FormatProvider.get -> System.IFormatProvider? +FixedWidthParser.FixedWidthReaderOptions.FormatProvider.init -> void +FixedWidthParser.FixedWidthReaderOptions.MaxLineLength.get -> int +FixedWidthParser.FixedWidthReaderOptions.MaxLineLength.init -> void +FixedWidthParser.FixedWidthReaderOptions.OnParseError.get -> System.Func? +FixedWidthParser.FixedWidthReaderOptions.OnParseError.init -> void +FixedWidthParser.FixedWidthReaderOptions.StringPool.get -> CommunityToolkit.HighPerformance.Buffers.StringPool? +FixedWidthParser.FixedWidthReaderOptions.StringPool.init -> void +FixedWidthParser.FixedWidthWriterOptions +FixedWidthParser.FixedWidthWriterOptions.Encoding.get -> System.Text.Encoding? +FixedWidthParser.FixedWidthWriterOptions.Encoding.init -> void +FixedWidthParser.FixedWidthWriterOptions.FixedWidthWriterOptions() -> void +FixedWidthParser.FixedWidthWriterOptions.FormatProvider.get -> System.IFormatProvider? +FixedWidthParser.FixedWidthWriterOptions.FormatProvider.init -> void +FixedWidthParser.FixedWidthWriterOptions.NewLine.get -> string! +FixedWidthParser.FixedWidthWriterOptions.NewLine.init -> void +FixedWidthParser.IFixedWidthConverter +FixedWidthParser.IFixedWidthConverter.TryFormat(T value, System.Span destination, in FixedWidthParser.FixedWidthContext context, out int bytesWritten) -> bool +FixedWidthParser.IFixedWidthConverter.TryParse(System.ReadOnlySpan column, in FixedWidthParser.FixedWidthContext context, out T value) -> bool +FixedWidthParser.IFixedWidthModel +FixedWidthParser.IFixedWidthModel.TryFormat(in TSelf model, System.Span destination, in FixedWidthParser.FixedWidthContext context, out int bytesWritten) -> bool +FixedWidthParser.IFixedWidthModel.TryParse(System.ReadOnlySpan line, in FixedWidthParser.FixedWidthContext context, out TSelf model) -> bool +FixedWidthParser.IRecordAccumulator +FixedWidthParser.IRecordAccumulator.Add(in TModel model) -> void +FixedWidthParser.IRecordAccumulator.Merge(TSelf following) -> void +FixedWidthParser.Infrastructure.AsyncRecordEnumerator +FixedWidthParser.Infrastructure.AsyncRecordEnumerator.Current.get -> TModel +FixedWidthParser.Infrastructure.AsyncRecordEnumerator.DisposeAsync() -> System.Threading.Tasks.ValueTask +FixedWidthParser.Infrastructure.AsyncRecordEnumerator.MoveNextAsync() -> System.Threading.Tasks.ValueTask +FixedWidthParser.Infrastructure.ColumnFormatOptions +FixedWidthParser.Infrastructure.ColumnFormatOptions.Alignment.get -> FixedWidthParser.Alignment +FixedWidthParser.Infrastructure.ColumnFormatOptions.ColumnFormatOptions() -> void +FixedWidthParser.Infrastructure.ColumnFormatOptions.ColumnFormatOptions(FixedWidthParser.Alignment alignment, char padding, string? format, FixedWidthParser.OverflowBehavior overflow) -> void +FixedWidthParser.Infrastructure.ColumnFormatOptions.Format.get -> string? +FixedWidthParser.Infrastructure.ColumnFormatOptions.Overflow.get -> FixedWidthParser.OverflowBehavior +FixedWidthParser.Infrastructure.ColumnFormatOptions.Padding.get -> char +FixedWidthParser.Infrastructure.ColumnParseOptions +FixedWidthParser.Infrastructure.ColumnParseOptions.ColumnParseOptions() -> void +FixedWidthParser.Infrastructure.ColumnParseOptions.ColumnParseOptions(char trimChar, FixedWidthParser.TrimMode trimMode, string? format) -> void +FixedWidthParser.Infrastructure.ColumnParseOptions.Format.get -> string? +FixedWidthParser.Infrastructure.ColumnParseOptions.TrimChar.get -> char +FixedWidthParser.Infrastructure.ColumnParseOptions.TrimMode.get -> FixedWidthParser.TrimMode +FixedWidthParser.Infrastructure.FixedWidthAsyncRecordEnumerable +FixedWidthParser.Infrastructure.FixedWidthAsyncRecordEnumerable.GetAsyncEnumerator(System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) -> FixedWidthParser.Infrastructure.AsyncRecordEnumerator! +FixedWidthParser.Infrastructure.FixedWidthRecordEnumerable +FixedWidthParser.Infrastructure.FixedWidthRecordEnumerable.GetEnumerator() -> FixedWidthParser.Infrastructure.RecordEnumerator +FixedWidthParser.Infrastructure.FixedWidthRuntime +FixedWidthParser.Infrastructure.GeneratedLineParser +FixedWidthParser.Infrastructure.GeneratedLineParser.GeneratedLineParser() -> void +FixedWidthParser.Infrastructure.GeneratedLineParser.TryParse(System.ReadOnlySpan line, in FixedWidthParser.FixedWidthContext context, out TModel model) -> bool +FixedWidthParser.Infrastructure.IRecordLineParser +FixedWidthParser.Infrastructure.IRecordLineParser.TryParse(System.ReadOnlySpan line, in FixedWidthParser.FixedWidthContext context, out TModel model) -> bool +FixedWidthParser.Infrastructure.LayoutLineParser +FixedWidthParser.Infrastructure.LayoutLineParser.LayoutLineParser() -> void +FixedWidthParser.Infrastructure.LayoutLineParser.TryParse(System.ReadOnlySpan line, in FixedWidthParser.FixedWidthContext context, out TModel model) -> bool +FixedWidthParser.Infrastructure.RecordEnumerator +FixedWidthParser.Infrastructure.RecordEnumerator.Current.get -> TModel +FixedWidthParser.Infrastructure.RecordEnumerator.Dispose() -> void +FixedWidthParser.Infrastructure.RecordEnumerator.MoveNext() -> bool +FixedWidthParser.Infrastructure.RecordEnumerator.RecordEnumerator() -> void +FixedWidthParser.Infrastructure.RecordEnumerator.Reset() -> void +FixedWidthParser.OverflowBehavior +FixedWidthParser.OverflowBehavior.Default = 0 -> FixedWidthParser.OverflowBehavior +FixedWidthParser.OverflowBehavior.Throw = 2 -> FixedWidthParser.OverflowBehavior +FixedWidthParser.OverflowBehavior.Truncate = 1 -> FixedWidthParser.OverflowBehavior +FixedWidthParser.TrimMode +FixedWidthParser.TrimMode.Both = 2 -> FixedWidthParser.TrimMode +FixedWidthParser.TrimMode.Leading = 1 -> FixedWidthParser.TrimMode +FixedWidthParser.TrimMode.Trailing = 0 -> FixedWidthParser.TrimMode +override FixedWidthParser.FixedWidthContext.GetHashCode() -> int +static FixedWidthParser.FixedWidth.AggregateAsync(System.IO.Stream! stream, FixedWidthParser.FixedWidthLayout! layout, bool leaveOpen = false, FixedWidthParser.FixedWidthReaderOptions? options = null, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.Task! +static FixedWidthParser.FixedWidth.AggregateAsync(System.IO.Stream! stream, bool leaveOpen = false, FixedWidthParser.FixedWidthReaderOptions? options = null, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.Task! +static FixedWidthParser.FixedWidth.AggregateFileAsync(string! path, FixedWidthParser.FixedWidthLayout! layout, FixedWidthParser.FixedWidthReaderOptions? options = null, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.Task! +static FixedWidthParser.FixedWidth.AggregateFileAsync(string! path, FixedWidthParser.FixedWidthReaderOptions? options = null, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.Task! +static FixedWidthParser.FixedWidth.Read(System.IO.Stream! stream, FixedWidthParser.FixedWidthLayout! layout, bool leaveOpen = false, FixedWidthParser.FixedWidthReaderOptions? options = null) -> FixedWidthParser.Infrastructure.FixedWidthRecordEnumerable>! +static FixedWidthParser.FixedWidth.Read(System.IO.Stream! stream, bool leaveOpen = false, FixedWidthParser.FixedWidthReaderOptions? options = null) -> FixedWidthParser.Infrastructure.FixedWidthRecordEnumerable>! +static FixedWidthParser.FixedWidth.ReadAsync(System.IO.Stream! stream, FixedWidthParser.FixedWidthLayout! layout, bool leaveOpen = false, FixedWidthParser.FixedWidthReaderOptions? options = null) -> FixedWidthParser.Infrastructure.FixedWidthAsyncRecordEnumerable>! +static FixedWidthParser.FixedWidth.ReadAsync(System.IO.Stream! stream, bool leaveOpen = false, FixedWidthParser.FixedWidthReaderOptions? options = null) -> FixedWidthParser.Infrastructure.FixedWidthAsyncRecordEnumerable>! +static FixedWidthParser.FixedWidth.ReadFile(string! path, FixedWidthParser.FixedWidthLayout! layout, FixedWidthParser.FixedWidthReaderOptions? options = null) -> FixedWidthParser.Infrastructure.FixedWidthRecordEnumerable>! +static FixedWidthParser.FixedWidth.ReadFile(string! path, FixedWidthParser.FixedWidthReaderOptions? options = null) -> FixedWidthParser.Infrastructure.FixedWidthRecordEnumerable>! +static FixedWidthParser.FixedWidth.ReadFileAsync(string! path, FixedWidthParser.FixedWidthLayout! layout, FixedWidthParser.FixedWidthReaderOptions? options = null) -> FixedWidthParser.Infrastructure.FixedWidthAsyncRecordEnumerable>! +static FixedWidthParser.FixedWidth.ReadFileAsync(string! path, FixedWidthParser.FixedWidthReaderOptions? options = null) -> FixedWidthParser.Infrastructure.FixedWidthAsyncRecordEnumerable>! +static FixedWidthParser.FixedWidth.TryFormat(in TModel model, FixedWidthParser.FixedWidthLayout! layout, System.Span destination, in FixedWidthParser.FixedWidthContext context, out int bytesWritten) -> bool +static FixedWidthParser.FixedWidth.TryFormat(in TModel model, System.Span destination, in FixedWidthParser.FixedWidthContext context, out int bytesWritten) -> bool +static FixedWidthParser.FixedWidth.TryParse(System.ReadOnlySpan line, FixedWidthParser.FixedWidthLayout! layout, in FixedWidthParser.FixedWidthContext context, out TModel model) -> bool +static FixedWidthParser.FixedWidth.TryParse(System.ReadOnlySpan line, in FixedWidthParser.FixedWidthContext context, out TModel model) -> bool +static FixedWidthParser.FixedWidth.Write(System.IO.Stream! stream, in TModel model, FixedWidthParser.FixedWidthLayout! layout, FixedWidthParser.FixedWidthWriterOptions? options = null) -> void +static FixedWidthParser.FixedWidth.Write(System.IO.Stream! stream, in TModel model, FixedWidthParser.FixedWidthWriterOptions? options = null) -> void +static FixedWidthParser.FixedWidth.WriteAsync(System.IO.Stream! stream, TModel model, FixedWidthParser.FixedWidthLayout! layout, FixedWidthParser.FixedWidthWriterOptions? options = null, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.Task! +static FixedWidthParser.FixedWidth.WriteAsync(System.IO.Stream! stream, TModel model, FixedWidthParser.FixedWidthWriterOptions? options = null, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.Task! +static FixedWidthParser.FixedWidth.WriteMany(System.IO.Stream! stream, System.Collections.Generic.IEnumerable! models, FixedWidthParser.FixedWidthLayout! layout, FixedWidthParser.FixedWidthWriterOptions? options = null) -> void +static FixedWidthParser.FixedWidth.WriteMany(System.IO.Stream! stream, System.Collections.Generic.IEnumerable! models, FixedWidthParser.FixedWidthWriterOptions? options = null) -> void +static FixedWidthParser.FixedWidth.WriteMany(System.IO.Stream! stream, System.ReadOnlySpan models, FixedWidthParser.FixedWidthLayout! layout, FixedWidthParser.FixedWidthWriterOptions? options = null) -> void +static FixedWidthParser.FixedWidth.WriteMany(System.IO.Stream! stream, System.ReadOnlySpan models, FixedWidthParser.FixedWidthWriterOptions? options = null) -> void +static FixedWidthParser.FixedWidth.WriteManyAsync(System.IO.Stream! stream, System.Collections.Generic.IEnumerable! models, FixedWidthParser.FixedWidthLayout! layout, FixedWidthParser.FixedWidthWriterOptions? options = null, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.Task! +static FixedWidthParser.FixedWidth.WriteManyAsync(System.IO.Stream! stream, System.Collections.Generic.IEnumerable! models, FixedWidthParser.FixedWidthWriterOptions? options = null, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.Task! +static FixedWidthParser.FixedWidthContext.operator !=(FixedWidthParser.FixedWidthContext left, FixedWidthParser.FixedWidthContext right) -> bool +static FixedWidthParser.FixedWidthContext.operator ==(FixedWidthParser.FixedWidthContext left, FixedWidthParser.FixedWidthContext right) -> bool +static FixedWidthParser.FixedWidthLayout.Build(System.Action!>! configure) -> FixedWidthParser.FixedWidthLayout! +static FixedWidthParser.FixedWidthLayout.FromAttributes() -> FixedWidthParser.FixedWidthLayout! +static FixedWidthParser.Infrastructure.FixedWidthRuntime.FormatConvert(TValue value, System.Span slice, in FixedWidthParser.FixedWidthContext context, TConverter converter, FixedWidthParser.Infrastructure.ColumnFormatOptions options, string! columnName) -> void +static FixedWidthParser.Infrastructure.FixedWidthRuntime.FormatString(string! value, System.Span slice, in FixedWidthParser.FixedWidthContext context, FixedWidthParser.Infrastructure.ColumnFormatOptions options, string! columnName) -> void +static FixedWidthParser.Infrastructure.FixedWidthRuntime.FormatUtf8Value(TValue value, System.Span slice, in FixedWidthParser.FixedWidthContext context, FixedWidthParser.Infrastructure.ColumnFormatOptions options, string! columnName) -> void +static FixedWidthParser.Infrastructure.FixedWidthRuntime.FormatValue(TValue value, System.Span slice, in FixedWidthParser.FixedWidthContext context, FixedWidthParser.Infrastructure.ColumnFormatOptions options, string! columnName) -> void +static FixedWidthParser.Infrastructure.FixedWidthRuntime.IsBlank(System.ReadOnlySpan column, in FixedWidthParser.Infrastructure.ColumnParseOptions options) -> bool +static FixedWidthParser.Infrastructure.FixedWidthRuntime.String(System.ReadOnlySpan column, in FixedWidthParser.FixedWidthContext context, in FixedWidthParser.Infrastructure.ColumnParseOptions options) -> string! +static FixedWidthParser.Infrastructure.FixedWidthRuntime.TryConvert(System.ReadOnlySpan column, in FixedWidthParser.FixedWidthContext context, TConverter converter, in FixedWidthParser.Infrastructure.ColumnParseOptions options, out TValue value) -> bool +static FixedWidthParser.Infrastructure.FixedWidthRuntime.TryDateOnlyExact(System.ReadOnlySpan column, in FixedWidthParser.FixedWidthContext context, in FixedWidthParser.Infrastructure.ColumnParseOptions options, out System.DateOnly value) -> bool +static FixedWidthParser.Infrastructure.FixedWidthRuntime.TryDateTimeExact(System.ReadOnlySpan column, in FixedWidthParser.FixedWidthContext context, in FixedWidthParser.Infrastructure.ColumnParseOptions options, out System.DateTime value) -> bool +static FixedWidthParser.Infrastructure.FixedWidthRuntime.TryDateTimeOffsetExact(System.ReadOnlySpan column, in FixedWidthParser.FixedWidthContext context, in FixedWidthParser.Infrastructure.ColumnParseOptions options, out System.DateTimeOffset value) -> bool +static FixedWidthParser.Infrastructure.FixedWidthRuntime.TryDouble(System.ReadOnlySpan column, in FixedWidthParser.FixedWidthContext context, in FixedWidthParser.Infrastructure.ColumnParseOptions options, out double value) -> bool +static FixedWidthParser.Infrastructure.FixedWidthRuntime.TryEnum(System.ReadOnlySpan column, in FixedWidthParser.FixedWidthContext context, in FixedWidthParser.Infrastructure.ColumnParseOptions options, out TEnum value) -> bool +static FixedWidthParser.Infrastructure.FixedWidthRuntime.TryFloat(System.ReadOnlySpan column, in FixedWidthParser.FixedWidthContext context, in FixedWidthParser.Infrastructure.ColumnParseOptions options, out float value) -> bool +static FixedWidthParser.Infrastructure.FixedWidthRuntime.TryParse(System.ReadOnlySpan column, in FixedWidthParser.FixedWidthContext context, in FixedWidthParser.Infrastructure.ColumnParseOptions options, out TValue value) -> bool +static FixedWidthParser.Infrastructure.FixedWidthRuntime.TryParseDecoded(System.ReadOnlySpan column, in FixedWidthParser.FixedWidthContext context, in FixedWidthParser.Infrastructure.ColumnParseOptions options, out TValue value) -> bool +static FixedWidthParser.Infrastructure.FixedWidthRuntime.TryTimeOnlyExact(System.ReadOnlySpan column, in FixedWidthParser.FixedWidthContext context, in FixedWidthParser.Infrastructure.ColumnParseOptions options, out System.TimeOnly value) -> bool +virtual FixedWidthParser.ColumnGetter.Invoke(in TModel model) -> TValue +virtual FixedWidthParser.ColumnSetter.Invoke(ref TModel model, TValue value) -> void +~override FixedWidthParser.FixedWidthContext.Equals(object obj) -> bool +~override FixedWidthParser.FixedWidthContext.ToString() -> string diff --git a/src/FixedWidthParser/PublicAPI.Unshipped.txt b/src/FixedWidthParser/PublicAPI.Unshipped.txt index c66e75a..7dc5c58 100644 --- a/src/FixedWidthParser/PublicAPI.Unshipped.txt +++ b/src/FixedWidthParser/PublicAPI.Unshipped.txt @@ -1,200 +1 @@ #nullable enable -FixedWidthParser.FixedWidth -FixedWidthParser.FixedWidthContext -FixedWidthParser.FixedWidthContext.Encoding.get -> System.Text.Encoding! -FixedWidthParser.FixedWidthContext.Encoding.init -> void -FixedWidthParser.FixedWidthContext.Equals(FixedWidthParser.FixedWidthContext other) -> bool -FixedWidthParser.FixedWidthContext.FixedWidthContext() -> void -FixedWidthParser.FixedWidthContext.FormatProvider.get -> System.IFormatProvider! -FixedWidthParser.FixedWidthContext.FormatProvider.init -> void -FixedWidthParser.FixedWidthContext.ReportFailedColumn(string! columnName, int start, int length) -> void -FixedWidthParser.FixedWidthContext.ReportShortLine(int requiredLength) -> void -FixedWidthParser.FixedWidthContext.StringPool.get -> CommunityToolkit.HighPerformance.Buffers.StringPool? -FixedWidthParser.FixedWidthContext.StringPool.init -> void -FixedWidthParser.FixedWidthParseException -FixedWidthParser.FixedWidthParseException.ColumnLength.get -> int -FixedWidthParser.FixedWidthParseException.ColumnName.get -> string? -FixedWidthParser.FixedWidthParseException.ColumnStart.get -> int -FixedWidthParser.FixedWidthParseException.FixedWidthParseException() -> void -FixedWidthParser.FixedWidthParseException.FixedWidthParseException(string! message) -> void -FixedWidthParser.FixedWidthParseException.FixedWidthParseException(string! message, System.Exception! innerException) -> void -FixedWidthParser.FixedWidthParseException.LineNumber.get -> long -FixedWidthParser.FixedWidthParseException.RawValue.get -> string? -FixedWidthParser.FixedWidthReaderOptions -FixedWidthParser.FixedWidthReaderOptions.BufferSize.get -> int -FixedWidthParser.FixedWidthReaderOptions.BufferSize.init -> void -FixedWidthParser.FixedWidthReaderOptions.Encoding.get -> System.Text.Encoding? -FixedWidthParser.FixedWidthReaderOptions.Encoding.init -> void -FixedWidthParser.FixedWidthReaderOptions.FixedWidthReaderOptions() -> void -FixedWidthParser.FixedWidthReaderOptions.FormatProvider.get -> System.IFormatProvider? -FixedWidthParser.FixedWidthReaderOptions.FormatProvider.init -> void -FixedWidthParser.FixedWidthReaderOptions.MaxLineLength.get -> int -FixedWidthParser.FixedWidthReaderOptions.MaxLineLength.init -> void -FixedWidthParser.FixedWidthReaderOptions.OnParseError.get -> System.Func? -FixedWidthParser.FixedWidthReaderOptions.OnParseError.init -> void -FixedWidthParser.FixedWidthReaderOptions.StringPool.get -> CommunityToolkit.HighPerformance.Buffers.StringPool? -FixedWidthParser.FixedWidthReaderOptions.StringPool.init -> void -FixedWidthParser.FixedWidthWriterOptions -FixedWidthParser.FixedWidthWriterOptions.Encoding.get -> System.Text.Encoding? -FixedWidthParser.FixedWidthWriterOptions.Encoding.init -> void -FixedWidthParser.FixedWidthWriterOptions.FixedWidthWriterOptions() -> void -FixedWidthParser.FixedWidthWriterOptions.FormatProvider.get -> System.IFormatProvider? -FixedWidthParser.FixedWidthWriterOptions.FormatProvider.init -> void -FixedWidthParser.FixedWidthWriterOptions.NewLine.get -> string! -FixedWidthParser.FixedWidthWriterOptions.NewLine.init -> void -FixedWidthParser.IFixedWidthModel -FixedWidthParser.IFixedWidthModel.TryFormat(in TSelf model, System.Span destination, in FixedWidthParser.FixedWidthContext context, out int bytesWritten) -> bool -FixedWidthParser.IFixedWidthModel.TryParse(System.ReadOnlySpan line, in FixedWidthParser.FixedWidthContext context, out TSelf model) -> bool -override FixedWidthParser.FixedWidthContext.GetHashCode() -> int -static FixedWidthParser.FixedWidth.TryFormat(in TModel model, System.Span destination, in FixedWidthParser.FixedWidthContext context, out int bytesWritten) -> bool -static FixedWidthParser.FixedWidth.TryParse(System.ReadOnlySpan line, in FixedWidthParser.FixedWidthContext context, out TModel model) -> bool -static FixedWidthParser.FixedWidthContext.operator !=(FixedWidthParser.FixedWidthContext left, FixedWidthParser.FixedWidthContext right) -> bool -static FixedWidthParser.FixedWidthContext.operator ==(FixedWidthParser.FixedWidthContext left, FixedWidthParser.FixedWidthContext right) -> bool -~override FixedWidthParser.FixedWidthContext.Equals(object obj) -> bool -~override FixedWidthParser.FixedWidthContext.ToString() -> string -FixedWidthParser.IRecordAccumulator -FixedWidthParser.IRecordAccumulator.Add(in TModel model) -> void -FixedWidthParser.IRecordAccumulator.Merge(TSelf following) -> void -FixedWidthParser.ColumnGetter -FixedWidthParser.ColumnSetter -FixedWidthParser.FixedWidthLayout -FixedWidthParser.FixedWidthLayout -static FixedWidthParser.FixedWidth.TryFormat(in TModel model, FixedWidthParser.FixedWidthLayout! layout, System.Span destination, in FixedWidthParser.FixedWidthContext context, out int bytesWritten) -> bool -static FixedWidthParser.FixedWidth.TryParse(System.ReadOnlySpan line, FixedWidthParser.FixedWidthLayout! layout, in FixedWidthParser.FixedWidthContext context, out TModel model) -> bool -static FixedWidthParser.FixedWidthLayout.FromAttributes() -> FixedWidthParser.FixedWidthLayout! -virtual FixedWidthParser.ColumnGetter.Invoke(in TModel model) -> TValue -virtual FixedWidthParser.ColumnSetter.Invoke(ref TModel model, TValue value) -> void -static FixedWidthParser.FixedWidth.Write(System.IO.Stream! stream, in TModel model, FixedWidthParser.FixedWidthLayout! layout, FixedWidthParser.FixedWidthWriterOptions? options = null) -> void -static FixedWidthParser.FixedWidth.Write(System.IO.Stream! stream, in TModel model, FixedWidthParser.FixedWidthWriterOptions? options = null) -> void -static FixedWidthParser.FixedWidth.WriteAsync(System.IO.Stream! stream, TModel model, FixedWidthParser.FixedWidthLayout! layout, FixedWidthParser.FixedWidthWriterOptions? options = null, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.Task! -static FixedWidthParser.FixedWidth.WriteAsync(System.IO.Stream! stream, TModel model, FixedWidthParser.FixedWidthWriterOptions? options = null, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.Task! -static FixedWidthParser.FixedWidth.WriteMany(System.IO.Stream! stream, System.Collections.Generic.IEnumerable! models, FixedWidthParser.FixedWidthLayout! layout, FixedWidthParser.FixedWidthWriterOptions? options = null) -> void -static FixedWidthParser.FixedWidth.WriteMany(System.IO.Stream! stream, System.Collections.Generic.IEnumerable! models, FixedWidthParser.FixedWidthWriterOptions? options = null) -> void -static FixedWidthParser.FixedWidth.WriteMany(System.IO.Stream! stream, System.ReadOnlySpan models, FixedWidthParser.FixedWidthLayout! layout, FixedWidthParser.FixedWidthWriterOptions? options = null) -> void -static FixedWidthParser.FixedWidth.WriteMany(System.IO.Stream! stream, System.ReadOnlySpan models, FixedWidthParser.FixedWidthWriterOptions? options = null) -> void -static FixedWidthParser.FixedWidth.WriteManyAsync(System.IO.Stream! stream, System.Collections.Generic.IEnumerable! models, FixedWidthParser.FixedWidthLayout! layout, FixedWidthParser.FixedWidthWriterOptions? options = null, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.Task! -static FixedWidthParser.FixedWidth.WriteManyAsync(System.IO.Stream! stream, System.Collections.Generic.IEnumerable! models, FixedWidthParser.FixedWidthWriterOptions? options = null, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.Task! -FixedWidthParser.FixedWidthColumnOptions -FixedWidthParser.FixedWidthColumnOptions.Alignment.init -> void -FixedWidthParser.FixedWidthColumnOptions.FixedWidthColumnOptions() -> void -FixedWidthParser.FixedWidthColumnOptions.Format.get -> string? -FixedWidthParser.FixedWidthColumnOptions.Format.init -> void -FixedWidthParser.FixedWidthColumnOptions.Overflow.init -> void -FixedWidthParser.FixedWidthColumnOptions.Padding.get -> char -FixedWidthParser.FixedWidthColumnOptions.Padding.init -> void -FixedWidthParser.FixedWidthColumnOptions.TrimChar.get -> char -FixedWidthParser.FixedWidthColumnOptions.TrimChar.init -> void -FixedWidthParser.FixedWidthColumnOptions.TrimMode.init -> void -FixedWidthParser.FixedWidthLayoutBuilder -FixedWidthParser.FixedWidthLayoutBuilder.Column(string! name, int start, int length, FixedWidthParser.ColumnGetter? get, FixedWidthParser.ColumnSetter? set, FixedWidthParser.FixedWidthColumnOptions? options = null) -> FixedWidthParser.FixedWidthLayoutBuilder! -FixedWidthParser.FixedWidthLayoutBuilder.Column(string! name, int start, int length, FixedWidthParser.ColumnGetter? get, FixedWidthParser.ColumnSetter? set, FixedWidthParser.FixedWidthColumnOptions? options = null) -> FixedWidthParser.FixedWidthLayoutBuilder! -FixedWidthParser.FixedWidthLayoutBuilder.DecodedColumn(string! name, int start, int length, FixedWidthParser.ColumnGetter? get, FixedWidthParser.ColumnSetter? set, FixedWidthParser.FixedWidthColumnOptions? options = null) -> FixedWidthParser.FixedWidthLayoutBuilder! -FixedWidthParser.FixedWidthLayoutBuilder.Factory(System.Func! factory) -> FixedWidthParser.FixedWidthLayoutBuilder! -FixedWidthParser.FixedWidthLayoutBuilder.NullableColumn(string! name, int start, int length, FixedWidthParser.ColumnGetter? get, FixedWidthParser.ColumnSetter? set, FixedWidthParser.FixedWidthColumnOptions? options = null) -> FixedWidthParser.FixedWidthLayoutBuilder! -static FixedWidthParser.FixedWidthLayout.Build(System.Action!>! configure) -> FixedWidthParser.FixedWidthLayout! -FixedWidthParser.FixedWidthLayoutBuilder.NullableDecodedColumn(string! name, int start, int length, FixedWidthParser.ColumnGetter? get, FixedWidthParser.ColumnSetter? set, FixedWidthParser.FixedWidthColumnOptions? options = null) -> FixedWidthParser.FixedWidthLayoutBuilder! -FixedWidthParser.Alignment -FixedWidthParser.Alignment.Left = 0 -> FixedWidthParser.Alignment -FixedWidthParser.Alignment.Right = 1 -> FixedWidthParser.Alignment -FixedWidthParser.FixedColumnAttribute -FixedWidthParser.FixedColumnAttribute.Alignment.get -> FixedWidthParser.Alignment -FixedWidthParser.FixedColumnAttribute.Alignment.set -> void -FixedWidthParser.FixedColumnAttribute.Converter.get -> System.Type? -FixedWidthParser.FixedColumnAttribute.Converter.set -> void -FixedWidthParser.FixedColumnAttribute.FixedColumnAttribute(int start, int length) -> void -FixedWidthParser.FixedColumnAttribute.Format.get -> string? -FixedWidthParser.FixedColumnAttribute.Format.set -> void -FixedWidthParser.FixedColumnAttribute.Length.get -> int -FixedWidthParser.FixedColumnAttribute.Overflow.get -> FixedWidthParser.OverflowBehavior -FixedWidthParser.FixedColumnAttribute.Overflow.set -> void -FixedWidthParser.FixedColumnAttribute.Padding.get -> char -FixedWidthParser.FixedColumnAttribute.Padding.set -> void -FixedWidthParser.FixedColumnAttribute.Start.get -> int -FixedWidthParser.FixedColumnAttribute.TrimChar.get -> char -FixedWidthParser.FixedColumnAttribute.TrimChar.set -> void -FixedWidthParser.FixedColumnAttribute.TrimMode.get -> FixedWidthParser.TrimMode -FixedWidthParser.FixedColumnAttribute.TrimMode.set -> void -FixedWidthParser.FixedWidthColumnOptions.Alignment.get -> FixedWidthParser.Alignment -FixedWidthParser.FixedWidthColumnOptions.Overflow.get -> FixedWidthParser.OverflowBehavior -FixedWidthParser.FixedWidthColumnOptions.TrimMode.get -> FixedWidthParser.TrimMode -FixedWidthParser.FixedWidthLayoutBuilder.ConvertedColumn(string! name, int start, int length, FixedWidthParser.ColumnGetter? get, FixedWidthParser.ColumnSetter? set, FixedWidthParser.IFixedWidthConverter! converter, FixedWidthParser.FixedWidthColumnOptions? options = null) -> FixedWidthParser.FixedWidthLayoutBuilder! -FixedWidthParser.FixedWidthLayoutBuilder.NullableConvertedColumn(string! name, int start, int length, FixedWidthParser.ColumnGetter? get, FixedWidthParser.ColumnSetter? set, FixedWidthParser.IFixedWidthConverter! converter, FixedWidthParser.FixedWidthColumnOptions? options = null) -> FixedWidthParser.FixedWidthLayoutBuilder! -FixedWidthParser.IFixedWidthConverter -FixedWidthParser.Infrastructure.AsyncRecordEnumerator -FixedWidthParser.Infrastructure.AsyncRecordEnumerator.Current.get -> TModel -FixedWidthParser.Infrastructure.AsyncRecordEnumerator.DisposeAsync() -> System.Threading.Tasks.ValueTask -FixedWidthParser.Infrastructure.AsyncRecordEnumerator.MoveNextAsync() -> System.Threading.Tasks.ValueTask -FixedWidthParser.Infrastructure.ColumnFormatOptions -FixedWidthParser.Infrastructure.ColumnFormatOptions.Alignment.get -> FixedWidthParser.Alignment -FixedWidthParser.Infrastructure.ColumnFormatOptions.ColumnFormatOptions() -> void -FixedWidthParser.Infrastructure.ColumnFormatOptions.Format.get -> string? -FixedWidthParser.Infrastructure.ColumnFormatOptions.Overflow.get -> FixedWidthParser.OverflowBehavior -FixedWidthParser.Infrastructure.ColumnFormatOptions.Padding.get -> char -FixedWidthParser.Infrastructure.FixedWidthAsyncRecordEnumerable -FixedWidthParser.Infrastructure.FixedWidthAsyncRecordEnumerable.GetAsyncEnumerator(System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) -> FixedWidthParser.Infrastructure.AsyncRecordEnumerator! -FixedWidthParser.Infrastructure.FixedWidthRecordEnumerable -FixedWidthParser.Infrastructure.FixedWidthRecordEnumerable.GetEnumerator() -> FixedWidthParser.Infrastructure.RecordEnumerator -FixedWidthParser.Infrastructure.FixedWidthRuntime -FixedWidthParser.Infrastructure.GeneratedLineParser -FixedWidthParser.Infrastructure.GeneratedLineParser.GeneratedLineParser() -> void -FixedWidthParser.Infrastructure.GeneratedLineParser.TryParse(System.ReadOnlySpan line, in FixedWidthParser.FixedWidthContext context, out TModel model) -> bool -FixedWidthParser.Infrastructure.IRecordLineParser -FixedWidthParser.Infrastructure.IRecordLineParser.TryParse(System.ReadOnlySpan line, in FixedWidthParser.FixedWidthContext context, out TModel model) -> bool -FixedWidthParser.Infrastructure.LayoutLineParser -FixedWidthParser.Infrastructure.LayoutLineParser.LayoutLineParser() -> void -FixedWidthParser.Infrastructure.LayoutLineParser.TryParse(System.ReadOnlySpan line, in FixedWidthParser.FixedWidthContext context, out TModel model) -> bool -FixedWidthParser.Infrastructure.RecordEnumerator -FixedWidthParser.Infrastructure.RecordEnumerator.Current.get -> TModel -FixedWidthParser.Infrastructure.RecordEnumerator.Dispose() -> void -FixedWidthParser.Infrastructure.RecordEnumerator.MoveNext() -> bool -FixedWidthParser.Infrastructure.RecordEnumerator.RecordEnumerator() -> void -FixedWidthParser.Infrastructure.RecordEnumerator.Reset() -> void -FixedWidthParser.OverflowBehavior -FixedWidthParser.OverflowBehavior.Default = 0 -> FixedWidthParser.OverflowBehavior -FixedWidthParser.OverflowBehavior.Throw = 2 -> FixedWidthParser.OverflowBehavior -FixedWidthParser.OverflowBehavior.Truncate = 1 -> FixedWidthParser.OverflowBehavior -FixedWidthParser.TrimMode -FixedWidthParser.TrimMode.Both = 2 -> FixedWidthParser.TrimMode -FixedWidthParser.TrimMode.Leading = 1 -> FixedWidthParser.TrimMode -FixedWidthParser.TrimMode.Trailing = 0 -> FixedWidthParser.TrimMode -static FixedWidthParser.FixedWidth.Read(System.IO.Stream! stream, FixedWidthParser.FixedWidthLayout! layout, bool leaveOpen = false, FixedWidthParser.FixedWidthReaderOptions? options = null) -> FixedWidthParser.Infrastructure.FixedWidthRecordEnumerable>! -static FixedWidthParser.FixedWidth.Read(System.IO.Stream! stream, bool leaveOpen = false, FixedWidthParser.FixedWidthReaderOptions? options = null) -> FixedWidthParser.Infrastructure.FixedWidthRecordEnumerable>! -static FixedWidthParser.FixedWidth.ReadAsync(System.IO.Stream! stream, FixedWidthParser.FixedWidthLayout! layout, bool leaveOpen = false, FixedWidthParser.FixedWidthReaderOptions? options = null) -> FixedWidthParser.Infrastructure.FixedWidthAsyncRecordEnumerable>! -static FixedWidthParser.FixedWidth.ReadAsync(System.IO.Stream! stream, bool leaveOpen = false, FixedWidthParser.FixedWidthReaderOptions? options = null) -> FixedWidthParser.Infrastructure.FixedWidthAsyncRecordEnumerable>! -static FixedWidthParser.FixedWidth.ReadFile(string! path, FixedWidthParser.FixedWidthLayout! layout, FixedWidthParser.FixedWidthReaderOptions? options = null) -> FixedWidthParser.Infrastructure.FixedWidthRecordEnumerable>! -static FixedWidthParser.FixedWidth.ReadFile(string! path, FixedWidthParser.FixedWidthReaderOptions? options = null) -> FixedWidthParser.Infrastructure.FixedWidthRecordEnumerable>! -static FixedWidthParser.FixedWidth.ReadFileAsync(string! path, FixedWidthParser.FixedWidthLayout! layout, FixedWidthParser.FixedWidthReaderOptions? options = null) -> FixedWidthParser.Infrastructure.FixedWidthAsyncRecordEnumerable>! -static FixedWidthParser.FixedWidth.ReadFileAsync(string! path, FixedWidthParser.FixedWidthReaderOptions? options = null) -> FixedWidthParser.Infrastructure.FixedWidthAsyncRecordEnumerable>! -static FixedWidthParser.Infrastructure.FixedWidthRuntime.FormatConvert(TValue value, System.Span slice, in FixedWidthParser.FixedWidthContext context, TConverter converter, FixedWidthParser.Infrastructure.ColumnFormatOptions options, string! columnName) -> void -static FixedWidthParser.Infrastructure.FixedWidthRuntime.FormatString(string! value, System.Span slice, in FixedWidthParser.FixedWidthContext context, FixedWidthParser.Infrastructure.ColumnFormatOptions options, string! columnName) -> void -static FixedWidthParser.Infrastructure.FixedWidthRuntime.FormatUtf8Value(TValue value, System.Span slice, in FixedWidthParser.FixedWidthContext context, FixedWidthParser.Infrastructure.ColumnFormatOptions options, string! columnName) -> void -static FixedWidthParser.Infrastructure.FixedWidthRuntime.FormatValue(TValue value, System.Span slice, in FixedWidthParser.FixedWidthContext context, FixedWidthParser.Infrastructure.ColumnFormatOptions options, string! columnName) -> void -FixedWidthParser.IFixedWidthConverter.TryFormat(T value, System.Span destination, in FixedWidthParser.FixedWidthContext context, out int bytesWritten) -> bool -FixedWidthParser.IFixedWidthConverter.TryParse(System.ReadOnlySpan column, in FixedWidthParser.FixedWidthContext context, out T value) -> bool -FixedWidthParser.FixedWidthReaderOptions.DegreeOfParallelism.get -> int -FixedWidthParser.FixedWidthReaderOptions.DegreeOfParallelism.init -> void -static FixedWidthParser.FixedWidth.AggregateAsync(System.IO.Stream! stream, FixedWidthParser.FixedWidthLayout! layout, bool leaveOpen = false, FixedWidthParser.FixedWidthReaderOptions? options = null, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.Task! -static FixedWidthParser.FixedWidth.AggregateAsync(System.IO.Stream! stream, bool leaveOpen = false, FixedWidthParser.FixedWidthReaderOptions? options = null, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.Task! -static FixedWidthParser.FixedWidth.AggregateFileAsync(string! path, FixedWidthParser.FixedWidthLayout! layout, FixedWidthParser.FixedWidthReaderOptions? options = null, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.Task! -static FixedWidthParser.FixedWidth.AggregateFileAsync(string! path, FixedWidthParser.FixedWidthReaderOptions? options = null, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.Task! -FixedWidthParser.Infrastructure.ColumnFormatOptions.ColumnFormatOptions(FixedWidthParser.Alignment alignment, char padding, string? format, FixedWidthParser.OverflowBehavior overflow) -> void -FixedWidthParser.Infrastructure.ColumnParseOptions -FixedWidthParser.Infrastructure.ColumnParseOptions.ColumnParseOptions() -> void -FixedWidthParser.Infrastructure.ColumnParseOptions.ColumnParseOptions(char trimChar, FixedWidthParser.TrimMode trimMode, string? format) -> void -FixedWidthParser.Infrastructure.ColumnParseOptions.Format.get -> string? -FixedWidthParser.Infrastructure.ColumnParseOptions.TrimChar.get -> char -FixedWidthParser.Infrastructure.ColumnParseOptions.TrimMode.get -> FixedWidthParser.TrimMode -static FixedWidthParser.Infrastructure.FixedWidthRuntime.IsBlank(System.ReadOnlySpan column, in FixedWidthParser.Infrastructure.ColumnParseOptions options) -> bool -static FixedWidthParser.Infrastructure.FixedWidthRuntime.String(System.ReadOnlySpan column, in FixedWidthParser.FixedWidthContext context, in FixedWidthParser.Infrastructure.ColumnParseOptions options) -> string! -static FixedWidthParser.Infrastructure.FixedWidthRuntime.TryConvert(System.ReadOnlySpan column, in FixedWidthParser.FixedWidthContext context, TConverter converter, in FixedWidthParser.Infrastructure.ColumnParseOptions options, out TValue value) -> bool -static FixedWidthParser.Infrastructure.FixedWidthRuntime.TryDateOnlyExact(System.ReadOnlySpan column, in FixedWidthParser.FixedWidthContext context, in FixedWidthParser.Infrastructure.ColumnParseOptions options, out System.DateOnly value) -> bool -static FixedWidthParser.Infrastructure.FixedWidthRuntime.TryDateTimeExact(System.ReadOnlySpan column, in FixedWidthParser.FixedWidthContext context, in FixedWidthParser.Infrastructure.ColumnParseOptions options, out System.DateTime value) -> bool -static FixedWidthParser.Infrastructure.FixedWidthRuntime.TryDateTimeOffsetExact(System.ReadOnlySpan column, in FixedWidthParser.FixedWidthContext context, in FixedWidthParser.Infrastructure.ColumnParseOptions options, out System.DateTimeOffset value) -> bool -static FixedWidthParser.Infrastructure.FixedWidthRuntime.TryDouble(System.ReadOnlySpan column, in FixedWidthParser.FixedWidthContext context, in FixedWidthParser.Infrastructure.ColumnParseOptions options, out double value) -> bool -static FixedWidthParser.Infrastructure.FixedWidthRuntime.TryFloat(System.ReadOnlySpan column, in FixedWidthParser.FixedWidthContext context, in FixedWidthParser.Infrastructure.ColumnParseOptions options, out float value) -> bool -static FixedWidthParser.Infrastructure.FixedWidthRuntime.TryParse(System.ReadOnlySpan column, in FixedWidthParser.FixedWidthContext context, in FixedWidthParser.Infrastructure.ColumnParseOptions options, out TValue value) -> bool -static FixedWidthParser.Infrastructure.FixedWidthRuntime.TryParseDecoded(System.ReadOnlySpan column, in FixedWidthParser.FixedWidthContext context, in FixedWidthParser.Infrastructure.ColumnParseOptions options, out TValue value) -> bool -static FixedWidthParser.Infrastructure.FixedWidthRuntime.TryTimeOnlyExact(System.ReadOnlySpan column, in FixedWidthParser.FixedWidthContext context, in FixedWidthParser.Infrastructure.ColumnParseOptions options, out System.TimeOnly value) -> bool -FixedWidthParser.FixedWidthLayoutBuilder.EnumColumn(string! name, int start, int length, FixedWidthParser.ColumnGetter? get, FixedWidthParser.ColumnSetter? set, FixedWidthParser.FixedWidthColumnOptions? options = null) -> FixedWidthParser.FixedWidthLayoutBuilder! -FixedWidthParser.FixedWidthLayoutBuilder.NullableEnumColumn(string! name, int start, int length, FixedWidthParser.ColumnGetter? get, FixedWidthParser.ColumnSetter? set, FixedWidthParser.FixedWidthColumnOptions? options = null) -> FixedWidthParser.FixedWidthLayoutBuilder! -static FixedWidthParser.Infrastructure.FixedWidthRuntime.TryEnum(System.ReadOnlySpan column, in FixedWidthParser.FixedWidthContext context, in FixedWidthParser.Infrastructure.ColumnParseOptions options, out TEnum value) -> bool diff --git a/src/FixedWidthParser/Readers/ParallelStreamAggregator.cs b/src/FixedWidthParser/Readers/ParallelStreamAggregator.cs index 7cf1fe6..422b57e 100644 --- a/src/FixedWidthParser/Readers/ParallelStreamAggregator.cs +++ b/src/FixedWidthParser/Readers/ParallelStreamAggregator.cs @@ -12,7 +12,9 @@ namespace FixedWidthParser.Readers /// breaks it hands over, so every chunk knows the number of its first line. Workers fold the chunks /// from a bounded queue into one accumulator each, merged in chunk order as they complete. Every chunk /// holds one of a few slots per worker from the moment it is produced until it is merged, which bounds - /// memory, chunks and unmerged accumulators alike, whatever the source's size. + /// memory, chunks and unmerged accumulators alike, whatever the source's size. The first failure in source + /// order ends the read: no later chunk can change the result, so the producer stops reading and the workers + /// stop taking chunks, instead of waiting for the end of a source that may never come. /// internal static class ParallelStreamAggregator { @@ -74,20 +76,22 @@ private static async Task PartitionedAsync(new BoundedChannelOptions(workerCount * 2) { SingleWriter = true }); - var merge = new OrderedMerge(slots); + var merge = new OrderedMerge(slots, stop); var workers = new Task[workerCount]; foreach (ref var worker in workers.AsSpan()) { #pragma warning disable IDISP013 // Every worker is awaited below, before the slots are disposed. - worker = Task.Run(() => ConsumeAsync(parser, queue.Reader, merge, options, ct), ct); + worker = Task.Run(() => ConsumeAsync(parser, queue.Reader, merge, options, stop.Token, ct), ct); #pragma warning restore IDISP013 } ExceptionDispatchInfo? producerFailure = null; try { - await ProduceAsync(source, queue.Writer, slots, options, options.ChunkSize > 0 ? options.ChunkSize : DefaultChunkSize, ct).ConfigureAwait(false); + await ProduceAsync(source, queue.Writer, slots, options, options.ChunkSize > 0 ? options.ChunkSize : DefaultChunkSize, stop.Token).ConfigureAwait(false); } // Cancellation too: the workers must finish before the chunks left in the queue are returned. #pragma warning disable CA1031 // Rethrown by merge.Result once the workers have finished. @@ -103,7 +107,14 @@ private static async Task PartitionedAsync( - TParser parser, ChannelReader queue, OrderedMerge merge, FixedWidthReaderOptions options, CancellationToken ct) + TParser parser, ChannelReader queue, OrderedMerge merge, FixedWidthReaderOptions options, + CancellationToken stop, CancellationToken ct) where TModel : allows ref struct where TParser : struct, IRecordLineParser where TAccumulator : IRecordAccumulator, new() { var context = options.Context; - await foreach (var chunk in queue.ReadAllAsync(ct).ConfigureAwait(false)) + await foreach (var chunk in queue.ReadAllAsync(stop).ConfigureAwait(false)) { Outcome outcome; try { - outcome = new(chunk.Index, Fold(parser, chunk, options, in context, ct), null); + outcome = new(chunk.Index, Fold(parser, chunk, options, in context, stop), null); } // Anything but the caller's cancellation, which stops everything, becomes this chunk's result: a chunk // that never completes would hold its slot, and the producer would wait for it forever. @@ -249,7 +262,7 @@ private static TAccumulator Fold( /// frees each chunk's slot then: a slow early chunk stalls the producer instead of letting later /// results pile up. /// - private sealed class OrderedMerge(SemaphoreSlim slots) + private sealed class OrderedMerge(SemaphoreSlim slots, CancellationTokenSource stop) where TModel : allows ref struct where TAccumulator : IRecordAccumulator, new() { @@ -260,11 +273,16 @@ private sealed class OrderedMerge(SemaphoreSlim slots) private TAccumulator? _merged; private ExceptionDispatchInfo? _failure; + /// True once a failure in source order decides the result. + public bool Failed => Volatile.Read(ref _failure) is not null; + public void Complete(Outcome outcome) { int released = 0; + bool failedNow; lock (_gate) { + bool failedBefore = _failure is not null; _pending.Add(outcome.Index, outcome); while (_pending.Remove(_next, out var ready)) { @@ -277,6 +295,12 @@ public void Complete(Outcome outcome) Accept(ready.Accumulator!); } } + failedNow = !failedBefore && _failure is not null; + } + // Outside the lock: cancelling runs the producer's and workers' callbacks. + if (failedNow) + { + stop.Cancel(); } if (released > 0) { diff --git a/src/FixedWidthParser/Writers/RecordWriter.cs b/src/FixedWidthParser/Writers/RecordWriter.cs index ff739a1..5306d30 100644 --- a/src/FixedWidthParser/Writers/RecordWriter.cs +++ b/src/FixedWidthParser/Writers/RecordWriter.cs @@ -19,8 +19,7 @@ public static void Write(Stream stream, in TModel model, TFo { var context = options.Context; Span record = stackalloc byte[StackRecordSize]; - int written = TryFormatRecord(in model, formatter, record, in context, options.NewLineBytes); - if (written > 0) + if (TryFormatRecord(in model, formatter, record, in context, options.NewLineBytes, out int written)) { stream.Write(record[..written]); return; @@ -104,8 +103,7 @@ public static async Task WriteManyAsync(Stream stream, IEnum int used = 0; foreach (var model in models) { - int written = TryFormatRecord(in model, formatter, buffer.AsSpan(used), in context, options.NewLineBytes); - if (written == 0) + if (!TryFormatRecord(in model, formatter, buffer.AsSpan(used), in context, options.NewLineBytes, out int written)) { await stream.WriteAsync(buffer.AsMemory(0, used), cancellationToken).ConfigureAwait(false); used = 0; @@ -125,8 +123,7 @@ private static int Append( Stream stream, in TModel model, TFormatter formatter, ref byte[] buffer, int used, in FixedWidthContext context, byte[] newLine) where TFormatter : struct, IRecordFormatter { - int written = TryFormatRecord(in model, formatter, buffer.AsSpan(used), in context, newLine); - if (written > 0) + if (TryFormatRecord(in model, formatter, buffer.AsSpan(used), in context, newLine, out int written)) { return used + written; } @@ -141,8 +138,7 @@ private static int FormatGrowing(in TModel model, TFormatter { while (true) { - int written = TryFormatRecord(in model, formatter, buffer, in context, newLine); - if (written > 0) + if (TryFormatRecord(in model, formatter, buffer, in context, newLine, out int written)) { return written; } @@ -152,16 +148,22 @@ private static int FormatGrowing(in TModel model, TFormatter } } - /// Formats a record and its line break into ; returns 0 when they do not fit. - private static int TryFormatRecord(in TModel model, TFormatter formatter, Span destination, in FixedWidthContext context, byte[] newLine) + /// + /// Formats a record and its line break into ; false when they do not fit. A record + /// with no columns and an empty line break fits in zero bytes, so the count alone cannot tell. + /// + private static bool TryFormatRecord( + in TModel model, TFormatter formatter, Span destination, in FixedWidthContext context, byte[] newLine, out int written) where TFormatter : struct, IRecordFormatter { - if (!formatter.TryFormat(in model, destination, in context, out int written) || destination.Length - written < newLine.Length) + if (!formatter.TryFormat(in model, destination, in context, out int record) || destination.Length - record < newLine.Length) { - return 0; + written = 0; + return false; } - newLine.CopyTo(destination[written..]); - return written + newLine.Length; + newLine.CopyTo(destination[record..]); + written = record + newLine.Length; + return true; } } } diff --git a/tests/FixedWidthParser.Generator.Tests/FixedWidthModelGeneratorTests.cs b/tests/FixedWidthParser.Generator.Tests/FixedWidthModelGeneratorTests.cs index 6ceb25b..ce9d1fd 100644 --- a/tests/FixedWidthParser.Generator.Tests/FixedWidthModelGeneratorTests.cs +++ b/tests/FixedWidthParser.Generator.Tests/FixedWidthModelGeneratorTests.cs @@ -274,6 +274,70 @@ public void InvalidModel_ReportsExpectedDiagnostic(string modelSource, string di Assert.Empty(result.GeneratedSources); } + [Fact] + public void KeywordNames_AreEscaped() + { + var result = Run(""" + public partial class @class : IFixedWidthModel<@class> + { + [FixedColumn(0, 3)] public int @event { get; set; } + [FixedColumn(3, 3)] public int? @int { get; set; } + } + """, ns: "Smoke.@namespace"); + + Assert.Empty(result.GeneratorDiagnostics); + Assert.Empty(result.CompilationErrors); + var generated = Assert.Single(result.GeneratedSources); + Assert.Contains("namespace Smoke.@namespace", generated, StringComparison.Ordinal); + Assert.Contains("ReportFailedColumn(\"event\"", generated, StringComparison.Ordinal); + } + + [Fact] + public void GenericModel_ReportsFwp013_AndOthersStillGenerate() + { + var result = Run(""" + public partial class Generic : IFixedWidthModel> + { + [FixedColumn(0, 3)] public int Value { get; set; } + } + + public partial class Plain : IFixedWidthModel + { + [FixedColumn(0, 3)] public int Value { get; set; } + } + """); + + var diagnostic = Assert.Single(result.GeneratorDiagnostics); + Assert.Equal("FWP013", diagnostic.Id); + Assert.Contains("Generic", diagnostic.GetMessage(System.Globalization.CultureInfo.InvariantCulture), StringComparison.Ordinal); + Assert.Contains("partial class Plain", Assert.Single(result.GeneratedSources), StringComparison.Ordinal); + } + + [Theory] + [InlineData(2147483646, 3, "FWP014")] + [InlineData(int.MaxValue, 1, "FWP014")] + [InlineData(2147483646, 1, null)] + public void ColumnEndingPastIntMaxValue_ReportsFwp014(int start, int length, string? diagnosticId) + { + var result = Run($$""" + public partial class FarModel : IFixedWidthModel + { + [FixedColumn({{start}}, {{length}})] public string Value { get; set; } = ""; + } + """); + + if (diagnosticId is null) + { + Assert.Empty(result.GeneratorDiagnostics); + Assert.Single(result.GeneratedSources); + } + else + { + Assert.Equal(diagnosticId, Assert.Single(result.GeneratorDiagnostics).Id); + Assert.Empty(result.GeneratedSources); + } + } + [Fact] public void AdjacentColumns_AreValid() { @@ -466,12 +530,12 @@ namespace GeneratorSmoke; .WithNullableContextOptions(NullableContextOptions.Enable)); } - private static GeneratorRunResult Run(string modelSource) + private static GeneratorRunResult Run(string modelSource, string ns = "GeneratorSmoke") { - string source = """ + string source = $""" using FixedWidthParser; - namespace GeneratorSmoke; + namespace {ns}; """ + modelSource; diff --git a/tests/FixedWidthParser.Generator.Tests/FixedWidthParser.Generator.Tests.csproj b/tests/FixedWidthParser.Generator.Tests/FixedWidthParser.Generator.Tests.csproj index 29760e0..246f10e 100644 --- a/tests/FixedWidthParser.Generator.Tests/FixedWidthParser.Generator.Tests.csproj +++ b/tests/FixedWidthParser.Generator.Tests/FixedWidthParser.Generator.Tests.csproj @@ -12,9 +12,9 @@ - + - + runtime; build; native; contentfiles; analyzers; buildtransitive all diff --git a/tests/FixedWidthParser.Tests/AllocationGuardTests.cs b/tests/FixedWidthParser.Tests/AllocationGuardTests.cs index 22edd22..9faf080 100644 --- a/tests/FixedWidthParser.Tests/AllocationGuardTests.cs +++ b/tests/FixedWidthParser.Tests/AllocationGuardTests.cs @@ -58,6 +58,42 @@ public void GeneratedPooledReader_AllocatesNothingPerLine() Assert.True(perLine <= 1, $"Generated pooled reader regressed to {perLine} B/line (expected ~0)."); } + // Generated, attribute and fluent layouts all parse enums through TryEnum. + [Theory] + [InlineData("-0001")] + [InlineData("-1")] + [InlineData("00001")] + public void ZeroPaddedEnumCode_AllocatesNothing(string text) + { + byte[] column = Encoding.ASCII.GetBytes(text); + var options = new Infrastructure.ColumnParseOptions(' ', TrimMode.Trailing, format: null); + + Assert.True(SettlesToZeroAllocation(() => Infrastructure.FixedWidthRuntime.TryEnum(column, default, in options, out Level _)), + $"Parsing enum code '{text}' still allocates once optimized."); + } + + /// + /// True once a round of calls allocates nothing. Unoptimized (tier-0) generic BCL code boxes where the + /// optimized code does not, so allocation is measured in rounds while the JIT tiers up, for about 3 s. + /// + private static bool SettlesToZeroAllocation(Action call) + { + for (int round = 0; round < 30; round++) + { + long before = GC.GetAllocatedBytesForCurrentThread(); + for (int i = 0; i < 1_000; i++) + { + call(); + } + if (GC.GetAllocatedBytesForCurrentThread() == before) + { + return true; + } + Thread.Sleep(100); + } + return false; + } + private static double PerCall(Func read) { const int n = 10_000; diff --git a/tests/FixedWidthParser.Tests/ColumnValidationTests.cs b/tests/FixedWidthParser.Tests/ColumnValidationTests.cs index 1620f2a..9d44e28 100644 --- a/tests/FixedWidthParser.Tests/ColumnValidationTests.cs +++ b/tests/FixedWidthParser.Tests/ColumnValidationTests.cs @@ -29,6 +29,28 @@ public void Parser_NegativeStart_Throws() Assert.Contains("Start", ex.Message, StringComparison.Ordinal); } + public sealed class FarEndModel + { + [FixedColumn(2147483646, 3)] public int Value { get; set; } + } + + public sealed class FarEdgeModel + { + [FixedColumn(2147483646, 1)] public int Value { get; set; } + } + + [Fact] + public void ColumnEndingPastIntMaxValue_Throws() + { + var attributes = Assert.Throws(() => FixedWidth.TryParse(ReadOnlySpan.Empty, FixedWidthLayout.FromAttributes(), Ctx, out _)); + var fluent = Assert.Throws(() => FixedWidthLayout.Build(b => b + .Column("Value", 2147483646, 3, (in FarEndModel m) => m.Value, (ref FarEndModel m, int v) => m.Value = v))); + + Assert.Contains("ends past int.MaxValue", attributes.Message, StringComparison.Ordinal); + Assert.Contains("ends past int.MaxValue", fluent.Message, StringComparison.Ordinal); + Assert.False(FixedWidth.TryParse(ReadOnlySpan.Empty, FixedWidthLayout.FromAttributes(), Ctx, out _)); + } + [Fact] public void Writer_ZeroLength_Throws() { diff --git a/tests/FixedWidthParser.Tests/ConverterAttributeTests.cs b/tests/FixedWidthParser.Tests/ConverterAttributeTests.cs index d521278..c184357 100644 --- a/tests/FixedWidthParser.Tests/ConverterAttributeTests.cs +++ b/tests/FixedWidthParser.Tests/ConverterAttributeTests.cs @@ -41,13 +41,26 @@ public void Converter_RejectingInput_ReturnsFalse() [Fact] public void MismatchedConverter_ParserConstruction_Throws() { - Assert.Throws(() => FixedWidth.TryParse(ReadOnlySpan.Empty, FixedWidthLayout.FromAttributes(), Ctx, out _)); + var ex = Assert.Throws(() => FixedWidth.TryParse(ReadOnlySpan.Empty, FixedWidthLayout.FromAttributes(), Ctx, out _)); + Assert.Contains("for column 'Value' must implement", ex.Message, StringComparison.Ordinal); } [Fact] public void MismatchedConverter_WriterConstruction_Throws() { - Assert.Throws(() => FixedWidth.WriteMany(Stream.Null, Array.Empty(), FixedWidthLayout.FromAttributes())); + var ex = Assert.Throws(() => FixedWidth.WriteMany(Stream.Null, Array.Empty(), FixedWidthLayout.FromAttributes())); + Assert.Contains("for column 'Value' must implement", ex.Message, StringComparison.Ordinal); + } + + // A converter may carry state, so the reader and the writer each get their own. + [Fact] + public void Converter_ReaderAndWriter_EachGetTheirOwnInstance() + { + var layout = FixedWidthLayout.FromAttributes(); + + Assert.True(FixedWidth.TryParse("42 "u8, layout, Ctx, out var parsed)); + Assert.Equal("42 ", WriteOne(layout, parsed)); + Assert.Equal(2, CountedConverter.Created); } } } diff --git a/tests/FixedWidthParser.Tests/CorrectnessTests.cs b/tests/FixedWidthParser.Tests/CorrectnessTests.cs index e06ad70..1c46a61 100644 --- a/tests/FixedWidthParser.Tests/CorrectnessTests.cs +++ b/tests/FixedWidthParser.Tests/CorrectnessTests.cs @@ -69,6 +69,31 @@ public void Nullable_BlankDetection_WithCustomTrimChar() Assert.Equal(padded, Format(generatedPadding)); } + [Theory] + [InlineData(" 0 0 * * *")] + [InlineData("0 0 0*** *")] + [InlineData(" 0 ** * ")] + public void Nullable_BlankDetection_MixedSpacesAndTrimChar(string text) + { + Assert.True(FixedWidth.TryParse(Bytes(text), FixedWidthLayout.FromAttributes(), Ctx, out var attributes)); + Assert.Null(attributes.Value); + Assert.Null(attributes.DoubleValue); + Assert.True(FixedWidth.TryParse(Bytes(text), Ctx, out var generated)); + Assert.Null(generated.Value); + Assert.Null(generated.DoubleValue); + } + + [Fact] + public void Generated_KeywordNames_ParseAndFormat() + { + Assert.True(FixedWidth.TryParse<@operator>(Bytes("42 7 "), Ctx, out var model)); + Assert.Equal(42, model.@event); + Assert.Equal(7, model.@int); + Assert.Equal("42 7 ", Format(model)); + Assert.True(FixedWidth.TryParse<@operator>(Bytes("42 "), Ctx, out var blank)); + Assert.Null(blank.@int); + } + [Fact] public void SignAware_ZeroPadding() { diff --git a/tests/FixedWidthParser.Tests/CultureTests.cs b/tests/FixedWidthParser.Tests/CultureTests.cs index f6ca75a..68d51ee 100644 --- a/tests/FixedWidthParser.Tests/CultureTests.cs +++ b/tests/FixedWidthParser.Tests/CultureTests.cs @@ -89,6 +89,43 @@ public void Parse_Double_CommaCulture_MatchesTheBcl(string cultureName) } } + // The fast paths read '+' as the positive sign, so a provider with another one must go to the BCL. + [Fact] + public void Parse_Numbers_CustomPositiveSign_MatchTheBcl() + { + var culture = (CultureInfo)CultureInfo.InvariantCulture.Clone(); + culture.NumberFormat.PositiveSign = "p"; + var context = CtxWith(culture); + var options = new Infrastructure.ColumnParseOptions(' ', TrimMode.Both, format: null); + + foreach (string text in new[] { "+1", "p1", "-1", "1.5", "+1.5", "p1.5" }) + { + bool doubleOk = Infrastructure.FixedWidthRuntime.TryDouble(Bytes(text), in context, in options, out double d); + bool floatOk = Infrastructure.FixedWidthRuntime.TryFloat(Bytes(text), in context, in options, out float f); + bool decimalOk = Infrastructure.FixedWidthRuntime.TryParse(Bytes(text), in context, in options, out decimal m); + + Assert.Equal(double.TryParse(text, NumberStyles.Float | NumberStyles.AllowThousands, culture, out double bclDouble), doubleOk); + Assert.Equal(float.TryParse(text, NumberStyles.Float | NumberStyles.AllowThousands, culture, out float bclFloat), floatOk); + Assert.Equal(decimal.TryParse(text, NumberStyles.Number, culture, out decimal bclDecimal), decimalOk); + Assert.Equal((bclDouble, bclFloat, bclDecimal), (d, f, m)); + } + Assert.False(Infrastructure.FixedWidthRuntime.TryDouble(Bytes("+1"), in context, in options, out _)); + } + + // A Latin-1 no-break space (0xA0) groups digits here; read as UTF-8 it is not a character at all. + [Fact] + public void Parse_Decimal_Latin1NoBreakSpaceGroups_IsDecoded() + { + var culture = (CultureInfo)CultureInfo.InvariantCulture.Clone(); + culture.NumberFormat.NumberGroupSeparator = " "; + var context = new FixedWidthContext { FormatProvider = culture, Encoding = System.Text.Encoding.Latin1 }; + var options = new Infrastructure.ColumnParseOptions(' ', TrimMode.Both, format: null); + + Assert.True(Infrastructure.FixedWidthRuntime.TryParse(System.Text.Encoding.Latin1.GetBytes("1 234.50"), in context, in options, out decimal value)); + Assert.Equal(1234.50m, value); + Assert.False(Infrastructure.FixedWidthRuntime.TryParse(System.Text.Encoding.Latin1.GetBytes("1 234.5x"), in context, in options, out decimal _)); + } + // The digit-only fast path for exact formats must agree with TryParseExact, including where it must // step aside: a Buddhist calendar (th-TH) and "/" or ":" standing for other separators (de-DE). [Theory] diff --git a/tests/FixedWidthParser.Tests/EnumColumnTests.cs b/tests/FixedWidthParser.Tests/EnumColumnTests.cs index 9b3b984..f7e93fa 100644 --- a/tests/FixedWidthParser.Tests/EnumColumnTests.cs +++ b/tests/FixedWidthParser.Tests/EnumColumnTests.cs @@ -38,6 +38,30 @@ public partial class GenEnumCodeModel : IFixedWidthModel [FixedColumn(0, 2, Format = "D", Padding = '0', Alignment = Alignment.Right)] public Side Side { get; set; } } + public enum Phase { Open = 1, Started = 1, Closed = 2 } + + public sealed class PhaseModel + { + [FixedColumn(0, 10)] public Phase Phase { get; set; } + } + + public partial class GenPhaseModel : IFixedWidthModel + { + [FixedColumn(0, 10)] public Phase Phase { get; set; } + } + + public enum Level { Low = -1, High = 1 } + + public sealed class LevelCodeModel + { + [FixedColumn(0, 5, Format = "D", Padding = '0', Alignment = Alignment.Right)] public Level Level { get; set; } + } + + public partial class GenLevelCodeModel : IFixedWidthModel + { + [FixedColumn(0, 5, Format = "D", Padding = '0', Alignment = Alignment.Right)] public Level Level { get; set; } + } + /// /// Enum columns, as ExcelReader reads them: a member's name in any case or its number, nothing the enum does not /// define. Attribute, generated and fluent layouts agree. @@ -75,6 +99,42 @@ public void NameOrNumber_Parses(string text, Side expected) Assert.Equal(expected, fluent.Side); } + [Theory] + [InlineData("Open")] + [InlineData("started")] + [InlineData("STARTED")] + [InlineData("01")] + public void Alias_ParsesLikeTheNameItShares(string text) + { + var fluent = FixedWidthLayout.Build(b => b + .Factory(() => new PhaseModel()) + .EnumColumn("Phase", 0, 10, (in PhaseModel m) => m.Phase, (ref PhaseModel m, Phase v) => m.Phase = v)); + byte[] line = Bytes(text.PadRight(10)); + + Assert.True(FixedWidth.TryParse(line, FixedWidthLayout.FromAttributes(), Ctx, out var attributes)); + Assert.True(FixedWidth.TryParse(line, Ctx, out var generated)); + Assert.True(FixedWidth.TryParse(line, fluent, Ctx, out var fromFluent)); + Assert.Equal(Phase.Open, attributes.Phase); + Assert.Equal(Phase.Open, generated.Phase); + Assert.Equal(Phase.Open, fromFluent.Phase); + } + + [Fact] + public void NegativeCode_ZeroPadded_RoundTrips() + { + var layout = FixedWidthLayout.FromAttributes(); + + Assert.Equal("-0001", WriteOne(layout, new LevelCodeModel { Level = Level.Low })); + Assert.Equal("-0001", Format(new GenLevelCodeModel { Level = Level.Low })); + Assert.True(FixedWidth.TryParse("-0001"u8, layout, Ctx, out var attributes)); + Assert.True(FixedWidth.TryParse("-0001"u8, Ctx, out var generated)); + Assert.Equal(Level.Low, attributes.Level); + Assert.Equal(Level.Low, generated.Level); + Assert.False(FixedWidth.TryParse("-0002"u8, layout, Ctx, out _)); + Assert.False(FixedWidth.TryParse("-0000"u8, layout, Ctx, out _)); + Assert.False(FixedWidth.TryParse("--001"u8, layout, Ctx, out _)); + } + [Theory] [InlineData("Hold")] [InlineData("3")] diff --git a/tests/FixedWidthParser.Tests/FixedWidthParser.Tests.csproj b/tests/FixedWidthParser.Tests/FixedWidthParser.Tests.csproj index 160ccbd..4a84942 100644 --- a/tests/FixedWidthParser.Tests/FixedWidthParser.Tests.csproj +++ b/tests/FixedWidthParser.Tests/FixedWidthParser.Tests.csproj @@ -10,6 +10,9 @@ false true Exe + + false true $(NoWarn);HLQ005;CA1707;CA1815;S3241;CA1051;S6640 @@ -21,9 +24,9 @@ - + - + runtime; build; native; contentfiles; analyzers; buildtransitive all diff --git a/tests/FixedWidthParser.Tests/GeneratedModels.cs b/tests/FixedWidthParser.Tests/GeneratedModels.cs index 1c94964..bcd59ff 100644 --- a/tests/FixedWidthParser.Tests/GeneratedModels.cs +++ b/tests/FixedWidthParser.Tests/GeneratedModels.cs @@ -193,4 +193,16 @@ public sealed partial class GenDerivedModel : GenBaseModel, IFixedWidthModelGenerated model whose type and members are C# keywords, declared with '@'. + public sealed partial class @operator : IFixedWidthModel<@operator> + { + [FixedColumn(0, 3)] public int @event { get; set; } + [FixedColumn(3, 3)] public int? @int { get; set; } + } + + /// Generated model with no columns: it formats as zero bytes. + public sealed partial class EmptyGenModel : IFixedWidthModel + { + } } diff --git a/tests/FixedWidthParser.Tests/LayoutWriteTests.cs b/tests/FixedWidthParser.Tests/LayoutWriteTests.cs index 73d85a3..f346cc2 100644 --- a/tests/FixedWidthParser.Tests/LayoutWriteTests.cs +++ b/tests/FixedWidthParser.Tests/LayoutWriteTests.cs @@ -140,6 +140,22 @@ public async Task WriteAsync_MatchesWrite() Assert.Equal(Decode(sync), Decode(async)); } + [Fact] + public async Task EmptyRecord_WithoutNewLine_WritesNothing() + { + var none = new FixedWidthWriterOptions { NewLine = "" }; + EmptyGenModel[] models = [new(), new()]; + await using var stream = new MemoryStream(); + + FixedWidth.Write(stream, models[0], none); + FixedWidth.WriteMany(stream, models, none); + FixedWidth.WriteMany(stream, (ReadOnlySpan)models, none); + await FixedWidth.WriteAsync(stream, models[0], none, TestContext.Current.CancellationToken); + await FixedWidth.WriteManyAsync(stream, models, none, TestContext.Current.CancellationToken); + + Assert.Equal(0, stream.Length); + } + [Fact] public void Write_NullNewLine_ThrowsArgumentException() { diff --git a/tests/FixedWidthParser.Tests/Native/NativeLayoutTests.cs b/tests/FixedWidthParser.Tests/Native/NativeLayoutTests.cs index b2108e1..ea4a108 100644 --- a/tests/FixedWidthParser.Tests/Native/NativeLayoutTests.cs +++ b/tests/FixedWidthParser.Tests/Native/NativeLayoutTests.cs @@ -57,6 +57,12 @@ public void Create_NegativeStart_Fails() AssertInvalid("negative Start (-1)", new Col("a", -1, 5, TString)); } + [Fact] + public void Create_EndPastIntMaxValue_Fails() + { + AssertInvalid("ends past int.MaxValue", new Col("a", 2147483646, 3, TString)); + } + [Fact] public void Create_ZeroLength_Fails() { diff --git a/tests/FixedWidthParser.Tests/Native/NativeLimitsTests.cs b/tests/FixedWidthParser.Tests/Native/NativeLimitsTests.cs index e3c986e..099ccb2 100644 --- a/tests/FixedWidthParser.Tests/Native/NativeLimitsTests.cs +++ b/tests/FixedWidthParser.Tests/Native/NativeLimitsTests.cs @@ -65,6 +65,31 @@ public void BufferGrowth_PastHalfOfInt_ClampsInsteadOfOverflowing() Assert.Throws(() => GrowableNativeBuffer.NextCapacity(int.MaxValue, (long)int.MaxValue + 1)); } + // A TableArena reserves a STRING column's full width on every row; trimmed text often uses a sliver of it. + [Fact] + public unsafe void TrimExcess_ShrinksOnlyAnOwnedMostlyEmptyBlock() + { + using var sparse = new GrowableNativeBuffer(); + sparse.Reserve(1000); + "OK"u8.CopyTo(sparse.Append(2)); + sparse.TrimExcess(2); + Assert.Equal(2, sparse.Capacity); + Assert.True(sparse.Span.SequenceEqual("OK"u8)); + + using var dense = new GrowableNativeBuffer(); + dense.Reserve(1000); + dense.Append(600); + dense.TrimExcess(600); + Assert.Equal(1000, dense.Capacity); + + byte* lent = stackalloc byte[1000]; + using var borrowed = new GrowableNativeBuffer(); + borrowed.Borrow(lent, 1000); + borrowed.TrimExcess(2); + Assert.Equal(1000, borrowed.Capacity); + Assert.True(borrowed.Pointer == lent); + } + [Fact] public void DecimalBuffer_SizeIsComputedIn64Bits() { diff --git a/tests/FixedWidthParser.Tests/Native/NativeReadTests.cs b/tests/FixedWidthParser.Tests/Native/NativeReadTests.cs index 4e9f565..ff5f73d 100644 --- a/tests/FixedWidthParser.Tests/Native/NativeReadTests.cs +++ b/tests/FixedWidthParser.Tests/Native/NativeReadTests.cs @@ -207,6 +207,42 @@ public void Read_RejectedLineMidway_LeavesNoPartialValues() } } + [Fact] + public void Read_NullableMixedSpacesAndTrimChar_IsNull() + { + var layout = Build(new Col("n", 0, 5, TInt64) { Nullable = true, TrimChar = '0' }); + + foreach (var outcome in Ok(" 0 0 \n0 0 0\n00 12\n", layout)) + { + Assert.Equal([null, null, 12L], outcome.Column(0)); + } + } + + /// A valid row, one whose value overflows once scaled, another valid row. + internal const string DecimalOverflow = " 0.000000000000001\n79228162514264337593543950335.0\n 0.000000000000002\n"; + + internal static NativeLayout DecimalOverflowLayout() + { + return Build(new Col("d", 0, 31, TDecimal) { Scale = 18 }); + } + + [Fact] + public void Read_DecimalOverflowingTheScale_IsSkippedOrParseError() + { + var layout = DecimalOverflowLayout(); + + foreach (var outcome in Ok(DecimalOverflow, layout, Options(skipInvalid: true))) + { + Assert.Equal([1000L, 2000L], outcome.Column(0)); + Assert.Equal(1, outcome.LinesSkipped); + } + foreach (var outcome in Both(DecimalOverflow, layout)) + { + Assert.Equal(NativeStatus.ParseError, outcome.Status); + Assert.Contains("column 'd'", outcome.Error, StringComparison.Ordinal); + } + } + [Fact] public void Read_ParseError_ReportsLineAndColumn() { diff --git a/tests/FixedWidthParser.Tests/Native/NativeReaderTests.cs b/tests/FixedWidthParser.Tests/Native/NativeReaderTests.cs index ae1ddd6..fc0a5a3 100644 --- a/tests/FixedWidthParser.Tests/Native/NativeReaderTests.cs +++ b/tests/FixedWidthParser.Tests/Native/NativeReaderTests.cs @@ -93,6 +93,21 @@ public void Reader_File_EqualsMemory() } } + [Fact] + public void Reader_DecimalOverflowingTheScale_IsSkipped() + { + byte[] content = Encoding.UTF8.GetBytes(NativeReadTests.DecimalOverflow); + var options = ReaderOptions(1) with { Read = Options(skipInvalid: true) }; + fixed (byte* data = content) + { + Assert.Equal(NativeStatus.Ok, ReaderApi.OpenMemory(data, content.Length, NativeReadTests.DecimalOverflowLayout(), &options, out nint reader)); + var batches = Drain(reader); + ReaderApi.Close(reader); + + Assert.Equal([1000L, 2000L], batches.SelectMany(b => b.Column(0))); + } + } + [Fact] public void Reader_AfterEof_ReturnsEof() { diff --git a/tests/FixedWidthParser.Tests/Native/NativeValidityMergeTests.cs b/tests/FixedWidthParser.Tests/Native/NativeValidityMergeTests.cs new file mode 100644 index 0000000..52b0bee --- /dev/null +++ b/tests/FixedWidthParser.Tests/Native/NativeValidityMergeTests.cs @@ -0,0 +1,86 @@ +using System.Runtime.InteropServices; +using FixedWidthParser.Native.Layouts; +using FixedWidthParser.Native.Reading; + +namespace FixedWidthParser.Tests.Native +{ + /// Merging the partitions' validity bitmaps must match a merge done one row at a time, at every bit offset. + public sealed unsafe class NativeValidityMergeTests + { + private static readonly NativeColumn Column = new() + { + Name = "n", + Length = 1, + Type = NativeType.Int64, + Nullable = true, + Parse = new(' ', TrimMode.Both, null), + }; + + private static bool[] Nulls(int rows, Func isNull) + { + return [.. Enumerable.Range(0, rows).Select(isNull)]; + } + + // The lead partition's length puts every later partition's first row at each bit offset in turn. + [Theory] + [InlineData(0)] + [InlineData(1)] + [InlineData(2)] + [InlineData(3)] + [InlineData(4)] + [InlineData(5)] + [InlineData(6)] + [InlineData(7)] + [InlineData(8)] + public void Concat_Validity_MatchesTheRowByRowMerge(int lead) + { + bool[][] partitions = + [ + Nulls(lead, r => r % 3 == 0), + Nulls(37, r => (r * 7 + lead) % 10 == 0), + Nulls(16, _ => false), + Nulls(24, _ => true), + Nulls(0, _ => false), + Nulls(65, r => r is 0 or 64), + Nulls(9, r => r == 8), + Nulls(8, r => r == 7), + ]; + var parts = partitions.Select(nulls => + { + var part = new ColumnBuilder(Column, int.MaxValue, nulls.Length); + foreach (bool isNull in nulls) + { + Assert.True(part.TryAppend(isNull ? " "u8 : "1"u8, default)); + } + return part; + }).ToArray(); + NativeColumnRaw merged = default; + try + { + ColumnBuilder.Concat(parts, &merged); + + bool[] all = [.. partitions.SelectMany(p => p)]; + byte[] expected = new byte[(all.Length + 7) / 8]; + expected.AsSpan().Fill(0xFF); + for (int row = 0; row < all.Length; row++) + { + if (all[row]) + { + expected[row >> 3] &= (byte)~(1 << (row & 7)); + } + } + Assert.Equal(all.Length, merged.Length); + Assert.Equal(expected, new ReadOnlySpan(merged.Validity, expected.Length).ToArray()); + } + finally + { + Marshal.FreeHGlobal((nint)merged.Values); + Marshal.FreeHGlobal((nint)merged.Validity); + foreach (var part in parts) + { + part.Dispose(); + } + } + } + } +} diff --git a/tests/FixedWidthParser.Tests/Native/NativeWriteTests.cs b/tests/FixedWidthParser.Tests/Native/NativeWriteTests.cs index 75abb9e..79f5aae 100644 --- a/tests/FixedWidthParser.Tests/Native/NativeWriteTests.cs +++ b/tests/FixedWidthParser.Tests/Native/NativeWriteTests.cs @@ -208,6 +208,56 @@ public void Write_File_EqualsMemory() } } + [Fact] + public void Write_FileFailure_KeepsTheExistingFile() + { + var read = Build(new Col("n", 0, 2, TInt64)); + var write = Build(new Col("n", 0, 1, TInt64)); + var table = ReadTable("12\n"u8.ToArray(), read); + string directory = Directory.CreateTempSubdirectory().FullName; + string path = Path.Combine(directory, "out.txt"); + try + { + File.WriteAllText(path, "ORIGINAL"); + + Assert.Equal(NativeStatus.Error, WriteApi.WriteFile(Encoding.UTF8.GetBytes(path), &table, write, null)); + + Assert.Equal("ORIGINAL", File.ReadAllText(path)); + Assert.Equal([path], Directory.GetFiles(directory)); + } + finally + { + NativeTable.Free(&table); + Directory.Delete(directory, recursive: true); + } + } + + [Theory] + [InlineData("out.txt")] + [InlineData(null)] + public void Write_File_ReplacesTheExistingFile(string? name) + { + var layout = EveryTypeLayout(); + var table = ReadTable(Encoding.UTF8.GetBytes(EveryType), layout); + string directory = Directory.CreateTempSubdirectory().FullName; + // null stands for the longest name a file may have: 255 characters on every supported platform. + string path = Path.Combine(directory, name ?? new string('a', 251) + ".txt"); + try + { + File.WriteAllText(path, new string('x', 1000)); + + Assert.Equal(NativeStatus.Ok, WriteApi.WriteFile(Encoding.UTF8.GetBytes(path), &table, layout, null)); + + Assert.Equal(WriteMemory(&table, layout).Bytes, File.ReadAllBytes(path)); + Assert.Equal([path], Directory.GetFiles(directory)); + } + finally + { + NativeTable.Free(&table); + Directory.Delete(directory, recursive: true); + } + } + [Theory] [InlineData(1, 0)] [InlineData(4, 1)] @@ -348,6 +398,108 @@ public void Write_TemporalWithoutFormat_IsIso() Assert.Equal("2023-10-24\n", Encoding.UTF8.GetString(RoundTrip("20231024\n", read, write))); } + /// Writes one string row whose offsets and declared length are given, over five allocated bytes "ABCDE". + private static (int Status, byte[] Bytes, string Error) WriteString(int start, int end, long dataLen) + { + int* offsets = stackalloc int[] { start, end }; + byte* data = stackalloc byte[] { (byte)'A', (byte)'B', (byte)'C', (byte)'D', (byte)'E' }; + var column = new NativeColumnRaw { Type = TString, Length = 1, Values = offsets, Data = data, DataLen = dataLen }; + var table = new NativeTableRaw { ColumnCount = 1, RowCount = 1, Columns = &column }; + return WriteMemory(&table, Build(new Col("s", 0, 5, TString))); + } + + [Theory] + [InlineData(0, 5, 1L)] + [InlineData(3, 1, 5L)] + [InlineData(-1, 2, 5L)] + [InlineData(0, 0, -1L)] + public void Write_StringOffsetsOutsideTheData_AreInvalidArgument(int start, int end, long dataLen) + { + var (status, _, error) = WriteString(start, end, dataLen); + + Assert.Equal(NativeStatus.InvalidArgument, status); + Assert.Contains("'s'", error, StringComparison.Ordinal); + } + + [Fact] + public void Write_StringOffsetsWithinTheData_Write() + { + var (status, bytes, error) = WriteString(1, 4, 5); + + Assert.True(status == NativeStatus.Ok, error); + Assert.Equal("BCD \n", Encoding.UTF8.GetString(bytes)); + } + + private static (int Status, byte[] Bytes, string Error) WriteDecimal(long unscaled, NativeLayout layout) + { + var column = new NativeColumnRaw { Type = TDecimal, Length = 1, Values = &unscaled }; + var table = new NativeTableRaw { ColumnCount = 1, RowCount = 1, Columns = &column }; + return WriteMemory(&table, layout); + } + + [Theory] + [InlineData(999_999_999_999_999_999L)] + [InlineData(-999_999_999_999_999_999L)] + public void Write_Decimal18Digits_ReadsBack(long unscaled) + { + var layout = Build(new Col("d", 0, 20, TDecimal)); + + var (status, bytes, error) = WriteDecimal(unscaled, layout); + + Assert.True(status == NativeStatus.Ok, error); + Assert.Equal([unscaled], FromMemory(bytes, layout).Column(0)); + } + + [Theory] + [InlineData(1_000_000_000_000_000_000L)] + [InlineData(-1_000_000_000_000_000_000L)] + public void Write_DecimalOver18Digits_IsError(long unscaled) + { + var (status, _, error) = WriteDecimal(unscaled, Build(new Col("d", 0, 20, TDecimal))); + + Assert.Equal(NativeStatus.Error, status); + Assert.Contains("\"d\"", error, StringComparison.Ordinal); + } + + private static (int Status, byte[] Bytes, string Error) WriteMicros(int type, long micros) + { + var column = new NativeColumnRaw { Type = type, Length = 1, Values = µs }; + var table = new NativeTableRaw { ColumnCount = 1, RowCount = 1, Columns = &column }; + string format = type == TTime ? "HHmmss" : "yyyyMMddHHmmss"; + return WriteMemory(&table, Build(new Col("t", 0, format.Length, type) { Format = format })); + } + + [Theory] + [InlineData(TTime, -1L)] + [InlineData(TTime, 86_400_000_000L)] + [InlineData(TTime, 1844674407370955162L)] + [InlineData(TTimestamp, 1844674407370955162L)] + [InlineData(TTimestamp, 253402300800000000L)] + [InlineData(TTimestamp, -62135596800000001L)] + [InlineData(TTimestamp, long.MaxValue)] + [InlineData(TTimestamp, long.MinValue)] + public void Write_TemporalOutOfRange_IsError(int type, long micros) + { + var (status, _, error) = WriteMicros(type, micros); + + Assert.Equal(NativeStatus.Error, status); + Assert.Contains("\"t\"", error, StringComparison.Ordinal); + } + + [Theory] + [InlineData(TTime, 0L, "000000\n")] + [InlineData(TTime, 86_399_999_999L, "235959\n")] + [InlineData(TTimestamp, 1698154200000000L, "20231024133000\n")] + [InlineData(TTimestamp, 253402300799999999L, "99991231235959\n")] + [InlineData(TTimestamp, -62135596800000000L, "00010101000000\n")] + public void Write_TemporalEndpoints_Format(int type, long micros, string expected) + { + var (status, bytes, error) = WriteMicros(type, micros); + + Assert.True(status == NativeStatus.Ok, error); + Assert.Equal(expected, Encoding.UTF8.GetString(bytes)); + } + [Fact] public void Write_EmptyTable_IsEmpty() { diff --git a/tests/FixedWidthParser.Tests/ParallelAggregateTests.cs b/tests/FixedWidthParser.Tests/ParallelAggregateTests.cs index 5510bf3..e007a10 100644 --- a/tests/FixedWidthParser.Tests/ParallelAggregateTests.cs +++ b/tests/FixedWidthParser.Tests/ParallelAggregateTests.cs @@ -98,6 +98,90 @@ public void Merge(CancelsItself following) } } + public sealed class ThrowsOnAdd : IRecordAccumulator + { + public void Add(in NameAgeModel model) + { + throw new InvalidOperationException("Add failed"); + } + + public void Merge(ThrowsOnAdd following) + { + } + } + + public sealed class ThrowsOnMerge : IRecordAccumulator + { + public void Add(in NameAgeModel model) + { + } + + public void Merge(ThrowsOnMerge following) + { + throw new InvalidOperationException("Merge failed"); + } + } + + /// A live, non-seekable source: hands over its prefix, then waits for data until cancelled. + private sealed class OpenEndedStream(byte[] prefix) : Stream + { + private int _position; + + public override bool CanRead => true; + + public override bool CanSeek => false; + + public override bool CanWrite => false; + + public override long Length => throw new NotSupportedException(); + + public override long Position { get => throw new NotSupportedException(); set => throw new NotSupportedException(); } + + public override async ValueTask ReadAsync(Memory buffer, CancellationToken cancellationToken = default) + { + if (_position < prefix.Length) + { + int count = Math.Min(buffer.Length, prefix.Length - _position); + prefix.AsMemory(_position, count).CopyTo(buffer); + _position += count; + return count; + } + await Task.Delay(Timeout.Infinite, cancellationToken).ConfigureAwait(false); + return 0; + } + + public override int Read(byte[] buffer, int offset, int count) + { + throw new NotSupportedException(); + } + + public override void Flush() + { + } + + public override long Seek(long offset, SeekOrigin origin) + { + throw new NotSupportedException(); + } + + public override void SetLength(long value) + { + throw new NotSupportedException(); + } + + public override void Write(byte[] buffer, int offset, int count) + { + throw new NotSupportedException(); + } + } + + public enum Failure + { + Parse, + Add, + Merge, + } + public static TheoryData SourcesAndChunkSizes() { var data = new TheoryData(); @@ -239,6 +323,32 @@ public async Task ParseFailure_ReportsTheSequentialLineNumber(Source source, int Assert.Equal(expected.Message, actual.Message); } + // Chunks of 16 bytes hold one line each: the source hands over three before it waits. A failure that decides + // the result must end the read there, not when the stream ends, which this one never does. + [Theory] + [InlineData(Failure.Parse)] + [InlineData(Failure.Add)] + [InlineData(Failure.Merge)] + public async Task FatalFailure_StopsReadingAnOpenStream(Failure failure) + { + byte[] prefix = Encoding.ASCII.GetBytes((failure == Failure.Parse ? "Name1 XX \n" : "Name1 001\n") + "Name2 002\nName3 003\nName4 004\n"); + using var timeout = CancellationTokenSource.CreateLinkedTokenSource(TestContext.Current.CancellationToken); + timeout.CancelAfter(TimeSpan.FromSeconds(10)); + var layout = FixedWidthLayout.FromAttributes(); + var options = new FixedWidthReaderOptions { DegreeOfParallelism = 2, ChunkSize = 16 }; + await using var stream = new OpenEndedStream(prefix); + + var thrown = await Record.ExceptionAsync(() => failure switch + { + Failure.Add => FixedWidth.AggregateAsync(stream, layout, leaveOpen: false, options, timeout.Token), + Failure.Merge => FixedWidth.AggregateAsync(stream, layout, leaveOpen: false, options, timeout.Token), + _ => FixedWidth.AggregateAsync(stream, layout, leaveOpen: false, options, timeout.Token), + }); + + Assert.False(timeout.IsCancellationRequested, "The failure waited for the stream to end."); + Assert.IsType(failure == Failure.Parse ? typeof(FixedWidthParseException) : typeof(InvalidOperationException), thrown); + } + [Theory] [MemberData(nameof(Sources))] public async Task OnParseError_SkipsEachBadLineOnce_WithItsLineNumber(Source source) diff --git a/tests/FixedWidthParser.Tests/TestModels.cs b/tests/FixedWidthParser.Tests/TestModels.cs index 9f97fe3..b0b8c38 100644 --- a/tests/FixedWidthParser.Tests/TestModels.cs +++ b/tests/FixedWidthParser.Tests/TestModels.cs @@ -40,6 +40,35 @@ public bool TryFormat(int value, Span destination, in FixedWidthContext co return value.TryFormat(destination, out bytesWritten, provider: context.FormatProvider); } } + + /// Counts its instances; used by one test only, so the count is that test's. + public sealed class CountedConverter : IFixedWidthConverter + { + private static int _created; + + public CountedConverter() + { + Interlocked.Increment(ref _created); + } + + public static int Created => Volatile.Read(ref _created); + + public bool TryParse(ReadOnlySpan column, in FixedWidthContext context, out int value) + { + return int.TryParse(column, context.FormatProvider, out value); + } + + public bool TryFormat(int value, Span destination, in FixedWidthContext context, out int bytesWritten) + { + return value.TryFormat(destination, out bytesWritten, provider: context.FormatProvider); + } + } + + public sealed class CountedConverterModel + { + [FixedColumn(0, 5, Converter = typeof(CountedConverter))] public int Value { get; set; } + } + /// Single custom-converted column, exercised by both the reflection parser/writer and the generator. public readonly record struct CentsConverterModel {