Parquet-on-FHIR: implement INT96 typed row writer for date range annotations

[cursor]

Summary

Parquet-on-FHIR specifies date range annotation columns (__field_start / __field_end) as deprecated INT96 physical type with TIMESTAMP(MILLIS) logical type. HAIStack currently exports these as INT64 TIMESTAMP(MILLIS), documented as an intentional deviation in docs/parquet-on-fhir-interop.md.

Why not implemented today

The FHIR parquet encoder uses parquet.NewGenericWriter[map[string]any] (pkg/parquetfhir/writer.go). parquet-go map writers cannot encode INT96 values; attempting to do so panics or fails at write time.

Impact

• Filtering / analytics: INT64 TIMESTAMP(MILLIS) is equivalent for range queries in DuckDB, Spark, and Trino.
• Strict spec compliance / lossless round-trip: Readers or validators that require literal INT96 physical type will reject or flag our files.
• Interop certification: Full aehrc Parquet-on-FHIR conformance may require INT96 unless explicitly waived.

Proposed approach

1. Introduce a typed row writer (struct-based or column-oriented) for annotation timestamp fields, or a custom parquet-go column writer path that emits INT96.
2. Keep INT64 as a fallback behind a feature flag or _parquetTimestampEncoding=int64|int96 export parameter for engines that prefer modern TIMESTAMP columns.
3. Add golden tests comparing INT96 column physical types against aehrc reference parquet fixtures once implemented.

References

• Compatibility matrix: docs/parquet-on-fhir-interop.md
• Spec: https://github.com/aehrc/parquet-on-fhir
• Related PR: Apache Parquet phases 1–4: flat, streaming, lakehouse, Parquet-on-FHIR interop #41 (Parquet-on-FHIR export)

Acceptance criteria

• Date annotation columns can be written as INT96 + TIMESTAMP(MILLIS) when configured
• Existing INT64 path remains default for parquet-go map writer compatibility
• Automated test asserts physical type for at least one Patient/Observation example
• Compatibility matrix updated with encoding switch behavior