BlobStore: streaming upload API to avoid O(file) memory at parquet export

[cursor]

Summary

Parquet export and lakehouse blob paths encode parquet in a streaming fashion (temp NDJSON spill + row-group encoding), but upload still loads the entire finished file into memory before calling store.BlobStore.Put. Peak RAM at upload time is therefore O(compressed parquet file size), not bounded by row-group size.

This is documented as an intentional limitation in pkg/view/README.md, docs/parquet-on-fhir-interop.md, and docs/CHANGELOG-parquet-analytics.md.

Current behavior

1. Parquet is written to a temp file on disk (writeLakehouseParquetBlob, writeParquetExportFile).
2. The full file is read back with os.ReadFile.
3. Bytes are passed to BlobStore.Put via store.BlobObject.Data []byte.

Affected call sites:

• pkg/analytics/lakehouse.go — writeLakehouseParquetBlob
• pkg/view/export_service.go — writeParquetExportFile → ExportFileStore.Put

store.BlobStore is a whole-object API:

type BlobObject struct {
    Key         string
    ContentType string
    Size        int64
    Data        []byte  // full payload required today
}

type BlobStore interface {
    Put(ctx context.Context, obj BlobObject) error
    // ...
}

Why not fixed in Parquet-on-FHIR follow-ups

Encoding was optimized (single-pass spill, row groups, no CollectMatchingResources for large exports). Fixing upload memory requires a cross-cutting BlobStore contract change and updates to every implementation (S3/GCS adapters, SQLite chunk store, in-memory test fakes, export artifact stores).

Impact

Scenario	Risk
Moderate exports (tens of MB parquet)	Acceptable in-process buffering
Multi-GB lakehouse blob uploads	Risk of OOM; must use filesystem partitions (LakehouseConfig.RootDir) today
$viewdefinition-export artifacts to blob-backed storage	Same O(file) peak at Put

Docs recommend ~2× compressed parquet size peak RAM at blob upload (file on disk + []byte for Put).

Proposed approach

1. Extend store.BlobStore (or add BlobStoreWithUpload) with a streaming put, e.g.:
   • PutStream(ctx, key, contentType, size int64, r io.Reader) error, or
   • PutFromPath(ctx, key, contentType, path string) error for backends that support file-based upload.
2. Optionally support multipart/resumable upload for S3/GCS adapters.
3. Update lakehouse blob and export artifact paths to stream from the temp parquet file without os.ReadFile.
4. Keep Put(BlobObject) for small inline payloads; deprecate or document size limits.

Workaround (today)

Use filesystem lakehouse partitions (LakehouseConfig.RootDir) for large exports — parquet streams directly to the final path with no full-file RAM buffer.

References

• Memory guidance: pkg/view/README.md (Parquet export sizing)
• Interop doc: docs/parquet-on-fhir-interop.md (Memory behavior table)
• Related PR stack: Apache Parquet phases 1–4: flat, streaming, lakehouse, Parquet-on-FHIR interop #41, Fix Parquet-on-FHIR review follow-up gaps #43, Close Parquet-on-FHIR ops review gaps #44, Fix HTTP run context and watermark test gaps #45

Acceptance criteria

• BlobStore (or companion interface) supports upload from io.Reader or file path without requiring full in-memory []byte
• writeLakehouseParquetBlob and export artifact upload use streaming put
• Existing Put(BlobObject) callers for small blobs remain supported
• Test asserts large blob upload does not allocate a []byte equal to file size (or documents backend-specific streaming behavior)
• Docs updated to mark blob O(file) limitation as resolved for streaming backends