Proof-of-conformance and proof-of-scale benchmark harness

[cursor]

Summary

HAIStack has strong unit and integration test coverage (~121 test files, Postgres via testcontainers) but no proof-of-conformance or proof-of-scale evidence. There are no Go benchmarks, no external conformance suites (Inferno, Touchstone, Crucible), no CapabilityStatement-vs-tested-endpoint matrix, and no published workload baselines. Server selection guidance emphasizes benchmarking your own data shape — HAIStack needs a reference workload and harness so adopters can evaluate fitness for their use case, not rely on generic FHIR server benchmarks.

Problem / gap

What we have today

Testing (correctness):

• CI: gofmt, go vet, golangci-lint, go test -race -count=1 -timeout 30m ./..., go build
• Integration tests: Postgres (testcontainers), SQLite, HTTP, sync, analytics, runtime
• pkg/testkit: fixtures, store fakes, sync scenarios, golden OperationOutcome helpers, FHIRPath assertions, AI harness

What is absent:

• Zero func Benchmark* tests in the repository
• No Touchstone, Inferno, Crucible, or HL7 validator conformance CI jobs
• No CapabilityStatement accuracy testing (GET /fhir/metadata vs actually supported operations)
• No load/stress/concurrency test suite
• No published compatibility or performance matrix
• No observability baseline (metrics, tracing) for benchmark runs
• No upgrade/migration regression testing under load

Why generic benchmarks are insufficient

The guidance explicitly states: optimization mechanisms differ across HAPI, Blaze, Aidbox, Smile, Azure, AWS, etc. — a generic benchmark number is a poor proxy for a particular workload. HAIStack's architecture (registry-driven search indexer, single WriteSession transactions, terminology projections, offline SQLite edge, Git-inspired sync) has unique performance characteristics that must be measured against representative workloads.

Minimum benchmark workloads (from guidance)

HAIStack should benchmark at least:

Workload	HAIStack packages	Current test coverage
Highest-cardinality searches	pkg/search, pkg/postgres	Functional only
Chained searches	pkg/search	Unknown completeness
_include / _revinclude	pkg/search, pkg/http	Functional only
Terminology expansion	pkg/terminology	Unit tests
Validation under real IG stack	pkg/validate, pkg/registry	Structural only
Transaction bundles	pkg/core, pkg/http	Integration tests
Conditional operations	pkg/core, pkg/http	Some tests
Patient _history reads	pkg/core, pkg/store	Integration tests
Bulk Data export	pkg/http	501 — not testable
Subscription/event loads	pkg/subscriptions, pkg/jobs	Unit/integration
Mixed concurrent reads/writes	pkg/runtime, pkg/sync	Limited
View refresh at scale	pkg/view, pkg/analytics	Integration only
Sync push/pull under load	pkg/sync	Scenario tests, no perf

Non-latency dimensions to compare:

• Operational observability (metrics, logs, trace spans)
• Upgrade behaviour (schema migrations during traffic)
• Reindex job impact on interactive latency
• SQLite edge vs Postgres edge vs cloud mode degradation modes

Proposed scope

Phase 1 — Reference workload and dataset

• Create benchmarks/ directory with:
  • Synthetic dataset generator (configurable cardinality: patients, observations, appointments)
  • Reference data shape documented (mimics clinic scheduling + vitals use case)
  • Seed scripts for SQLite and Postgres
  • Versioned dataset manifests (size, resource counts, search cardinality)
• Document minimum hardware profiles for benchmark runs (CI vs local)

Phase 2 — Go benchmark harness

• Add func Benchmark* tests (or benchmarks/ main packages) for:
  • Single-resource CRUD
  • Simple search (one parameter, high cardinality)
  • Chained search
  • _include / _revinclude
  • Terminology lookup and ValueSet expansion
  • Validation (structural + future profile validation)
  • Transaction bundle (mixed create/update)
  • Conditional create/update/delete
  • _history read (version chain)
  • View execution + analytics refresh
  • Sync push/pull round-trip
  • Concurrent mixed read/write (configurable goroutine count)
• Output: ns/op, allocations, optional custom metrics (p50/p95/p99 via benchstat)
• CI job: run benchmarks on main (non-blocking initially), store results as artifacts
• Optional: benchmark regression gate (fail if p95 regresses >X% vs baseline)

Phase 3 — Conformance matrix

• Generate CapabilityStatement from pkg/registry snapshot
• Build tested-endpoint matrix:
  • CRUD per resource type
  • Search parameters (supported vs declared)
  • Operations (SDC, custom, bulk)
  • _history, _search, bundles
• Automated test proving CapabilityStatement claims match actual handler behaviour
• Publish matrix as docs/conformance-matrix.md (generated, not hand-maintained)

Phase 4 — External conformance evaluation

• Evaluate and document fit for:
  • Inferno (RESTful API tests)
  • Touchstone (if applicable to library/server profile)
  • Crucible basic suite
• Add optional CI workflow (may be nightly, not per-PR) running Inferno against reference pkg/http deployment
• Track pass/fail scorecard over time
• Depends on: Bulk Data server, profile validation, auth scenarios

Phase 5 — Observability and upgrade benchmarks

• Add minimal metrics hooks for benchmark runs:
  • Request latency histogram
  • Search index write lag
  • Job queue depth
  • DB connection pool utilization
• Upgrade benchmark: run migration N→N+1 under read load, measure downtime/latency spike
• Reindex benchmark: measure interactive query impact during full reindex

Phase 6 — Comparison documentation (not horse-race)

• Document HAIStack benchmark methodology (not "we're faster than HAPI")
• Provide adapter notes for running same workload against external servers (optional comparison script)
• Publish results for: SQLite edge, Postgres edge, Postgres cloud mode

Acceptance criteria

• Reference workload dataset generator exists and is documented
• ≥10 benchmark functions covering the minimum workload list
• CI stores benchmark artifacts on main branch merges
• CapabilityStatement vs tested-endpoint matrix is generated and tested
• README links to benchmark guide and explains how to interpret results
• At least one concurrency benchmark (mixed R/W) exists
• Upgrade-under-load test exists for Postgres migrations

Out of scope (for this issue)

• Claiming superiority over HAPI/Firely/Blaze (methodology only)
• Production APM integration (Prometheus exporter is separate)
• Load testing SaaS infrastructure (k6 cloud, etc.) — local/CI only initially

Affected packages / files

Area	Path
New benchmarks	benchmarks/
Search	pkg/search/
HTTP	pkg/http/
Core	pkg/core/
Terminology	pkg/terminology/
Sync	pkg/sync/
Analytics	pkg/analytics/
Runtime	pkg/runtime/
CI	.github/workflows/benchmarks.yml
Docs	docs/benchmarks.md, docs/conformance-matrix.md

Dependencies

• Blocked partially by: Bulk Data server (cannot benchmark export until implemented)
• Blocked partially by: Profile validation (validation-under-IG benchmark needs conformance artefacts)
• Complements: Authorization scenario tests (concurrent authz under load)

References

• Inferno Framework
• Crucible
• Go benchmarking: testing.B, benchstat
• HAIStack README — "not a monolithic FHIR server" positioning

Related issues

• Conformance artefacts (IG + validator for validation benchmarks)
• Bulk Data server (export benchmarks)
• Authorization semantics (authz under load)
• Research: vendor-neutral benchmark suite contribution