Research contributions: reproducible FHIR pipelines, policy semantics, and evaluation corpora

[cursor]

Summary

Recent FHIR literature identifies five high-impact research gaps: a vendor-neutral benchmark suite, validated R4/R5/R6 semantic-conversion corpora, computable cross-vendor consent/policy semantics, terminology-mapping quality and provenance methods, and reproducible pipelines joining FHIR/Bulk Data/SQL-on-FHIR to clinical AI while preserving provenance and validation. HAIStack's architecture aligns directly with several of these gaps — particularly policy-governed AI, ViewDefinition analytics, terminology projections, and modular edge/cloud composition — but the repository contains no research artefacts, evaluation scripts, reproducible datasets, or published methodology.

Problem / gap

HAIStack's research-relevant strengths (unrealized)

Research gap	HAIStack component	Current state
Vendor-neutral FHIR benchmark suite	Modular Go libraries, SQLite + Postgres modes	No benchmarks, no published workloads
R4/R5/R6 semantic-conversion corpora	pkg/proto, ResourceEnvelope, Google FHIR adapter	R4 only; no conversion corpus
Computable consent/policy semantics	pkg/auth policy DSL + pkg/smart scope narrowing	Stub patient scope; no formal semantics doc
Terminology mapping quality/provenance	pkg/terminology, ConceptMap as FHIR resources	Projections exist; no quality metrics or provenance model
Reproducible FHIR → AI pipelines	pkg/ai, pkg/view, pkg/audit, pkg/fhirpath	Example examples/ai-authz; no reproducible pipeline or paper artefacts

What is missing

1. No research/ or docs/research/ directory with reproducible experiments
2. No public evaluation datasets (synthetic or de-identified) for benchmarking
3. No comparison baselines against other FHIR implementations
4. No provenance chain from raw FHIR → validated view rows → AI tool input → audit log
5. No semantic conversion test corpus for version migration (R4→R5→R6)
6. No terminology mapping evaluation (precision/recall of ConceptMap translations, provenance tracking)
7. No FAIR metadata for published artefacts (DOI, license, citation file)
8. No documented research agenda connecting HAIStack roadmap to literature gaps

Why this matters

• Differentiation: HAIStack's "safe AI access" story is research-grade in concept but not in evidence.
• Adoption: Researchers and health informatics teams need reproducible pipelines, not just libraries.
• Community contribution: The five gaps are explicitly vendor-neutral — HAIStack could contribute without becoming a server horse-race.
• FAIRness: FHIR data infrastructure for ML requires provenance, validation, and reproducibility — core HAIStack packages address this but do not demonstrate it.
• Funding and collaboration: Research artefacts unlock grants, academic partnerships, and standards participation.

Proposed scope

Track A — Vendor-neutral FHIR benchmark suite (ties to benchmark issue)

• Publish reference workload under research/benchmarks/ with:
  • Dataset generator + fixed seed datasets (small/medium/large)
  • Workload definitions (YAML/JSON) portable across implementations
  • HAIStack runner (go test / CLI)
  • Optional adapter template for external servers (HAPI, etc.)
• Document methodology emphasizing workload-specific evaluation, not single score
• Target: citable artefact (Zenodo/figshare) with CITATION.cff

Track B — Semantic conversion corpora (R4/R5/R6)

• Create research/semantic-conversion/:
  • Paired resource instances: R4 input → expected R5/R6 output (where mapping is defined)
  • Edge cases: renamed elements, cardinality changes, type changes, CodeableConcept shifts
  • Validation expectations per version
• Build on pkg/proto adapter and ResourceEnvelope codec
• Scoring: structural equivalence, semantic equivalence (FHIRPath assertions), information loss flags
• Start with 50–100 resource pairs across Patient, Observation, Condition, MedicationRequest
• Document relationship to HL7 version conversion packages

Track C — Computable consent and policy semantics

• Formalize pkg/auth policy DSL semantics:
  • Document decision algorithm (deny-by-default, first-match, patient scope overlay)
  • Define interaction with SMART scopes (scope grants ∩ policy allows)
  • Provide "policy narrowing" examples matching SMART spec warnings
• Create research/policy-semantics/:
  • Cross-vendor policy scenario catalogue (vendor-neutral test cases)
  • Consent resource patterns (R4 Consent, future R5/R6 Permission)
  • Machine-readable scenarios: principal + consent state + request → expected decision
• Implement scenario runner in pkg/testkit/authztest (shared with auth issue)
• Target: submission to HL7 or publish as open test suite

Track D — Terminology mapping quality and provenance

• Extend pkg/terminology evaluation:
  • ConceptMap translation quality metrics (exact match, narrow, broad, unmatched)
  • Provenance model: which ConceptMap version, which source CodeSystem version, translation timestamp
  • Audit trail for terminology decisions affecting validation or analytics
• Create research/terminology-evaluation/:
  • Reference ConceptMaps with gold-standard translations
  • Scripts to score mapping pipelines
  • Document known limitations of finite ValueSet expansion
• Link terminology provenance to pkg/audit events

Track E — Reproducible FHIR → AI pipeline

• Create research/ai-pipeline/ end-to-end reproducible example:

  FHIR resources (validated against IG)
    → ViewDefinition projection (pkg/view)
    → permissioned row set (pkg/auth)
    → AI tool invocation (pkg/ai)
    → structured response
    → audit log with full provenance chain (pkg/audit)
  

• Requirements:
  • Fixed dataset (synthetic patients + observations)
  • Pinned IG/conformance versions (depends on conformance issue)
  • Deterministic or seeded LLM stub for reproducibility without API keys
  • Exportable provenance bundle: input hashes, view version, policy version, tool version, output
• Document FAIR metadata for the pipeline artefact
• Jupyter notebook or Makefile target: make research-ai-pipeline

Cross-cutting research infrastructure

• Add research/README.md — research agenda, how to cite, how to reproduce
• Add CITATION.cff for the repository
• Add research/LICENSE note if datasets differ from Apache 2.0 code
• CI: research-* workflows run reproducibility checks (non-blocking initially)
• No PHI — all datasets synthetic or explicitly licensed

Acceptance criteria

• research/ directory exists with README and citation guidance
• At least one track (A–E) has a fully reproducible artefact with make or script entry point
• AI pipeline demo produces auditable provenance chain from FHIR → view → tool → audit
• Policy semantics document describes scope ∩ policy interaction with ≥10 examples
• Semantic conversion corpus has ≥50 paired instances with scoring script
• Terminology evaluation has ≥1 ConceptMap gold set with quality metrics
• CITATION.cff present

Out of scope (for this issue)

• Publishing academic papers (artefacts enable; writing is separate)
• Real PHI datasets
• Production LLM API integration in CI (use stubs)
• Full R6 support (corpus can start R4→R5)

Priority recommendation

Priority	Track	Rationale
P0	E — AI pipeline	Core product differentiation; uses existing packages
P0	C — Policy semantics	Unblocks auth issue; SMART ecosystem need
P1	A — Benchmark suite	Shared with benchmark issue; high community value
P1	D — Terminology evaluation	Supports conformance and analytics credibility
P2	B — Semantic conversion	Longer horizon; depends on multi-version support

Affected packages / files

Area	Path
AI	pkg/ai/
Views	pkg/view/
Audit	pkg/audit/
Auth / policy	pkg/auth/
SMART	pkg/smart/
Terminology	pkg/terminology/
Proto / codec	pkg/proto/, pkg/types/
Test kit	pkg/testkit/
New	research/
Examples	examples/ai-authz (extend)

References

• FHIR heterogeneity, research integration, FAIRness, and ML standardisation gaps (recent literature)
• FAIR Principles
• HAIStack README — safe AI access, ViewDefinitions, terminology projections
• Related HL7 work: SMART scope narrowing, Bulk Data, SQL-on-FHIR, IPS

Related issues

• Conformance artefacts (IG pinning for AI pipeline)
• Authorization semantics (policy scenario catalogue)
• Data engineering paths (Bulk Data + analytics in pipeline)
• Benchmark / proof-of-conformance (Track A overlap)