Runtime IG & terminology platform: global catalog, tenant opt-in, async install, and FHIR tx API

[cursor]

Summary

HAIStack needs a platform-level conformance and terminology layer that mirrors the registry’s existing Postgres hybrid model: global canonical stores for heavy shared artefacts (IG StructureDefinitions, LOINC, ICD CodeSystems) with per-tenant opt-in overlays (ValueSets, enablement, bindings). Today, IG/terminology installation is CLI-only, synchronous, not hot-reloaded, and terminology is fully duplicated per tenant.

This issue tracks the runtime/admin API, async install pipeline, global-vs-tenant scoping, remote terminology, and FHIR terminology HTTP surface.

Problem / gap

Current state (as of 2026-03)

Capability	Status	Notes
Install IG at runtime	⚠️ Partial	haistack module install only; local pre-compiled JSON; blocks synchronously
Upload IG over HTTP	❌	No admin/FHIR endpoint
Install from packages.fhir.org	❌	No NPM/package registry client
FHIR $npm / Package registry install	❌	Not implemented
HTTP/admin API for package install	❌	Module lifecycle is CLI-only
Hot-reload validation profile catalog	❌	validate.Engine + RegistryProfileCatalog built once at startup (pkg/runtime/wire.go)
POST StructureDefinition via FHIR API → updates validation catalog	❌	Stored as ordinary resource only; does not register profiles
Global terminology catalog	❌	All terminology is scope_id-scoped (tenant ID on Postgres)
Tenant terminology opt-in / overlay	❌	No subscription model; LOINC/ICD would be duplicated per tenant
Remote terminology server (tx.fhir.org, etc.)	❌	terminology.Chain exists but is not wired in runtime
HTTP $lookup / $expand / $validate-code	❌	Internal terminology.LocalService only
Async IG/terminology install	❌	Module install is synchronous; no job integration

Architectural mismatch

Registry (Postgres) already does hybrid scoping correctly:

postgres.DB.DefinitionStore()          → global canonical FHIR definition JSON
postgres.TenantDB.RegistryInstallStore() → per-tenant enablement overlay

Terminology does not:

scope_id + canonical_url + version   → fully tenant-scoped; no shared layer

Desired model for IGs + terminology:

┌──────────────────────────────────────────────────────────────┐
│  Global conformance catalog (platform, read-mostly)          │
│  • IG StructureDefinitions, SearchParameters, OperationDefs  │
│  • LOINC / ICD / SNOMED CodeSystems (canonical + projections)│
│  • Installed once; shared across tenants                     │
└──────────────────────────────────────────────────────────────┘
                          ▲
                          │ fallback / compose
┌─────────────────────────┴────────────────────────────────────┐
│  Per-tenant terminology & conformance overlay                │
│  • ValueSets (compose from global CodeSystems)               │
│  • Tenant-specific CodeSystems / ConceptMaps (optional)      │
│  • Opt-in subscriptions: which global packs are enabled    │
│  • Profile/terminology bindings active for this tenant       │
└──────────────────────────────────────────────────────────────┘

Rationale: IGs, LOINC, and ICD are large, versioned, and identical across tenants. ValueSets express which codes matter for this tenant’s workflows and should remain tenant-scoped. This avoids storing the same LOINC expansion 50× while still letting tenants differ on bindings.

Runtime install gaps

• Module install persists to DB but a running HTTP server does not hot-reload RegistryProfileCatalog or validate.Engine.
• No way to trigger install from HTTP, Package Registry, or FHIR $npm.
• Install is synchronous — large IGs/terminology packs should run as durable jobs (pkg/jobs) with progress, retry, and failure reporting.

Proposed scope

Phase 1 — Global catalog + tenant overlay (data model)

• Add global terminology store on postgres.DB (parallel to global DefinitionStore):
  • Global CodeSystem canonical JSON + compiled projections (LOINC, ICD, HL7 base)
  • Global IG definitions remain in global DefinitionStore (already exists)
• Add per-tenant terminology overlay on postgres.TenantDB:
  • Tenant ValueSet resources (compose/include from global CodeSystems)
  • Tenant opt-in records: which global terminology packs and IG modules are enabled
  • Optional tenant-local CodeSystems / overrides
• Wire terminology.Chain in runtime:

  tenant LocalService → global LocalService → remote Provider (optional)
  

• Migration path from current all-tenant-scoped terminology tables
• SQLite/edge: single scope with optional reference to bundled global packs (no multi-tenant duplication on device)

Phase 2 — Async install pipeline (jobs)

• New job types: ig.install, terminology.install, package.fetch
• Job handler flow:
  1. Fetch/validate package (local path, uploaded tarball, or packages.fhir.org)
  2. Parse NPM IG manifest / FHIR package index
  3. Install global definitions + compile terminology projections
  4. Record tenant opt-in if scoped to a tenant
  5. Rebuild registry snapshot + warm profile catalog
  6. Publish completion event / OperationOutcome for polling
• Admin API: POST /admin/packages/install (returns job ID)
• Idempotent installs keyed by package name + version + tenant scope

Phase 3 — HTTP surfaces

• Package install API (admin, authorized):
  • Upload IG tarball (multipart/form-data)
  • Install from URL / packages.fhir.org package name+version
  • Optional FHIR Package $npm compatibility layer
• Hot-reload hook after successful install job:
  • registry.Manager.RebuildSnapshot()
  • RegistryProfileCatalog.Warm() or replace catalog on validate.Engine
  • Invalidate terminology cache
  • Update CapabilityStatement enabled types/operations
• FHIR terminology operations (tenant-scoped, SMART-gated):
  • CodeSystem/$lookup
  • ValueSet/$expand
  • CodeSystem/$validate-code / ValueSet/$validate-code
• StructureDefinition registration path:
  • POST/PUT StructureDefinition optionally registers into global or tenant catalog (not just resource store)
  • Or restrict profile registration to package install only (document decision)

Phase 4 — Remote terminology provider

• Implement terminology.RemoteProvider adapter for FHIR tx servers (tx.fhir.org, on-prem Snowstorm, etc.)
• Configurable per deployment: global remote fallback, tenant override, offline-only mode
• Cache remote expansions with TTL; respect force-system-version semantics
• Rate limiting and circuit breaker for tx server calls

Acceptance criteria

• LOINC (or a representative large CodeSystem) can be installed once at platform level and referenced by multiple tenants without duplicating projection storage
• Tenants can create/enable ValueSets that compose global CodeSystems without re-importing LOINC
• IG package install can be triggered via HTTP admin API and runs as a background job (not blocking request thread)
• After install job completes, running server picks up new profiles for $validate and write-path validation without restart
• Install from a local upload and from packages.fhir.org (at least one pilot package) demonstrated end-to-end
• terminology.Chain is wired in pkg/runtime with tenant → global → remote precedence
• FHIR $lookup / $expand / $validate-code exposed for enabled tenants
• Install failures surface as job status + OperationOutcome; partial installs are compensated (existing module installer pattern)
• Documentation: global vs tenant scoping, opt-in model, and admin workflows

Out of scope (follow-up issues)

• Public publication to packages.fhir.org (we consume, not publish)
• Full Touchstone/Inferno server certification
• R5/R6 terminology or IG support
• Per-tenant custom LOINC subsets (could be a ValueSet-only concern)

Affected packages / files

Area	Path
Runtime wiring	pkg/runtime/wire.go, pkg/runtime/builder.go
Registry hybrid model	pkg/registry/, pkg/postgres/definition_store.go, pkg/postgres/registry_install_store.go
Terminology	pkg/terminology/, pkg/postgres/terminology_store.go, pkg/sqlite/terminology_store.go
Module installer	pkg/modules/
Jobs	pkg/jobs/
HTTP	pkg/http/ (new admin + terminology operation routes)
Validation	pkg/validate/profile_catalog_registry.go
Core writes	pkg/core/service.go (compileTerminology)
CLI (retain)	cmd/haistack/command/module.go

Design notes

Align with existing registry hybrid

Reuse the proven pattern:

Layer	Global	Per-tenant
StructureDefinitions (IG)	DefinitionStore	RegistryInstallStore enablement
CodeSystems (LOINC, ICD)	new global terminology catalog	—
ValueSets	—	tenant terminology overlay
Profile validation	global canonical lookup	tenant enablement + bindings

Job-based install (not synchronous)

Large terminology packs (LOINC ~100MB+) and IG compiles must not block HTTP handlers. Use pkg/jobs with:

• progress reporting (installed N/M definitions)
• retry with backoff on transient fetch failures
• compensating rollback on partial failure (existing modules.Installer pattern)

Hot-reload contract

Define a ConformanceRefresher interface called after install job success:

type ConformanceRefresher interface {
    Refresh(ctx context.Context) error // rebuild snapshot, warm catalog, invalidate terminology cache
}

Inject into job handler and optionally expose POST /admin/conformance/refresh for manual trigger.

References

• Registry hybrid model: pkg/registry/README.md
• Terminology scoping: pkg/terminology/README.md
• Module installer: pkg/modules/README.md
• terminology.Chain: pkg/terminology/chain.go
• Related: Conformance artefacts as first-class source code (FSH, SUSHI, IG, validator CI) #7 (Conformance artefacts as first-class source code — FSH/SUSHI/CI; this issue covers runtime platform for those artefacts)

Context from recent PR work

• PR perf(validate): fast-mode slicing and profile validation optimizations #15: validation performance (profile catalog warm at startup)
• PR feat(http): wire FHIR Resource/$validate operation #16: FHIR Resource $validate (full mode by default; depends on profile catalog loaded at startup)

Both reinforce the need for hot-reload after runtime IG install — today a restart is required for new StructureDefinitions to affect validation.