enhancement: add a health-aware load-balancing layer for API workers

[CoreyLeath-code]

Finding

The local Compose stack exposes the ingestion API (8080), drift engine (7070), and LLM guard (8000) directly on host ports. It has health checks but no stable Layer-7 entry point, upstream pool, or replica-failure handling for API/inference traffic.

Requested work

Add a production-quality load-balancing layer appropriate to SentinelAI's existing service boundaries.

Scope

• Identify which externally reachable API(s) can safely run as stateless replicas; do not duplicate heavyweight model initialization without measuring its impact.
• Place an NGINX or equivalent Layer-7 proxy in front of the selected API worker pool and expose the proxy, not individual replicas, in the local Compose demonstration.
• Wire health/readiness into upstream selection; preserve existing authentication, metrics, and dashboard behavior.
• Propagate forwarded headers and request IDs; preserve client identity for any existing rate-limit/security controls.
• Add observability for proxy requests, latency, errors, upstream availability, and worker identity only through a safe test/debug mechanism.
• Add automated configuration, multi-replica, proxy-routing, and failed-replica integration validation.
• Document measured load-test commands and failure demonstration steps. Do not commit or claim results that were not measured.
• Integrate Kubernetes Service/Ingress only where it matches the existing manifests; document the distinction from local NGINX.

Acceptance criteria

• Three compatible API replicas can run behind one stable local endpoint without host-port collisions.
• Requests reach more than one healthy replica in an automated test.
• A stopped/unready replica is removed from traffic and service remains available.
• The proxy configuration, Compose stack, existing tests, and relevant CI jobs validate without weakening existing checks.
• README documents algorithm selection, health/readiness, failure behavior, observability, and reproducible load-test commands.
• No secrets or unsupported production-performance claims are added.

Evidence

• docker-compose.yml maps direct host ports for ingestion-service, drift-engine, and llm-guard.
• Existing service health checks are present, but no upstream proxy/load-balancer service is defined.
• No open issue mentioning NGINX, load balancing, or horizontal scaling exists as of 2026-08-12.

Dependencies / decisions

The implementation must choose a load-balanced boundary after verifying which APIs are stateless and whether model loading is safe to replicate. It must not replace asynchronous telemetry or downstream service communication with HTTP proxying.