feat: add a health-aware ingestion load balancer

[CoreyLeath-code]

Summary

Implements issue #21 and retains the dated tracking entry in Dailylog.md.

• Adds an NGINX gateway at the existing public ingestion port (8080) and keeps Go ingestion replicas internal.
• Separates process liveness (/health) from Postgres-aware readiness (/ready).
• Adds an optional, disabled-by-default replica header used solely by the CI scaling test.
• Adds gateway telemetry scraping through the NGINX Prometheus exporter.

Files changed

• nginx/ingestion-load-balancer.conf — least-connections upstream, health/readiness proxy policy, forwarded-request headers, internal NGINX status.
• docker-compose.yml — removes direct host exposure of ingestion replicas; adds the public gateway and exporter; points the dashboard at the gateway.
• ingestion-service/main.go — exposes liveness/readiness separately and optional test-only replica identity.
• ingestion-service/main_test.go — verifies the replica-identity header is opt-in.
• monitoring/prometheus.yml — scrapes gateway exporter metrics.
• .github/workflows/ci-cd.yml — validates NGINX syntax and exercises three replicas, multi-replica routing, and one-replica failure resilience.
• README.md — documents the gateway, scaling command, and probe behavior.
• Dailylog.md — preserves the dated record and links enhancement: add a health-aware load-balancing layer for API workers #21.

Validation performed

• Compared the branch against current main: 8 files changed, branch is 8 commits ahead and 0 behind.
• Reviewed the Compose topology to confirm only the gateway publishes port 8080.
• Added committed Go unit coverage for the opt-in observability contract.
• Added a dedicated GitHub Actions smoke job that runs nginx -t, starts three replicas, probes /ready, checks multiple X-Instance-ID values, then stops one replica and rechecks readiness.

The connected GitHub workspace cannot execute Docker or Go locally; the new CI job is the authoritative runtime validation and this PR remains draft until it reports.

Risks

• The NGINX upstream depends on Docker Compose DNS, so this configuration is scoped to Compose rather than Kubernetes or a managed load balancer.
• Write requests are intentionally not retried by NGINX because the ingestion endpoint does not provide an idempotency-key contract.
• No load/performance metric is claimed by this PR.

Follow-up recommendations

• Run a measured load test against the gateway in the target deployment environment before selecting replica counts or timeouts.
• Add authentication/rate limiting at the gateway when the ingestion API is exposed beyond a trusted network.
• Promote the same readiness contract into the production orchestrator before deployment.

Summary by CodeRabbit

• New Features

  • Added a load-balanced ingestion gateway that distributes requests across multiple service replicas.
  • Added separate liveness and database-backed readiness checks.
  • Added optional instance identification in response headers.
  • Added gateway metrics for monitoring and updated the dashboard to use the gateway.

• Bug Fixes

  • Improved resilience by routing health and readiness checks around unavailable replicas.

• Documentation

  • Documented the load-balanced ingestion architecture, scaling, and readiness behavior.

• Tests

  • Added automated coverage for gateway routing, replica failure handling, readiness, and instance headers.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 8299c7eb-710f-4a69-a89e-f95c18c359eb

📥 Commits

Reviewing files that changed from the base of the PR and between 8c97571 and b1ff20e.

📒 Files selected for processing (8)

• .github/workflows/ci-cd.yml
• Dailylog.md
• README.md
• docker-compose.yml
• ingestion-service/main.go
• ingestion-service/main_test.go
• monitoring/prometheus.yml
• nginx/ingestion-load-balancer.conf

---

📝 Walkthrough

Walkthrough

Changes

Ingestion load balancing

Layer / File(s)	Summary
Service health and instance identity ingestion-service/main.go, ingestion-service/main_test.go	The service separates liveness from database-backed readiness. Optional X-Instance-ID middleware covers service routes. Tests verify enabled and disabled behavior.
Gateway and replica deployment nginx/ingestion-load-balancer.conf, docker-compose.yml, monitoring/prometheus.yml	NGINX routes requests across ingestion replicas, retries health checks, and exposes status metrics. Compose uses readiness checks and routes Streamlit traffic through the gateway.
Deployment validation and documentation .github/workflows/ci-cd.yml, README.md, Dailylog.md	CI validates NGINX configuration, replica routing, and readiness after a replica stops. Documentation and the daily log describe the gateway architecture and current implementation status.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Streamlit
  participant NGINX
  participant IngestionReplicas
  participant Prometheus
  Streamlit->>NGINX: Send ingestion request
  NGINX->>IngestionReplicas: Route request across replicas
  IngestionReplicas-->>NGINX: Return response with instance identity
  NGINX-->>Streamlit: Return response
  Prometheus->>NGINX: Scrape gateway metrics

Loading

Possibly related issues

• CoreyLeath-code/SentinelAI issue 21: The PR implements its health-aware NGINX load-balancing objectives.
• CoreyLeath-code/Scalable-Event-Driven-Ride-Sharing-Platform issue 17: The PR covers the same gateway, replica routing, readiness, resilience testing, and documentation objectives.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch docs/log-load-balancer-gap

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}