Upgrade Loki module to upstream 3.7.x (from 3.5.12)

[DavidePrincipi]

Loki upstream has moved to 3.7.x while the ns8-loki module is still pinned to grafana/loki:3.5.12 in build-images.sh. 3.5 is EOL since months.

Staying two minor versions behind means missing bugfixes/security patches already backported through the 3.6.x and 3.7.x release branches, and the gap will only grow more painful to close the longer it's deferred.

Proposed solution

Bump the pinned image in build-images.sh from grafana/loki:3.5.12 to grafana/loki:3.7.1 (or latest 3.7.x patch at merge time).

Checked against the current imageroot/loki-config.yaml and deployment mode (single-binary, inmemory ring/kvstore, filesystem object store, boltdb-shipper + tsdb/v13 schema, structured metadata enabled), none of the 3.6/3.7 breaking changes require a config or schema migration:

• AWS SDK v1→v2 (3.6.0): not applicable, filesystem backend
• Scheduler capacity/worker-thread changes (3.7.0): internal to single-binary mode, no explicit scheduler config in use
• Parsed labels no longer override structured metadata (3.7.0): behavioral fix, worth a post-upgrade query sanity check
• Promtail removed from the Loki image (3.7.3): not applicable, NS8 ships Promtail as a separate module/image
• BusyBox/shell removed from the Loki image (3.5.8+): not applicable, no exec-based healthchecks in loki-server.service

update-module.d/10config doesn't need changes — it only backfills env defaults on first upgrade from very old versions; this is a straight image-tag swap + restart.

Plan: bump the tag, rebuild/publish, then validate on a test instance carrying real historical boltdb-shipper + tsdb data (WAL replay, ingestion continuity, existing LogQL queries/dashboards across both index periods) before rolling out.

Alternative solutions

Stay on 3.5.x and upgrade later — not preferred, since skipping more versions increases upgrade risk and forfeits current bugfixes/security patches in the interim.

Additional context

Upstream release notes: https://grafana.com/docs/loki/latest/release-notes/v3-7/ and https://grafana.com/docs/loki/latest/setup/upgrade/

[DavidePrincipi]

Test case

In testing loki 1.5.0-dev.1

1. Restore a recent production backup (single-node cluster) onto a test NS8 instance, still on Loki 3.5.12 (stable version). You have it as unactive instance.
2. Sanity-check retention_days in the Settings > System logs page for the unactive instance
3. In the System Logs page, run a couple of searches over the last few days and note they return expected results — this is your baseline.
4. Upgrade the unactive module to 3.7.x. (testing version)
5. Watch journalctl -f _UID=$(id -u loki2) through the restart for clean startup (no WAL/config errors).
6. Re-run the same System Logs searches on the unactive instance — confirm same results as baseline.
7. Check also the active instance is receiving the logs and searches work too. Confirm new log lines keep appearing in System Logs a few minutes after upgrade (ingestion still flowing).