Skip to content

roadmap.md

Latest commit

 

History

History
125 lines (114 loc) · 7.39 KB

roadmap.md

File metadata and controls

125 lines (114 loc) · 7.39 KB
Raw

Roadmap

Open work, extracted from the per-VLAN task lists that used to live inside the inventory. Ordered roughly by how much it matters.

Security

  • Retire the MokerLink switch's previous SNMP community. All four devices are rotated, but neo still accepts its old community alongside the new one: its firmware does not persist a deletion from the community table, and each attempt drops the SNMP agent until the switch is rebooted. Low priority and accepted for now — the community is read-only and reachable only from the management VLAN. Overwrite the row with a fresh value rather than deleting it, next time the switch is down anyway. → runbook
  • Move to SNMPv3 authPriv where the hardware supports it. pfSense, the APC and iLO all do; the MokerLink switch does not, which is the blocker for doing it uniformly.
  • Decide whether the lab VLAN needs egress filtering before the deliberately-vulnerable playground exists.

Monitoring

  • Add ifXTable (64-bit counters) to the mokerlink SNMP module. The current ifTable counters are 32-bit and wrap in roughly 34 seconds at gigabit line rate, so sustained high-throughput ports under-report. The SwitchCounterWrapSuspected alert detects this but does not fix it. Now actionable — the switch has only been polling since the three faults in #22 were cleared. Mind the trap documented in generator.yaml: this switch returns ifSpecific with a zero-length value that makes snmp-exporter discard the whole response, so the walk lists individual columns and must stay clear of column 22.
  • Deploy Alloy to the remaining hosts — currently only the monitoring host and one other report in. Saruman and oracle are next.
  • Turn on remote logging on morpheus. The receiver, parsing and three firewall alert rules are deployed; the firewall is not yet sending. Until it does, FirewallLogsStopped is the only one of them that can fire. → runbook
  • Enable Suricata on morpheus. Parsing and two alert rules are deployed; the IDS is not yet installed. IDS-only on the IoT and guest interfaces, and expect several days of tuning before the alerts mean anything. → runbook · ADR-0006
  • Detect Suricata being dead. A quiet IDS and a stopped IDS produce identical log output, so no log rule can tell them apart — unlike FirewallLogsStopped, which works because a firewall is never silent. This needs a process or heartbeat metric from morpheus, which the SNMP module does not currently expose.
  • Add blackbox-exporter for uptime and TLS-expiry checks on internal services.
  • Capture dashboard screenshots for the README once the stack has a few days of real data. → images/README.md

Infrastructure

  • Copy the firewall backup off prometheus, and buy a spare. make backup-firewall now exports and encrypts morpheus's config, and restore-the-firewall.md documents the restore — but a backup on the same shelf as the thing it protects is not a backup, and the runbook is a hypothesis until it has been restored onto a spare once. The spare should be the same ProDesk model: pfSense stores interface assignments by device name, so identical hardware restores straight through and anything else drops you into the console dialogue.
  • Replace the UPS battery. mjolnir currently has none, so a mains loss is an immediate hard shutdown of the rack. A replacement APCRBC115 cartridge is on order. Most rules in ups.rules.yaml still report on a UPS that cannot hold the load — but UpsSelfTestFailed now detects it, keyed on the one metric the management card does not fabricate. Once the pack is fitted, enable scheduled self-tests on the NMC so that rule stays live evidence rather than a stale last-known result.
  • Decide what oracle (10.0.99.30) is for. It is a dual-core AMD A6-9200 with 4 GB and a 5400 rpm disk — considerably less machine than this list previously claimed, and too little for anything demanding.
  • Plan and build the NAS on VLAN 40.
  • Procure a second server ("ifrit") for the isolated playground network.
  • Build the playground — only after the main network is finished.
  • Work out DNS for the MokerLink management UI so it is not reached by IP.

Automation

  • Home Assistant integration with the eero API, so device joins and leaves show up as events rather than being discovered by accident.
  • Move stack deployment from make up over SSH to something pull-based, so the monitoring host converges on the repo rather than being pushed to.
  • Automate the Grafana dashboard export step — the current loop (edit in UI, copy JSON, commit) is manual and therefore skipped under pressure.

Done

  • Settle the 10.0.30.10 question. It is the iLO BMC on its dedicated port; the Proxmox host is Saruman at 10.0.30.110. The SNMP target's hypervisor-bmc role label was correct all along — the inventory was not
  • Bridge mode on the ISP gateway
  • Lock down guest VLAN firewall rules
  • Move IoT devices onto their own SSID and VLAN
  • Stand up Prometheus, Grafana, Loki, snmp-exporter and Alloy
  • Consolidate five broken compose files into one working stack
  • Provision Grafana datasources and dashboards from files
  • Add alerting (32 rules) and Alertmanager routing
  • Move secrets to SOPS + age
  • Add CI: lint, config validation, secret scanning
  • Pin every image by digest, not just tag, with drift detection in CI
  • Add SECURITY.md with a disclosure policy and known-exposure summary
  • Loki alerting rules (8) for auth, SSH brute force and disk/OOM events, validated in CI by booting the pinned Loki image against them
  • Replace the CA and leaf certificates that leaked, and add tooling so issuing one is a command rather than a research project
  • Serve Grafana over TLS with that CA, verified end to end — Prometheus scrapes it with ca_file and server_name rather than insecure_skip_verify
  • Point Alertmanager at a real receiver. The webhook was the ntfy.example.invalid placeholder for the entire life of the stack, so no alert had ever been delivered
  • Surface firing alerts on the dashboards. Forty rules and one routing tree existed with nothing showing them; four dashboards now carry a table of their own component's alerts
  • Stop the UPS dashboard reporting a battery that is not there — the management card fabricates charge, runtime and status
  • Purge the shared SNMP community, the inline Grafana password and the TLS private keys under certificates/ from git history, and delete the .gitleaksignore that acknowledged them
  • Give every SNMP device its own community and rotate all four on the hardware, confirming pfSense, the APC NMC and iLO each refuse the old shared string. The switch accepts its new one but also still holds its previous community — an accepted residual, recorded in SECURITY.md