diff --git a/README.md b/README.md index fcbdbcb..2d55b55 100644 --- a/README.md +++ b/README.md @@ -76,7 +76,7 @@ graph TB WS["workstations"] end subgraph V30["VLAN 30 · Lab"] - HV["shiva · Proxmox"] + HV["Saruman · Proxmox
BMC: shiva"] end subgraph Terminal["VLANs 40 / 20 / 10 · egress only"] UNTRUSTED["media · IoT · guest"] diff --git a/docs/architecture.md b/docs/architecture.md index 0e19a46..f9018af 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -29,7 +29,7 @@ graph TB end subgraph V30["VLAN 30 · ImaginationLAN · Lab"] - HV["shiva · ProLiant DL360 Gen9
Proxmox VE"] + HV["Saruman · ProLiant DL360 Gen9
Proxmox VE
BMC: shiva"] end subgraph V20["VLAN 20 · Skids · IoT"] @@ -131,7 +131,7 @@ hole from the monitoring VLAN into the monitored one. | Host | VLAN | Stack | Contents | | --- | --- | --- | --- | | `prometheus` (10.0.99.20) | 99 | [`stacks/observability`](../stacks/observability) | Prometheus, Alertmanager, Loki, Grafana, snmp-exporter, Alloy | -| `shiva` (10.0.30.10) | 30 | *(none yet)* | Proxmox VE — see [roadmap](roadmap.md) | +| `Saruman` (10.0.30.110) | 30 | *(none yet)* | Proxmox VE 9.2.11, no guests — see [roadmap](roadmap.md) | | `oracle` (10.0.99.30) | 99 | *(none yet)* | Undecided | One directory per stack, not one per service. A stack is the unit that gets diff --git a/docs/hardware.md b/docs/hardware.md index 2dc2e55..cbc84ce 100644 --- a/docs/hardware.md +++ b/docs/hardware.md @@ -9,7 +9,7 @@ landfill. | U | Device | Role | | --- | --- | --- | | U1–U2 | APC Smart-UPS[^UPS] | Power | -| U3 | HPE ProLiant DL360 Gen9[^Shiva] | Proxmox hypervisor (`shiva`) | +| U3 | HPE ProLiant DL360 Gen9[^Shiva] | Proxmox hypervisor (`Saruman`, BMC `shiva`) | | U5 | HP ProDesk 600 G4 Mini[^ProDesk] | pfSense firewall (`morpheus`) | | U6 | MT-VIKI 8-port KVM[^KVM] | Console access | | U7 | Jadol 24-port patch panel[^Panel] | Cabling | @@ -25,14 +25,30 @@ through the house. | Host | Hardware | CPU | RAM | Storage | OS | | --- | --- | --- | --- | --- | --- | | `morpheus` | HP ProDesk 600 G4 Mini | i5-8500T | 32 GB | 1 TB SSD | FreeBSD 15.0 (pfSense) | -| `shiva` | HPE ProLiant DL360 Gen9 | 2× Xeon E5 v3/v4 | — | — | Proxmox VE | -| `prometheus` | Apple MacBook Pro (2012) | i5/i7 | — | SSD | Ubuntu Server 24.04.3 | -| `oracle` | Dell Inspiron 15 | i5-1235U | 32 GB | 2 TB SSD | Ubuntu Server 24.04.3 | +| `Saruman` | HPE ProLiant DL360 Gen9 | 2× Xeon E5-2680 v3 (48 threads) | 128 GB | 2× 1 TB SAS HDD, RAID 1 | Proxmox VE 9.2.11 | +| `prometheus` | Apple MacBook Pro (2012, Retina 13") | i5/i7 | 8 GB | 256 GB SSD | Ubuntu Server 24.04.3 | +| `oracle` | Dell Inspiron 15-3565 | AMD A6-9200 (2 cores) | 4 GB | 500 GB HDD | Ubuntu Server 24.04.3 | The observability stack runs on a thirteen-year-old MacBook. It handles four SNMP devices at a 60-second interval, two Alloy agents, and 30 days of metric retention without complaint — which is a useful thing to know before spending -money on a monitoring host. +money on a monitoring host. Its RAM is soldered at 8 GB and it has no built-in +Ethernet, so it reaches the network over a USB NIC. + +`oracle` was previously recorded here as an i5-1235U with 32 GB and a 2 TB SSD. +It is not: it is a dual-core AMD A6-9200 with 4 GB and a 5400 rpm disk. The +older entry described a machine that does not exist, which is worth stating +plainly because it was load-bearing in planning. + +## Management + +| Host | BMC | Address | Notes | +| --- | --- | --- | --- | +| `Saruman` | `shiva` — HPE iLO 4, firmware 2.82 | `10.0.30.10` | iLO Advanced licensed. Dedicated network port. DHCP with a reservation | + +The BMC and the host it manages carry different names and different addresses: +`shiva` is the iLO, `Saruman` is the hypervisor at `10.0.30.110`. Earlier +revisions of this repository treated `shiva` as the hypervisor itself. ## Accessories diff --git a/docs/network.md b/docs/network.md index c03d255..c9a9b49 100644 --- a/docs/network.md +++ b/docs/network.md @@ -22,7 +22,7 @@ allowed to reach. | [Degens](#degens--vlan-10--guest) | 10 | `10.0.10.0/24` | Guest Wi-Fi | Internet | Hostnames are thematic rather than functional — `morpheus` is the firewall, -`mjolnir` the UPS, `shiva` the hypervisor. The Role column is the source of +`mjolnir` the UPS, `Saruman` the hypervisor. The Role column is the source of truth for what a box actually does. --- @@ -77,7 +77,7 @@ only one Hicks is permitted to reach for management. | morpheus | `10.0.99.1` | `02:26:26:xx:xx:xx` | HP ProDesk 600 G4 Mini | FreeBSD 15.0 | Rack U5 | Firewall | | mjolnir | `10.0.99.10` | `28:29:86:xx:xx:xx` | APC Smart-UPS[^UPS] | — | Rack U1–U2 | UPS | | prometheus | `10.0.99.20` | `00:05:1b:xx:xx:xx` | Apple MacBook Pro (2012)[^MacBookPro] | Ubuntu 24.04.3 | Shelf | **Observability stack** | -| oracle | `10.0.99.30` | `58:8a:5a:xx:xx:xx` | Dell Inspiron 15[^Dell] | Ubuntu 24.04.3 | Shelf | Spare / undecided | +| oracle | `10.0.99.30` | `58:8a:5a:xx:xx:xx` | Dell Inspiron 15-3565[^Dell] | Ubuntu 24.04.3 | Shelf | Spare / undecided | ### Notes @@ -172,20 +172,26 @@ Where things get broken on purpose. | Hostname | IP | MAC (OUI) | Device | OS | Location | Role | | --- | --- | --- | --- | --- | --- | --- | | morpheus | `10.0.30.1` | `02:26:26:xx:xx:xx` | HP ProDesk 600 G4 Mini | FreeBSD 15.0 | Rack U5 | Firewall | -| shiva | `10.0.30.10` | `94:57:a5:xx:xx:xx` | HPE ProLiant DL360 Gen9[^Shiva] | Proxmox VE | Rack U3 | Hypervisor | +| shiva | `10.0.30.10` | `94:57:a5:xx:xx:xx` | HPE iLO 4 (DL360 Gen9 BMC)[^Shiva] | iLO 2.82 | Rack U3 | Out-of-band management | +| Saruman | `10.0.30.110` | — | HPE ProLiant DL360 Gen9[^Shiva] | Proxmox VE 9.2.11 | Rack U3 | Hypervisor | ### Notes - Reachable from Hicks only; outbound internet permitted. -- `shiva` will host services for the main network once physical hosts run out. +- `shiva` and `Saruman` are the same physical box: `shiva` is the iLO BMC on its + dedicated port, `Saruman` is the Proxmox install. They are separate addresses + and separate names, and conflating them is a mistake this document previously + made. +- `Saruman` currently runs no guests. - A second server ("ifrit") is planned to carry the deliberately-vulnerable playground, isolated from everything here. > [!NOTE] -> `10.0.30.10` is polled over SNMP as the ProLiant's iLO. If the iLO ever moves -> to a dedicated address, update -> `stacks/observability/prometheus/targets/snmp.yaml` to match — the host and -> its BMC currently share this address. +> `10.0.30.10` is the iLO BMC, not the hypervisor, and it is what +> `stacks/observability/prometheus/targets/snmp.yaml` polls — the `hypervisor-bmc` +> role label there is accurate. The BMC takes its address by DHCP, so it is held +> by a reservation on pfSense; without one, a new lease would silently break the +> SNMP target, which hard-codes the address. [^Shiva]: [HPE ProLiant DL360 Gen9](https://buy.hpe.com/us/en/servers/rack-servers/proliant-dl300-servers/proliant-dl360-server/p/1010026922) diff --git a/docs/roadmap.md b/docs/roadmap.md index b8a7ba3..7af7a07 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -32,12 +32,9 @@ inventory. Ordered roughly by how much it matters. response, so the walk lists individual columns and must stay clear of column 22. - [ ] Deploy Alloy to the remaining hosts — currently only the monitoring host - and one other report in. `shiva` and `oracle` are next. + and one other report in. `Saruman` and `oracle` are next. - [ ] Add blackbox-exporter for uptime and TLS-expiry checks on internal services. -- [ ] Confirm whether `10.0.30.10` is genuinely both the Proxmox host and its - iLO, or whether one of the two records is stale. The SNMP target and the - inventory currently agree on the address but describe different things. - [ ] Capture dashboard screenshots for the README once the stack has a few days of real data. → [`images/README.md`](images/README.md) @@ -47,8 +44,9 @@ inventory. Ordered roughly by how much it matters. is an immediate hard shutdown of the rack. Every rule in `ups.rules.yaml` is currently reporting on a UPS that cannot actually hold the load. -- [ ] Decide what `oracle` (10.0.99.30) is for. It is a 32 GB / 2 TB machine - sitting idle on the management VLAN. +- [ ] Decide what `oracle` (10.0.99.30) is for. It is a dual-core AMD A6-9200 + with 4 GB and a 5400 rpm disk — considerably less machine than this list + previously claimed, and too little for anything demanding. - [ ] Plan and build the NAS on VLAN 40. - [ ] Procure a second server ("ifrit") for the isolated playground network. - [ ] Build the playground — **only** after the main network is finished. @@ -65,6 +63,9 @@ inventory. Ordered roughly by how much it matters. ## Done +- [x] Settle the `10.0.30.10` question. It is the iLO BMC on its dedicated port; + the Proxmox host is `Saruman` at `10.0.30.110`. The SNMP target's + `hypervisor-bmc` role label was correct all along — the inventory was not - [x] Bridge mode on the ISP gateway - [x] Lock down guest VLAN firewall rules - [x] Move IoT devices onto their own SSID and VLAN diff --git a/docs/security.md b/docs/security.md index 8242497..5b6b52e 100644 --- a/docs/security.md +++ b/docs/security.md @@ -15,7 +15,7 @@ What this network is actually built to survive: | A corporate laptop carrying something in from outside | Sits on VLAN 50 but has no management access | | A lab VM escaping into the house | VLAN 30 reachable only *from* trusted, never *to* it | | Losing visibility of a failure | 32 alert rules, 30 days of metrics and logs | -| Mains power loss | UPS on the management VLAN, monitored, alerts on `category=power` | +| Mains power loss | **Not currently defended.** `mjolnir` has no battery installed — see below | What it explicitly does **not** defend against: a determined attacker with physical access to the rack, a supply-chain compromise in an upstream container @@ -24,11 +24,21 @@ filtering by domain, and no MFA on the internal services. ## Segmentation -Default deny between every segment. Two exceptions: +Default deny between every segment. Three exceptions: 1. Specific hosts on **Hicks (50)** may reach **Winterfell (99)** on management ports. Without this there is no way to administer anything. 2. **Hicks (50)** may reach **ImaginationLAN (30)** so the lab is usable. +3. **Winterfell (99)** may reach `10.0.30.10` on **ImaginationLAN (30)** for + SNMP. This is how `snmp-exporter` polls the ProLiant's iLO, and it has been + in place — and answering — for as long as that target has existed. + +[ADR-0002](adr/0002-vlan-segmentation-strategy.md) records two rules, which was +accurate when the decision was made. The third arrived with the monitoring +stack and was never written down. It is the safe direction — management +initiating into the lab, never the reverse — but an undocumented rule is still +an undocumented rule, and a document that overstates a control is worse than +one that admits the exception. Everything else — IoT, media, guest — gets internet and nothing more. @@ -79,6 +89,22 @@ gets ignored — and then a genuinely new leak goes unnoticed alongside it. The purge removed what it acknowledged, so the file was deleted. A history scan that passes with no exceptions is the evidence the purge worked. +### The UPS reports a battery it does not have + +`mjolnir` has no battery installed. Its Network Management Card nonetheless +reports 100% state of charge, 48.0 VDC, a battery temperature, an hour of +runtime, a 2030 replacement date, and `upsAlarmsPresent = 0`. Every one of those +values is derived rather than measured. + +The single honest signal it emits is the self-test result, which returns +**Refused — internal fault**. Any alert rule keyed on charge, runtime or alarm +count will therefore never fire, no matter how bad things get. Rules for this +device must key on `upsAdvTestDiagnosticsResults` and the age of the last +successful test instead. + +This is worth stating carefully: the monitoring did not fail, and neither did +the rules. The device lied, and the rules trusted it. + ### Why SNMPv2c is still a weak point The devices are polled with SNMPv2c, which transmits the community string in