Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ graph TB
WS["workstations"]
end
subgraph V30["VLAN 30 · Lab"]
HV["shiva · Proxmox"]
HV["Saruman · Proxmox<br/>BMC: shiva"]
end
subgraph Terminal["VLANs 40 / 20 / 10 · egress only"]
UNTRUSTED["media · IoT · guest"]
Expand Down
4 changes: 2 additions & 2 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ graph TB
end

subgraph V30["VLAN 30 · ImaginationLAN · Lab"]
HV["shiva · ProLiant DL360 Gen9<br/>Proxmox VE"]
HV["Saruman · ProLiant DL360 Gen9<br/>Proxmox VE<br/>BMC: shiva"]
end

subgraph V20["VLAN 20 · Skids · IoT"]
Expand Down Expand Up @@ -131,7 +131,7 @@ hole from the monitoring VLAN into the monitored one.
| Host | VLAN | Stack | Contents |
| --- | --- | --- | --- |
| `prometheus` (10.0.99.20) | 99 | [`stacks/observability`](../stacks/observability) | Prometheus, Alertmanager, Loki, Grafana, snmp-exporter, Alloy |
| `shiva` (10.0.30.10) | 30 | *(none yet)* | Proxmox VE — see [roadmap](roadmap.md) |
| `Saruman` (10.0.30.110) | 30 | *(none yet)* | Proxmox VE 9.2.11, no guests — see [roadmap](roadmap.md) |
| `oracle` (10.0.99.30) | 99 | *(none yet)* | Undecided |

One directory per stack, not one per service. A stack is the unit that gets
Expand Down
26 changes: 21 additions & 5 deletions docs/hardware.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ landfill.
| U | Device | Role |
| --- | --- | --- |
| U1–U2 | APC Smart-UPS[^UPS] | Power |
| U3 | HPE ProLiant DL360 Gen9[^Shiva] | Proxmox hypervisor (`shiva`) |
| U3 | HPE ProLiant DL360 Gen9[^Shiva] | Proxmox hypervisor (`Saruman`, BMC `shiva`) |
| U5 | HP ProDesk 600 G4 Mini[^ProDesk] | pfSense firewall (`morpheus`) |
| U6 | MT-VIKI 8-port KVM[^KVM] | Console access |
| U7 | Jadol 24-port patch panel[^Panel] | Cabling |
Expand All @@ -25,14 +25,30 @@ through the house.
| Host | Hardware | CPU | RAM | Storage | OS |
| --- | --- | --- | --- | --- | --- |
| `morpheus` | HP ProDesk 600 G4 Mini | i5-8500T | 32 GB | 1 TB SSD | FreeBSD 15.0 (pfSense) |
| `shiva` | HPE ProLiant DL360 Gen9 | 2× Xeon E5 v3/v4 | — | — | Proxmox VE |
| `prometheus` | Apple MacBook Pro (2012) | i5/i7 | — | SSD | Ubuntu Server 24.04.3 |
| `oracle` | Dell Inspiron 15 | i5-1235U | 32 GB | 2 TB SSD | Ubuntu Server 24.04.3 |
| `Saruman` | HPE ProLiant DL360 Gen9 | 2× Xeon E5-2680 v3 (48 threads) | 128 GB | 2× 1 TB SAS HDD, RAID 1 | Proxmox VE 9.2.11 |
| `prometheus` | Apple MacBook Pro (2012, Retina 13") | i5/i7 | 8 GB | 256 GB SSD | Ubuntu Server 24.04.3 |
| `oracle` | Dell Inspiron 15-3565 | AMD A6-9200 (2 cores) | 4 GB | 500 GB HDD | Ubuntu Server 24.04.3 |

The observability stack runs on a thirteen-year-old MacBook. It handles four
SNMP devices at a 60-second interval, two Alloy agents, and 30 days of metric
retention without complaint — which is a useful thing to know before spending
money on a monitoring host.
money on a monitoring host. Its RAM is soldered at 8 GB and it has no built-in
Ethernet, so it reaches the network over a USB NIC.

`oracle` was previously recorded here as an i5-1235U with 32 GB and a 2 TB SSD.
It is not: it is a dual-core AMD A6-9200 with 4 GB and a 5400 rpm disk. The
older entry described a machine that does not exist, which is worth stating
plainly because it was load-bearing in planning.

## Management

| Host | BMC | Address | Notes |
| --- | --- | --- | --- |
| `Saruman` | `shiva` — HPE iLO 4, firmware 2.82 | `10.0.30.10` | iLO Advanced licensed. Dedicated network port. DHCP with a reservation |

The BMC and the host it manages carry different names and different addresses:
`shiva` is the iLO, `Saruman` is the hypervisor at `10.0.30.110`. Earlier
revisions of this repository treated `shiva` as the hypervisor itself.

## Accessories

Expand Down
22 changes: 14 additions & 8 deletions docs/network.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ allowed to reach.
| [Degens](#degens--vlan-10--guest) | 10 | `10.0.10.0/24` | Guest Wi-Fi | Internet |

Hostnames are thematic rather than functional — `morpheus` is the firewall,
`mjolnir` the UPS, `shiva` the hypervisor. The Role column is the source of
`mjolnir` the UPS, `Saruman` the hypervisor. The Role column is the source of
truth for what a box actually does.

---
Expand Down Expand Up @@ -77,7 +77,7 @@ only one Hicks is permitted to reach for management.
| morpheus | `10.0.99.1` | `02:26:26:xx:xx:xx` | HP ProDesk 600 G4 Mini | FreeBSD 15.0 | Rack U5 | Firewall |
| mjolnir | `10.0.99.10` | `28:29:86:xx:xx:xx` | APC Smart-UPS[^UPS] | — | Rack U1–U2 | UPS |
| prometheus | `10.0.99.20` | `00:05:1b:xx:xx:xx` | Apple MacBook Pro (2012)[^MacBookPro] | Ubuntu 24.04.3 | Shelf | **Observability stack** |
| oracle | `10.0.99.30` | `58:8a:5a:xx:xx:xx` | Dell Inspiron 15[^Dell] | Ubuntu 24.04.3 | Shelf | Spare / undecided |
| oracle | `10.0.99.30` | `58:8a:5a:xx:xx:xx` | Dell Inspiron 15-3565[^Dell] | Ubuntu 24.04.3 | Shelf | Spare / undecided |

### Notes

Expand Down Expand Up @@ -172,20 +172,26 @@ Where things get broken on purpose.
| Hostname | IP | MAC (OUI) | Device | OS | Location | Role |
| --- | --- | --- | --- | --- | --- | --- |
| morpheus | `10.0.30.1` | `02:26:26:xx:xx:xx` | HP ProDesk 600 G4 Mini | FreeBSD 15.0 | Rack U5 | Firewall |
| shiva | `10.0.30.10` | `94:57:a5:xx:xx:xx` | HPE ProLiant DL360 Gen9[^Shiva] | Proxmox VE | Rack U3 | Hypervisor |
| shiva | `10.0.30.10` | `94:57:a5:xx:xx:xx` | HPE iLO 4 (DL360 Gen9 BMC)[^Shiva] | iLO 2.82 | Rack U3 | Out-of-band management |
| Saruman | `10.0.30.110` | — | HPE ProLiant DL360 Gen9[^Shiva] | Proxmox VE 9.2.11 | Rack U3 | Hypervisor |

### Notes

- Reachable from Hicks only; outbound internet permitted.
- `shiva` will host services for the main network once physical hosts run out.
- `shiva` and `Saruman` are the same physical box: `shiva` is the iLO BMC on its
dedicated port, `Saruman` is the Proxmox install. They are separate addresses
and separate names, and conflating them is a mistake this document previously
made.
- `Saruman` currently runs no guests.
- A second server ("ifrit") is planned to carry the deliberately-vulnerable
playground, isolated from everything here.

> [!NOTE]
> `10.0.30.10` is polled over SNMP as the ProLiant's iLO. If the iLO ever moves
> to a dedicated address, update
> `stacks/observability/prometheus/targets/snmp.yaml` to match — the host and
> its BMC currently share this address.
> `10.0.30.10` is the iLO BMC, not the hypervisor, and it is what
> `stacks/observability/prometheus/targets/snmp.yaml` polls — the `hypervisor-bmc`
> role label there is accurate. The BMC takes its address by DHCP, so it is held
> by a reservation on pfSense; without one, a new lease would silently break the
> SNMP target, which hard-codes the address.

[^Shiva]: [HPE ProLiant DL360 Gen9](https://buy.hpe.com/us/en/servers/rack-servers/proliant-dl300-servers/proliant-dl360-server/p/1010026922)

Expand Down
13 changes: 7 additions & 6 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,12 +32,9 @@ inventory. Ordered roughly by how much it matters.
response, so the walk lists individual columns and must stay clear of
column 22.
- [ ] Deploy Alloy to the remaining hosts — currently only the monitoring host
and one other report in. `shiva` and `oracle` are next.
and one other report in. `Saruman` and `oracle` are next.
- [ ] Add blackbox-exporter for uptime and TLS-expiry checks on internal
services.
- [ ] Confirm whether `10.0.30.10` is genuinely both the Proxmox host and its
iLO, or whether one of the two records is stale. The SNMP target and the
inventory currently agree on the address but describe different things.
- [ ] Capture dashboard screenshots for the README once the stack has a few days
of real data. → [`images/README.md`](images/README.md)

Expand All @@ -47,8 +44,9 @@ inventory. Ordered roughly by how much it matters.
is an immediate hard shutdown of the rack. Every rule in
`ups.rules.yaml` is currently reporting on a UPS that cannot actually hold
the load.
- [ ] Decide what `oracle` (10.0.99.30) is for. It is a 32 GB / 2 TB machine
sitting idle on the management VLAN.
- [ ] Decide what `oracle` (10.0.99.30) is for. It is a dual-core AMD A6-9200
with 4 GB and a 5400 rpm disk — considerably less machine than this list
previously claimed, and too little for anything demanding.
- [ ] Plan and build the NAS on VLAN 40.
- [ ] Procure a second server ("ifrit") for the isolated playground network.
- [ ] Build the playground — **only** after the main network is finished.
Expand All @@ -65,6 +63,9 @@ inventory. Ordered roughly by how much it matters.

## Done

- [x] Settle the `10.0.30.10` question. It is the iLO BMC on its dedicated port;
the Proxmox host is `Saruman` at `10.0.30.110`. The SNMP target's
`hypervisor-bmc` role label was correct all along — the inventory was not
- [x] Bridge mode on the ISP gateway
- [x] Lock down guest VLAN firewall rules
- [x] Move IoT devices onto their own SSID and VLAN
Expand Down
30 changes: 28 additions & 2 deletions docs/security.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ What this network is actually built to survive:
| A corporate laptop carrying something in from outside | Sits on VLAN 50 but has no management access |
| A lab VM escaping into the house | VLAN 30 reachable only *from* trusted, never *to* it |
| Losing visibility of a failure | 32 alert rules, 30 days of metrics and logs |
| Mains power loss | UPS on the management VLAN, monitored, alerts on `category=power` |
| Mains power loss | **Not currently defended.** `mjolnir` has no battery installed — see below |

What it explicitly does **not** defend against: a determined attacker with
physical access to the rack, a supply-chain compromise in an upstream container
Expand All @@ -24,11 +24,21 @@ filtering by domain, and no MFA on the internal services.

## Segmentation

Default deny between every segment. Two exceptions:
Default deny between every segment. Three exceptions:

1. Specific hosts on **Hicks (50)** may reach **Winterfell (99)** on management
ports. Without this there is no way to administer anything.
2. **Hicks (50)** may reach **ImaginationLAN (30)** so the lab is usable.
3. **Winterfell (99)** may reach `10.0.30.10` on **ImaginationLAN (30)** for
SNMP. This is how `snmp-exporter` polls the ProLiant's iLO, and it has been
in place — and answering — for as long as that target has existed.

[ADR-0002](adr/0002-vlan-segmentation-strategy.md) records two rules, which was
accurate when the decision was made. The third arrived with the monitoring
stack and was never written down. It is the safe direction — management
initiating into the lab, never the reverse — but an undocumented rule is still
an undocumented rule, and a document that overstates a control is worse than
one that admits the exception.

Everything else — IoT, media, guest — gets internet and nothing more.

Expand Down Expand Up @@ -79,6 +89,22 @@ gets ignored — and then a genuinely new leak goes unnoticed alongside it. The
purge removed what it acknowledged, so the file was deleted. A history scan
that passes with no exceptions is the evidence the purge worked.

### The UPS reports a battery it does not have

`mjolnir` has no battery installed. Its Network Management Card nonetheless
reports 100% state of charge, 48.0 VDC, a battery temperature, an hour of
runtime, a 2030 replacement date, and `upsAlarmsPresent = 0`. Every one of those
values is derived rather than measured.

The single honest signal it emits is the self-test result, which returns
**Refused — internal fault**. Any alert rule keyed on charge, runtime or alarm
count will therefore never fire, no matter how bad things get. Rules for this
device must key on `upsAdvTestDiagnosticsResults` and the age of the last
successful test instead.

This is worth stating carefully: the monitoring did not fail, and neither did
the rules. The device lied, and the rules trusted it.

### Why SNMPv2c is still a weak point

The devices are polled with SNMPv2c, which transmits the community string in
Expand Down
Loading