otaru

Over-Engineering at Its Finest

Bare-metal k3s home lab and technical playground.

The only rule: Raspberry Pi 🍓. Why? Why not ¯\_(ツ)_/¯

Architecture

Current cluster layout:

• Dedicated 192.168.10.0/24 VLAN for cluster nodes and service virtual IPs
• k3s with embedded etcd on three control-plane nodes
• Flannel wireguard-native for pod networking
• MetalLB + Envoy Gateway for service and ingress virtual IPs, with the Envoy Gateway controller in envoy-gateway-system and the ingress proxy in gateway
• Istio ambient mesh with Kiali for service mesh observability
• MCP servers (Kubernetes and UniFi) are JWT-gated at Envoy Gateway; local agents reach them through a workstation agentgateway that injects short-lived Hydra tokens (see MCP authentication)

Hardware

Node	Device	Role	Storage
raspberrypi-00	Raspberry Pi 5 8GB	Control plane	Lexar NM620 256GB
raspberrypi-01	Raspberry Pi 5 8GB	Control plane	Crucial P3 Plus 4TB
raspberrypi-02	Raspberry Pi 5 8GB	Control plane	Crucial P3 Plus 4TB
raspberrypi-03	Raspberry Pi 5 8GB	Worker	Crucial P2 500GB
ai-00	AI inference host	Worker	-
nuc-001	Intel NUC Mini PC Core i3-3217U DC3217IYE 8GB	Worker	120 GB SSD
ucg-ultra	UniFi Cloud Gateway Ultra	Router/Gateway	-
usw-lite-8-poe	UniFi Switch Lite 8 PoE	PoE switch	-
usw-ultra	UniFi Switch Ultra	PoE switch	-
u7-pro-front	UniFi U7 Pro	Access point	-
u7-pro-back	UniFi U7 Pro	Access point	-
rackmate-t1	GeeekPi DeskPi RackMate T1	Rack enclosure	-
rack-mount	GeeekPi 10" 2U Rack Mount	Pi rack mount	-

Three nodes form the control plane. Three nodes are workers, including ai-00 for inference and temporary nuc-00.

Network Layout

The UniFi gateway routes eight managed networks. Detailed firewall behavior and verification steps are documented in Infrastructure operations.

VLAN	Network	Subnet	Purpose
1	Default	192.168.1.0/24	Untagged and VLAN-unaware IoT devices
3	Guest	192.168.3.0/24	Visitor devices
4	Client	192.168.4.0/24	Trusted personal clients and media receivers
5	IoT Public	192.168.5.0/24	Internet-connected IoT devices
6	IoT Private	192.168.6.0/24	IoT devices that must remain offline
7	Work	192.168.7.0/24	Work devices
8	Unrestricted	192.168.8.0/24	Unrestricted trusted devices
10	Server	192.168.10.0/24	Cluster nodes and service VIPs

Key addresses on the Server network:

IP	Role
192.168.10.1	VLAN gateway
192.168.10.50	Kubernetes API VIP
192.168.10.51	Internal ingress VIP
192.168.10.60	raspberrypi-00
192.168.10.61	raspberrypi-01
192.168.10.62	raspberrypi-02
192.168.10.63	raspberrypi-03
192.168.10.70	ai-00
192.168.10.80	nuc-00

Cluster Components

Category	Name	Description
Application	Blocky	Stateless DNS server
Application	changedetection.io	Self-hosted website change detection and alerting
Application	CyberChef	The Cyber Swiss Army Knife by GCHQ
Application	Excalidraw	Virtual whiteboard for sketching hand-drawn like diagrams
Application	Firecrawl	Self-hosted web page extraction
Application	Hermes	Self-hosted AI agent runtime with local model integration
Application	Home Assistant	Home automation
Application	Inference	Self-hosted inference server
Application	JSON Crack	JSON, YAML, etc. visualizer and editor
Application	Kubernetes MCP Server	Model Context Protocol server for Kubernetes cluster operations
Application	SearXNG	Self-hosted web search
Application	TeslaMate	Self-hosted data logger for Tesla
Application	Umami	Self-hosted web analytics
Application	UniFi MCP	Model Context Protocol server for UniFi Network controller operations
Application	冗PowerBot	Telegram bot tracks and counts individual message counts in groups
CI/CD	Argo CD	GitOps, drift detection, and reconciliation
Connectivity	Cloudflare Tunnel	Cloudflare Zero Trust edge
Connectivity	CoreDNS	Cluster DNS managed separately from the disabled bundled k3s add-on
Connectivity	Envoy Gateway	Gateway API ingress controller with controller in envoy-gateway-system and proxy in gateway; TCP/HTTP ingress can use the mesh path, UDP remains outside Istio mTLS
Connectivity	Flannel	wireguard-native encrypted pod networking for k3s
Connectivity	Gateway API	Kubernetes standard CRDs for managing network traffic
Connectivity	httpbin	Generic echo server for health checks
Connectivity	Istio	Service mesh control plane and ambient dataplane (istiod, istio-cni, ztunnel)
Connectivity	k3s-apiserver-loadbalancer	Operator that keeps the kubernetes service exposed as a LoadBalancer
Connectivity	MetalLB	Bare metal LoadBalancer implementation for service virtual IP allocation and L2 advertisement
Connectivity	Tailscale	Kubernetes operator that publishes the Envoy Gateway VIP onto a Tailscale tailnet
Database	CloudNativePG	A Kubernetes operator that manages PostgreSQL clusters
Database	CloudNativePG Barman Cloud Plugin	Backup plugin for CloudNativePG
Monitoring	Fluent Bit	Node log collection and shipping to Loki
Monitoring	Grafana	Dashboards and visualization for cluster metrics and logs
Monitoring	Heartbeats	Kubernetes operator for heartbeat monitoring
Monitoring	Kiali	Service mesh observability UI for Istio
Monitoring	Kubernetes Metrics Server	Scalable, efficient source of container resource metrics for Kubernetes built-in autoscaling pipelines
Monitoring	Loki	Log aggregation and query backend
Monitoring	Prometheus	Metrics collection and query backend
Scheduling	Descheduler	Evicts pods that restart too often, repaves old Deployments, and moves pods off nodes that are over the utilisation target
Scheduling	k8s-cleaner	Automated failed pod cleanup and periodic workload repaving
Scheduling	KEDA	Event Driven Autoscaler
Scheduling	Reloader	Watch changes in ConfigMap and Secret and do rolling upgrades
Scheduling	VPA	Vertical Pod Autoscaler for automatic CPU/memory right-sizing
Security	1Password Connect	Proxy service for 1Password; acts as a secret provider
Security	amazon-eks-pod-identity-webhook	Amazon EKS Pod Identity Webhook for IRSA in bare metal Kubernetes clusters
Security	cert-manager	Manages TLS certificates via Let's Encrypt and ACME protocol
Security	External Secrets Operator	Extracts secrets from a secret provider
Security	Falco	Cloud-native runtime security
Security	Kyverno	Kubernetes policy engine
Security	oidc-provider	Kubernetes OIDC provider and JWKS endpoint
Security	Ory Hydra	OAuth 2.0 and OpenID Connect issuer for machine clients and JWT validation
Security	Trivy Operator	Vulnerability and config scans for running workloads
Storage	CSI Snapshot Controller	Kubernetes CSI VolumeSnapshot CRDs and controller used by Longhorn
Storage	Longhorn	Distributed block storage system; backup and restore from/to remote destinations

IaaS, PaaS, and SaaS

Category	Name	Service	Description
AI	Meta	Muse	Personal AI agent
CI/CD	Anthropic	Claude Code	AI code review
CI/CD	GitHub	Actions	Repository validation and release automation
CI/CD	Renovate	Dependency Updates	Automated dependency maintenance
Connectivity	Cloudflare	Access	Edge access control
Connectivity	Cloudflare	DNS	Authoritative DNS service
Connectivity	Cloudflare	Tunnel	Edge connectivity
Connectivity	Tailscale	Tailscale	Private overlay network
Connectivity	UniFi	Network	Gateway, VLAN, WLAN, and firewall management
Messaging	Telegram	Bot API	Bot messaging for Hermes and 冗PowerBot
Monitoring	Heartbeats	Heartbeats Operator	Kubernetes operator for heartbeat monitoring
Monitoring	WebGazer	Uptime Monitoring	Health check
Security	1Password	Connect	Secrets automation
Security	AWS	IAM	OIDC/IRSA permissions and token exchange
Security	Let's Encrypt	Let's Encrypt	Certificate authority
Security	Snyk	Snyk	Vulnerability detection
Storage	AWS	DynamoDB	Database backend for 冗PowerBot
Storage	AWS	SQS	Queue backend for 冗PowerBot
Storage	AWS	S3	OpenTofu remote state
Storage	Backblaze	B2	Longhorn and CloudNativePG backups

Cluster Bootstrap

1. Install Tooling

   brew install mise

2. Configure Tooling

   Install the tools in mise.toml.

   mise trust
   mise install

   Activate Mise in your shell so direct tool commands use the versions in mise.toml.

   Authenticate the GitHub CLI before Terraform/Terragrunt or Helm OCI dependency updates need GitHub access.

   gh auth login

3. Add SSH Keys to known_hosts

   KH=~/.ssh/known_hosts && touch "$KH" && for ip in 192.168.10.{60..63} 192.168.10.70 192.168.10.80; do ssh-keygen -f "$KH" -R "$ip"; ssh-keyscan "$ip" >> "$KH"; done

4. Set Up Service Credentials

   Create the local files documented in Secrets before running cluster bootstrap. Keep actual credentials outside this repo.

5. Bootstrap Cluster

   make setup

Oopsy

Update host packages and reboot the entire cluster.

make maintenance

Upgrade the k3s Kubernetes version and restart workloads.

make upgrade

Unlock a LUKS node after boot.

make unlock <node-name>

Wipe everything and start from scratch.

make nuke

Set up Raspberry Pi nodes and the k3s cluster.

make setup

Footnotes

1. Yes, yes, I know I said Raspberry Pi only, but have you seen Raspberry Pi 5 prices lately? ¯\_(ツ)_/¯ This is a temporary worker node until the damage-to-wallet ratio improves. ↩