docs(install): add fleet install guide and Ansible playbook

CHANGELOG.md

@@ -20,6 +20,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   improved Go TLS capture.
 - Oracle Cloud instance-metadata support.
 - Pressure Stall Information (PSI) cgroup metrics.
+- Fleet install guide and an Ansible playbook for installing the agent on many hosts at once.
 
 ### Changed
 

README.md

@@ -75,6 +75,8 @@ Pass `-v vX.Y.Z` to pin to a specific release. The script writes a
 systemd unit at `/etc/systemd/system/nudgebee-node-agent.service` and
 starts it.
 
+To install on many hosts at once, see [installing on a fleet](docs/fleet-install.md).
+
 Kubernetes is not required. With no Kubernetes API available, the agent
 names connections by systemd unit, container, host and DNS name instead
 of by pod and service; see

docs/fleet-install.md

@@ -0,0 +1,124 @@
+# Installing on a fleet of hosts
+
+`install.sh` installs the agent on one host. To put it on many hosts in one go, run
+it from a tool that already reaches them. This page covers Ansible (recommended) and
+an SSH loop.
+
+## Before you start
+
+- **Hosts:** Linux with systemd, amd64 or arm64, kernel 5.8 or newer. Windows and
+  macOS aren't supported.
+- **Where the data goes:** the agent pushes to storage you run. Have these ready:
+  - a Prometheus remote-write receiver for metrics (`METRICS_ENDPOINT`)
+  - an OTLP/HTTP endpoint for traces and logs (`TRACES_ENDPOINT`, `LOGS_ENDPOINT`)
+  - an `API_KEY`, if your endpoints need one
+- **Network:** each host needs outbound access to GitHub to download the release, and to
+  your storage endpoints.
+- **Access:** SSH to each host as a user with passwordless `sudo`.
+- **What is tracked:** each systemd unit and each Docker or containerd container is a
+  service. A process started outside a unit or container, such as with `nohup`, is not.
+
+## Settings
+
+The agent reads its settings from environment variables. On a host they live in
+`/etc/default/nudgebee-node-agent`, which the systemd unit loads on start. Both methods
+below write that file, so any setting works, and the key stays out of the process list
+and `sudo` logs. The variable names are listed in [`flags/flags.go`](../flags/flags.go).
+
+The ones a fleet usually sets:
+
+| Variable | What it does |
+|---|---|
+| `METRICS_ENDPOINT` | Prometheus remote-write URL. |
+| `TRACES_ENDPOINT`, `LOGS_ENDPOINT` | OTLP/HTTP URLs. |
+| `API_KEY` | Sent to the endpoints. |
+| `TRACES_SAMPLING` | Fraction of traces kept, `0.0` to `1.0`. Default `1.0`. |
+| `CA_FILE` | CA bundle for endpoints signed by a private CA. |
+
+Set `METRICS_ENDPOINT`, `TRACES_ENDPOINT` and `LOGS_ENDPOINT` individually. `COLLECTOR_ENDPOINT`
+sets all of them from one base URL, but it sends metrics to `<base>/v1/metrics` as Prometheus
+remote-write, which an otel-collector doesn't accept.
+
+When a metrics endpoint is set, the agent serves its own `/metrics` on `127.0.0.1:10300`.
+This can't be changed with `LISTEN`.
+
+## Ansible
+
+The files are in [`examples/ansible`](../examples/ansible).
+
+1. Copy `inventory.example.ini` to `inventory.ini` and list your hosts under `[node_agent]`.
+2. Copy `vars.example.yml` to `vars.yml`. Set your endpoints and a release version.
+3. Keep `API_KEY` in [ansible-vault](https://docs.ansible.com/ansible/latest/vault_guide/index.html).
+   Create the vaulted value without leaving the key in your shell history:
+
+   ```sh
+   ansible-vault encrypt_string --stdin-name api_key
+   ```
+
+   Paste the result into a file such as `secrets.yml`, and uncomment the `API_KEY` line in `vars.yml`.
+4. Try one host first, then run on the fleet:
+
+   ```sh
+   cd examples/ansible
+   ansible-playbook -i inventory.ini -e @vars.yml -e @secrets.yml --ask-vault-pass install-node-agent.yml --limit orders-web
+   ansible-playbook -i inventory.ini -e @vars.yml -e @secrets.yml --ask-vault-pass install-node-agent.yml
+   ```
+
+   Leave out `-e @secrets.yml --ask-vault-pass` if you don't use a vault.
+
+The playbook checks each host meets the requirements, writes the settings file, runs the
+installer, and then checks the agent is running and hasn't restarted. It works through the
+fleet in batches of 20% and stops if more than 10% of a batch fails. Change that with
+`-e node_agent_serial=5`. Variables can also come from `group_vars` or `host_vars`, which
+lets you set per-host values such as `REGION`.
+
+**Upgrade:** change `node_agent_version` and run the playbook again.
+
+**Remove:**
+
+```sh
+ansible-playbook -i inventory.ini -e @vars.yml install-node-agent.yml -e node_agent_state=absent
+```
+
+**Check the fleet:**
+
+```sh
+ansible node_agent -i inventory.ini -b -m command -a 'systemctl is-active nudgebee-node-agent'
+```
+
+## Without Ansible
+
+Any tool that can run a command over SSH works. Put the settings in a file, one
+`NAME='value'` per line, and send it to each host along with the installer:
+
+```sh
+# node-agent.env
+METRICS_ENDPOINT='http://metrics.example.com:8428/api/v1/write'
+TRACES_ENDPOINT='http://otel.example.com:4318/v1/traces'
+LOGS_ENDPOINT='http://otel.example.com:4318/v1/logs'
+TRACES_SAMPLING='0.1'
+```
+
+```sh
+VERSION=v0.1.9
+while read -r host <&3; do
+  ssh "$host" "sudo sh -c 'umask 077; cat > /etc/default/nudgebee-node-agent' &&
+    curl -fsSL https://raw.githubusercontent.com/nudgebee/node-agent/$VERSION/install.sh |
+    sudo sh -s -- -v $VERSION" < node-agent.env
+done 3< hosts.txt
+```
+
+The installer comes from the release tag, so the script and the binary always match. `sudo` must
+not ask for a password, because there's no terminal to type one into.
+
+For hosts you create from a template or autoscaling group, run the same steps from
+cloud-init `runcmd` or the instance user data, so every new host installs itself on first boot.
+
+## Good to know
+
+- **Pin a version** for a fleet. With `latest`, hosts installed on different days run
+  different releases, and every host queries the GitHub API on its own, which can hit
+  GitHub's rate limit when many share one public IP.
+- **Re-running restarts the agent** on every host, even when nothing changed, which leaves a
+  short gap in metrics. With the default `latest` it also upgrades.
+- **Troubleshooting a host:** `journalctl -u nudgebee-node-agent`.

examples/ansible/install-node-agent.yml

@@ -0,0 +1,141 @@
+---
+# Install, upgrade or remove nudgebee-node-agent on a fleet of Linux hosts.
+# See docs/fleet-install.md.
+#
+#   ansible-playbook -i inventory.ini -e @vars.yml install-node-agent.yml
+#   ansible-playbook -i inventory.ini -e @vars.yml install-node-agent.yml -e node_agent_state=absent
+#
+# Variables (set them in vars.yml, group_vars or host_vars; every one has a default):
+#   node_agent_state    present | absent          (default: present)
+#   node_agent_version  a release tag, or latest  (default: latest; pin one for a fleet)
+#   node_agent_env      agent settings, written to /etc/default/nudgebee-node-agent
+#   node_agent_serial   hosts per batch           (default: 20%)
+- name: Nudgebee node-agent
+  hosts: "{{ node_agent_hosts | default('node_agent') }}"
+  become: true
+  # Roll through the fleet in batches and stop if a batch goes bad.
+  serial: "{{ node_agent_serial | default('20%') }}"
+  max_fail_percentage: 10
+  tasks:
+    - name: Resolve settings
+      ansible.builtin.set_fact:
+        na_state: "{{ node_agent_state | default('present') }}"
+        na_version: "{{ node_agent_version | default('latest') }}"
+        na_env: "{{ node_agent_env | default({}) }}"
+
+    - name: Check the settings
+      ansible.builtin.assert:
+        that: na_state in ['present', 'absent']
+        fail_msg: "node_agent_state must be 'present' or 'absent', not '{{ na_state }}'"
+
+    # Values are written in single quotes, where systemd keeps backslashes as they
+    # are. A single quote or newline in a value can't be represented.
+    - name: Check the agent settings can be written
+      ansible.builtin.assert:
+        that: (na_env.values() | map('string') | select('search', "['\n]") | list | length) == 0
+        fail_msg: node_agent_env values can't contain a single quote or a newline
+
+    - name: Check the host can run the agent
+      ansible.builtin.assert:
+        that:
+          - ansible_facts['system'] == 'Linux'
+          - ansible_facts['service_mgr'] == 'systemd'
+          - ansible_facts['architecture'] in ['x86_64', 'aarch64']
+          - ansible_facts['kernel'].split('-')[0] is version('5.8', '>=')
+        fail_msg: >-
+          node-agent needs Linux with systemd, amd64 or arm64, and kernel 5.8 or newer
+          (this host: {{ ansible_facts['system'] }}, {{ ansible_facts['service_mgr'] }},
+          {{ ansible_facts['architecture'] }}, {{ ansible_facts['kernel'] }})
+      when: na_state == 'present'
+
+    - name: Install node-agent
+      when: na_state == 'present'
+      block:
+        # The agent reads this file through its systemd unit. Settings go here
+        # instead of on the installer's command line, so API_KEY stays out of
+        # sudo logs and the process list, and every agent setting is honoured.
+        - name: Write the agent settings
+          ansible.builtin.copy:
+            dest: /etc/default/nudgebee-node-agent
+            owner: root
+            group: root
+            mode: "0600"
+            content: |
+              {% for key, value in na_env.items() %}
+              {{ key }}='{{ value }}'
+              {% endfor %}
+          no_log: true  # the settings can hold API_KEY
+          register: settings
+
+        # Fetch the installer from the same ref as the binary, so a pinned
+        # version never picks up installer changes from a moving branch.
+        - name: Download the installer
+          ansible.builtin.get_url:
+            url: "https://raw.githubusercontent.com/nudgebee/node-agent/{{ 'main' if na_version == 'latest' else na_version }}/install.sh"
+            dest: /var/tmp/nudgebee-node-agent-install.sh
+            owner: root
+            group: root
+            mode: "0700"
+            force: true
+
+        - name: Install or upgrade node-agent
+          ansible.builtin.command: sh /var/tmp/nudgebee-node-agent-install.sh -v {{ na_version }}
+          register: install
+          changed_when: true
+
+        # The installer restarts the service only when its own files change,
+        # so restart for a settings-only change.
+        - name: Make sure the service is running with the current settings
+          ansible.builtin.systemd:
+            name: nudgebee-node-agent
+            state: "{{ 'restarted' if settings.changed else 'started' }}"
+            enabled: true
+          when: not ansible_check_mode
+
+        # Restart=always hides a crash-loop: the unit is briefly "active" before
+        # the agent exits on a bad endpoint, an old kernel or an eBPF load error.
+        # Wait out two restart intervals (5s each) and check it never restarted.
+        - name: Let the agent run for a moment
+          ansible.builtin.pause:
+            seconds: 12
+          when: not ansible_check_mode
+
+        - name: Read the service state
+          ansible.builtin.systemd:
+            name: nudgebee-node-agent
+          register: service
+          when: not ansible_check_mode
+
+        - name: Check the agent is up and stayed up
+          ansible.builtin.assert:
+            that:
+              - service.status.ActiveState == 'active'
+              - service.status.NRestarts | default('0') == '0'
+            fail_msg: >-
+              nudgebee-node-agent is {{ service.status.ActiveState }} and has restarted
+              {{ service.status.NRestarts | default('?') }} time(s). On the host:
+              journalctl -u nudgebee-node-agent
+          when: not ansible_check_mode
+      always:
+        - name: Remove the installer
+          ansible.builtin.file:
+            path: /var/tmp/nudgebee-node-agent-install.sh
+            state: absent
+
+    - name: Remove node-agent
+      when: na_state == 'absent'
+      block:
+        - name: Look for the uninstall script
+          ansible.builtin.stat:
+            path: /usr/bin/nudgebee-node-agent-uninstall.sh
+          register: uninstall_script
+
+        - name: Uninstall node-agent
+          ansible.builtin.command: /usr/bin/nudgebee-node-agent-uninstall.sh
+          changed_when: true
+          when: uninstall_script.stat.exists
+
+        - name: Remove the agent settings
+          ansible.builtin.file:
+            path: /etc/default/nudgebee-node-agent
+            state: absent

examples/ansible/inventory.example.ini

@@ -0,0 +1,8 @@
+# Copy to inventory.ini. Hosts need SSH access and a user that can sudo.
+[node_agent]
+orders-web ansible_host=192.0.2.10
+orders-api ansible_host=192.0.2.11
+db-1       ansible_host=192.0.2.20
+
+[node_agent:vars]
+ansible_user=ubuntu

examples/ansible/vars.example.yml

@@ -0,0 +1,18 @@
+---
+# Copy to vars.yml.
+node_agent_version: v0.1.9  # pin a release; "latest" follows new releases on every run
+
+# Written to /etc/default/nudgebee-node-agent on each host. Any agent setting works
+# here (see flags/flags.go for the names). Quote every value.
+node_agent_env:
+  # Metrics: Prometheus remote-write receiver (VictoriaMetrics, Prometheus, Mimir).
+  METRICS_ENDPOINT: "http://metrics.example.com:8428/api/v1/write"
+  # Traces and logs: OTLP/HTTP, normally an otel-collector.
+  TRACES_ENDPOINT: "http://otel.example.com:4318/v1/traces"
+  LOGS_ENDPOINT: "http://otel.example.com:4318/v1/logs"
+  TRACES_SAMPLING: "0.1"
+  # Keep the key in ansible-vault, not in this file. Create the vaulted value with
+  #   ansible-vault encrypt_string --stdin-name api_key
+  # (it reads the key from stdin, so it stays out of shell history), paste the result
+  # into a vault-vars file, and run the playbook with --ask-vault-pass.
+  # API_KEY: "{{ api_key }}"