diff --git a/CHANGELOG.md b/CHANGELOG.md index 6ae28c97..2ca30290 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,6 +20,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 improved Go TLS capture. - Oracle Cloud instance-metadata support. - Pressure Stall Information (PSI) cgroup metrics. +- Fleet install guide and an Ansible playbook for installing the agent on many hosts at once. ### Changed diff --git a/README.md b/README.md index 997495f6..99a23c6e 100644 --- a/README.md +++ b/README.md @@ -75,6 +75,8 @@ Pass `-v vX.Y.Z` to pin to a specific release. The script writes a systemd unit at `/etc/systemd/system/nudgebee-node-agent.service` and starts it. +To install on many hosts at once, see [installing on a fleet](docs/fleet-install.md). + Kubernetes is not required. With no Kubernetes API available, the agent names connections by systemd unit, container, host and DNS name instead of by pod and service; see diff --git a/docs/fleet-install.md b/docs/fleet-install.md new file mode 100644 index 00000000..14eb52e1 --- /dev/null +++ b/docs/fleet-install.md @@ -0,0 +1,124 @@ +# Installing on a fleet of hosts + +`install.sh` installs the agent on one host. To put it on many hosts in one go, run +it from a tool that already reaches them. This page covers Ansible (recommended) and +an SSH loop. + +## Before you start + +- **Hosts:** Linux with systemd, amd64 or arm64, kernel 5.8 or newer. Windows and + macOS aren't supported. +- **Where the data goes:** the agent pushes to storage you run. Have these ready: + - a Prometheus remote-write receiver for metrics (`METRICS_ENDPOINT`) + - an OTLP/HTTP endpoint for traces and logs (`TRACES_ENDPOINT`, `LOGS_ENDPOINT`) + - an `API_KEY`, if your endpoints need one +- **Network:** each host needs outbound access to GitHub to download the release, and to + your storage endpoints. +- **Access:** SSH to each host as a user with passwordless `sudo`. +- **What is tracked:** each systemd unit and each Docker or containerd container is a + service. A process started outside a unit or container, such as with `nohup`, is not. + +## Settings + +The agent reads its settings from environment variables. On a host they live in +`/etc/default/nudgebee-node-agent`, which the systemd unit loads on start. Both methods +below write that file, so any setting works, and the key stays out of the process list +and `sudo` logs. The variable names are listed in [`flags/flags.go`](../flags/flags.go). + +The ones a fleet usually sets: + +| Variable | What it does | +|---|---| +| `METRICS_ENDPOINT` | Prometheus remote-write URL. | +| `TRACES_ENDPOINT`, `LOGS_ENDPOINT` | OTLP/HTTP URLs. | +| `API_KEY` | Sent to the endpoints. | +| `TRACES_SAMPLING` | Fraction of traces kept, `0.0` to `1.0`. Default `1.0`. | +| `CA_FILE` | CA bundle for endpoints signed by a private CA. | + +Set `METRICS_ENDPOINT`, `TRACES_ENDPOINT` and `LOGS_ENDPOINT` individually. `COLLECTOR_ENDPOINT` +sets all of them from one base URL, but it sends metrics to `/v1/metrics` as Prometheus +remote-write, which an otel-collector doesn't accept. + +When a metrics endpoint is set, the agent serves its own `/metrics` on `127.0.0.1:10300`. +This can't be changed with `LISTEN`. + +## Ansible + +The files are in [`examples/ansible`](../examples/ansible). + +1. Copy `inventory.example.ini` to `inventory.ini` and list your hosts under `[node_agent]`. +2. Copy `vars.example.yml` to `vars.yml`. Set your endpoints and a release version. +3. Keep `API_KEY` in [ansible-vault](https://docs.ansible.com/ansible/latest/vault_guide/index.html). + Create the vaulted value without leaving the key in your shell history: + + ```sh + ansible-vault encrypt_string --stdin-name api_key + ``` + + Paste the result into a file such as `secrets.yml`, and uncomment the `API_KEY` line in `vars.yml`. +4. Try one host first, then run on the fleet: + + ```sh + cd examples/ansible + ansible-playbook -i inventory.ini -e @vars.yml -e @secrets.yml --ask-vault-pass install-node-agent.yml --limit orders-web + ansible-playbook -i inventory.ini -e @vars.yml -e @secrets.yml --ask-vault-pass install-node-agent.yml + ``` + + Leave out `-e @secrets.yml --ask-vault-pass` if you don't use a vault. + +The playbook checks each host meets the requirements, writes the settings file, runs the +installer, and then checks the agent is running and hasn't restarted. It works through the +fleet in batches of 20% and stops if more than 10% of a batch fails. Change that with +`-e node_agent_serial=5`. Variables can also come from `group_vars` or `host_vars`, which +lets you set per-host values such as `REGION`. + +**Upgrade:** change `node_agent_version` and run the playbook again. + +**Remove:** + +```sh +ansible-playbook -i inventory.ini -e @vars.yml install-node-agent.yml -e node_agent_state=absent +``` + +**Check the fleet:** + +```sh +ansible node_agent -i inventory.ini -b -m command -a 'systemctl is-active nudgebee-node-agent' +``` + +## Without Ansible + +Any tool that can run a command over SSH works. Put the settings in a file, one +`NAME='value'` per line, and send it to each host along with the installer: + +```sh +# node-agent.env +METRICS_ENDPOINT='http://metrics.example.com:8428/api/v1/write' +TRACES_ENDPOINT='http://otel.example.com:4318/v1/traces' +LOGS_ENDPOINT='http://otel.example.com:4318/v1/logs' +TRACES_SAMPLING='0.1' +``` + +```sh +VERSION=v0.1.9 +while read -r host <&3; do + ssh "$host" "sudo sh -c 'umask 077; cat > /etc/default/nudgebee-node-agent' && + curl -fsSL https://raw.githubusercontent.com/nudgebee/node-agent/$VERSION/install.sh | + sudo sh -s -- -v $VERSION" < node-agent.env +done 3< hosts.txt +``` + +The installer comes from the release tag, so the script and the binary always match. `sudo` must +not ask for a password, because there's no terminal to type one into. + +For hosts you create from a template or autoscaling group, run the same steps from +cloud-init `runcmd` or the instance user data, so every new host installs itself on first boot. + +## Good to know + +- **Pin a version** for a fleet. With `latest`, hosts installed on different days run + different releases, and every host queries the GitHub API on its own, which can hit + GitHub's rate limit when many share one public IP. +- **Re-running restarts the agent** on every host, even when nothing changed, which leaves a + short gap in metrics. With the default `latest` it also upgrades. +- **Troubleshooting a host:** `journalctl -u nudgebee-node-agent`. diff --git a/examples/ansible/install-node-agent.yml b/examples/ansible/install-node-agent.yml new file mode 100644 index 00000000..093b85f8 --- /dev/null +++ b/examples/ansible/install-node-agent.yml @@ -0,0 +1,141 @@ +--- +# Install, upgrade or remove nudgebee-node-agent on a fleet of Linux hosts. +# See docs/fleet-install.md. +# +# ansible-playbook -i inventory.ini -e @vars.yml install-node-agent.yml +# ansible-playbook -i inventory.ini -e @vars.yml install-node-agent.yml -e node_agent_state=absent +# +# Variables (set them in vars.yml, group_vars or host_vars; every one has a default): +# node_agent_state present | absent (default: present) +# node_agent_version a release tag, or latest (default: latest; pin one for a fleet) +# node_agent_env agent settings, written to /etc/default/nudgebee-node-agent +# node_agent_serial hosts per batch (default: 20%) +- name: Nudgebee node-agent + hosts: "{{ node_agent_hosts | default('node_agent') }}" + become: true + # Roll through the fleet in batches and stop if a batch goes bad. + serial: "{{ node_agent_serial | default('20%') }}" + max_fail_percentage: 10 + tasks: + - name: Resolve settings + ansible.builtin.set_fact: + na_state: "{{ node_agent_state | default('present') }}" + na_version: "{{ node_agent_version | default('latest') }}" + na_env: "{{ node_agent_env | default({}) }}" + + - name: Check the settings + ansible.builtin.assert: + that: na_state in ['present', 'absent'] + fail_msg: "node_agent_state must be 'present' or 'absent', not '{{ na_state }}'" + + # Values are written in single quotes, where systemd keeps backslashes as they + # are. A single quote or newline in a value can't be represented. + - name: Check the agent settings can be written + ansible.builtin.assert: + that: (na_env.values() | map('string') | select('search', "['\n]") | list | length) == 0 + fail_msg: node_agent_env values can't contain a single quote or a newline + + - name: Check the host can run the agent + ansible.builtin.assert: + that: + - ansible_facts['system'] == 'Linux' + - ansible_facts['service_mgr'] == 'systemd' + - ansible_facts['architecture'] in ['x86_64', 'aarch64'] + - ansible_facts['kernel'].split('-')[0] is version('5.8', '>=') + fail_msg: >- + node-agent needs Linux with systemd, amd64 or arm64, and kernel 5.8 or newer + (this host: {{ ansible_facts['system'] }}, {{ ansible_facts['service_mgr'] }}, + {{ ansible_facts['architecture'] }}, {{ ansible_facts['kernel'] }}) + when: na_state == 'present' + + - name: Install node-agent + when: na_state == 'present' + block: + # The agent reads this file through its systemd unit. Settings go here + # instead of on the installer's command line, so API_KEY stays out of + # sudo logs and the process list, and every agent setting is honoured. + - name: Write the agent settings + ansible.builtin.copy: + dest: /etc/default/nudgebee-node-agent + owner: root + group: root + mode: "0600" + content: | + {% for key, value in na_env.items() %} + {{ key }}='{{ value }}' + {% endfor %} + no_log: true # the settings can hold API_KEY + register: settings + + # Fetch the installer from the same ref as the binary, so a pinned + # version never picks up installer changes from a moving branch. + - name: Download the installer + ansible.builtin.get_url: + url: "https://raw.githubusercontent.com/nudgebee/node-agent/{{ 'main' if na_version == 'latest' else na_version }}/install.sh" + dest: /var/tmp/nudgebee-node-agent-install.sh + owner: root + group: root + mode: "0700" + force: true + + - name: Install or upgrade node-agent + ansible.builtin.command: sh /var/tmp/nudgebee-node-agent-install.sh -v {{ na_version }} + register: install + changed_when: true + + # The installer restarts the service only when its own files change, + # so restart for a settings-only change. + - name: Make sure the service is running with the current settings + ansible.builtin.systemd: + name: nudgebee-node-agent + state: "{{ 'restarted' if settings.changed else 'started' }}" + enabled: true + when: not ansible_check_mode + + # Restart=always hides a crash-loop: the unit is briefly "active" before + # the agent exits on a bad endpoint, an old kernel or an eBPF load error. + # Wait out two restart intervals (5s each) and check it never restarted. + - name: Let the agent run for a moment + ansible.builtin.pause: + seconds: 12 + when: not ansible_check_mode + + - name: Read the service state + ansible.builtin.systemd: + name: nudgebee-node-agent + register: service + when: not ansible_check_mode + + - name: Check the agent is up and stayed up + ansible.builtin.assert: + that: + - service.status.ActiveState == 'active' + - service.status.NRestarts | default('0') == '0' + fail_msg: >- + nudgebee-node-agent is {{ service.status.ActiveState }} and has restarted + {{ service.status.NRestarts | default('?') }} time(s). On the host: + journalctl -u nudgebee-node-agent + when: not ansible_check_mode + always: + - name: Remove the installer + ansible.builtin.file: + path: /var/tmp/nudgebee-node-agent-install.sh + state: absent + + - name: Remove node-agent + when: na_state == 'absent' + block: + - name: Look for the uninstall script + ansible.builtin.stat: + path: /usr/bin/nudgebee-node-agent-uninstall.sh + register: uninstall_script + + - name: Uninstall node-agent + ansible.builtin.command: /usr/bin/nudgebee-node-agent-uninstall.sh + changed_when: true + when: uninstall_script.stat.exists + + - name: Remove the agent settings + ansible.builtin.file: + path: /etc/default/nudgebee-node-agent + state: absent diff --git a/examples/ansible/inventory.example.ini b/examples/ansible/inventory.example.ini new file mode 100644 index 00000000..9662e920 --- /dev/null +++ b/examples/ansible/inventory.example.ini @@ -0,0 +1,8 @@ +# Copy to inventory.ini. Hosts need SSH access and a user that can sudo. +[node_agent] +orders-web ansible_host=192.0.2.10 +orders-api ansible_host=192.0.2.11 +db-1 ansible_host=192.0.2.20 + +[node_agent:vars] +ansible_user=ubuntu diff --git a/examples/ansible/vars.example.yml b/examples/ansible/vars.example.yml new file mode 100644 index 00000000..09a0dcf9 --- /dev/null +++ b/examples/ansible/vars.example.yml @@ -0,0 +1,18 @@ +--- +# Copy to vars.yml. +node_agent_version: v0.1.9 # pin a release; "latest" follows new releases on every run + +# Written to /etc/default/nudgebee-node-agent on each host. Any agent setting works +# here (see flags/flags.go for the names). Quote every value. +node_agent_env: + # Metrics: Prometheus remote-write receiver (VictoriaMetrics, Prometheus, Mimir). + METRICS_ENDPOINT: "http://metrics.example.com:8428/api/v1/write" + # Traces and logs: OTLP/HTTP, normally an otel-collector. + TRACES_ENDPOINT: "http://otel.example.com:4318/v1/traces" + LOGS_ENDPOINT: "http://otel.example.com:4318/v1/logs" + TRACES_SAMPLING: "0.1" + # Keep the key in ansible-vault, not in this file. Create the vaulted value with + # ansible-vault encrypt_string --stdin-name api_key + # (it reads the key from stdin, so it stays out of shell history), paste the result + # into a vault-vars file, and run the playbook with --ask-vault-pass. + # API_KEY: "{{ api_key }}"