Skip to content
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 69 additions & 0 deletions docs/DEPLOY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
### Copyright (c) 2026

Author(s):
* The Jamulus Development Team

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License
along with this program. If not, see [<https://www.gnu.org/licenses/>](https://www.gnu.org/licenses/).

---
Comment on lines +3 to +19

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suppose this is not right - we need the full header.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed — full AGPL header now (also applied the same fix to docs/ARCHITECTURE.md in #3791).


# Deploying a Jamulus Server

[COMPILING.md](../COMPILING.md) ends when the binary exists. This document covers the step after that: putting a self-built headless server binary on a production host and verifying it actually runs. Most self-inflicted server outages happen in this step, and every rule below corresponds to a real-world failure.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm. I think we should link the website directly. But I'll need to have a thought on this.


For configuring and operating a server (registration, recording, welcome message, etc.), see the [Server manual](https://jamulus.io/wiki/Running-a-Server).

## Build for the target, not for the build machine

- **CPU architecture must match.** A binary built on x86-64 will not run on an ARM host (and vice versa) — the service crash-loops with `Exec format error`. Before copying, compare `file ./Jamulus` on the build machine with `uname -m` on the target.
- **Build on the oldest OS release you deploy to.** Binaries depend on the glibc/libstdc++ of the build machine. A binary built on Ubuntu 24.04 fails on 22.04 with `GLIBCXX_3.4.32 not found`, while a 22.04 build runs fine on 24.04 and later. Newer hosts run older binaries; the reverse never holds.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is clearly just targeted at AI.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's targeted at running a fleet, which AI makes much more realistic. some of my Linux instances are really tiny, and compiling on them is fraught with peril. So I've sorta taught it to build certain binaries in certain places and copy them along when done. All that version-specific gotcha stuff has been very useful in this. still seems like something a human would also want to be aware of.

- **Check shared libraries after every copy.** `ldd /path/to/jamulus | grep "not found"` must print nothing. This catches a missing Qt runtime package before systemd shows you a crash loop. The minimal headless runtime needs the Qt core, network, concurrent and xml libraries (see [COMPILING.md](../COMPILING.md)).
- **Low-memory hosts:** on machines with ≤ 1 GB RAM, build with `make -j1`, or build on a bigger machine of the same OS/architecture and copy the binary. If you must add temporary swap to survive a build, remove it afterwards — see below.

## Run under systemd

Use the unit shipped in [`linux/debian/jamulus-headless.service`](../linux/debian/jamulus-headless.service) as your starting point — it already encodes hard-won defaults (dedicated `jamulus` user, `Nice=-20`, real-time I/O scheduling, `MemorySwapMax=0`, `Restart=on-failure`). Manage the server only through `systemctl`; never kill the process by PID.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a README.md to linux/debian explaining this then (if it doesn't exist yet)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in #3799 — one new file, linux/debian/README.md.


## Host tuning

- **Enlarge the UDP receive buffer.** Under load, default kernel buffers drop packets, which musicians hear as dropouts. Set in `/etc/sysctl.d/99-jamulus.conf`:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sou you'd use an agent to deploy Jamulus to a machine automatically???

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. I run 81 servers on 26 instances. I paste in a single authentication secret and then never visit the Linux instance again.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TBH I'd rather use something deterministic like ansible for this.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TBH I'd rather use something deterministic like ansible for this.

Yes - or MicroK8s against a revision controlled deployment state. I would trust an AI Agent any more than I'd trust a human agent. You end believing both when they tell you something is working. If you have to fix it in Git to get it working, it tends to stay working or you know why due to the revision history.


```
net.core.rmem_max=4194304
net.core.rmem_default=4194304
```

- **No swap on a live server.** Swapping causes latency spikes audible to everyone connected. Keep swap off (the shipped systemd unit sets `MemorySwapMax=0` for the service; better still, don't enable swap on the host at all).
- **Don't compete with the audio process.** Never compile, or run other CPU-heavy work, on a host while musicians are connected — CPU contention causes dropouts just like network loss does.

## Firewall

- Inbound UDP on the server port (default 22224) must be open. Remember that cloud providers filter *in front of* the host (AWS security groups, OCI security lists) in addition to any host firewall, and some images run `firewalld` by default — you may need to open the port in two places.
- If you enable JSON-RPC (`--jsonrpcport`), never expose that TCP port to the internet. Bind it to localhost or firewall it to specific trusted addresses, and always use `--jsonrpcsecretfile`.
- Corollary: a TCP probe of a firewalled port from outside proves nothing about the service. Verify RPC from an allowed host, not from the internet.

## Verify every deploy

A deploy is finished when all of these pass on the target host — not when the file lands:

```bash
file /usr/bin/jamulus-headless # architecture matches `uname -m`
ldd /usr/bin/jamulus-headless | grep "not found" # no output
/usr/bin/jamulus-headless --version # the version you just built
systemctl status jamulus-headless # active (running)
```

Then check the restart counter is stable (`systemctl show -p NRestarts jamulus-headless`, again a minute later — a crash loop can look "active" in a single snapshot), and finally do an end-to-end test: if the server is registered with a directory, confirm it appears in the listing, and connect a client to confirm audio passes.

When a change "isn't working" on a server, check the binary's timestamp against the commit you think it contains *before* debugging — a stale binary explains most such mysteries.