Repository navigation
venari: run self-hosted Convex beside Ghost - #173
Merged
Merged
Conversation
Contributor
⊕ Entity-level changesCLAUDE.md
tools/just/personal.just
tools/venari/README.md
tools/venari/convex/.env.example
tools/venari/convex/.gitignore
tools/venari/convex/docker-compose.yml
tools/venari/ghost/docker-compose.yml
Summary: 12 added, 6 modified, 1 deleted across 7 files functions and classes, not lines · sem |
alycda
added this pull request to stack #174
September 22, 2026 19:55
Ghost does not replace Convex, and the reverse is also true. Ghost hands out disposable Postgres databases, so it is the test database. Convex is an app backend: server functions, auth, and live queries that push to subscribers. It is what create-epoch-app is built on. This runs the open-source backend and dashboard on venari, the same way Soft Serve and Ghost run there: a /srv/<name>/ twin in this repo, loopback-only ports, and ssh -L from the laptop. Decisions: - SQLite in data/, not Postgres. Upstream's README recommends SQLite as the start. Putting it in the Ghost cluster would give Convex's data the lifecycle of a cluster that is disposable by design. A separate Postgres would cost RAM a 4 GB box does not have spare. The move later is export, set POSTGRES_URL, import. This differs from what I first suggested (the Ghost cluster, with its own role); I changed it after reading upstream's docs. - Both images are pinned to 0cf49cb, the revision :latest pointed at on 2026-09-22. Upstream does not guarantee that a mismatched backend and dashboard work together. - No .env and no agenix secret. The backend generates its instance secret into data/credentials/. Admin keys are signed with it, every generate call yields a different valid key, and each project keeps its own key in a gitignored .env.local, as upstream documents. - DISABLE_BEACON and NEXT_PUBLIC_LOAD_MONACO_INTERNALLY are set, so the images make no outbound calls. - The tunnel recipes put their forwards inline rather than use an ssh alias like venari-ghost, so nothing outside the repo has to exist first. Checked in a sandbox container, not on venari, against the pinned images: - A project deployed with a schema, a mutation, a query and a "use node" action. - A ConvexClient subscription saw 252 pushed updates, from 1 row to 501, across 500 concurrent mutations. - Peak memory was 150 MiB for the backend and 270 MiB for the dashboard. That sets the limits at 512m and 384m, so the box's container limits total about 2 GB. - After a restart the data was intact and the old admin key still worked; a bogus key was rejected (BadAdminKey). - The ports bind to 127.0.0.1 only. `docker compose config` validates, and personal.just parses (rust-just --list). Not checked: the ssh tunnel itself (the sandbox has no sshd), and anything on the box. Specific changes: - tools/venari/convex/: docker-compose.yml and .gitignore (twin of /srv/convex) - tools/venari/README.md: table row, a "Convex" section (storage, measurements, access, admin keys, the loopback-only limit, setup, upgrades), and Convex's data/ added to the backups gap - tools/just/personal.just: convex-tunnel, convex-tunnel-close, convex-admin-key - CLAUDE.md: venari's service list in the structure tree Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Tw8tuRUyKtkcjmN4upeawF
The first cut used SQLite, arguing that Convex's data should not share a lifecycle with a cluster built for disposable databases. Alyssa chose Postgres in Ghost's cluster instead: one Postgres on a 4 GB box, not two. This commit makes that safe and records what it cost. Isolation, from reading the fork (internal/server/store.go) and then testing it: - ghost-server only acts on databases in its own ghost.databases table. Sizes, forks, pauses, DROP and pg_terminate_backend all join on it, so `ghost list` never shows convex_self_hosted and nothing Ghost does can drop it. The storage limit is reported, not enforced. - Postgres grants CONNECT on new databases to PUBLIC. tsdbadmin is the shared role Ghost hands to agents, so CONNECT is revoked; tsdbadmin now gets "User does not have CONNECT privilege". Convex has its own role, so `ghost password` does not touch it. - Both projects join venari-postgres, an external network owned by neither. `docker compose down` in /srv/ghost then removes only ghost_default. TLS turned out to be the surprise. The backend verifies Postgres certificates, and Ghost's is self-signed (CaUsedAsEndEntity). DO_NOT_REQUIRE_SSL alone still attempts TLS when the server offers it; sslmode=disable alone is refused. It takes both, and the hop is plaintext (pg_stat_ssl.ssl = f). I accepted that: container to container on a private bridge, never off the host, and readable only by root, which can read .env anyway. Re-issuing Ghost's certificate from a CA the backend trusts is the alternative, and it is untested. A cost that remains: state is now in two places, the database in the cluster and files, modules and credentials in data/. Only `npx convex export` captures both at one moment. The Ghost section no longer calls the whole cluster disposable. Checked in a sandbox with ghost-server built from the fork and Ghost set up per its README: - Deploy, mutation, query and "use node" action all work. A subscription saw 271 pushed updates across 500 mutations. - ghost create/fork/delete/list were unaffected. - Outages: Ghost `down` and back up, and Convex never restarted. On a cold start with Convex first, it crash-looped 7 times, then recovered by itself. - Peak memory: backend 132 MiB, dashboard 88 MiB, ghost-postgres 213 MiB of its 768m. - The README's Convex setup block ran verbatim, starting from Ghost on its current compose (no shared network), and ended in "Connected to Postgres". Specific changes: - tools/venari/ghost/docker-compose.yml: postgres joins the external venari-postgres network - tools/venari/convex/docker-compose.yml: POSTGRES_URL to ghost-postgres as role convex, with the plaintext settings and the reasons for them - tools/venari/convex/.env.example and .gitignore: the role password - tools/venari/README.md: Convex section rewritten for the shared cluster (isolation, network, TLS, split state, measurements, setup including the role and REVOKE); the Ghost setup creates the network; Ghost's and the box's backup notes now name Convex's data Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Tw8tuRUyKtkcjmN4upeawF
Ghost's compose file declared venari-postgres as external, so Ghost depended on a network that exists only for Convex. Copy the updated twin to /srv/ghost on a box where nobody ran `docker network create`, and `just ghost-deploy` (`docker compose up -d server`) fails with "network venari-postgres declared as external, but could not be found", even with no Convex anywhere. Ghost now creates the network itself, with a fixed `name:` so Convex can refer to it. Convex keeps it external. That puts the dependency in the right direction: Ghost runs alone, and Convex needs Ghost up first, which it needs for its database anyway. What I was worried about, ownership tearing the network out from under Convex, does not happen (Docker Compose 2, 2026-10-16). `docker compose down` in the owning project while another project is attached prints "Network venari-postgres Resource is still in use", exits 0, and keeps the network; the attached container keeps running and still resolves the service by name after the owner comes back up. Only when both are down is the network removed. Checked with the real files: - (a) A fresh box, no network, ghost-deploy's `up -d server`: exit 0, network created. - (b) Ghost on main's compose file, then the README's Convex setup block run verbatim: Ghost and Convex both healthy, and "Connected to Postgres". Both manual `docker network create` steps are gone from the README. Found by /code-review (finding 7). Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Tw8tuRUyKtkcjmN4upeawF
Convex and every agent's Ghost databases share one cluster, and nothing
divided its connections. The reviewer assumed Postgres' default of 100;
the real limit is 25 (TimescaleDB tunes max_connections from the
container's 768m), with 3 reserved for superusers. The Convex
backend's pool defaults to 128 (POSTGRES_MAX_CONNECTIONS in
crates/common/src/knobs.rs at the pinned 0cf49cb). Either side could
exhaust the 22 shared slots ("sorry, too many clients already"),
failing agents' connections or Convex's mutations.
Two caps, one in each process:
- POSTGRES_MAX_CONNECTIONS=8 in Convex's compose. The backend logs
"Postgres connection pool max size 8".
- CONNECTION LIMIT 10 on the convex role, in the README's setup SQL.
Postgres enforces it whatever the knob says, with 2 slots of headroom
for reconnects. Tested: with 10 convex sessions open, the next login
gets FATAL: too many connections for role "convex".
That leaves about 12 slots for agents. ghost-server itself connects as
the superuser, so it can fall back on the reserved slots.
Is 8 enough? Under the 500-mutation burst plus a live subscription,
Convex peaked at 7 connections whether the pool was 128 or 8. The
burst took 5.5 s at 128 and 6.1 s at 8 in this sandbox, which is
within its noise. These are not load figures for venari.
Found by /code-review (finding 4).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Tw8tuRUyKtkcjmN4upeawF
The tunnel bound laptop ports 3210/3211, the same ports a local
`npx convex dev` deployment starts at (convex 1.46.0,
localDeployment/utils.js: startPort 3210). Two failures follow:
- Local backend running first: the tunnel dies on a bind error.
Reproduced against a stand-in sshd ("cannot listen to port", exit
255).
- Tunnel open first: the CLI quietly moves the local backend to the
next free port. A project whose .env.local says 127.0.0.1:3210 then
deploys to venari instead of the local backend, with no error.
Because CONVEX_CLOUD_ORIGIN hardcoded 127.0.0.1:3210, the tunnel could
not simply be moved.
The laptop ends are now the box ports plus 10000 (13210 -> 3210,
13211 -> 3211, 16791 -> 6791), the convention Ghost already uses
(15432 -> 5432). Changed together, because they must agree:
- CONVEX_CLOUD_ORIGIN, CONVEX_SITE_ORIGIN and
NEXT_PUBLIC_DEPLOYMENT_URL in compose
- the forwards in personal.just
- the README's tunnel line, dashboard URL and .env.local example
The box side is unchanged: 3210/3211/6791 on loopback.
Checked through a real ssh -L tunnel to a stand-in sshd, with the
backend on 3210 and the laptop ends on 13210/13211/16791:
- `npx convex deploy` over 13210 succeeds.
- An HTTP action answers on 13211.
- ctx.storage.getUrl returns http://127.0.0.1:13210/api/storage/...,
and that URL serves the file through the tunnel, so the advertised
origin is the one clients use.
- A live subscription pushed updates across 500 mutations, and the
dashboard loads on 16791.
Found by /code-review (finding 2).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Tw8tuRUyKtkcjmN4upeawF
personal.just reaches every profile, including the Linux devcontainer
(common.nix -> global.justfile), and the tunnel recipes broke there:
- `pgrep -qf`: procps-ng pgrep has no -q ("invalid option -- 'q'",
exit 2, checked on 4.0.4), so "already open" was never detected and
a second tunnel was always attempted.
- `pkill -f '<pattern>'`: the pattern is also in the argv of the
recipe's own `sh -c`, so close killed its own shell. Reproduced:
exit 144, and neither message printed.
The recipes no longer look for processes at all. convex-tunnel starts
ssh as a control master on ~/.ssh/venari-convex.sock; `ssh -O check`
answers "open?" and `ssh -O exit` closes it. That works the same on
macOS and Linux. Each recipe is now a bash script with
`set -euo pipefail`, the convention justfile already uses.
Checked against a stand-in sshd, on Linux:
- close with nothing open -> "no convex tunnel open", exit 0
- open -> open, exit 0; curl through 13210 -> 200
- open again -> "already open", exit 0
- close -> "closed"; close again -> "no tunnel", exit 0
- master killed with -9, socket left behind -> open recovers (the
stale socket is removed first) and the tunnel works
- 13210 taken by another listener -> exit 255 with ssh's bind error,
not a false "open"
Not checked on macOS here; nothing in it is platform-specific beyond
OpenSSH's control-socket options.
ghost-tunnel on main uses the same pgrep -q / pkill -f pattern and has
the same bugs. That recipe is outside this PR, so it is left alone.
Found by /code-review (finding 3).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Tw8tuRUyKtkcjmN4upeawF
`ssh ... | tail -1` without pipefail exits with tail's status, so the recipe always exited 0. Reproduced against a stand-in sshd: with the backend stopped, and with ssh unreachable, the old recipe printed an empty line and exited 0. That empty line, or an error message's last line, is what would be pasted into .env.local as CONVEX_SELF_HOSTED_ADMIN_KEY. The recipe is now a bash script with `set -euo pipefail`, and it also refuses any output not shaped like a key (<instance>|<hex>; the "Admin key:" banner goes to stderr, so stdout is only the key). Checked: - working: prints the key, exit 0 - backend stopped: "service "backend" is not running", exit 1 - ssh unreachable: "Connection refused", exit 255 - the shape check alone accepts a real key and rejects an error line, an empty string and a bare "<instance>|" The two failure runs used a copy of the recipe with /srv/convex pointed at the test deployment. Found by /code-review (finding 5). Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Tw8tuRUyKtkcjmN4upeawF
Three passages disagreed. The Ghost section says the cluster rides the provider's box backups. The Convex section said two separate copies of the cluster and data/ cannot be consistent. The backups gap said `npx convex export` is "the only backup". A whole-disk provider snapshot is neither separate copies nor nothing: it takes both halves at one moment. So a reader could skip box backups for Convex, or trust them without knowing what they do and do not give. The account now, the same in both Convex passages: - A whole-disk snapshot captures the cluster and data/ together, and is crash-consistent like a power cut, if the provider snapshots the disk atomically. Nobody has checked that, and the text says so. - `npx convex export` is consistent by construction and is the only copy that leaves the box. - Copying the halves separately (pg_dump, then data/) is what is not consistent. The planned restic setup must export, or copy with the backend stopped. Found by /code-review (finding 6). Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Tw8tuRUyKtkcjmN4upeawF
Two things the review grouped together: - personal.just said the forwards are inline "so nothing outside this repo has to exist first". That was false: the recipes ssh to the `venari` and `venari-root` aliases, which live only in ~/.ssh/config. On a machine without them they fail with "Could not resolve hostname venari". The comment now names both aliases and points to the README's "Access" section, which describes them. The true part stays: unlike Ghost, Convex needs no third alias. - The design reasoning was written out twice, in the compose header and in the README, and the two would drift on the next edit to either. The README's "Convex" section is now its only home; it already held every point. The compose file keeps where it lives and what it depends on, plus one line per setting saying what must stay true when editing (origins match the tunnel, plaintext needs both settings, pool below the role limit). 104 -> 75 lines. Comments only. `docker compose config` resolves to the same output before and after, and personal.just is unchanged apart from comment lines. Found by /code-review (finding 10). Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Tw8tuRUyKtkcjmN4upeawF
alycda
force-pushed
the
claude/convex-venari
branch
from
October 16, 2026 08:12
56c27b4 to
1e804a5
Compare
alycda
marked this pull request as ready for review
October 16, 2026 08:12
On macOS, -L 13210:... binds both 127.0.0.1 and [::1]. With another process on 127.0.0.1:13210, ssh still bound [::1], ExitOnForwardFailure counted that as success, and the recipe reported the tunnel open while CONVEX_SELF_HOSTED_URL (127.0.0.1:13210) reached the other process. Binding 127.0.0.1 explicitly makes the conflict fail (exit 255, no socket left), which is what the recipe's comment promised. Co-Authored-By: Claude Opus 5.5 <[email protected]>
The pinned images no longer pull from ghcr.io/get-convex: the registry answers "denied" even for :latest, and the package API says Not Found. The org was renamed to convex (github.com/get-convex/convex-backend now redirects to convex/convex-backend), and the images moved with it. The same 0cf49cb revision tags exist at ghcr.io/convex/, so the pin itself is unchanged. Upstream's self-hosted/docker/docker-compose.yml still names the old path, so following it verbatim fails the same way. The compose comment records that, so nobody "fixes" this back to match upstream. Found running the README setup on venari for the first time. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Ghost does not replace Convex, and the reverse is also true. Ghost hands out disposable Postgres databases, so it is the test database. Convex is an app backend: server functions, auth, and live queries that push to subscribers. It is what create-epoch-app is built on. This PR runs the open-source backend and dashboard on venari, like Soft Serve and Ghost:
/srv/<name>/twin in this repossh -Lfrom the laptopIt stores its data in Ghost's Postgres cluster, so the box runs one Postgres, not two.
What
tools/venari/convex/:docker-compose.yml(twin of/srv/convex/),.env.example(role password),.gitignoretools/venari/ghost/docker-compose.yml:postgresalso joinsvenari-postgres, which this project creates (fixedname:)tools/venari/README.md:tools/just/personal.just:convex-tunnel/convex-tunnel-close(an ssh control socket, so they work on macOS and Linux) andconvex-admin-key(fails instead of printing a non-key)CLAUDE.md: venari's service list in the structure treeNo Nix changes and no agenix secret.
Decisions to review
convex_self_hosted, and its own role,convex.CONNECTis revoked fromPUBLIC, sotsdbadmin, the shared role Ghost gives agents, cannot open the database.ghost-serveronly acts on databases in its ownghost.databasestable (checked in the fork'sinternal/server/store.go), so it cannot list, size or drop Convex's database.venari-postgres; Convex's declares it external. Ghost runs with or without Convex.docker compose downin/srv/ghostwhile Convex is attached keeps the network ("Resource is still in use", exit 0), and Convex keeps running.POSTGRES_MAX_CONNECTIONS, default 128), and its role hasCONNECTION LIMIT 10, leaving about 12 for agents. Convex used 7 under a 500-mutation burst.CaUsedAsEndEntity. The fix takes bothDO_NOT_REQUIRE_SSL=1and?sslmode=disable; either alone still fails. The hop never leaves the host. The alternative, not tested, is to re-issue Ghost's certificate from a CA the backend trusts.data/(files, modules, credentials). A whole-disk box snapshot takes both at once (crash-consistent, if the provider snapshots atomically, which is unchecked).npx convex exportis consistent by construction and is the only copy that leaves the box. Copying the two halves separately is not consistent.0cf49cb, the revision:latestpointed at on 2026-09-22, pulled fromghcr.io/convex/. The oldghcr.io/get-convex/path stopped working when the org was renamed: it answersdeniedeven for:latest. Upstream's own compose file still names the old path.DISABLE_BEACONandNEXT_PUBLIC_LOAD_MONACO_INTERNALLYare set.http://127.0.0.1:13210, the laptop's end of the tunnel (box port + 10000, like Ghost's 15432), because 3210/3211 are where a localnpx convex devdeployment starts. This serves one person over ssh. A frontend other people use needs a second firewall port and a TLS proxy. That is a separate decision.The first commit used SQLite. The second moves to the shared cluster, at the author's call.
Verification
In a sandbox, not on venari, with
ghost-serverbuilt from the fork and Ghost set up per its README:"use node"action (convex 1.46.0). A subscription saw 271 pushed updates across 500 concurrent mutations.tsdbadminis refused onconvex_self_hosted.ghost create,fork,deleteandlistare unaffected.downand back up, Convex never restarted and kept its data. On a cold start with Convex first, it crash-looped 7 times, then recovered by itself.ghost-postgrespeaked at 213 MiB of 768m. The box's container limits total about 2 GB of 4 GB.Connected to Postgres.Since the code review (10 findings, one commit each): the tunnel, admin key and deploy were run through a real
ssh -Lto a stand-in sshd on Linux. Storage URLs come back on 13210 and serve through the tunnel. The tunnel recipes handle open, re-open, close, a stale socket and a bind conflict. The README setup ran verbatim again with the final files.On venari (2026-10-16)
The README setup ran on the box, starting from Ghost on
main's compose file. It found one bug: the images had moved toghcr.io/convex/(510d04f). After that fix:ghost-postgresre-created ontovenari-postgres, so Ghost was down for a few seconds. The role and database were created, and the backend loggedConnected to Postgres database: convex-self-hosted.tsdbadminis refused onconvex_self_hosted("User does not have CONNECT privilege"). From the laptop,ghost listshows Ghost's 2 databases and not Convex's.convex-tunnelopens, a second call reports "already open", andconvex-tunnel-closecloses it.127.0.0.1only.convex-admin-keyreturns aconvex-self-hosted|<hex>key.docker compose restart backend.ghost-postgres67 MiB.Not verified:
restart: unless-stoppedacross compose projects)The smoke-test functions and their
pingstable (3 rows) are still deployed on venari.🤖 Generated with Claude Code
https://claude.ai/code/session_01Tw8tuRUyKtkcjmN4upeawF
Generated by Claude Code