feat(dev): run agentex locally without Docker #353
Conversation
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Stand up the full backend as host processes with embedded datastores — no Docker daemon required — as a lighter alternative to the container stack. `./dev.sh local` (also `make dev-local` / `python -m scripts.dev_local`) provisions: - Postgres via bundled pgserver (unix socket) and Redis via bundled redislite - a Temporal dev server + UI and the agentex worker (--no-temporal to skip) - a local mongod, required for the full stack (--no-mongo / --lean to skip) - an optional OpenTelemetry collector (--no-otel to skip) then runs migrations, supervises uvicorn + the worker, and tears everything down cleanly on SIGINT/SIGTERM. --lean is a minimal Postgres+Redis+API stack; --ephemeral uses a throwaway data dir. The runner is a small scripts/dev_local package (config / services / supervise / runner) so the pure config/env layer stays testable. App-side changes to make the no-Docker path robust (all no-ops when Mongo is configured, as it always is in Docker/prod): - The Temporal worker no longer crashes when MongoDB is unavailable — the Mongo CRUD adapter tolerates an unset database and errors only on real use, so the worker degrades like the API instead of taking down the stack. - Skip the Mongo connection entirely when MONGODB_URI is unset, removing a ~20s startup hang against the implicit localhost:27017 default. - In local mode the backend rewrites agents' host.docker.internal ACP host to loopback (AGENTEX_ACP_HOST_OVERRIDE), so default-scaffolded agents work without manifest edits. Also fix a frontend dev-server process leak in dev.sh (kill the whole make→npm→next tree and sweep orphans; report status by listening port), correct the MongoDB and OpenTelemetry install commands, and document local mode in README and CLAUDE.md.
356b3eb to
aa8bb61
Compare
| raw = acp_url_override | ||
|
|
||
| # Prefer the production deployment's URL when there's no explicit override. | ||
| if raw is None and agent.production_deployment_id: |
There was a problem hiding this comment.
@smoreinis can you take a look here just make sure it doesn't conflict with the preview workflow?
| """ | ||
| # In docker-free local mode, rewrite host.docker.internal -> the host-reachable | ||
| # override so the healthcheck matches how the request path dials the agent. | ||
| acp_url = resolve_acp_url(acp_url) |
There was a problem hiding this comment.
just making sure, no other places we need to do conversion right?
| if acp_url_override: | ||
| return acp_url_override | ||
| """Resolve the ACP URL for an agent, optionally overriding with a specific URL. | ||
|
|
There was a problem hiding this comment.
did we lose the override?
There was a problem hiding this comment.
hi, so ./dev.sh no-docker will pass _ACP_HOST_OVERRIDE_ENV variable, so the function resolve_acp_url will resolve this. Besides this, the logic of this function is kept the same
| Returns the URL unchanged when the override env var is unset or the URL does | ||
| not use the Docker sentinel host, so it is safe to call on every ACP dial. | ||
| """ | ||
| override = os.environ.get(_ACP_HOST_OVERRIDE_ENV) |
There was a problem hiding this comment.
is this where the override moved?
There was a problem hiding this comment.
yes, here we override the environment variable _ACP_HOST_OVERRIDE_ENV. This environment is set and passed from config.py file
| # Development Server | ||
| # | ||
|
|
||
| dev: install-dev ## Start development server with Docker Compose |
There was a problem hiding this comment.
these are both technically local maybe rename to something more meaningful?
|
|
||
| > **MongoDB is required for the full local stack** and is always started — the Temporal | ||
| > worker builds Mongo-backed repositories at startup, so a missing/unreachable Mongo | ||
| > makes the runner fail fast (with an install message) rather than crash the worker. |
danielmillerp
left a comment
danielmillerp
left a comment
There was a problem hiding this comment.
lots of small questions! also as far as testing on PC, not super sure. A lot of our clients do use PCs and I know there are Scaliens who have PCs. What problems do you anticipate?
There was a problem hiding this comment.
Overall this is rad.
One UX suggestion: in addition dev.sh, I think this would be really nice to have in the agentex CLI. I.E. you can run a single agent inline. therefore to start running a single agent you can start quick rather than having both the agentex backend and cli running. then this setup would be suitable for someone testing a multiagent flow.
Not sure if that is possible given the abstraction and integration, but wanted to throw it out there as a north star.
+1 to that! |
@danielmillerp So Im testing this on Windows (via AWS Workspaces) but the embedded Redis uses (the redislite package has no native Windows build (Redis ships no supported Windows server) so the dependency required to run locally without Docker won't even install on native Windows. |
Some findings from testing local dev on Windows:
@NiteshDhanpal had a clear view on how to scope this. His take: don't treat Windows-native local mode as a hard requirement for now (given the redislite/Redis limitation), and use a support matrix instead:
His main concern is a reliable fallback so customers aren't blocked when local setup is painful or impossible, and he thinks a cloud/dev-environment fallback is the most reliable option for that. |
I'm aligned @levilentz does that make sense for customers? Also CC: @lucyakoroleva for context per today's standup discussion |
@danielmillerp @aringuyen3 I think broadly this is a good first step. I think assuming that the customer will not have WSL or docker is the way to test this as that it normally the problem. can we just disable redis in this mode? |
I like that! not necessary to go from 0 to 1 to stream |
3 participants
What
Adds a docker-free mode that runs the full agentex backend as host processes with embedded datastores — a lighter alternative to the Docker Compose stack, closer to a one-command
langgraph dev-style workflow.The bare
./dev.sh(Docker) is unchanged and now also accepts an explicit./dev.sh dockeralias. The docker-free mode is also available asmake dev-no-dockerandpython -m scripts.dev_nodocker.Why
Standing up a local environment previously required the full Docker stack. This lets a developer run the backend with a single command and no Docker daemon:
pgserver(unix socket) and Redis via bundledredislitemongod— always started; the stack requires it (the Temporal worker builds Mongo-backed repositories at boot)It runs migrations, supervises uvicorn + the worker, and tears everything down cleanly on Ctrl-C / SIGTERM.
--ephemeraluses a throwaway data dir;--mongo-uripoints at an existing MongoDB instead of launching a localmongod.App-side change
Safe no-op wherever the backend runs in Docker / staging / prod (the env var it keys on is unset there):
ACP host rewrite for docker-free mode. Agents register their ACP URL at
host.docker.internal(the SDK default, so a Docker backend can reach an agent on the host), which a host-process backend can't resolve. The runner setsAGENTEX_ACP_HOST_OVERRIDE=127.0.0.1and the backend rewrites only that sentinel host to the override when dialing agents — in the ACP request path, the agent-API-key proxy path, and the Temporal healthcheck. When the env var is unset, the stored URL is used verbatim. Default-scaffolded agents work without manifest edits.Also
dev.sh(kill the wholemake → npm → nexttree and sweep orphans; report status by listening port).brew trustfirst) and OpenTelemetry (release binary; not in Homebrew) install commands.dev-no-dockeruv dependency group (pgserver,redislite,greenlet) pulled only for docker-free mode.README.mdandCLAUDE.md.Contract note
The runner is a package (
scripts/dev_nodocker/), so the direct invocation ispython -m scripts.dev_nodocker(notpython scripts/dev_nodocker.py)../dev.sh no-dockerandmake dev-no-dockerare unchanged for callers.Testing
Manually exercised on macOS (Docker stopped):
--lean— the API passes/healthzwithin ~1s and/readyzreports Postgres, Redis, and MongoDB all healthy; the Temporal worker stays up; teardown frees all ports with no orphaned processes.mongodabsent aborts with an actionable install message (or point at an external instance with--mongo-uri).python -m scripts.dev_nodocker; agents scaffolded byagentex initconnect without manifest edits (ACP host rewrite).Platform support: macOS and Linux only. Not supported on native Windows — the embedded Redis (
redislite) ships no Windows server build; run it under WSL2, whereredislite/pgserver/mongodbehave as on Linux (WSL2 path not yet verified).Greptile Summary
This PR adds a docker-free local development mode (
./dev.sh no-docker) that runs the full agentex backend as host processes with embedded Postgres (pgserver), Redis (redislite), a Temporal dev server, a localmongod, and an optional OTel collector — no Docker daemon required. A companionresolve_acp_urlutility rewriteshost.docker.internalACP URLs to127.0.0.1whenAGENTEX_ACP_HOST_OVERRIDEis set (a no-op in Docker/staging/prod), applied consistently across the ACP request path, the agent-API-key proxy, and the Temporal healthcheck activity.scripts/dev_nodockerpackage: pureconfig.py, provisioning inservices.py, subprocess plumbing insupervise.py, and orchestration inrunner.py; returns exit code 1 on unexpected process crash and 0 on clean Ctrl-C/SIGTERM.dev.shextended: addsno-dockersubcommand,kill_treefor cleaner frontend teardown, mode-file tracking sostop/status/restartwork correctly regardless of which mode was last used, and a best-effortotelcol-contribrelease-binary installer.resolve_acp_urlis inserted at all three agent-dial sites; the env var is unset everywhere except the no-docker runner subprocess environment, making the change a true no-op on Docker/prod paths.Confidence Score: 5/5
Safe to merge — the app-side changes are a genuine no-op in Docker/staging/prod and the new dev tooling is well-isolated.
The resolve_acp_url rewrite only activates when AGENTEX_ACP_HOST_OVERRIDE is set, which happens exclusively inside the no-docker runner subprocess environment. All three ACP dial sites are covered consistently. The runner returns the correct non-zero exit code on unexpected crashes and tears down embedded datastores cleanly. The remaining findings are minor style and dev-ergonomics nits that do not affect runtime correctness.
No files require special attention beyond the noted nits in .gitignore (dead path entry), Makefile (bare TODO), supervise.py (fire-and-forget task reference), and acp_url.py (INFO vs DEBUG log level).
Important Files Changed
Sequence Diagram
sequenceDiagram participant devsh as dev.sh no-docker participant runner as scripts.dev_nodocker.runner participant svc as services.py participant sup as supervise.py participant api as uvicorn (API) participant worker as Temporal worker participant agent as Agent process devsh->>runner: uv run python -m scripts.dev_nodocker runner->>svc: provision_postgres() runner->>svc: provision_redis() runner->>svc: provision_mongo() runner->>svc: provision_temporal() runner->>sup: run_migrations() runner->>sup: spawn(api, uvicorn) activate api runner->>sup: spawn(worker, run_worker.py) activate worker runner->>sup: wait_for_health(:5003/healthz) sup-->>runner: healthy Note over api,agent: ACP host rewrite active (AGENTEX_ACP_HOST_OVERRIDE=127.0.0.1) api->>agent: resolve_acp_url(host.docker.internal:port) to 127.0.0.1:port worker->>agent: healthcheck via resolve_acp_url() devsh->>runner: SIGTERM (Ctrl-C or dev.sh stop) runner->>sup: terminate(worker) deactivate worker runner->>sup: terminate(api) deactivate api runner->>svc: temporal.shutdown() runner->>svc: teardown_redis() runner->>svc: pg_server.cleanup()Reviews (6): Last reviewed commit: "fix commnet" | Re-trigger Greptile
Context used:
Learned From
scaleapi/scaleapi#126926