[rennf93]

Follow-up to trail #32 (change 118): give each event loop its own database engines and RAG service, closed by the loop that opened them, and make pytest fail on ResourceWarning too. This closes the leaks #1264 left open (it landed through train 4, #1265), and a journal search bug that one of them was hiding.

Why

#1264 left ResourceWarning exempt from filterwarnings = ["error"]. The e2e smoke still had 31 unclosed connections. The full suite had another leak: 18 unraisable ResourceWarnings, which failed whichever later test the garbage collector happened to run in.

Both came from process-wide async resources in a process that runs several event loops: each test's loop, each asyncio.run, the e2e harness's uvicorn threads, the eval bench's servers. asyncpg connections belong to the loop that opened them.

• DB engines. One engine slot per pool was shared by every loop. A loop that found another loop's engine disposed it, while that loop's fire-and-forget hooks still held its connections. Engines on loops that had ended were never disposed: run_db's asyncio.run, the eval bench's servers.
• The RAG service. OptimalService was one per process. tests/integration/test_journal_routes.py::test_search_my_entries_returns_list built the real one on its own loop, and nothing closed it. About 900 files later, test_optimal_grounding.py closed it from another loop. Its vector-store pools couldn't close their sockets there, and VectorStore.close swallowed the Event loop is closed error on the grounds that "the connections died with the loop". They hadn't: they stayed open until garbage collection. These are the 18 warnings; the 15 that pytest printed were 9 sockets and 6 transports to Postgres.
• The one-shot CLIs. python -m roboco.cli (with or without --db-only), python -m roboco.vault rebuild and the decisions export run their work under their own asyncio.run, and never closed the engine it opened. The connections died with the process, without a Terminate to Postgres.

Rewriting the journal route test exposed a production bug. JournalService.search_entries filtered RAG matches through get_journal(agent_id), which looks a journal up by its own id, not its agent's. It never found one, so journal search always returned an empty list. The route test accepted a 500 the route can't return. One service test patched get_journal, and the other documented the "near-miss in production" and asserted only that a list came back.

What changed

• roboco/db/base.py (commit 1):
  • _PoolEngines keeps one engine per event loop, for each of the primary and background pools. A loop never takes over another's engine.
  • close_db closes only the running loop's engines. An engine built with no loop running is claimed by the first loop to use it.
  • _DbHolder, _BackgroundDbHolder, _rebind_holder_to_current_loop and _dispose_on_its_loop are gone.
• The one-shot CLIs (commit 3): a new run_then_close_db awaits the work, then close_db, before the loop ends. roboco/cli.py, roboco/vault.py and roboco/services/decisions/export.py wrap their asyncio.run work in it.
• roboco/services/optimal.py (commit 6): OptimalService is one per event loop (_OptimalServiceHolder), and so is its init lock. close_optimal_service closes only the running loop's. Production runs one loop, so it still has one service.
• VectorStore.close no longer swallows Event loop is closed. A close can only reach another loop's pool through a missing close on the right one, and that should fail loudly.
• close_shared_embedder closes the embedder's sync HTTP client; before, it only dropped the reference.
• JournalService.search_entries (commit 7) uses get_journal_by_agent.
• Each loop's owner closes its engines and RAG service before the loop ends:
  • production: the app lifespan, which already did;
  • the one-shot CLIs: run_then_close_db;
  • each test's loop: the root conftest's _close_this_loops_db_and_rag;
  • the e2e app: a new lifespan, _close_db_and_rag_on_shutdown;
  • the e2e run_db: run_then_close_db.
• A guard in the root conftest, _nothing_left_open_on_an_ended_loop, fails any test that leaves an engine or RAG service on a loop that has ended, and names it. Without it, such a leak would be silent: the per-loop registries keep the objects alive, so their connections never reach the garbage collector and never warn.
• The guard's first full run found two test files reaching the real database by accident. Each opened the global engine on TestClient's loop, which ends with the request and never closes it:
  • test_middleware.py: its flow-verb requests wrote their latency sample through the real engine. A module fixture skips that write; test_verb_latency_telemetry.py covers the write itself.
  • test_i_will_plan_rich_required.py: its first test used the real choreographer dependency, which opens a database session even for a request its validation rejects. Its app now always gets a mocked one.
• In the e2e smoke, the guard found a leak the earlier "0 unclosed" run had missed, because the registry kept it alive. test_video_pipeline.py runs the real render step under its own asyncio.run, which opened the background pool and left a live connection on it. It now runs through run_then_close_db.
• pyproject.toml: filterwarnings = ["error"]. ResourceWarning is no longer exempt.
• The journal route test now stubs RAG and inserts its entry directly: the create route would start indexing it into the real RAG.
• Pre-existing problems fixed in their own commits: trailing comments the conventions check flagged in the touched files, lifted (commits 2, 4 and 8). Every file's AST is identical.
• Commit 9 corrects the leak count in the map's note to the one above.
• Commit 5 renames the new CLI test to test_cli_entrypoint.py. As test_cli.py it shared a basename with tests/unit/conventions/test_cli.py, which stops collection, since neither directory is a package. My targeted runs never collected both; the full suite did.
• Map: rows for _PoolEngines, get_engine, close_db, run_then_close_db, get_optimal_service, close_optimal_service, search_entries and the two conftest fixtures, with dated notes. The stale "event-loop-bound singleton" note is rewritten.

Verification

• How the leaks were found:
  • e2e: PYTHONASYNCIODEBUG=1 makes each unclosed asyncpg connection report where it was opened.
  • Full suite: a garbage collection after every test file pinned the leak to test_optimal_grounding.py, and a detector watching the RAG holder named the test that built the service.
  • Then the guard: its first full runs found the three test files above, six tests in all. All are fixed.
• Tests, run serially as CI runs them, with filterwarnings = ["error"] and no exemption:
  • full suite: 19,492 passed, 560 skipped, 0 warnings, coverage 91.22%. Before this branch, the same setting failed it with 18 unraisable ResourceWarnings;
  • e2e smoke: 76 passed, 6 skipped, 0 warnings, nothing left open. Before: 31 unclosed connections;
  • both ran on a tree byte-identical to commit 7. Commit 8 only moves comments, with identical ASTs, and commit 9 changes only the map.
  • The in-image suites' files, with Postgres pointed at a closed port as that job runs: 33 passed, 50 skipped (they skip on this host), no notice. The job itself needs Docker, which my environment can't reach; its CI run is the check.
• These new tests fail on the old code:
  • the four CLI entry-point tests;
  • the two per-loop RAG tests;
  • VectorStore.close raising on an ended loop;
  • the embedder's client being closed;
  • journal search returning the agent's own match, through the route and the service.
• Static checks: clean: ruff format and check, mypy roboco/ tests/, mypy roboco/runtime/orchestrator.py, vulture, xenon, bandit at CI's threshold, import-linter. The codebase map is 0 errors and 0 warnings, with the concat in sync and reflow clean. Conventions has no findings in the touched files except three god classes, JournalService, NotificationService and OptimalService. They predate this branch and belong to later waves of the god-class plan.

🤖 Generated with Claude Code

[github-actions]

Thanks for opening your first pull request on RoboCo!

Quick checklist before review (most of these are enforced by CI, but worth a glance):

• make quality — ruff format check, ruff check, mypy, pytest (≥80% coverage), and the rest of the gate
• Panel changes pass pnpm lint and pnpm exec tsc --noEmit (run from panel/)
• No # noqa / # type: ignore shortcuts; pre-existing violations in touched files are fixed
• Added an entry under ## [Unreleased] in CHANGELOG.md
• Signed the CLA (the bot will prompt you on this PR)
• Signed your commits — master requires verified signatures (SSH signing setup)
• Updated any affected docs under docs/

See CONTRIBUTING.md for the full workflow and the Code of Conduct for the community standards we follow.

Welcome aboard — a maintainer will review shortly.