Skip to content

feat: memory_base.eval.sandbox serves the real REST API on a throwaway Postgres - #390

Merged
yw0nam merged 4 commits into
mainfrom
feat/eval-sandbox
Oct 21, 2026
Merged

yw0nam merged 4 commits into
mainfrom
feat/eval-sandbox

Conversation

@yw0nam

@yw0nam yw0nam commented Oct 21, 2026

Copy link
Copy Markdown
Owner

Summary

uv run python -m memory_base.eval.sandbox serves the real REST API on a throwaway Postgres and prints its URL, a key, the author, and an example curl. A PR's runtime evidence then comes from real HTTP requests without touching the deployment and without a temporary test file.

main(wait)
  install _stop for SIGINT and SIGTERM          (first signal: ignore both, raise KeyboardInterrupt)
  with throwaway_postgres() as database:        (eval/longmemeval.py, db.Dockerfile on tmpfs)
      key = asyncio.run(_prepare(url))          (schema; keys.new_key("sandbox"); close_pool in finally)
      with EvalApi() as api:                    (eval/mcp_writer.py, uvicorn on a free loopback port)
          print URL / KEY / AUTHOR / curl … /notes
          wait()
  KeyboardInterrupt → "sandbox stopped", exit 0; previous handlers restored

The API subprocess inherits the environment: DB_URL points at the throwaway DB, and the embedder, reranker and chat settings come from .env. EvalApi.__enter__ now cleans up (terminates the subprocess, removes its temp dir) on any exception during boot, including a failed Popen and Ctrl-C; its timeout RuntimeError is unchanged.

Owner decisions: reuse the eval parts (throwaway_postgres, EvalApi, keys.new_key) rather than a new bash/compose path; the API uses the real .env services.

Review notes (GPT spec review ×3, two-axis review, a fix round, final two-axis review):

  • Applied: both signals installed explicitly (a background start inherits SIGINT as ignored), a second stop signal cannot abort cleanup, argv dropped, the README line moved into the Development code block.
  • Not changed: the boot sequence repeats mcp_writer.run's three lines rather than gaining a shared helper; _prepare_schema stays private.
  • Known limit: a stop during throwaway_postgres's own docker run can leave a container in the Created state (--rm never applies to a container that did not start). Remove it with docker rm -f $(docker ps -aq --filter label=memory-base-longmemeval).

Related issues

None.

Runtime evidence (required for ingest / retrieval / serving behavior)

Background starts from the worktree (implementer and spec reviewer, independently):

run /notes with key without key exit stderr labelled containers left
kill -INT 200 401 0 sandbox stopped none
kill -TERM 200 401 0 sandbox stopped none
INT twice, 1 s apart 200 401 0 sandbox stopped none
TERM twice — — 0 sandbox stopped none
INT during docker build — — 0 sandbox stopped none
INT during docker run — — 0 sandbox stopped one Created container (known limit above)

Printed by a run (the key lives only in that throwaway DB):

URL http://127.0.0.1:33799
KEY xuNyDjwvDj2CUm-8Gxv40zUv1G_D7J0al0W1sSWRUuU
AUTHOR claude-code
curl -s -H 'X-API-Key: xuNy…' -H 'X-Author: claude-code' http://127.0.0.1:33799/notes   → []

RED before each change: ImportError: cannot import name 'sandbox', assert [] == [True] (EvalApi boot interrupt); then AttributeError: … no attribute '_stop', assert not True (temp dir left after a failed Popen).

Verification

  • uv run pytest passes — tests/eval unit: 124 passed
  • uv run ruff format --check . and uv run ruff check . pass
  • docker compose build mcp passes (Dockerfile / dependency changes) — N/A
  • Runtime evidence attached above, or N/A justified

Checklist (core principles)

  • Selective storage — notes stay distilled; verbatim text lives only in doc_rows and the turns lane
  • Source-specific logic stays out of the source-agnostic core (ingest pipeline, retrieval, serving)
  • No hardcoded endpoints/models — configuration lives in .env
  • New/changed behavior ships a test in this PR (or skip-tests label justified)

@yw0nam
yw0nam merged commit 97acdc9 into main Oct 21, 2026
4 checks passed
@yw0nam
yw0nam deleted the feat/eval-sandbox branch October 21, 2026 06:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant