feat(dimsim): add closed-loop agent evaluations

[michaellee8]

Contribution path

• Linked issue or discussion: QA Benchmark with in-house agent #3069

Related to #3069.

Problem

DimSim workflows could score browser state, but they could not run a
deterministic closed-loop evaluation against a live DimOS agent. There was no
correlated reset → dispatch → scoring lifecycle, authoritative server-physics
reset, bounded infrastructure failure handling, or task-scoped way to validate
an agent declaration.

This also allowed find-and-go-to-bathtub to produce false positives: the agent
could claim it found the bathtub without being physically close to it.

Solution

Add an explicit single-workflow agent mode:

dimsim eval apartment/find-and-go-to-bathtub --agent

The implementation:

• Adds a
  runEval → evalReady → evalReset → resetAck → agent_send → evalStart → evalResult
  protocol correlated by runId.
• Resets authoritative Rapier position, yaw, velocity, and command state before
  dispatch.
• Dispatches the workflow task exactly once through the existing MCP
  agent_send tool.
• Starts scoring only after reset and successful dispatch.
• Adds bounded timeouts and classifies task failures separately from
  infrastructure errors.
• Performs terminal cleanup by stopping exploration, perception watching, and
  navigation and by clearing bridge velocity.
• Adds JSON/JUnit reporting and exit codes for pass, task failure, and
  infrastructure failure.
• Adds apartment bathtub workflows, including an outside-start discovery task.
• Adds a read-only /agent output sidecar for the discovery workflow.
• Requires find-and-go-to-bathtub to satisfy both:
  • a tool-free assistant response exactly equal to FOUND_BATHTUB
  • a final distance of at most 1 metre from the bathtub
• Keeps model credentials and model configuration entirely in DimOS.
• Adds Deno, browser-harness, bridge-physics, reporting, rubric, and Python
  sidecar tests.
• Runs the DimSim Deno suite in CI.

The branch also improves the live DimSim runtime by:

• forwarding and deduplicating odometry, LiDAR, and video through
  DimSimConnection
• rejecting low-similarity semantic-navigation matches
• keeping the visual-language model alive across perception-watcher cycles

No shared system prompt is changed. The FOUND_BATHTUB instruction belongs only
to the relevant workflow.

How to Test

Start the live stack:

dimos --transport lcm --simulation dimsim \
  --dimsim-scene apartment \
  --dimsim-port 8092 \
  --no-dimsim-headless \
  --viewer none \
  run unitree-go2-agentic

Then run the evaluation:

deno run -A --unstable-net misc/DimSim/cli/cli.ts \
  eval apartment/find-and-go-to-bathtub \
  --agent --port 8092

Automated validation:

cd misc/DimSim
deno check --config cli/deno.json cli/cli.ts
deno test -A --unstable-net --config cli/deno.json cli evals

cd ../..
uv run pytest dimos/simulation/dimsim/test_agent_output_sidecar.py -q

Results:

• DimSim Deno suite: 40 passed
• Python sidecar tests: 3 passed
• Deno type-check: passed
• Live agent smoke test: passed in 118.4 seconds
• Final bathtub distance: 0.961 metres
• Exact FOUND_BATHTUB declaration captured
• Terminal cleanup confirmed

AI assistance

OpenAI Codex with GPT-5.6-Sol was used extensively to inspect the codebase, implement
the feature, write tests and documentation, operate the live stack, and analyze
evaluation behavior. I made the technical decisions, reviewed the resulting behavior and live
evaluation outcome.

Checklist

• I have read and approved the
  CLA.

[michaellee8]

Screen.Recording.2026-07-28.at.8.47.18.PM.mp4

Demo screen recording demostrating navigation from outside the house to the bathtub

[michaellee8]

Ran all 5 scenarios and passed including one of them doing a fresh longer navigation task from outside the house to the innermost bathtub.