Agent Framework Load Benchmark

Benchmarks AI agent frameworks against a plain-Python baseline under a deterministic fake LLM, so the results measure framework CPU overhead only.

Status: scaffolded, not yet run. Every component and all six adapters are written, but the services, the framework adapters and the SLA benchmark have not been run against real installs. This repository therefore contains no benchmark results. See Status and limits.

The question

On a single CPU core, how many concurrent users can each framework serve before it breaks the SLA, and how does that compare with a hand-written baseline?

Teams usually compare agent frameworks on features (tool-calling syntax, memory, graph versus agent abstraction) and rarely on runtime cost. Two frameworks that look identical in a demo can behave very differently under load: some keep more state per conversation, some re-validate and re-serialise messages, some add telemetry that costs CPU, some use heavier async abstractions. This benchmark makes that difference visible before a framework choice is locked in.

How the noise is removed

Source of noise	Replaced by
Real LLM	a fake OpenAI-compatible server with fixed delays: 3 x 250 ms of "thinking", then exactly 100 tokens streamed at 50 tokens per second, so an idle conversation takes about 2.8 s
Real database or API	two mock tools (get_order_details, check_return_policy) with a fixed 20 ms delay each
Multi-core scaling	the server under test runs one worker on one CPU core (cpus: "1.0" in docker-compose.yml)
Tracing, caching, logging	switched off in every adapter

Whatever latency appears above the 2.8 s floor as load rises is queuing plus framework overhead: event-loop scheduling, validation, state management, object allocation.

Architecture

flowchart LR
    LT[Locust<br/>virtual users + correctness checks] -->|POST /chat/framework| AS
    subgraph AS[Agent server · FastAPI · 1 worker · 1 CPU]
      direction TB
      R[reference<br/>plain Python]
      A1[openai_agents]
      A2[msaf<br/>Microsoft Agent Framework]
      A3[pydantic_ai]
      A4[langgraph]
      A5[google_adk<br/>via LiteLLM]
    end
    AS -->|OpenAI-compatible<br/>chat completions| FL[Fake LLM server<br/>250 ms x 3, then 100 tokens at 50/s]
    AS -->|direct calls| MT[Mock tools<br/>20 ms each]
    SLA[sla_benchmark.py<br/>ramp-up and grading] -->|drives| LT

Loading

Each conversation is 3 model calls and 2 tool calls. Every adapter maps its framework's native stream onto the same server-sent events (token events, then a final done event with the refund amount and the call counts), so the load generator treats all frameworks identically.

Component	Where
Fake LLM server	fake_llm/server.py
Agent server and adapters	agent_server/main.py, agent_server/adapters/
Shared mock business logic	common/
Load script and SLA automation	load_test/locustfile.py, load_test/sla_benchmark.py
Containers	docker/, docker-compose.yml

What is measured

Metric	Meaning
P95 and P99 latency	how long 95% and 99% of conversations take
Error rate	crashes, timeouts and dropped connections
Correctness	the exact refund text, exactly 100 tokens, exactly 3 model calls and 2 tool calls. A fast but wrong answer does not count
Max sustainable concurrent users	the highest load that passes everything above; the number used to rank frameworks against the plain-Python ceiling

The SLA rule

The benchmark starts at 10 concurrent users and doubles (20, 40, 80, 160, ...) until a violation. A load level passes only if three separate 10-minute runs all hold:

• P95 latency under 8 s and P99 under 12 s
• error rate under 1%
• at least 99% of completed conversations pass the correctness checks
• no crash, out-of-memory or sign of a memory leak

Each virtual user pauses 3 to 10 seconds between conversations.

Frameworks compared

Plain Python (the performance ceiling), OpenAI Agents SDK, Microsoft Agent Framework, Pydantic AI, LangGraph, and Google ADK (routed through LiteLLM). A longer discussion is in BENCHMARK-OVERVIEW.md; the original requirements are in agent-benchmark-spec.md.

Quickstart (not yet run)

python -m venv .venv && source .venv/bin/activate        # Windows: .venv\Scripts\Activate.ps1
pip install -r requirements.txt

uvicorn fake_llm.server:app --port 9000                   # terminal 1
FAKE_LLM_URL=http://localhost:9000 uvicorn agent_server.main:app --port 8000 --workers 1   # terminal 2
locust -f load_test/locustfile.py --host http://localhost:8000                              # terminal 3

Or with Docker:

docker compose up --build
docker compose --profile loadtest run load_test

Run the full SLA ramp and grading for one framework (this takes hours, because every load level needs three 10-minute runs):

python load_test/sla_benchmark.py --host http://localhost:8000 --framework reference

Only reference has no extra dependency. The other adapters need their SDKs from requirements/agent_server.txt. Each adapter is imported lazily, so the server still starts without them and only that framework's endpoint returns 503.

Results

None yet. After a real run, record the maximum sustainable users per framework, with the machine, the CPU model and the exact SDK versions.

Status and limits

• Not run: the Docker stack, the framework adapters against their real SDKs, the Locust script and the SLA benchmark.
• Not verified: the SDK version ranges in requirements/agent_server.txt, and the package names of the newer frameworks.
• Scope: a fake LLM isolates framework overhead but does not predict real-world latency. The workload is one fixed, tool-calling conversation on one core, so it says little about other workloads.
• Opinions, not measurements: the pros and cons of each framework in BENCHMARK-OVERVIEW.md are written judgements.
• Origin: the project was built from the written requirements in agent-benchmark-spec.md.

License

MIT