Documentation | Quickstart | PyPI | Issues
SMOLTRACE is a benchmarking and evaluation framework for Smolagents, Hugging Face's lightweight agent library. It runs ToolCallingAgent and CodeAgent against curated or custom task datasets, auto-instruments everything with OpenTelemetry, and publishes results, traces, metrics, and a leaderboard — with zero configuration.
pip install smoltrace
export HF_TOKEN=hf_your_token
export OPENAI_API_KEY=sk-your_key # or MISTRAL_API_KEY, ANTHROPIC_API_KEY, ...
smoltrace-eval --model openai/gpt-4.1-nano --provider litellm --agent-type both --enable-otelThat's it. One command evaluates both agent types and automatically creates 4 datasets on the Hugging Face Hub:
{username}/smoltrace-results-{timestamp}— per-test-case outcomes{username}/smoltrace-traces-{timestamp}— OpenTelemetry spans{username}/smoltrace-metrics-{timestamp}— GPU / CO2 / cost time-series{username}/smoltrace-leaderboard— aggregate stats, updated every run
No repo names to configure — everything is derived from your HF_TOKEN.
- Zero configuration — only
HF_TOKENrequired; dataset names are auto-generated from your username - Multi-provider — LiteLLM (OpenAI, Anthropic, Mistral, Groq, ...), HF Inference API, local Transformers models on GPU, and Ollama
- OTEL native — auto-instrumented via genai-otel-instrument: spans, token counts, cost tracking, CO2 emissions
- GPU metrics — utilization, memory, temperature, and power tracked by default for local models, flattened for dashboards
- 20+ agent tools — web search, webpage reading, file system, text processing (
grep/sed/sort), and process/system tools (ps/curl/ping), with working-directory sandboxing - 3 benchmark datasets — quick tasks (13), comprehensive benchmark (132: GAIA/Math/SimpleQA), and an ops/SRE benchmark (24)
- Flexible output — push to HF Hub, save local JSON, or export to OpenSearch with typed mappings
- Parallel execution —
--parallel-workers 8for 10-50x faster API-model evaluations - Dataset lifecycle —
smoltrace-copy-datasetsto bootstrap,smoltrace-cleanupto safely prune old runs - HF Jobs ready — run on Hugging Face cloud hardware (CPU to A100) with pay-as-you-go billing
pip install smoltrace # core
pip install smoltrace[gpu] # + GPU metrics for local models
pip install smoltrace[opensearch] # + OpenSearch exportRequires Python 3.11+. For development installs and full requirements, see the installation guide.
# API model via LiteLLM (default provider)
smoltrace-eval --model openai/gpt-4 --provider litellm --agent-type tool
# Local GPU model via Transformers (GPU metrics on by default)
smoltrace-eval --model meta-llama/Llama-3.1-8B --provider transformers --agent-type both
# Local model via Ollama
smoltrace-eval --model qwen2.5-coder:3b --provider ollama --agent-type tool --enable-otel
# HF Inference API
smoltrace-eval --model meta-llama/Llama-3.1-70B-Instruct --provider inference --agent-type both
# Enable agent tools, filter by difficulty, run in parallel
smoltrace-eval \
--model openai/gpt-4.1-nano \
--provider litellm \
--enable-tools visit_webpage read_file grep \
--working-directory ./workspace \
--difficulty easy \
--parallel-workers 8 \
--enable-otelGeneration settings (sent to the model on every call; unset = the provider's default): --temperature, --top-p, --top-k, --max-new-tokens, --reasoning-effort none|minimal|low|medium|high, --enable-thinking true|false. A setting the provider cannot apply is printed as [GENERATION] not applied for <provider>: ... and recorded in the leaderboard row's generation_settings; it is never dropped silently. --enable-thinking is the chat-template switch of open-weight models (OpenAI-compatible endpoints, Ollama, transformers); OpenAI, Anthropic and Gemini natives use --reasoning-effort.
Web search: --enable-tools google_search --search-provider duckduckgo|serper|brave|tavily (SERPER_API_KEY, BRAVE_API_KEY, TAVILY_API_KEY; duckduckgo needs none). A provider without its key stops the run instead of running without search.
Other useful flags: --dataset-name (custom task dataset), --model-args key=value (variables handed to the agent, not to the model), --prompt-yml (custom prompts: a smolagents prompt-template file, e.g. prompt_template.tool.yaml, merged over the defaults; a lone plain system_prompt is prepended to the default; a missing or unparseable file stops the run rather than falling back to the default prompts), --mcp-server-url (MCP tools), --private (private datasets), --output-format hub|json|opensearch.
Full flag reference: CLI documentation.
import os
from smoltrace.core import run_evaluation
all_results, trace_data, metric_data, dataset_used, run_id = run_evaluation(
model="openai/gpt-4",
provider="litellm",
agent_type="both",
enable_otel=True,
hf_token=os.getenv("HF_TOKEN"),
)Lower-level building blocks (generate_dataset_names, push_results_to_hf, compute_leaderboard_row, update_leaderboard, ...) are documented in the Python API reference.
| Dataset | Size | Purpose |
|---|---|---|
kshitijthakkar/smoltrace-tasks (default) |
13 tasks | Quick validation and development |
kshitijthakkar/smoltrace-benchmark-v1 |
132 tasks | Comprehensive evaluation (GAIA, Math, SimpleQA) |
kshitijthakkar/smoltrace-ops-benchmark |
24 tasks | APM / AIOps / SRE / DevOps scenarios |
Copy them to your own account with smoltrace-copy-datasets, or bring your own tasks as a HF dataset or local JSON. Schemas, the ops-benchmark sample-data setup, and custom task creation are covered in the datasets guide.
Community task datasets: the TraceMind-AI collection bundles 40+ ready-to-run task datasets spanning real-world domains — travel, e-commerce, healthcare, finance, DevOps/SRE, AIOps, cybersecurity, and more — for evaluating agents on domain-specific workloads. Point --dataset-name at any of them, e.g. --dataset-name MCP-1st-Birthday/smoltrace-finance-tasks.
--output-format hub # default: 4 datasets on the HF Hub
--output-format json # 5 local JSON files (results, traces, metrics, leaderboard row, metadata)
--output-format opensearch # 4 OpenSearch indexes with typed mappings and bulk indexingDetails and OpenSearch connection options: output formats guide.
Run evaluations on HF cloud hardware without a local GPU (Pro / Team / Enterprise):
hf jobs run \
--flavor cpu-basic \
-s HF_TOKEN=hf_your_token \
-s OPENAI_API_KEY=your_key \
python:3.12 \
bash -c "pip install smoltrace ddgs && smoltrace-eval --model openai/gpt-4 --provider litellm --enable-otel"GPU flavors (t4-small through a100-large), the Python API, and scheduled evaluations are covered in the HF Jobs guide.
Each run creates 3 timestamped datasets, so periodic cleanup keeps your profile tidy:
smoltrace-cleanup --older-than 30d # dry-run preview (default)
smoltrace-cleanup --keep-recent 5 --no-dry-run # actually deleteCleanup is dry-run by default, never touches your leaderboard, and permanently protects the benchmark/tasks datasets. See the dataset management guide.
Every run appends aggregate stats (success rate, pass@1, avg steps, tokens, duration, cost, CO2, GPU utilization) to your personal leaderboard, with links back to the run's results, traces, and metrics datasets. SMOLTRACE computes pass@1 from the first recorded completion for each logical (agent_type, test_id) task and publishes the rule and denominator alongside the score. Community rankings live at huggingface/smolagents-leaderboard — contribute your runs! More in the leaderboard guide.
- Quickstart — first evaluation in minutes
- Configuration — environment variables and options
- Running evaluations — providers, difficulty filters, parallelism
- Agent tools — all 20+ tools with security notes
- GPU metrics & cost — what's collected and how it's aggregated
- Changelog
- llms.txt — machine-readable docs index for LLM agents
- Discord — chat with the community (shared with genai-otel-instrument)
- TraceVerse Community — an open org on the Hub for evaluation & observability datasets, spaces, and real-world MCP apps; join and share your own
- TraceMind-AI collection — 40+ domain-specific task datasets to evaluate against
- GitHub Issues — bug reports and feature requests
Contributions welcome! See the contributing guide.
git clone https://github.com/Mandark-droid/SMOLTRACE.git && cd SMOLTRACE
pip install -e .[dev]
pytestApache-2.0. See LICENSE.
⭐ Star this repo to support Smolagents! Questions? Open an issue.