Building AgentMeasure — check whether your agent metrics mean what their labels claim.
Run the checks · Project website · 中文说明 · My blog
One declared operation has a failed attempt, then a successful retry:
| Measurement | Result |
|---|---|
| Logical operations | 1 |
| Execution attempts | 2 |
| Operation success | 1 / 1 = 100% |
| Attempt success | 1 / 2 = 50% |
Both success rates describe the same execution, at different grains. If a trace cannot establish the grouping or outcome, the result is UNPROVABLE.
AgentMeasure turns these distinctions into local checks and CI results: PASS / FAIL / UNPROVABLE. It is useful for agent-framework maintainers, MCP/API providers, and teams comparing agent evaluations.
- Merged upstream: my OpenLIT token-accounting fix prevents reasoning tokens from being added to an output total that already includes them.
- Independent test cases: Urusilla’s external conformance vectors exposed measurement defects, including an operation-grain metric aggregated at assignment grain. The report records the fix and regression fixture.
- Explicit limits: the Conformance Pack checks execution grain, retry reconciliation, cost preservation, operation grain, and evidence boundaries. Token and cache checks are not yet supported by the pack; the OpenLIT fix is a separate upstream contribution.
With Git and Python 3 installed, run the included fixtures:
git clone https://github.com/roy-tong/AgentMeasure.git
cd AgentMeasure
python3 conformance/pack/agentmeasure selftestThis checks the repository’s fixtures. To check your own measurements, prepare an FMT-002 fixture and optionally provide metric claims. Raw data stays local unless you choose to share it.
Help make the checks better: contribute a counterexample or discuss a measurement. Start with a synthetic or public fixture and the number you expected. If the project is useful, star or share AgentMeasure.
When agents become software consumers, what counts as one use, one completed task, or one useful outcome? I am building AgentMeasure to make those units inspectable.
Read the essay · 中文：当软件的消费者变成 Agent
Other research tools and agent-readable catalog
| Tool | What it does |
|---|---|
| iRead Research Monitor | Ongoing source discovery and evidence-aware research digests |
| User Demand Research (SURE) | Auditable demand evidence and research reports (Skill + CLI + MCP) |
| Bilibili Video to Transcript | Public videos to timestamped, searchable research text |
| Roy's AI Product Research Library | Public research, essays, and a read-only agent index |
Machine-readable catalog: https://raw.githubusercontent.com/roy-tong/roy-tong/main/agent-tools.json
gh skill preview roy-tong/roy-tong find-research-tool
gh skill install roy-tong/roy-tong find-research-tool --agent codex --scope userThe router helps select a tool; it does not perform the selected tool’s work.
I also write about AI, products, embodied intelligence, and building companies. Personal blog · About · X · Contact