fix(early-stop): decide skill activation on the tool call, not its result

[mohsen-uipath]

Summary

Armed skill-activation rows that clearly trigger a skill were not early-stopping and burned their full turn budget. Two facts combined to break the stop at scale:

1. The live watcher evaluated armed criteria on the tool result (ToolEndEvent). When a turn is cut short (e.g. a timeout), the Skill tool call is left unresolved — the deadline check discards the pending result and finalize force-closes the call as UNRESOLVED, which the watcher (correctly) ignores. So the decision, though fully determined by the call itself, was never observed and the run kept going.
2. The final skill_triggered scorer used "any engagement" while the live verdict used "first engagement". Because the run kept going, the agent could engage additional skills, and each extra engagement was counted as a false positive against its own criterion.

Changes

• orchestration/early_stop.py — decide on the tool call. The watcher now evaluates armed criteria on ToolStartEvent (the call) in addition to the result. For an observable criterion the verdict is fully determined by the call's inputs (which skill / which command), so the watcher latches the instant the call is dispatched. The agent polls should_stop immediately after dispatching each message, so the loop breaks before a cut-short turn can strip the result. The in-flight call (not yet in the collector, which reduces commands from ToolEndEvent) is appended to the partial trajectory and reported at tool_call_index + 1; the counter still advances on the resolved ToolEndEvent. UNRESOLVED ends remain ignored, so an orphaned tool closed post-loop can never latch a false stop.

• criteria/skill_triggered.py — first-engagement scoring. The final check now scores the FIRST engaged skill, matching the live verdict, via a shared _first_engaged_skill_names helper. A second skill invoked alongside or after the first is no longer counted as a competing activation, so it is not a false positive. The live verdict and the authoritative score now agree by construction, whether or not the run stopped early. Removes the now-unused _engaged_skill helper.

• formatting.py — quiet the rate-limit warning. Skips the SDK RateLimitEvent (an out-of-band throttling notice) in format_messages alongside StreamEvent, so it no longer logs a spurious "unhandled SDK message type" warning. Cosmetic; does not affect scoring.

Tests

Adds coverage for:

• the tool-call decision (call fires before/without a result, latches before a later UNRESOLVED end, tool_call_index accounting);
• the orchestrator cut on a resultless call;
• first-engagement scoring, including the parallel-call case;
• the rate-limit event skip.

All existing early-stop / skill-triggered tests pass unchanged.

Verification

• ruff format + ruff check clean
• pyright — 0 errors
• full test suite — all green
• custom architectural lint (CE001–CE025) — all green

[github-actions]

Claude finished @mohsen-uipath's task in 1m 12s —— View job

---

Code Review in Progress

Todo List:

• Read code review guidelines from .github/code_review.md
• Read project conventions from CLAUDE.md
• Analyze PR diff and understand changes
• Read full files for context (not just diffs)
• Review implementation correctness
• Check cross-file consistency
• Perform "what's missing" analysis
• Design-level scrutiny
• Format and submit final review