[api][plan][python][examples] Framework-managed LLM-as-judge routing (Strategies.llm)

[purushah]

What

Implements the Strategies.llm(...) follow-up promised in discussion #897: the engine — not the strategy — executes the judge call, on the same durable, metered, observable chat path as any model call. Tracking: #1062.

Revised after review (thanks @wenjin272, @weiqingy): the PR now adopts the declaration/executor architecture proposed in review — one serializable RoutingStrategy declaration at the API layer, executors in Plan, dispatch by a language-neutral strategy type — plus all eleven review items. The user-facing builder is unchanged:

env.addResource("judge", ResourceType.CHAT_MODEL, judgeModelDescriptor);  // any plain chat model

env.addResource("router", ResourceType.MODEL_ROUTER,
    ModelRouter.of("small", "big")
        .describe("small", "fast and cheap: chit-chat, lookups")
        .describe("big",   "expensive: code, SQL, multi-step reasoning")
        .strategy(Strategies.llm("judge"))          // optional: .withMaxContextChars(n)
        .defaultModel("small")
        .fallback(true)
        .build());

Architecture (revised per review)

• RoutingStrategy is a declaration, not behavior: RoutingStrategyType (RULE_BASED | LLM_JUDGE | CUSTOM) + arguments (+ executor class for CUSTOM). There is no route() at the API layer — the previous UnsupportedOperationException-throwing judge class is gone.
• Execution lives in Plan, dispatched by type: rule matching in the resolver over patterns pre-compiled by the router; judge prompt/verdict logic in LlmJudgeRoutingExecutor (package-private pure helpers). No instanceof on concrete classes.
• Plan JSON is language-neutral: strategy_type: "llm_judge" + strategy_args — no Java class names for built-ins. A Java-emitted snapshot is committed and the Python cross-language test deserializes it verbatim, so Python parity ([Umbrella] Pluggable model routing (MODEL_ROUTER): delivery status, Python parity, documentation, and follow-ups #1062) needs only Python-side executors.
• The resolver owns the single durable boundary (route:<router>) for every strategy type — no executor can skip persistence; replay short-circuits execution, so neither the strategy nor the judge is re-invoked on recovery.
• Custom logic implements CustomRoutingExecutor with the data-only RoutingContext (v1's select-don't-delegate fence, verbatim): a custom executor cannot make hidden model calls. The engine-facing signature stays plan-internal.

Judge semantics (unchanged from the original design, per #897)

• Judge chat under durable id "judge:<router>" (engine retries, trace events, token attribution); decision persisted with source and judge-inclusive decision_ms under "route:<router>"; abstains persist as abstains, so replay after a candidate-set change resolves to the current default.
• Verdicts constrained to candidate names; unparseable / ambiguous / non-candidate verdicts abstain to the default with the cause recorded.
• Failure policy: exhausted retries honor the error-handling strategy (FAIL loud, IGNORE abstain-with-cause). Cancellation is determined from the thread's interrupt state plus explicit cancellation types (isCancellation); a bare InterruptedIOException (the shape Okio uses for ordinary HTTP timeouts) follows the normal failure policy — with a regression test.
• Judge input is full-fidelity (review): the complete message list, plus the rendered request when the anchor candidate binds a prompt — via the same BaseChatModelSetup.prepareRequestMessages the chat path uses, so the judge's view cannot drift from what the selected model receives. Opt-in cost cap: withMaxContextChars(n) (rendered request + SYSTEM pinned, newest-first fill, truncation flagged as judge_context_truncated).
• Fail-fast validation: strategy argument rules live in the declaration constructor (bad configs fail at Strategies.llm(...)/plan construction — no reflective validation path to bypass). Plan construction verifies the judge model is registered and its descriptor binds no prompt/tools/skills; a slim runtime backstop catches instance-level bindings plan-time cannot see. Custom executor classes are checked (exists / implements / constructor) without instantiation.
• Python: add_resource(..., MODEL_ROUTER) raises an explicit not-yet-supported error.

Deliberate trade-offs (reviewed, kept)

• The judge path performs a second durable write (route: in addition to judge:) to keep the route-record schema uniform and preserve replayed decision_ms.
• The judge's rendered view anchors to the default candidate's bound prompt. Candidates binding different prompts see their own rendering only at answer time — rendering per-candidate before the decision is circular. Documented in the resolver javadoc.
• Full judge context means judge cost scales with conversation length; the cap is deliberately opt-in rather than a default, so the out-of-box behavior matches what the selected model sees.

Claims → tests

claim	test
verdict routes to named candidate, durable ids, metadata	llmJudgeVerdictRoutesToNamedCandidate
unparseable / non-candidate / ambiguous verdicts abstain	llmJudgeUnparseableVerdictAbstainsToDefault, llmJudgeVerdictOutsideCandidatesAbstainsToDefault, parseVerdictAcceptsOnlyCandidates
replay: stored abstain → current default; strategy not re-run	storedAbstainReplaysToCurrentDefaultWithoutRerunningStrategy
replay: original decision_ms reported	storedDecisionReplaysWithOriginalDecisionMs
judge timeout (InterruptedIOException) under IGNORE abstains, thread not left interrupted	judgeInterruptedIOTimeoutAbstainsToDefaultUnderIgnore
judge sees full conversation / rendered bound prompt	judgeSeesFullConversation, judgeSeesRenderedBoundPrompt
cap pins SYSTEM + rendered template + newest; flags truncation	maxContextCharsCapsJudgeInputAndFlagsTruncation, contextCapPinsRenderedTemplateMessages
SYSTEM + empty USER + promptArgs does not route blind	judgeMessagesFallBackToPromptArgsWithSystemAndEmptyUser
judge token keys actually produced	judgeTokenCountsLandInDecisionMetadata
bad-args judge / missing judge model / non-plain judge fail at plan construction	judgeWithMissingJudgeModelFailsAtPlanConstruction, typoedJudgeModelFailsAtPlanConstruction, judgeWithBoundPrompt/ToolsFailsAtPlanConstruction
instance-level bound prompt caught at runtime (FAIL loud / IGNORE degrade)	llmJudgeRejectsInstanceLevelPromptBoundJudgeSetupUnderFail, llmJudgePromptBoundSetupAbstainsToDefaultUnderIgnore
custom executors never instantiated at plan time	customExecutorIsNotInstantiatedDuringPlanConstruction
Java plan deserializes in Python	committed Java snapshot + test_python_can_deserialize_java_plan_with_judge_router

Compatibility impact

• API change (0.4.0 is unreleased, so no released users affected): custom strategies now implement CustomRoutingExecutor.route(strategy, context) instead of RoutingStrategy.route(context); Strategies.of(...) is now Strategies.custom(...); the plan wire format uses strategy_type/strategy_args (+ strategy_executor_class for CUSTOM) instead of strategy_clazz.
• Plain-chat agents and rule-routed agents are behaviorally untouched.
• Event consumers see the decision_source value llm_judge and optional metadata keys judge_model, judge_prompt_tokens, judge_completion_tokens, judge_context_truncated. (decision_source is no longer duplicated inside the metadata map — it contradicted the top-level key on the judge+fallback path.)

Test evidence

• 702 Java tests green (api 378, plan 324; 13 pre-existing skips) including the new replay-path, cancellation, context-fidelity, and plan-validation suites; Python suites green including the Java-snapshot cross-language test.
• End-to-end re-run post-restructure on a standalone Flink 2.3.0 cluster (dist jar + flink run of ModelRoutingJudgeExample, local Ollama): judge verdicts routed with full metadata; fallback composition exercised live.

Generative AI was used for this change.

Generated-by: Claude Code 2.1.239 (Claude Fable 5)

[wenjin272]

Thanks for opening the follow-up PR @purushah. Could you please create a corresponding github tracking issue and link the design discussion, #964, and this PR? The issue should capture the complete scope: what was delivered in V1, the planned V2 implementation, Python-side parity, and user documentation. Any additional follow-ups can then be linked to it so we can track the feature end to end.

I’ll start reviewing this PR as soon as possible.

[purushah]

Done — opened #1062 to track the feature end to end (design discussion #897, v1 #964, this PR, plus Python parity and user documentation as follow-ups). @wenjin272

[wenjin272]

Thanks for the work on this—the implementation looks solid overall. I’ve left a few minor comments.

Since LLM-based routing has a special framework-managed execution model, it also prompted some broader thoughts about the current routing API design, particularly the separation between API-level strategy declarations and Plan-level execution. It would be good to align on this design before finalizing the PR.

[wenjin272]

Could we move STRATEGY_CLAZZ_KEY and STRATEGY_ARGS_KEY to the beginning of the class, before the instance fields? That would make the class-level constants easier to discover and follow the usual Java class layout.

[purushah]

Agreed — will move STRATEGY_CLAZZ_KEY and STRATEGY_ARGS_KEY above the instance fields in the next push.

[wenjin272]

Could we avoid treating every non-SocketTimeoutException InterruptedIOException as cancellation? Okio may use a plain InterruptedIOException("timeout") for ordinary HTTP timeouts. Under IGNORE, this branch sets the thread’s interrupt flag and rethrows, bypassing the intended abstain-to-default behavior, so no candidate model is invoked.

I suggest determining cancellation from the thread state and explicit cancellation types instead:

if (Thread.currentThread().isInterrupted()) {
    return true;
}

if (t instanceof InterruptedException
        || t instanceof ClosedByInterruptException
        || t instanceof CancellationException) {
    return true;
}

A bare InterruptedIOException should follow the normal FAIL/RETRY/IGNORE handling. It may also be clearer to rename containsInterrupt() to isCancellation().

Could we also add a regression test where the judge throws new InterruptedIOException("timeout") under IGNORE, verifying that the default model is invoked and the thread is not marked as interrupted?

[purushah]

Great catch — will fix as you suggested, including the rename and the regression test.

[wenjin272]

The judge call can persist an interruption before it is identified as cancellation, but this comes from the existing shared durable execution path and also affects direct chat calls. I’ve opened #1070 to track the common fix, so this does not need to block this PR.

[purushah]

Thanks for scoping this and opening #1070 — agreed it's the shared durable-execution path (direct chat calls hit it the same way) and best fixed there. Happy to help on that issue after this PR settles.

[wenjin272]

The judge currently keeps only the last USER message and serializes promptArgs as raw key: value pairs. This can lose context in two ways:

• Message history: given SYSTEM: "You are reviewing Java concurrency code", USER: "Focus on race conditions", and USER: "<code>", the judge sees only <code>, while the selected model receives the complete message list.
• Bound prompt semantics: given a prompt template Review this SQL for performance issues: {input}, an empty user message, and promptArgs = {input: "SELECT ..."}, the judge sees only input: SELECT ..., while the selected model receives the fully rendered review instruction.

The judge may therefore route based on a materially different request from what the selected model receives. Could we preserve the complete message list and include the effective prompt/template semantics when constructing the judge input?

[purushah]

You're right. Honestly, sending only the last message was me trying to keep the judge cheap — the whole point of a cheap judge is that the decision costs less than the answer, and I was worried about long conversations and tool outputs blowing that up. But your examples show the flip side: the judge can end up routing on something very different from what the model actually gets, and the bound-prompt case is just broken — the task lives in the template and the judge never sees it.

So let's do it your way by default: the judge gets the complete message list, and the rendered request (template + args) when a prompt is bound. For anyone who does need to cap the routing cost, I'll add an optional knob — something like Strategies.llm("judge").withMaxContextChars(n). Not set → everything goes to the judge. Set → we keep the rendered request and the SYSTEM message, fill the rest newest-first, and mark the decision metadata when anything got dropped, so the event log stays honest about what the judge saw.

[wenjin272]

validateLlmJudgeReferences() already verifies that the referenced judge model exists. Could we also validate there that its descriptor has no bound prompt, tools, or skills, and remove judgeSetupMisconfiguration() from the request path?

These are static configuration constraints and should fail when constructing the AgentPlan, independently of the runtime error-handling strategy.

[purushah]

Agreed — these are static constraints and should fail at plan construction. Will move the no-prompt/tools/skills checks into validateLlmJudgeReferences() and remove judgeSetupMisconfiguration() from the request path.

[wenjin272]

I think the current abstraction mixes two different concepts:

1. RoutingStrategy is documented as executable selection logic whose primary contract is route(RoutingContext).
2. LlmJudgeRoutingStrategy is actually framework-managed configuration plus prompt/verdict helpers. Its route() method always throws UnsupportedOperationException.

Because of this mismatch:

• ModelRouter instantiates every strategy as a RoutingStrategy, even when it cannot execute route().
• ModelRouter.route() is not valid for every successfully constructed router.
• ModelRoutingResolver must inspect the concrete implementation with instanceof LlmJudgeRoutingStrategy and bypass the public strategy contract.
• Future framework-managed strategies would require more concrete-type branches.
• Built-in strategy identity is represented by Java implementation class names, which is difficult to align with a future native Python implementation.

Could we separate declaration from execution more explicitly?

API layer: RoutingStrategy as an immutable declaration

The current RoutingStrategyDescriptor already serves this purpose, so I suggest renaming it to RoutingStrategy and removing route() from the API layer entirely.

For example:

public final class RoutingStrategy implements Serializable {

    private final RoutingStrategyType type;
    private final Map<String, Object> arguments;

    // Only present for CUSTOM.
    @Nullable private final String executorClass;
}

The built-in factories would continue returning this API object:

Strategies.rules(rules);
Strategies.llm("judge");
Strategies.llm("judge", promptTemplate);
Strategies.custom(executorClass, arguments);

ModelRouter would store this declaration directly instead of reflectively instantiating an executable strategy:

private final RoutingStrategy strategy;

public RoutingStrategy getStrategy() {
    return strategy;
}

Therefore, ModelRouter.instantiateStrategy(), ModelRouter.route(), and the current API-level LlmJudgeRoutingStrategy implementation would no longer be needed.

Plan layer: RoutingExecutor as the execution contract

Execution belongs in Plan because this layer has access to RunnerContext, ChatModelInvoker, durable execution, retry configuration, metrics, tracing, and error handling.

For example:

Provide RoutingExecutor interface in API layer:

public interface RoutingExecutor {

    RoutingDecision route(
            RoutingStrategy strategy,
            RoutingContext routingContext,
            RunnerContext runnerContext)
            throws Exception;
}

The built-in implementations would live in Plan:

RuleBasedRoutingExecutor
LlmJudgeRoutingExecutor

RuleBasedRoutingExecutor would own the current rule evaluation and durable route-decision call.

LlmJudgeRoutingExecutor would own:

• construction of the judge messages;
• invocation through ChatModelInvoker;
• judge retries and error handling;
• verdict parsing;
• judge and route durable IDs;
• routing metrics and tracing metadata.

This also keeps buildJudgeMessages() and parseVerdict() next to the framework-managed judge execution instead of exposing them through an API class that pretends to be directly executable.

Executor resolution

Plan could resolve built-in executors through a registry keyed by a language-neutral strategy type:

RoutingExecutor executor =
        routingExecutorRegistry.get(router.getStrategy().getType());

RoutingDecision decision =
        executor.route(router.getStrategy(), routingContext, runnerContext);

For example:

RULE_BASED -> RuleBasedRoutingExecutor
LLM_JUDGE  -> LlmJudgeRoutingExecutor
CUSTOM     -> executor class carried by the strategy declaration

This removes the concrete instanceof branch from ModelRoutingResolver. It also avoids encoding built-in strategies using Java FQCNs; Java and Python can serialize the same strategy type and provide their own Plan-level executor.

Custom executors

A custom strategy declaration can carry the user-provided executor class and constructor arguments:

Strategies.custom(
        CostAwareRoutingExecutor.class,
        Map.of("threshold", 1000));

The custom implementation would implement the Plan-level contract:

public class CostAwareRoutingExecutor implements RoutingExecutor {

    public CostAwareRoutingExecutor(Map<String, Object> arguments) {
        // Initialize custom configuration.
    }

    @Override
    public RoutingDecision route(
            RoutingStrategy strategy,
            RoutingContext context,
            RunnerContext runnerContext)
            throws Exception {
        // Custom routing execution.
    }
}

During AgentPlan construction, the framework should validate that:

• the custom executor class exists;
• it implements RoutingExecutor;
• it has the supported Map<String, Object> or no-argument constructor;
• built-in strategy configuration and judge-model references are valid.

A Plan-side typed factory can accept Class<? extends RoutingExecutor> while still returning the API-level RoutingStrategy.

The user-facing router declaration remains essentially unchanged:

ModelRouter.of("small", "large")
        .describe("small", "Fast model for simple requests")
        .describe("large", "Stronger model for complex requests")
        .strategy(Strategies.llm("judge"))
        .defaultModel("small")
        .build();

This gives the two concepts clear responsibilities:

• RoutingStrategy: serializable API declaration of what routing behavior is configured.
• RoutingExecutor: Plan-level implementation of how that behavior is executed.

It removes the unsupported route() method, avoids concrete-type dispatch, keeps built-in execution in the correct module, allows user-defined executors, and provides a cleaner path for the future Python implementation.

[purushah]

You're right — I worked the proposal through in code and it's the better architecture: declaration at the API layer, executors in Plan, dispatch by a language-neutral type. It removes the throwing route(), the instanceof, and the FQCN identity that blocks Python parity (#1062). Since 0.4.0 is unreleased, I'll adopt it in this PR.

Two amendments within the design, both to keep #964's guarantees structural: (1) the resolver keeps the single durableExecute wrap around the executor dispatch, so no executor can skip persistence and replay never re-invokes the judge; (2) custom executors get the data-only RoutingContext instead of RunnerContext — otherwise a custom executor can call a chat model directly, an unmetered cascade inside the decision step, which v1 deliberately made impossible.

If that works for you: RoutingStrategy becomes the serializable declaration, Strategies.* return it, both built-in executors move to Plan (prompt/verdict helpers package-private), plan JSON gets strategy_type tags, user-facing builder unchanged. Your other comments ride along in the same push.

[weiqingy]

Thanks for taking this on. A few questions inline.

[weiqingy]

This checks the durable ids. But FakeRunnerContext (:212-221) always calls through in both durableExecute and durableExecuteAsync, and there is no way to seed a stored result. So no test takes the replay path.

That leaves two things unproven: a stored abstain resolving to the router's current default (ModelRoutingResolver.java:279-281), and decision_ms surviving replay (:290-292), which is the reason the PR body gives for the second durable write.

A small map in the fake keyed on callable.getId() would open that path. Would that be worth adding here, or is replay meant to be covered end-to-end?

[purushah]

Great catch — yes, worth covering here. Will add a seedable store to FakeRunnerContext and tests for both replay behaviors.

[weiqingy]

This catch treats "not a judge" and "a judge with bad arguments" the same way. A judge subclass without judge_model throws from super(args), lands here, and returns null. So validateLlmJudgeReferences skips that router, including the judge-model check. build() misses it too, because the guard at ModelRouter.java:266 compares the exact class name. judgeSubclassIsValidatedByAssignability passes valid args, so this case is untested.

Under the default FAIL you get a loud error per record. Under IGNORE the request is dropped with a warning, so routing is quietly off for the whole job. That is what the javadoc at :699-703 is trying to prevent.

Would it help for instantiateIfLlmJudge to tell the two apart, returning null only when the class is absent or not a judge, and letting a bad-args failure through?

[purushah]

Great find — agreed. Will split the two cases so a judge subclass with bad arguments fails at plan construction, and add the missing test.

[weiqingy]

ModelRoutingResolver.java:272-273 and :283-284 already put decision_source into the decision metadata, and that map goes in under "metadata" five lines below. So when a judge-routed request falls back, the same block says decision_source = "fallback" here and metadata.decision_source = "llm_judge" inside. The fallback event at ChatModelAction.java:413-424 splits the same way.

Nothing in production reads the nested one. Reads go to the event attribute (ModelRoutingEvent.java:153) or to this top-level key (ChatModelActionRoutingTest.java:702, :909). The only assertion on the nested map is the containsKey at :377.

Since the PR body treats decision_source as consumer-visible, would dropping the two resolver puts be enough? Or does the judge path want its own key there, under a name that cannot clash?

[purushah]

Good catch — dropping the two resolver puts is enough; the event attribute and top-level key stay the single consumer-visible story. Will fix.

[weiqingy]

judge_prompt_tokens and judge_completion_tokens are new here and appear only at these two writes. FakeChatModel.chat (ChatModelActionRoutingTest.java:95-107) never sets extraArgs, so both branches are dead in every test and neither key has been produced once.

The PR body lists both under compatibility impact, and the comment at :376 says every shipped payload key is read or asserted somewhere.

One FakeChatModel outcome with extraArgs = Map.of("promptTokens", 12, "completionTokens", 3) plus an assertion would close it. Would you rather do that, or narrow what the comment claims?

[purushah]

Good catch — will add the extraArgs outcome and assertions so both keys are actually exercised.

[weiqingy]

This goes Python out and Python back in: model_dump_json() then model_validate_json(). Java is never involved. If Java emitted a different key or shape for strategy_clazz / strategy_args, this would still pass. So it does not yet back the PR body's line about proving Java plans deserialize in Python.

The snapshot helper in this file (_SNAPSHOT_DIR at :52, used by the test at :293) could give that claim something real. Is pointing this at a Java-produced plan practical, or would narrowing the claim be easier?

[purushah]

Fair point — will commit a Java-produced plan snapshot for the Python test to deserialize (cut from the new strategy_type format once the restructure in the other thread lands, so we don't snapshot an obsolete shape).

[weiqingy]

Thanks for working through the five earlier points. All five look closed to me. A few new questions inline on the restructure.

[weiqingy]

The rule evaluator lost its multi-turn test in the move. RoutingTest.ruleMatchesLatestUserMessageNotFirst (old RoutingTest.java:64) matched the regex on the oldest USER turn and asserted abstain. Its replacement (RoutingTest.java:69-82) only checks the RoutingContext accessors, so nothing runs the rule evaluator now.

Every rule-routed request left in the suite has one user message, so firstUserMessage() and lastUserMessage() return the same string. Nothing pins the choice on this line. Would one multi-turn case be worth adding back here?

[purushah]

Great catch — the restructure dropped exactly the test that pinned last-vs-first. Will add a multi-turn case that runs the rule evaluator end-to-end and asserts the match is against the latest user message.

[weiqingy]

RULE_BASED falls to default: break here, so rule keys are never checked at plan time. The javadoc at :701 says static routing constraints fail at plan construction and never per record, and a rule key naming a non-candidate is just as static.

Today it only surfaces at request time, at the candidate check in ModelRoutingResolver.java:153, inside the durable call. IGNORE then drops every matching record.

The note at RoutingStrategy.java:111-112 says the builder validates rule keys "where the candidates are in hand". That covers the fluent path, but a descriptor read back from a plan skips the builder. Worth a check in this arm too?

[purushah]

You're right — a rule key naming a non-candidate is just as static as a missing judge model, and the descriptor path skips the builder check. Will add a RULE_BASED arm that validates rule keys against the candidates at plan construction, with a test for the descriptor-built case.

[weiqingy]

"cached per TaskManager" does not match the lifecycle. ResourceCache is an instance field of ActionExecutionOperator.java:101 on main, assigned at :187, so the scope is one subtask. At parallelism 8 on one TaskManager a user gets 8 executor instances, not 1.

This is the only place documenting the new extension point's lifetime, and the same sentence invites executors to keep instance state. :152 repeats it. Is "cached per subtask" the accurate phrase for both?

[purushah]

Good catch — "per subtask" is the accurate phrase. Will fix both spots; since that sentence is what tells executor authors how instance state behaves, it should be precise.

[weiqingy]

The two new replay tests (ChatModelActionRoutingTest.java:1062, :1090) seed a custom-strategy router, so they take the durable call at :122. The judge path stores its decision here instead, and the judge work runs before this call. Nothing seeds route:<router> for a judge router. Does the judge path want its own test, or do you read it as covered by the shared code?

[purushah]

Fair question — it deserves its own test. The judge path has enough of its own orchestration before the shared store that "covered by shared code" is exactly the kind of claim I shouldn't leave untested. Will add a replay test that seeds route: for a judge router and asserts the judge is never invoked.

[weiqingy]

nit: these two guards check != null only, so a non-Number value under promptTokens or completionTokens goes into the durable metadata as-is. The existing reader of the same two extraArgs keys checks the type first: ChatModelAction.java:225-232 uses instanceof Number. The new test passes Integer, so it does not reach the gap. Should this line and :293 match that guard?

[purushah]

Good catch — will match the instanceof Number guard the reader already uses, and extend the test with a non-Number value.

[weiqingy]

nit: this guard is byte-identical to agent.py:168-175, message text included, but only the Agent path has a test (test_model_router_not_supported.py:28-30). The examples reach this copy through env.add_resource, for example rag_agent_example.py:42. One shared helper plus a parametrized test over both entry points would cover it. Would you rather do that here, or leave it for the Python routing work?

[purushah]

Nice find. The shared helper is cheap, so I'll do it in this PR — one helper plus a parametrized test over both entry points, rather than leaving a known duplication for the Python routing work to trip over.

[weiqingy]

Thanks for working through these. All six from the last round check out. Three small things inline, all in the new code.

[weiqingy]

This return also skips compileRules, and those pattern checks (empty key, non-String value, bad regex at ModelRouter.java:173-196) never look at candidates. The javadoc you added at :718-720 promises them for RULE_BASED.

What made me look twice is :742-751 just above: a missing strategy_type throws there, on the same "the constructor rejects it anyway" reasoning. Reachability is low either way. Would hoisting the compile above the guard be worth it?

Something like:

var ruleKeys = ModelRouter.compileRules(strategy).keySet();
if (!(candidates instanceof List)) {
    return;
}
for (String ruleKey : ruleKeys) { ... }

[purushah]

Good catch — the guard order defeats the exact promise the javadoc makes. Will hoist compileRules above the guard as you wrote it, and add a test for the mis-shaped-candidates + bad-pattern case.

[weiqingy]

Does the last clause hold? recordChatTokenMetrics adds three conditions this site does not: null metric group (ChatModelAction.java:222), non-empty model_name (:229-230), both counts > 0 (:235). A judge reply with promptTokens: 0 records no metric, but both keys still land in the metadata.

The instanceof Number half reads right. Worth trimming just the "always agree" sentence?

[purushah]

You're right, "always agree" overclaims — the metrics side has conditions this site doesn't. Will trim the sentence to just the type-guard parity.

[weiqingy]

nit: does the last sentence still fit? The first one says compileRules is the single validation path, and build() does validate by calling it (:319). So "Patterns were validated at build(); this re-validates" seems to argue with it.

[purushah]

Stale sentence from before the rewrite — will drop it.