Skip to content

Name the unmet option, not just the capability, in resolution failures - #298

Draft
sarthak-19 wants to merge 1 commit into
WordPress:trunkfrom
sarthak-19:fix/resolver-diagnose-unmet-options
Draft

sarthak-19 wants to merge 1 commit into
WordPress:trunkfrom
sarthak-19:fix/resolver-diagnose-unmet-options

Conversation

@sarthak-19

Copy link
Copy Markdown
Contributor

Summary

ModelResolver::resolve() built its "no models found" failure message from the primary required capability only. When resolution actually failed because a required option (e.g. webSearch, inputModalities) eliminated every candidate, the message still blamed the capability — which those models did in fact support. That makes the message actively misleading: checking whether the capability is supported confirms that it is.

ModelRequirements::getUnmetRequirements() already exists specifically so calling code can report why a model is unsuitable, but ModelResolver's failure path never called it.

Change

On an empty candidate map, before throwing:

  • Re-run the lookup using only the required capabilities (dropping required options).
  • If that also returns nothing, the capability itself is unsupported and the message is unchanged.
  • If it returns models, intersect getUnmetRequirements()['options'] across them and append a clause naming the option(s) that every one of them fails to support.

The extra lookup only happens on a path that's already about to throw, so it costs nothing in the normal case.

Before:

No models found for provider "acme" that support text_generation.

After (when webSearch is what actually eliminated every candidate):

No models found for provider "acme" that support text_generation. The following requested option is not supported by any of those models: webSearch.

Test plan

  • composer test:unit — 1205 tests pass, including new ModelResolverTest cases: option named (unscoped and provider-scoped), several options named, and capability genuinely unsupported (no option named).
  • PromptBuilderTest::testGenerateResultWithProviderNoModelsThrowsException updated: the registry mock's expectation widens from once() to exactly(2) since fromPromptData() always adds inputModalities as a required option, so the diagnostic re-lookup fires; the thrown message is unchanged since that second lookup also finds nothing.
  • composer lint (phpcs + phpstan) — clean.

ModelResolver::resolve() built its "no models found" message from the
primary required capability only. When resolution actually failed
because of a required option that every otherwise-suitable model
lacked, the message blamed the capability, which those models did
support, misleading anyone debugging the failure.

On an empty candidate map, re-run the lookup using only the required
capabilities. If that also returns nothing, the capability itself is
unsupported and the message is unchanged. If it returns models,
intersect their unmet options and name the ones every one of them
fails to support.
@codecov

codecov Bot commented Sep 22, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.05882% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 86.66%. Comparing base (77a5995) to head (b726973).

Files with missing lines Patch % Lines
src/Providers/ModelResolver.php 97.05% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##              trunk     #298      +/-   ##
============================================
+ Coverage     86.58%   86.66%   +0.07%     
- Complexity     1383     1394      +11     
============================================
  Files            69       69              
  Lines          4449     4483      +34     
============================================
+ Hits           3852     3885      +33     
- Misses          597      598       +1     
Flag Coverage Δ
unit 86.66% <97.05%> (+0.07%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant