[ispringer]

This PR makes two related but independent changes to phrase field (pf) boosting in the DisMax and eDisMax query parsers.

(co-authored with Claude Code: Opus 4.8, Opus 5.5 and Fable 5.1)

1. DisMax: skip the pf boost for single-term queries (changed)

The DisMax parser no longer applies the pf phrase boost to single-term queries, matching eDisMax (which already skips them via its minClauseSize gate). A one-position phrase imposes no adjacency constraint, so it is just a term boost.

A query counts as single-term only when both:

• the user query is one whitespace-free term, and
• its analyzed phrase is a single position in every pf field: a TermQuery, or single-position synonyms as built by synonymQueryStyle=as_same_term (SynonymQuery) or pick_best (DisjunctionMaxQuery). The BooleanQuery that as_distinct_terms builds is kept, because eDisMax's minClauseSize gate also keeps it — the two parsers agree for every style.

So these keep their boost:

• a single input term that analyzes into multiple tokens (CJK, ngram, WordDelimiterGraph, multi-word synonyms), and
• a multi-word query that analyzes to one token on a keyword or string pf field (an exact-match boost).

This may change relevancy scores for single-term DisMax queries, including those whose pf fields are not also in qf (such fields previously contributed a term boost via pf alone).

2. eDisMax: don't emit an empty pf clause (fixed)

When no phrase qualifies for the boost (single-term queries, q=*:*), minClauseSize discards every phrase and parsing returns an empty BooleanQuery, which was added to the main query as an empty SHOULD clause. It did not affect scoring but defeated the single-clause MatchAllDocsQuery optimization; it is now omitted, so q=*:* reduces to a MatchAllDocsQuery even when pf is set. Fielded-only queries (e.g. q=subject:foo) also now return early instead of building and parsing an empty phrase.

Notes

• The primary value here is correct relevancy. An earlier approach skipped phrase analysis entirely for single-term queries, but that could not be done correctly from the raw query string — it wrongly dropped boosts for multi-token analyses — so the phrase is now always analyzed and then inspected. Remaining performance effects (expected, not benchmarked):
  • DisMax single-term queries execute one fewer DisjunctionMaxQuery clause, avoiding a second postings read and scoring pass per pf field for every matching doc.
  • eDisMax q=*:* with pf now reaches Solr's MatchAllDocsQuery fast paths (answered from the live-docs set, bypassing the filterCache) instead of scoring a BooleanQuery over every doc.
  • eDisMax fielded-only queries skip creating a parser and analyzing an empty phrase per pf slop group.
• The reference guide's DisMax pf section now notes that single-term queries get no pf boost and that a pf-only field therefore contributes nothing to a single-term query's score (use qf or bq instead); the same note is in major-changes-in-solr-11.adoc since this is a relevancy change with no opt-out.
• Multi-word queries are deliberately left untouched by this PR (hence the raw-term check): DisMax always boosted them before and still does, even where eDisMax would drop e.g. a string-field exact match via minClauseSize. Converging the two parsers' multi-word behavior is out of scope.

Tests

Tests added

TestDisMaxQParser (new, SolrTestCase) — unit tests for DisMaxQParser.isEffectivelySingleTerm (package-private via @VisibleForTesting), exercising the query-shape logic directly without a running core:

• Single-term shapes → boost dropped: bare and BoostQuery-wrapped TermQuery; one-position synonyms as built by as_same_term (SynonymQuery) and pick_best (DisjunctionMaxQuery); degenerate one-term PhraseQuery/MultiPhraseQuery; an all-single-term DisjunctionMaxQuery across fields.
• Multi-term shapes → boost kept: two-term PhraseQuery, two-position MultiPhraseQuery, a sloppy graph phrase (BooleanQuery of per-path phrases), and a mixed DisjunctionMaxQuery where one field is multi-term.
• Unrecognized / null shapes → conservatively kept: MatchAllDocsQuery, the as_distinct_terms all-SHOULD BooleanQuery (eDisMax keeps it too), a BooleanQuery with a required clause, an empty DisjunctionMaxQuery, and null (must not throw).

DisMaxRequestHandlerTest.testSingleTermPhraseBoost (new) — end-to-end through the classic DisMax parser:

• q=wireless (single analyzed token) → the qf clause is present and the pf clause (subword:wireless)^10.0 is not.
• q=wi-fi (WordDelimiter splits into wi fi) → pf phrase boost retained.
• q=wi-fi on a field with autoGeneratePhraseQueries="false" (the default since schema 1.4) → the qf clause is a plain disjunction, so the pf phrase boost is the only adjacency signal and is retained. Added a subword_noautophrase field to the shared collection1 schema to drive this.
• q=usa (multi-word synonym → united states of america) → pf phrase boost retained. Added a text_syn field (existing synonym-enabled type) to the shared collection1 schema.
• q=hello world with pf=foo_s^10 (string field) → the exact-match boost (foo_s:hello world)^10.0 is retained even though it analyzes to a single term.
• The pf-retained assertions match the pf clause specifically (its ps slop and pf boost), since with autoGeneratePhraseQueries="true" the qf clause is the same phrase.

TestExtendedDismaxParser.testSingleTermPhraseBoost (new) — end-to-end through eDisMax:

• q=wireless → no pf boost and no empty () clause; q=wi-fi and q=usa → boost retained (via minClauseSize).
• q=*:* → reduces to MatchAllDocsQuery (regression guard for the empty-clause fix).

(CJK was intentionally not given a dedicated test — a single CJK input term yields the same multi-term PhraseQuery the wi-fi/usa cases already exercise, so it adds no code-path coverage while requiring a CJK analyzer in the test schema.)

Regression checks run

All green (319 tests across 18 suites):

• TestDisMaxQParser, DisMaxRequestHandlerTest, TestExtendedDismaxParser, TestMaxScoreQueryParser, QueryEqualityTest — the parser suites directly touched.
• Every other solr/core suite that exercises the DisMax parser: QueryElevationComponentTest, HighlighterTest, RankFieldTest, DistributedQParserTest, TestFunctionQuery, TestAuxIndexJoinQParserPlugin, TestScoreJoinQPNoScore, TestQueryTypes, TestSearchPerf, SpellCheckCollatorTest, TestJoin, SolrPluginUtilsTest, plus BasicFunctionalityTest for the additive text_syn / subword_noautophrase fields in the shared collection1 schema.
• ./gradlew tidy and ./gradlew check -x test — clean.

Checklist

Please review the following and check all that apply:

• I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
• I have created a Jira issue and added the issue ID to my pull request title.
• I have given Solr maintainers access to contribute to my PR branch. (optional but recommended, not available for branches on forks living under an organisation)
• I have developed this patch against the main branch.
• I have run ./gradlew check.
• I have added tests for my changes.
• I have added documentation for the Reference Guide
• I have added a changelog entry for my change

🤖 Generated with Claude Code

[dsmiley]

I'm rather surprised to see this had no impact on the tests. There should definitely be a test.

I suppose some people will notice a scoring impact... but this seems like the correct behavior here -- so be it. The parser shouldn't be frozen.

[dsmiley]

I had some doubts so I shared them with Claude Opus for analysis. Here is the report.

---

I checked two things here: (a) whether pf2/pf3 already short-circuit so that this is merely the same rule extended to pf, and (b) whether the changelog's relevancy caveat is accurate. (a) holds; (b) does not, and the counterexample also invalidates the premise of the change itself.

The pf2/pf3 short-circuit does already exist

ExtendedDismaxQParser.java:628:

if (null == fields || fields.isEmpty() || null == clauses || clauses.size() < shingleSize) return;

shingleSize is the wordGram, assigned at :1741-1745 — pf→0, pf2→2, pf3→3. So:

• pf2: 1 clause → 1 < 2 → returns. Already short-circuited.
• pf3: 1–2 clauses → returns. Already short-circuited.
• pf: clauses.size() < 0 never fires; then 0 is resolved to clauses.size(). The new shingleSize < 2 is therefore equivalent to clauses.size() < 2 — structurally the same rule pf2/pf3 already get.

So the shape of the change is consistent. It additionally covers clauses.size() == 0 (e.g. q=*:*, or a query made only of fielded/phrase clauses), which previously built an empty "" phrase. That's a real bonus: the resulting empty SHOULD clause was defeating the clauses().size() == 1 MatchAllDocsQuery optimization at :197 whenever pf was configured.

But "single term" is being detected as "no whitespace", which is not the same thing

Parsed queries on current main (pre-patch), with qf and pf both set to the same field, where subject is fieldType text (WordDelimiterGraphFilter, autoGeneratePhraseQueries="true"):

edismax q=wi-fi      +((subject:wi subject:fi)) (subject:"wi fi")   <- pf boost is REAL
edismax q=wireless   +(subject:wireless) ()                          <- pf boost is a no-op
dismax  q=wi-fi      +(subject:"wi fi") (subject:"wi fi")
dismax  q=wireless   +(subject:wireless) (subject:wireless)          <- REAL (no minClauseSize in DisMax)

wi-fi contains no whitespace, so it is a single clause and this patch short-circuits it. Yet today it produces subject:"wi fi" — an adjacency constraint that the qf clause (subject:wi subject:fi) does not impose. Same field, same boost weight, and ranking still changes.

The "minClauseSize=2 discarded it anyway" premise only holds when analysis emits exactly one token. Line 2 above is that case — and note the boost there shows up as (), an empty BooleanQuery, because SolrQueryParserBase:277 returns newBooleanQuery().build() rather than null, so the phrase != null guard at :679 never suppressed it. It contributes nothing to scoring, so for that case this patch is a legitimate (cosmetic + minor perf) cleanup.

Analyzer chains where a whitespace-single-token query still yields a meaningful phrase boost:

• WordDelimiterGraphFilter — wi-fi, iPhone12, SKUs and part numbers
• CJK — the serious one. Chinese/Japanese/Korean text has no spaces, so an entire query is one clause. With CJKBigramFilter or an ngram tokenizer that yields many tokens and a real pf phrase boost today. This patch would silently disable pf for essentially every CJK query. (pf2/pf3 were already lost to :628; pf was the one still working.)
• Multi-word synonyms, German decompounders, ngram tokenizers

The changelog caveat is too narrow

This may affect relevancy scores for single-term queries if pf fields differ from qf fields or use different boost weights

The wi-fi case above changes ranking with identical pf and qf fields and identical boosts, so the stated condition is not the real one. There is also a third path not mentioned: eDisMax's pf parse sets setRemoveStopFilter(true) (:651), so pf and qf can differ in stopword handling even for identical field lists.

The caveat is correct in the narrow case it was presumably written for — identical fields, identical boosts, and one-token-in/one-token-out analysis. There the boost clause is identical to the qf clause, the score doubles uniformly, and rank order is preserved.

The DisMax and eDisMax halves are not the same risk

These are described together, but DisMax has no minClauseSize at all — it exists only in ExtendedDismaxQParser; SolrPluginUtils.DisjunctionMaxQueryParser has no such gate. Line 4 of the output above shows DisMax applying a real TermQuery pf boost for a plain single term. So the DisMax change is never an optimization of a no-op; it always removes a live boost. That deserves separate treatment (and a separate, blunter changelog note) from the eDisMax half.

Suggestion

If the goal is the optimization, gate on the analyzed token count rather than on whitespace. For eDisMax specifically, minClauseSize already does this correctly and safely — the early return only buys you skipping the parse, at the cost of being wrong for multi-token analysis. The current guard is sound only for whitespace-tokenized, one-token-in-one-token-out analysis chains.

[dsmiley]

Idea: I'm thinking that we want to know how many clauses are are in the BooleanQuery of the parsed userQuery. If it's one (or no BQ) then there were no whitespace separated components, and no need for phrase boost.

At this stage right here, we only have the String version of userQuery, so we don't know. Rather than do parser-like things here by inspecting the string, we can probably avoid this as redundant/hacky given we're going to already parse teh userQuery anyway. We can inspect a parsedQuery. Inspecting a String is not idea.

So where I'm going with this is, update the caller (who has the parsedUserQuery) to inspect it and conditionally call getPhraseQuery. Then I think you won't even be modifying getPhraseQuery.

[ispringer]

Opus advises against reusing the parsed q / counting BQ clauses for two reasons:

1. Clause count ≈ whitespace count. The qf parser has setSplitOnWhitespace(true), so BQ clause count is really the number of whitespace-separated words, not analyzed terms. That skips the boost for wi-fi, CJK, WordDelimiterGraph, multi-word synonyms, etc. — all "one word" but multiple analyzed terms that genuinely want the phrase boost.
2. q is the qf parse, the boost is the pf parse. parsedUserQuery is analyzed with the qf fields; the phrase boost is analyzed with the pf fields, force-quoted. When pf ≠ qf they can differ in fields/analyzers (and shape, since one's free-form and one's a forced phrase), so the qf query's term count isn't a reliable proxy.

From me: It seems better to favor correctness for all the corner cases over the perf gain from skipping the pp parse. Do you agree?

[dsmiley]

1. I don't get the problem. "so BQ clause count is really the number of whitespace-separated words, not analyzed terms. " -- so what? I think we indeed want to know the clause count of 'q' (i.e. BQ clause count) and don't care what the analyzed term shape looks like. If the query is wi-fi then this is one clause and will end up producing a PhraseQuery of "wi fi" because the non-whitespace span of characters implies positional adjacency to analyzed component terms.
2. "so the qf query's term count isn't a reliable proxy." I agree and didn't mean to imply using that. I only suggest examining the clause count of the parsedUserQuery, which is loosely the number of whitespace separated parts. Technically, maybe not exactly this, since the query string may have syntax like quotes, thus the whitespace inside quoted parts don't count. But the parser accounts for that, emitting a BooleanQuery clause for each component, thus it'd just be a PhraseQuery if a user quoted their query already.

[ispringer]

You're right on (2), I misread your suggestion, sorry about that.

On (1), the piece I failed to spell out is autoGeneratePhraseQueries. The "wi-fi already becomes "wi fi"" behavior only happens when the qf field type has it set to true. It has defaulted to false since schema 1.4 (TextField), and in the _default configset only text_en_splitting* enable it; text_general and text_cjk don't. With it false, the qf clause for wi-fi is (wi fi), a plain disjunction, so the pf boost is the only adjacency signal, and gating on clause count would drop it. CJK with CJKBigram is the worst case: every query is a single clause, so clause-count gating would disable pf for essentially all CJK queries.

Checking the analyzed pf phrase also keeps DisMax consistent with eDisMax, whose minClauseSize gate counts analyzed terms in the pf phrase rather than clauses, so eDisMax keeps pf for wi-fi too.

I did take your structural suggestion: the check now lives in addMainQuery, and getPhraseQuery is unchanged from main (6590729). I also added a DisMax test case on a field with autoGeneratePhraseQueries="false" showing wi-fi keeps its pf boost, and confirmed it fails if the gate is switched to clause count. The existing wi-fi/usa assertions were also matching the qf clause (the test schema is version 1.0, so the flag defaults to true there), so I tightened them to match the pf clause specifically.

(Analysis and changes co-authored with Claude Code, Opus 5.5.)

[github-actions]

This PR has had no activity for 60 days and is now labeled as stale. Any new activity will remove the stale label. To attract more reviewers, please tag people who might be familiar with the code area and/or notify the [email protected] mailing list. To exempt this PR from being marked as stale, make it a draft PR or add the label "exempt-stale". If left unattended, this PR will be closed after another 60 days of inactivity. Thank you for your contribution!

[ispringer]

@dsmiley - can you take another look when you have a chance? thanks!