QueryUnderstandLab

When does LLM-based query understanding actually help search?

QueryUnderstandLab is a compact research project for studying how local LLMs understand, normalize, and rewrite search queries — and whether those changes improve downstream retrieval.

The project rule is simple:

Improve the query only if the retrieval system benefits.

At a Glance

• Research question: when should a search system preserve the original query, and when should it rewrite or enrich it?
• Primary local LLM: qwen3:8b through Ollama
• Query-understanding tasks: intent classification, entity/concept extraction, robustness, and canonical rewriting
• Lexical retrieval: BM25
• Dense retrieval: sentence-transformers/all-MiniLM-L6-v2
• Real-data evaluation: MS MARCO v1.1 candidate ranking
• Main finding: unconditional LLM rewriting hurt retrieval on the small MS MARCO evaluation
• Key pattern: dense retrieval was more tolerant of rewriting than BM25, but raw queries still performed better overall

Detailed experiment notes and limitations live in RESULTS.md.

Why This Project Exists

Search systems often add increasingly sophisticated query-understanding layers:

• intent prediction
• entity extraction
• spelling and normalization
• query expansion
• semantic rewriting
• retrieval routing

It is easy to assume that a cleaner, more explicit, or more semantic query must be better for search.

That assumption is not always true.

A rewrite can preserve human meaning while changing the lexical or semantic signals a retriever relies on. QueryUnderstandLab therefore separates two questions:

Did the model understand the query?

from:

Did that understanding improve retrieval?

The second question is the primary evaluation target.

Research Questions

The project focuses on four small questions:

1. Can a local LLM reliably produce structured query-understanding output?
2. Is that understanding stable under typos, abbreviations, dropped words, and light paraphrases?
3. Does canonical LLM rewriting improve lexical retrieval?
4. Does rewriting behave differently for lexical and dense retrieval?

The project deliberately avoids turning these questions into a large benchmark or production search stack.

Experimental Flow

raw query
   |
   +--> structured understanding
   |      - intent
   |      - entities
   |      - concepts
   |      - ambiguity
   |
   +--> controlled perturbations
   |      - typo
   |      - abbreviation
   |      - word drop
   |      - light paraphrase
   |
   +--> canonical rewrite
          |
          +--> BM25
          |
          +--> dense retrieval

The same principle is used throughout: keep the experiment small enough that individual failures can be inspected manually.

Key Findings

1. Valid JSON Is Not Enough

The first local-model outputs were syntactically valid JSON but did not reliably follow the intended schema.

Examples included:

• free-form intent descriptions instead of controlled labels
• dictionary-shaped entities instead of typed entity lists
• missing fields

Strict parsing and schema validation made these failures visible rather than silently accepting them.

For qwen3:8b, disabling thinking mode was also important for short, structured extraction.

2. Intent Taxonomy Definitions Matter

A small 10-query gold audit initially produced:

Setting	Intent Accuracy	Schema Success
Initial prompt	0.60	10 / 10
Clarified taxonomy	1.00	10 / 10

The second number should not be interpreted as benchmark-level performance. The audit is tiny, and the prompt refinement was informed by its errors.

The useful result is that intent boundaries such as content_discovery vs. product_discovery need explicit definitions.

3. Robustness Is Different From Correctness

The project generates deterministic query variants using:

• adjacent-character typos
• abbreviations
• word deletion
• light lexical paraphrases

Clear queries often remained intent-stable, while abbreviations and wording changes sometimes changed the predicted intent or concepts.

This exposed an important distinction:

A model can be consistent and still be wrong.

Robustness metrics therefore complement, rather than replace, a small gold correctness audit.

4. Toy Retrieval Made Rewriting Look Promising

On a tiny hand-built BM25 corpus:

Method	MRR
Raw query	0.83
Canonical rewrite	1.00
Entity/concept expansion	0.83

For example:

best coffee restaurants brooklyn

became:

best coffee shops in brooklyn

The rewrite removed a misleading lexical term and moved the relevant document from rank 2 to rank 1.

That result looked promising — until the same idea was evaluated on real search queries.

5. Rewriting Hurt BM25 on MS MARCO

On 30 MS MARCO v1.1 candidate-ranking examples:

BM25 Condition	MRR
Raw query	0.565
LLM rewrite	0.463

Rewrite outcomes:

• 6 wins
• 16 ties
• 8 losses

The rewritten queries were often semantically reasonable, but lexical normalization could remove terms that BM25 was successfully matching.

Examples:

what causes genetic alterations in normal cells
-> causes of genetic mutations in normal cells

how long can you keep chicken salad in your fridge
-> shelf life of chicken salad in refrigerator

A semantically cleaner query is not necessarily a better lexical retrieval query.

6. Dense Retrieval Was More Tolerant — But Rewriting Still Did Not Help

Using sentence-transformers/all-MiniLM-L6-v2 on the same 30-query setup:

Dense Condition	MRR
Raw query	0.601
LLM rewrite	0.567

Rewrite outcomes:

• 2 wins
• 23 ties
• 5 losses

Dense retrieval was less sensitive to semantic normalization than BM25, but the raw query still performed better overall.

Main Takeaway

The experiments do not support unconditional query rewriting.

A better mental model is:

query
  |
  v
query analysis
  |
  +-- already clear --------------------> retrieve directly
  |
  +-- noisy / ambiguous / poorly formed -> selectively rewrite or expand

The next production-oriented question would be how to predict when rewriting is worth the risk.

That question is intentionally left outside this version of the project.

Repository Structure

QueryUnderstandLab/
├── examples/
│   ├── data/
│   ├── gold_intent_experiment.py
│   ├── msmarco_candidate_experiment.py
│   ├── msmarco_dense_experiment.py
│   ├── query_understanding_smoke.py
│   ├── retrieval_experiment.py
│   └── robustness_experiment.py
├── src/queryunderstandlab/
│   ├── benchmark.py
│   ├── dense.py
│   ├── gold_eval.py
│   ├── io.py
│   ├── ollama.py
│   ├── perturbations.py
│   ├── retrieval.py
│   ├── rewrite.py
│   ├── robustness.py
│   ├── schema.py
│   └── understanding.py
├── tests/
├── RESULTS.md
├── pyproject.toml
└── requirements-bench.txt

Setup

Python 3.12 is used for the project.

python3.12 -m venv .venv
source .venv/bin/activate

python -m pip install --upgrade pip
python -m pip install -e ".[dev]"

Run the test suite:

pytest

For MS MARCO and dense-retrieval experiments:

python -m pip install -r requirements-bench.txt

Local LLM Setup

The experiments use Ollama and expect a local model.

The primary model used in the reported experiments is:

ollama pull qwen3:8b

Confirm that Ollama is available:

ollama list

Then run the structured-understanding smoke test:

python examples/query_understanding_smoke.py

Reproducing the Experiments

Query Robustness

python examples/robustness_experiment.py

Gold Intent Audit

python examples/gold_intent_experiment.py

Toy Retrieval

python examples/retrieval_experiment.py

MS MARCO + BM25

python examples/msmarco_candidate_experiment.py \
  --limit 30 \
  --output results/msmarco_v11_candidate_eval_30.jsonl

MS MARCO + Dense Retrieval

python examples/msmarco_dense_experiment.py \
  --limit 30 \
  --output results/msmarco_v11_dense_eval_30.jsonl

MS MARCO data and the sentence-transformer weights are downloaded on first use.

What This Project Taught

1. JSON validity is not schema correctness.
2. Schema correctness is not semantic correctness.
3. Intent taxonomies need explicit boundary definitions.
4. Robustness and correctness measure different things.
5. Query rewriting changes the retrieval distribution.
6. A semantically cleaner query can be worse for lexical retrieval.
7. Dense retrieval is more tolerant of rewriting, but not automatically helped by it.
8. Query understanding should be evaluated through downstream search impact, not only through model-output quality.
9. Rewriting is better viewed as a selective intervention than universal query preprocessing.

Scope and Limitations

This is a learning-oriented research project, not a production benchmark.

• The gold intent audit contains only 10 queries.
• MS MARCO experiments use 30 candidate-ranking examples.
• Evaluation is candidate ranking rather than full-corpus retrieval.
• The primary LLM is qwen3:8b.
• The dense experiment uses one embedding model.
• No statistical significance testing is reported.
• The prompt was refined using errors from the small gold audit.

These constraints are intentional.

Stop Point

The project stops after demonstrating:

• structured local-LLM query understanding
• strict schema validation
• robustness testing
• a small correctness audit
• canonical rewriting
• BM25 retrieval evaluation
• dense retrieval evaluation
• failure analysis

Not included in this version:

• large-scale benchmarking
• fine-tuning
• personalization
• multilingual evaluation
• hybrid retrieval
• learned rewrite routing
• production serving

The goal was to learn the core concepts quickly, obtain a few falsifiable results, and stop once the central research question had been answered.