MarketRank is a second-stage e-commerce search reranker that learns to order candidate products by relevance using a combination of lexical similarity, dense semantic representations, marketplace-specific constraint features, and LambdaMART Learning-to-Rank.
Built on the Amazon Shopping Queries ESCI dataset, MarketRank achieves 0.7050 NDCG@10 on 8,956 held-out search queries, outperforming:
- TF-IDF cosine similarity by 8.72%
- MiniLM semantic similarity by 3.06%
The project emphasizes not just final model performance, but also query-safe evaluation, feature ablation, segment analysis, error analysis, and reproducible experimentation.
| Model | NDCG@5 | NDCG@10 | NDCG@20 |
|---|---|---|---|
| TF-IDF cosine similarity | 0.5893 | 0.6485 | 0.7484 |
| MiniLM semantic similarity | 0.6341 | 0.6841 | 0.7743 |
| MarketRank | 0.6622 | 0.7050 | 0.7895 |
MarketRank vs TF-IDF : +8.72%
MarketRank vs MiniLM : +3.06%
All feature and model-selection decisions were made using a separate validation set.
The official test queries were kept untouched until the final feature set, LambdaMART configuration, and stopping point were frozen.
Product search is fundamentally a ranking problem, not just a text-classification problem.
Consider:
Query: iphone 15 case
Possible candidate products might include:
iPhone 15 silicone case
iPhone 14 silicone case
iPhone 15 screen protector
Samsung Galaxy phone case
A useful search system must determine the relative ordering of these candidates for the same query.
MarketRank therefore formulates search as:
Shopper Query
│
▼
Candidate Products
│
▼
Lexical + Semantic + Marketplace Features
│
▼
LambdaMART
│
▼
Relevance Score
│
▼
Ranked Product List
MarketRank is intentionally scoped as a reranker.
It does not retrieve candidates from the entire product catalog. Candidate products are supplied by the ESCI benchmark, and MarketRank learns how to reorder them.
MarketRank uses the Amazon Shopping Queries Dataset (ESCI).
Dataset repository:
Amazon Science — ESCI Shopping Queries Dataset
The project uses:
product_locale = "us"
small_version = 1
resulting in:
| Dataset property | Value |
|---|---|
| Query-product judgments | 601,354 |
| Unique queries | 29,844 |
| Unique products in modeling table | 482,105 |
| Development queries | 20,888 |
| Held-out test queries | 8,956 |
Each query-product pair contains one of four ESCI relevance labels:
| Label | Meaning |
|---|---|
| E | Exact |
| S | Substitute |
| C | Complement |
| I | Irrelevant |
For LambdaRank training, the labels are encoded ordinally while using relevance gains aligned with the ESCI ranking objective:
I → 0.00
S → 0.01
C → 0.10
E → 1.00
The source ESCI data consists primarily of two Parquet datasets:
Query-product relevance judgments
+
Product catalog metadata
DuckDB is used to query these files directly using SQL.
The modeling dataset is formed using the composite product key:
(product_locale, product_id)
and joins relevance judgments to attributes such as:
product_title
product_brand
product_color
product_description
product_bullet_point
SQL is also used for dataset profiling, including:
- locale distributions;
- label distributions;
- query counts;
- train/test statistics;
- candidate-set sizes;
- product-field coverage;
- join validation.
This keeps relational data preparation separate from the downstream NLP and ranking pipeline.
A search-ranking dataset cannot safely be split by individual rows.
For example, a naive row-level split could produce:
query 123 → products A/B/C → train
query 123 → products D/E → validation
The same query context would then appear on both sides of the split.
MarketRank instead splits at the query level.
Every query belongs entirely to exactly one split:
16,711 queries → train
4,177 queries → validation
8,956 queries → final test
The final pipeline explicitly verifies:
development/test query overlap = 0
The official test set remains untouched during:
- feature design;
- ablation experiments;
- model selection;
- stopping-point selection.
The first ranking baseline uses word-level TF-IDF over product titles.
Configuration:
Unigrams + bigrams
100,000 maximum features
L2-normalized vectors
Cosine similarity ranking
Because TF-IDF vectors are L2 normalized:
cosine(query, product)
=
query_vector · product_vector
This baseline is especially useful for exact marketplace terminology such as:
brands
product names
model numbers
dimensions
technical specifications
NDCG@5 = 0.5829
NDCG@10 = 0.6417
NDCG@20 = 0.7459
The second baseline uses:
sentence-transformers/all-MiniLM-L6-v2
to generate dense embeddings for queries and product titles.
Embeddings are normalized and ranked using cosine similarity.
This enables semantic matches such as:
wireless earphones
≈
Bluetooth earbuds with charging case
even when lexical overlap is weak.
NDCG@5 = 0.6269
NDCG@10 = 0.6793
NDCG@20 = 0.7720
MiniLM improved validation NDCG@10 by approximately 5.87% over TF-IDF, confirming that dense semantic representations provide substantial ranking signal.
The final MarketRank model combines eight features, each representing a distinct relevance hypothesis.
Measures exact lexical similarity between the query and product title.
Useful for:
brands
product terminology
model identifiers
specifications
MiniLM cosine similarity captures semantic relationships even when shopper and catalog terminology differ.
Example:
query:
wireless earphones
product:
Bluetooth earbuds with charging case
Measures the fraction of unique query tokens appearing in the candidate title.
Conceptually:
matched query tokens
────────────────────
total query tokens
This estimates how completely the product satisfies the constraints expressed in the search.
Measures token-set overlap:
|Q ∩ P|
Jaccard = ─────────
|Q ∪ P|
Unlike query coverage, Jaccard also penalizes unrelated title vocabulary.
Detects whether the normalized search query occurs directly in the normalized product title.
This provides a high-precision lexical signal.
Detects whether the candidate product's brand was explicitly specified by the shopper.
Example:
query:
nike running shoes
product brand:
Nike
→ brand_match = 1
This feature does not encode whether a brand is intrinsically better; it only captures whether the product satisfies an explicitly requested brand constraint.
Captures explicitly requested product colors when catalog color metadata is available.
Example:
query:
black nike shoes
product color:
Black
→ color_match = 1
A marketplace-specific feature designed to preserve model numbers, capacities, dimensions, and other specification-sensitive tokens.
Examples:
iphone 15
RTX 4060
WH-1000XM5
65W charger
55 inch OLED
16 oz bottle
Consider:
Query:
iphone 15 case
Product A:
iPhone 15 silicone case
Product B:
iPhone 14 silicone case
Dense embeddings may consider A and B highly similar because almost all vocabulary is identical.
Commercially, however, the model-number mismatch may be critical.
The numeric/model feature explicitly measures whether digit-containing query specifications also appear in the candidate title.
Hyphenated product identifiers such as:
WH-1000XM5
are canonicalized so they can match variants such as:
wh1000xm5
Semantic embeddings are powerful, but marketplace search frequently depends on exact constraints.
MarketRank therefore combines:
semantic meaning
+
lexical similarity
+
query constraint coverage
+
brand/color constraints
+
model/specification matching
rather than replacing lexical retrieval signals with embeddings.
This allows the ranker to learn interactions such as:
high semantic similarity
+
matching product specification
→ strong relevance
versus:
high semantic similarity
+
incorrect model number
→ potential mismatch
The final ranking model uses LightGBM LambdaMART.
Each query forms one ranking group.
For example:
Query A → 20 candidates
Query B → 17 candidates
Query C → 24 candidates
is represented to the ranker as:
group = [20, 17, 24, ...]
This ensures candidate comparisons occur within the same query, rather than between unrelated products.
Objective LambdaRank
Primary metric NDCG@10
Learning rate 0.05
Number of leaves 31
Maximum depth 6
Minimum child samples 100
LambdaRank truncation 13
Final boosting iterations 183
The number of trees was selected using validation NDCG@10.
The final model was subsequently retrained using all 20,888 development queries with the frozen configuration before evaluating the held-out test set.
The primary metric is Normalized Discounted Cumulative Gain at rank 10 (NDCG@10).
Ranking quality depends not only on whether relevant products appear, but also where they appear.
DCG gives higher reward when high-relevance products appear near the top:
gain_i
DCG = Σ ───────────
log₂(i + 1)
NDCG normalizes this against the best possible ranking:
DCG
NDCG = ─────
IDCG
A perfect ranking receives:
NDCG = 1
MarketRank reports:
NDCG@5
NDCG@10
NDCG@20
with NDCG@10 used for model selection.
Controlled ablation experiments were conducted by retraining LambdaMART while progressively adding feature families.
| Feature configuration | Validation NDCG@10 |
|---|---|
| Lexical features | 0.6713 |
| + Semantic similarity | 0.6925 |
| + Brand / color | 0.6963 |
| + Numeric / model overlap | 0.7038 |
Adding semantic similarity produced the largest incremental gain:
0.6713 → 0.6925
Structured brand and color constraints further improved ranking:
0.6925 → 0.6963
The custom numeric/model feature then improved performance to:
0.7038 NDCG@10
representing an additional:
+0.0074 absolute NDCG@10
over the otherwise-full model.
Final LambdaMART gain importance:
| Feature | Gain importance |
|---|---|
| Semantic similarity | 53.0% |
| Query coverage | 20.9% |
| Token Jaccard | 9.2% |
| Numeric/model overlap | 6.1% |
| TF-IDF similarity | 3.9% |
| Brand match | 3.6% |
| Color match | 2.3% |
| Exact phrase match | 1.0% |
Semantic similarity is the dominant feature, but approximately half of the model's accumulated split gain comes from other signals.
Feature importance is interpreted as accumulated tree-split gain and not as a causal estimate of feature value.
The ablation experiments therefore provide stronger evidence about incremental feature contributions.
After the architecture was frozen, the final LambdaMART model was trained on:
419,653 development query-product judgments
20,888 development queries
and evaluated on:
181,701 held-out judgments
8,956 unseen queries
with zero query overlap.
| Model | NDCG@5 | NDCG@10 | NDCG@20 |
|---|---|---|---|
| TF-IDF | 0.5893 | 0.6485 | 0.7484 |
| MiniLM | 0.6341 | 0.6841 | 0.7743 |
| MarketRank | 0.6622 | 0.7050 | 0.7895 |
vs TF-IDF : +8.72%
vs MiniLM : +3.06%
The validation score was:
0.7038 NDCG@10
while held-out test performance was:
0.7050 NDCG@10
showing consistent generalization between development and unseen queries.
The held-out test queries were also analyzed according to whether the query contained numeric/model specifications.
| Query type | Queries | MarketRank NDCG@10 | MiniLM NDCG@10 | Gain over MiniLM |
|---|---|---|---|---|
| No numeric token | 7,122 | 0.7105 | 0.6988 | +0.0117 |
| Numeric/model query | 1,834 | 0.6840 | 0.6272 | +0.0568 |
MarketRank's advantage over the pure semantic baseline is substantially larger on queries containing numeric constraints.
This is consistent with the motivation for explicit marketplace constraint features.
However, the segment result reflects the complete MarketRank model, so it should not be interpreted as the causal effect of the numeric feature alone.
The separate validation ablation provides direct experimental evidence for the numeric/model feature's incremental contribution.
MarketRank computes query-level NDCG to identify:
- difficult queries;
- queries where LambdaMART substantially improves over MiniLM;
- segments where exact constraints matter;
- cases where semantic similarity alone fails.
Examples of queries showing especially large gains over MiniLM include:
8 1/2 x 11 sticker sheet
3m mask n99
75 ft telephone cords for landline phones
14 gallon trash can
hydro flask 16 oz wide mouth
seventh generation toilet paper 48 rolls
These examples suggest that product quantities, dimensions, and specifications can be difficult for generic semantic similarity models to rank correctly.
Observed failure cases also include:
- ambiguous queries;
- sparse product descriptions;
- unusual product terminology;
- multilingual queries;
- cases where catalog metadata provides insufficient disambiguation.
Amazon ESCI Parquet Data
│
▼
DuckDB + SQL Profiling
│
▼
US Ranking Subset
│
▼
Product Metadata Join
│
▼
Query-Disjoint Train / Validation / Test
│
▼
Feature Engineering
┌──────┼────────┐
│ │ │
TF-IDF MiniLM Marketplace
Signals
│ │ │
└──────┼────────┘
▼
Cached Feature Tables
│
▼
LambdaMART
│
▼
NDCG@5 / @10 / @20
│
▼
Ablation Study
│
▼
Segment + Error Analysis
MarketRank/
│
├── README.md
├── requirements.txt
├── .gitignore
│
├── sql/
│ ├── 01_build_modeling_dataset.sql
│ └── 02_validate_modeling_dataset.sql
│
├── src/
│ ├── check_setup.py
│ ├── check_data.py
│ ├── profile_data.py
│ ├── build_dataset.py
│ ├── validate_dataset.py
│ ├── create_splits.py
│ ├── metrics.py
│ ├── baseline_tfidf.py
│ ├── baseline_semantic.py
│ ├── text_utils.py
│ ├── features.py
│ ├── build_features.py
│ ├── train_lambdamart.py
│ ├── run_ablations.py
│ ├── build_test_features.py
│ ├── final_evaluation.py
│ ├── error_analysis.py
│ └── plot_results.py
│
├── tests/
│ ├── test_metrics.py
│ └── test_features.py
│
├── results/
│ ├── tfidf_baseline.json
│ ├── semantic_baseline.json
│ ├── lambdamart_validation.json
│ ├── final_test_metrics.json
│ ├── ablations.csv
│ ├── final_feature_importance.csv
│ ├── segment_analysis.csv
│ └── worst_queries.csv
│
└── figures/
├── model_comparison.png
├── ablation_study.png
└── feature_importance.png
Large generated files are intentionally excluded from version control:
raw datasets
processed Parquet files
trained LightGBM models
fitted TF-IDF vectorizers
embedding artifacts
git clone https://github.com/yetiwannacode/MarketRank.git
cd MarketRankpython -m venv .venvWindows:
.venv\Scripts\activatemacOS/Linux:
source .venv/bin/activateInstall dependencies:
pip install -r requirements.txtClone the official Amazon Science repository:
git clone https://github.com/amazon-science/esci-data.git external/esci-dataThe dataset uses Git LFS.
If required:
git lfs pullMarketRank expects:
external/esci-data/shopping_queries_dataset/
├── shopping_queries_dataset_examples.parquet
└── shopping_queries_dataset_products.parquet
python src/build_dataset.py
python src/validate_dataset.py
python src/create_splits.pyTF-IDF:
python src/baseline_tfidf.pyMiniLM:
python src/baseline_semantic.pypython src/build_features.pyThis stage computes:
lexical features
semantic embeddings
marketplace-specific features
and stores the resulting feature table locally.
MiniLM encoding is the most compute-intensive stage.
python src/train_lambdamart.pypython src/run_ablations.pypython src/build_test_features.pypython src/final_evaluation.pypython src/error_analysis.pypython src/plot_results.pyRun the metric tests:
python tests/test_metrics.pyRun the handcrafted feature tests:
python tests/test_features.pyThe tests cover:
custom NDCG calculation
query coverage
Jaccard similarity
phrase matching
attribute matching
numeric/model matching
hyphenated product identifiers
Python
SQL
DuckDB
Pandas
NumPy
PyArrow
scikit-learn
SentenceTransformers
LightGBM
Matplotlib
Joblib
The target problem is relative product ordering within a query, not independent prediction of ESCI classes.
Products belonging to the same search query form one ranking problem. Allowing one query across multiple splits would contaminate evaluation.
Product titles had essentially complete coverage in the selected dataset, while fields such as description and color contained substantially more missing values.
TF-IDF preserves exact terminology and specifications, while MiniLM captures semantic similarity. Marketplace relevance benefits from both.
Product ranking frequently depends on explicit constraints—brand, color, quantity, capacity, size, or model number—that generic dense embeddings may blur.
Semantic encoding is significantly more expensive than LambdaMART training. Caching the feature table makes ranking experiments and ablations inexpensive and reproducible.
MarketRank is deliberately scoped as an offline second-stage relevance reranker.
It does not currently model:
- first-stage retrieval;
- user personalization;
- historical clicks;
- add-to-cart behavior;
- purchases and conversions;
- price;
- inventory;
- delivery speed;
- seller quality;
- product images;
- multilingual embeddings;
- online learning.
The ESCI benchmark provides relevance judgments rather than marketplace behavioral logs, so MarketRank focuses on offline product relevance.
A production marketplace search stack could extend MarketRank in several directions.
BM25 / ANN retrieval
↓
MarketRank reranking
First-stage retrieval could efficiently select a few hundred candidates from a catalog containing millions of products.
Additional features could include:
click-through rate
add-to-cart probability
conversion rate
purchase history
revenue per impression
product popularity
Clicks are affected by where products were previously displayed.
A production LTR system therefore needs techniques such as:
propensity weighting
counterfactual learning-to-rank
randomized exploration
to reduce position bias.
Product-image embeddings could be combined with text representations to improve relevance for visual or attribute-heavy queries.
A more expensive query-product transformer could rerank only the highest-scoring MarketRank candidates.
For example:
Retriever
↓
Top 100 candidates
↓
MarketRank
↓
Top 20
↓
Cross-encoder
↓
Final ranking
Offline NDCG improvements would ultimately need confirmation through controlled marketplace experiments.
Potential online metrics include:
CTR
successful-search rate
add-to-cart rate
conversion rate
revenue per search
query reformulation rate
MarketRank demonstrates that strong product-search ranking benefits from combining representation learning with explicit marketplace constraints.
Dense semantic similarity provides the largest individual ranking signal, but lexical overlap, query coverage, product attributes, and specification-aware matching add complementary information that semantic similarity alone does not capture.
The resulting LambdaMART system achieves:
0.7050 held-out NDCG@10
across:
8,956 previously unseen search queries
while outperforming both lexical and dense-semantic baselines and supporting its design choices through controlled ablations, feature importance, segment analysis, and query-level error analysis.