Skip to content

fix(search): match stop words in message bodies - #703

Merged
maathimself merged 2 commits into
maathimself:mainfrom
salmonumbrella:fix/search-stop-word-body-match-upstream
Oct 29, 2026
Merged

maathimself merged 2 commits into
maathimself:mainfrom
salmonumbrella:fix/search-stop-word-body-match-upstream

Conversation

@salmonumbrella

Copy link
Copy Markdown
Contributor

Summary

Search now finds messages by common words in their body. Reproduced on current main: a message with the body How are we doing Orion? doesn't come back for how are we doing Orion, because PostgreSQL's English dictionary drops we, are and how and the empty query never matches. With this change it's found.

fix-search-match-stop-words-in-hydrated-message-bodies.mp4

Changes

  • When a term has nothing left after the English dictionary, the body match uses the simple dictionary for that term only. Terms with real words keep English stemming, including mixed ones like the running.
  • Negated stop words now exclude bodies that contain them (Orion -we). Punctuation-only terms like ??? still match nothing in the body, so they don't become wildcards.
  • The simple-dictionary match reads the same 600,000-character body prefix as the English one. Sender, subject and stored search-vector matching are unchanged.

One commit based on current main. No migrations or new dependencies. Stop words that only appear in the snippet or stored search vector, and full-body indexing, are left for separate work.

Testing

  • Reproduced before the fix: the new PostgreSQL regression suite fails 7 of its tests against main's search.js, then all 26 search tests pass with this patch.
  • Node 22, PostgreSQL 16: backend npm test passes 3,982 tests (2 existing optional tests skipped). The 3 smtpTransport.stage.test.js failover cases time out on my macOS machine on unpatched main as well; they don't touch search.
  • Backend lint, plugin boundary, syntax check and production dependency audit at the high-severity threshold pass.
  • Regression coverage includes positive and negated stop words, account, unified-inbox and Trash scoping, English stemming, stored vectors, metadata substring matching, the character cap boundary, and a generated check against literal word membership.

The demo uses fictional data and illustrates the behavior; it is not a live-provider test.

Closes #702
Refs #667


Contributor License Agreement

By submitting this pull request I confirm that:

  • I have read and agree to the Contributor License Agreement.
  • My contribution is my own original work (or I have identified any third-party material and confirmed it is compatible with the CLA).
  • I have the right to submit this contribution under the terms of the CLA.

salmonumbrella and others added 2 commits October 28, 2026 12:19
PostgreSQL's English dictionary drops words like "we", so a term made only of
stop words could never match a message body. Fall back to the simple
dictionary for that term only; stemming and stored vectors are unchanged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

@maathimself maathimself left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified locally: the full backend suite passes with PostgreSQL (the new suite fails 4 of 7 against main's search.js and passes with this change), and timing on 20,000 synthetic bodies is unchanged for both stop words and ordinary words, since the body tsvector was already computed per row. Merging, thanks.

@maathimself
maathimself merged commit 9ce36d5 into maathimself:main Oct 29, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Search misses common words like "we" in message bodies

2 participants