Repository navigation
fix(search): match stop words in message bodies - #703
Merged
maathimself merged 2 commits intoOct 29, 2026
Merged
maathimself merged 2 commits into
maathimself merged 2 commits into
Conversation
PostgreSQL's English dictionary drops words like "we", so a term made only of stop words could never match a message body. Fall back to the simple dictionary for that term only; stemming and stored vectors are unchanged. Co-Authored-By: Claude Opus 5.5 <[email protected]>
maathimself
approved these changes
Oct 29, 2026
maathimself
left a comment
Owner
There was a problem hiding this comment.
Verified locally: the full backend suite passes with PostgreSQL (the new suite fails 4 of 7 against main's search.js and passes with this change), and timing on 20,000 synthetic bodies is unchanged for both stop words and ordinary words, since the body tsvector was already computed per row. Merging, thanks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Search now finds messages by common words in their body. Reproduced on current main: a message with the body
How are we doing Orion?doesn't come back forhow are we doing Orion, because PostgreSQL's English dictionary dropswe,areandhowand the empty query never matches. With this change it's found.fix-search-match-stop-words-in-hydrated-message-bodies.mp4
Changes
simpledictionary for that term only. Terms with real words keep English stemming, including mixed ones likethe running.Orion -we). Punctuation-only terms like???still match nothing in the body, so they don't become wildcards.One commit based on current main. No migrations or new dependencies. Stop words that only appear in the snippet or stored search vector, and full-body indexing, are left for separate work.
Testing
search.js, then all 26 search tests pass with this patch.npm testpasses 3,982 tests (2 existing optional tests skipped). The 3smtpTransport.stage.test.jsfailover cases time out on my macOS machine on unpatched main as well; they don't touch search.The demo uses fictional data and illustrates the behavior; it is not a live-provider test.
Closes #702
Refs #667
Contributor License Agreement
By submitting this pull request I confirm that: