Skip to content

feat(FR-3558): generate a build-time search index of pages, tabs, and setting items - #8811

Open
nowgnuesLee wants to merge 5 commits into
mainfrom
FR-3558-search-index
Open

feat(FR-3558): generate a build-time search index of pages, tabs, and setting items#8811
nowgnuesLee wants to merge 5 commits into
mainfrom
FR-3558-search-index

Conversation

@nowgnuesLee

@nowgnuesLee nowgnuesLee commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

Resolves #8807 (FR-3558)

What

  • react/scripts/build-search-index.mjs (pnpm run search-index) walks routes.tsx, type-strips each module with vite.transformWithEsbuild, and follows value imports to collect the t('…') keys reachable from every route entry.
  • Emits react/src/generated/searchIndex.json — 34 entries, ?tab= deep links for 8 pages / 24 tabs, 4 setting pages / 54 items, 1,701 body keys after dropping keys shared by ≥10 entries (140 kB raw, 18.5 kB gzip).
  • scripts/verify.sh gains a "Search index" drift gate next to the Relay one; scripts/dev.mjs runs the build once on start.
  • The index is committed and marked generated (.gitattributes, .prettierignore, .claude/rules/review-ignored-paths.md).

Why

The app is an SPA whose other routes are React.lazy chunks — they are not mounted when the palette opens, so their text does not exist in the DOM to search. The index is therefore derived from source at build time, and it stores i18n keys, not strings, so all 20 locales work and switching language needs no re-index.

Verification

  • bash scripts/verify.sh=== ALL PASS ===
  • vitest: 4 tests (searchIndex.test.ts)
  • Two consecutive builds are byte-identical; the drift gate fails when the committed index is stale.

Review fixes

Applied in fix(FR-3558): keep page-owned keys, index optional-param routes, and harden the extractor (8f78dfa).

Blockers

  1. The ≥10-entry noise filter deleted page-owned vocabulary. The walk is transitive with no depth limit, so session.launcher.SharedMemory reaches 19 entries exactly as button.Cancel does — a count-only filter dropped both. Measured before the fix: session.launcher.{SharedMemory,ClusterMode,Memory} absent from the session launcher, data.{Foldername,UsageMode,Host} absent from Data, session.SessionName absent everywhere. Placement now uses the depth already recorded in keyMap: an entry owns a key when it declares it within OWNER_DEPTH (4) import hops of the route component. A key owned by ≥ NOISE_THRESHOLD entries is chrome and dropped everywhere; any other shared key survives only on its owners; a key owned by nobody survives where it is shallowest. All seven measured keys are back on their pages, and button.*, time.*, errorBoundary.*, general.ExtendLoginSession are gone from all entries.
  2. chat/:id? was skipped as parametrised, so Chat — a first-class sidebar item — had no entry. Optional (?-suffixed) params no longer disqualify a route; it is indexed at the path with the optional segment omitted (/project/:projectName/chat).

Should-fix

  1. Tab recognizer accepts value as a key alias, so radio-strip tabs ({ label, value }) get their labels: 26/53 → 41/53 tabs labelled. TAB_OVERRIDES was re-checked and kept — the Data page's statusCategory is parseAsString with labels built by _.map, so nothing is statically detectable (verified: removing the override leaves the page with 0 tabs). It is now guarded by an assertion instead.
  2. Startup assertions for the hardcoded config: every configured path (CUT_PATH_*, EXTERNAL_PREFIXES, SHIM_COMPONENT_FILES, ROUTE_CHROME_COMPONENTS, routes.tsx, en.json) must exist, and every TAB_OVERRIDES key must match an indexed route — otherwise the build throws with the stale entry named. All of these previously failed silently on rename.
  3. Both can't-fail tests replaced. The route-coverage test compared the extractor's output to the extractor's own route list; it now checks VALID_MENU_KEYS against the index, with a documented exclusion list (summary/job aliases, plugin-rendered pipeline) that is itself asserted not to be stale. "Drops keys on every page" was structurally always true; it is now two tests that assert the ownership rule directly — named page keys present on their pages, named chrome keys absent from all.
  4. --check was dead (never called, and it ignored --out). Deleted, along with diffSummary and the unused search-index:check script — verify.sh's rebuild-then-git status gate is the real check, mirroring Relay.

Nits

  • (9) The component file-path array is no longer emitted; it is --verbose report output only. (10) Dead fileCount removed. (11) revisionTab removed from TAB_LIKE_PARAMS — only reachable from a skipped parametrised route.
  • (12) missingFromEnJson no longer warns on t('key', 'Fallback'), which renders the fallback: 4 warnings → 1, and that one (storageHost.permission.SelectionLimitExceeded, called as t(key, { options })) is a genuine miss.
  • (15) objectElements takes the right operand only for && BinaryExpressions, not for ??/||/+. (16) dedupeSettings keys on key+group, so the same title in two groups no longer collapses.

Measured effect

before after
entries 34 35 (/project/:projectName/chat added)
distinct body keys 1,701 1,940
key placements 2,840 3,913
keys dropped 229 (by count) 39 (chrome, by ownership)
tabs labelled 26/53 41/53
raw / gzip 141 kB / 18.3 kB 178 kB / 22.8 kB

The artifact grows because page-owned vocabulary that was being deleted is now retained; dropping the component array offsets part of it.

Deliberately not applied

  • (13) dev.mjs rewriting the committed index on every dev-server start. Accepted as-is: the build is ~0.7 s, never blocks startup, and a developer who changes a route wants the palette to match without a manual step. The risk it raises is a merge conflict on the regenerated artifact, which is addressed instead by a new .claude/rules/search-index-conflicts.md — on conflict, take either side and re-run pnpm run search-index; never hand-merge.
  • (14) Cross-machine determinism. The index is deterministic for a given source tree (asserted byte-for-byte by searchIndex.test.ts), but it is derived by walking the filesystem, so it is only reproducible across machines to the extent that the checkout and the resolved typescript/vite versions match — both are pinned by the lockfile, and verify.sh re-derives it in CI, so a machine-specific build would fail the gate rather than ship.
  • (8) Renaming the script — skipped as churn.

@github-actions

github-actions Bot commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

Coverage Report for root-coverage

Status Category Percentage Covered / Total
🔵 Lines 18.79% 28 / 149
🔵 Statements 20% 32 / 160
🔵 Functions 37.5% 6 / 16
🔵 Branches 26.47% 18 / 68
File CoverageNo changed files found.
Generated in workflow #115 for commit 23a34ed by the Vitest Coverage Report Action

@github-actions

github-actions Bot commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

Coverage Report for react-coverage (./react)

Status Category Percentage Covered / Total
🔵 Lines 15.54% 5659 / 36394
🔵 Statements 13.04% 6894 / 52868
🔵 Functions 13.62% 883 / 6482
🔵 Branches 9.41% 4761 / 50560
File Coverage
File Stmts Branches Functions Lines Uncovered Lines
Changed Files
react/src/components/GlobalSearchPalette/searchIndex.types.ts 100% 100% 100% 100%
Generated in workflow #501 for commit 23a34ed by the Vitest Coverage Report Action

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a source-derived search index for future global palette functionality.

Changes:

  • Extracts route, tab, setting, and i18n metadata.
  • Adds deterministic generation tests and drift verification.
  • Regenerates during development and documents generated-file handling.

Reviewed changes

Copilot reviewed 11 out of 12 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
react/scripts/build-search-index.mjs Implements index extraction and serialization.
react/src/generated/searchIndex.json Stores the generated index artifact.
searchIndex.types.ts Defines the index’s TypeScript interface.
searchIndex.test.ts Tests coverage, ownership, and determinism.
scripts/verify.sh Adds the index drift gate.
scripts/dev.mjs Regenerates the index at startup.
react/package.json Adds the workspace generator command.
package.json Exposes the root generator command.
.gitattributes Marks the index as generated.
.prettierignore Excludes generated JSON from formatting.
search-index-conflicts.md Documents conflict resolution.
review-ignored-paths.md Excludes generated index content from review.

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

Comment thread react/scripts/build-search-index.mjs Outdated
Comment thread react/package.json Outdated
Comment thread react/src/components/GlobalSearchPalette/searchIndex.test.ts Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 12 changed files in this pull request and generated no new comments.

Suppressed comments (1)

react/scripts/build-search-index.mjs:370

  • Flattening both sides of runtime conditionals makes gated search surfaces unconditional in the generated index. For example, EnvironmentPage.tsx:68-71 exposes registry only to superadmins, StatisticsPage.tsx:39-42 gates user-session-history on backend support, and UserSettingsPage.tsx:217-258 gates setting items on configuration; the emitted tab/setting shapes contain no corresponding guard. The palette can therefore surface hits that render no tab or setting for the current user. Preserve availability metadata for query-time filtering, or otherwise exclude/explicitly register guarded surfaces instead of flattening them unconditionally.
    if (ts.isBinaryExpression(x)) {
      if (x.operatorToken.kind === ts.SyntaxKind.AmpersandAmpersandToken)
        push(x.right);
      return;
    }
    if (ts.isConditionalExpression(x)) {
      push(x.whenTrue);
      push(x.whenFalse);

@agatha197 agatha197 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please resolve conflicts

nowgnuesLee and others added 5 commits September 3, 2026 13:23
… setting items

The global search palette cannot search other pages' DOM — every route is a
React.lazy chunk that is not mounted when the palette opens. So the index is
derived from source at build time and stores i18n KEYS, not strings, which
makes all 20 locales and language switching free.

`react/scripts/build-search-index.mjs` (zero new deps: the react workspace's
`typescript` + `vite.transformWithEsbuild`) parses `routes.tsx` with the TS AST,
walks each entry's value-import graph at unlimited depth, and collects `t()` /
`i18nKey=` keys plus structured tabs (nuqs enum parsers bound to `?tab=` and the
tab-like params) and `SettingList` items with their description keys. Edges into
other route entries, `MainLayout/**` and `DefaultProviders` are cut, BUI is
excluded, plain keys present on >= 10 entries are dropped as chrome, and dynamic
template keys are recorded but never emitted so the index holds no key `t()`
cannot resolve. Output: 34 entries, 8 `?tab=` pages / 24 keys, 4 setting pages /
54 items, 1,701 body keys, 141 kB / 18 kB gzip, byte-identical across runs.

The index lives at `react/src/generated/searchIndex.json`, NOT under
`__generated__` as the plan said: relay-compiler deletes every file it does not
own in its artifact directory. `.gitattributes`, `.prettierignore` and the
review-ignored-paths rule are extended to the new directory instead.

`scripts/verify.sh` gains `check_search_index_drift` next to the Relay gate, and
`scripts/dev.mjs` refreshes the index once at startup (non-fatal on failure).

Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01KBDYpkXSSpvZJyLsyCfEdc
…harden the extractor

BLOCKER — the >=10-entry noise filter deleted page-owned vocabulary. The walk
is transitive with no depth limit, so `session.launcher.SharedMemory` reaches 19
entries exactly as `button.Cancel` does, and a count-only filter dropped both.
`session.launcher.{SharedMemory,ClusterMode,Memory}` were absent from the session
launcher, `data.{Foldername,UsageMode,Host}` from Data, `session.SessionName` from
everywhere. Placement is now decided by OWNERSHIP, using the depth `keyMap`
already recorded: an entry owns a key when it declares it within OWNER_DEPTH (4)
import hops of the route component. A key owned by >= NOISE_THRESHOLD entries is
chrome and is dropped everywhere; otherwise a shared key survives only on its
owners. All seven measured keys are back on their pages; `button.*`,
`time.*`, `errorBoundary.*` and `general.ExtendLoginSession` are gone from all.
1701 -> 1940 distinct body keys, 229 count-dropped keys -> 39 chrome keys.

BLOCKER — `hasEntityParam` treated the optional `:id?` in `chat/:id?` as an
entity param, so Chat — a first-class sidebar item — had no entry. Optional
params no longer disqualify a route; it is indexed at the path with the optional
segment omitted (`/project/:projectName/chat`). 34 -> 35 entries.

Also:
- Tab recognizer accepts `value` as a `key` alias, so radio-strip tabs get their
  labels: 26/53 -> 41/53 tabs labelled. TAB_OVERRIDES is still required — the
  Data page's `statusCategory` is `parseAsString`, so nothing is detectable
  statically — and is now asserted against the indexed route set.
- Startup assertions: every configured file path (CUT_*, SHIM_COMPONENT_FILES,
  ROUTE_CHROME_COMPONENTS, EXTERNAL_PREFIXES, routes.tsx, en.json) must exist and
  every TAB_OVERRIDES key must match an indexed route, or the build throws.
  These failed silently on rename before.
- Tests: the route-coverage test compared the extractor to itself; it now checks
  VALID_MENU_KEYS against the index with a documented exclusion list
  (summary/job aliases, plugin-rendered pipeline) and asserts the exclusions are
  not stale. The "drops keys on every page" test was structurally always true;
  replaced with direct assertions on the ownership rule.
- Dropped the dead `--check` mode and the `search-index:check` script: verify.sh's
  rebuild-then-`git status` gate is the real check, mirroring Relay.
- Dropped the `component` file-path array from the emitted JSON (now --verbose
  logging only) and the unused `fileCount`; removed `revisionTab` from
  TAB_LIKE_PARAMS (only reachable from a skipped parametrised route).
- `missingFromEnJson` no longer warns on `t('key', 'Fallback')`, which renders;
  4 warnings -> 1, and that one is a genuine miss.
- `objectElements` only takes the right operand of `&&`, not of every binary
  operator; `dedupeSettings` keys on key+group so a title can repeat per group.
- New rule `.claude/rules/search-index-conflicts.md`: regenerate on conflict,
  never hand-merge.

Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01KBDYpkXSSpvZJyLsyCfEdc
- normalize `rel()` to POSIX separators so the CONFIG path comparisons
  (cuts, shims, chrome, external prefixes) and emitted IDs hold on Windows
- run `search-index` in `build:only` so a production build regenerates the
  index from the source being built, matching the Relay treatment
- drop a diagnostic console.log from searchIndex.test.ts; the values are
  asserted immediately below

Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CsM4qV5xYywwVzbYFPP6gS
…deleted chrome entry

main moved the admin session page's `statusCategoryValues` into
`helper/sessionStatusBuckets.ts`, and `resolveArrayArg` only looked for a
same-file declaration, so `/admin/session` silently lost both status tabs
from the index. Follow a named import into its module (two hops, no barrels).

main also deleted `FlexActivityIndicator.tsx` (FR-3662), which the
CONFIG-integrity assert correctly refused; remove it from
`ROUTE_CHROME_COMPONENTS`. Index regenerated.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

@agatha197 agatha197 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ran the extractor, the vitest file, and a byte-identical rebuild locally: all clean. One latent mismatch in the extractor.

if (ts.isArrayLiteralExpression(n)) return n;
if (ts.isCallExpression(n)) {
const arg = n.arguments[0];
if (arg) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The doc comment says [...].filter(...) is unwrapped, but a call with any argument always descends into arguments[0] — for .filter(fn) that is the callback, so the n.expression.expression branch is never reached and the array comes back null (the whole tab/setting list is silently dropped). Check PropertyAccessExpression before the argument, or drop that case from the comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Global search palette (Cmd+K) with a build-time page/tab/setting index

3 participants