Repository navigation
feat(lsp): resolution-first semantic tokens rework - #3867
Merged
Merged
Conversation
Rework the BAML semantic-tokens classifier to be resolution-driven (rust-analyzer style): walk the lossless CST for token positions and syntactic tokens (keywords/operators/punctuation), and resolve identifier meaning through the HIR/inference rather than guessing from the CST. - tokens/index.rs: span-keyed resolution index built over every inference-bearing scope (function / lambda / let) via the new uniform tir::scope_body lookup, so names in spawn/block/closure bodies classify with their own scope's inference. - Interface members now resolve like class members: resolve_interface_member records MemberResolution::InterfaceMethod / InterfaceField. MIR-safe (new variants lower to None item-refs and fall through to interface dispatch). - Highlighting fixes: as / with / is keywords, generators, generic type args and params (typeParameter), associated types (decl + binding), self member accesses, methods after .as<Interface>, string escapes (escapeSequence), byte strings, qualified type paths, true/false/null literals, object/map keys vs values. - bitflags-backed ModifierSet; classifier shared with `baml describe`. - tools_semantic_tokens: standalone web viewer for the classifier. 78 fixtures. See crates/baml_lsp2_actions/src/tokens/GAPS.md for the scaling next-steps (on-demand resolution, range/viewport, delta) and the remaining known gaps.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (21)
🚧 Files skipped from review as they are similar to previous changes (16)
📝 WalkthroughWalkthroughThis PR expands BAML LSP2 semantic-token classification with resolution-aware types, modifiers, escape sequences, range queries, and delta responses. It adds compiler support for contextual keywords and catch bindings, broad semantic-token fixtures, definition-resolution tests, and a local semantic-token viewer tool. ChangesSemantic-token engine and compiler integration
LSP transport and validation
Semantic-token viewer
Estimated code review effort: 5 (Critical) | ~120 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Warning Tools execution failed with the following error: Failed to run tools: 13 INTERNAL: Received RST_STREAM with code 2 (Internal server error) Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 20
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
baml_language/crates/baml_compiler2_mir/src/lower.rs (1)
6111-6145: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winHandle captured roots in interface-method value lowering.
This path only runs when the first segment is a local. A captured receiver in
let f = x.eqskips the candidate-switch branch and falls through tolower_multi_segment_path_as_field_chain, so the method value gets lowered like a field/map lookup instead of a bound-method dispatch.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@baml_language/crates/baml_compiler2_mir/src/lower.rs` around lines 6111 - 6145, The interface-method value lowering in the path that starts by checking self.locals for the first segment only handles plain locals, so captured receivers like let f = x.eq bypass bound-method dispatch and get treated as field/map chains. Update the logic in lower.rs around the method-candidate resolution branch to recognize captured roots as valid receivers, then route them through lower_path_receiver_to_local and emit_bound_method_candidate_switch just like local roots, instead of falling through to lower_multi_segment_path_as_field_chain.
🧹 Nitpick comments (1)
baml_language/crates/tools_semantic_tokens/src/analysis.rs (1)
46-244: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd unit tests around the offset/parser helpers.
This file is mostly pure string/offset logic, so a small
#[cfg(test)]module for UTF-16 offsets, escaped snapshot lines, and text-only diff cases would give faster and more targeted coverage than end-to-end fixtures alone. As per coding guidelines, "Prefer writing Rust unit tests over integration tests where possible."🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@baml_language/crates/tools_semantic_tokens/src/analysis.rs` around lines 46 - 244, Add a focused #[cfg(test)] module in analysis.rs to unit test the pure helpers instead of relying only on fixture flows. Cover offset_to_line_col, utf16_offset, parse_expected, and diff_count with cases for non-ASCII/UTF-16 positioning, escaped or quoted snapshot text lines, and text-only token diffs. Use the existing helper symbols and Token shape to keep the tests local and resilient to line changes.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@baml_language/crates/baml_compiler2_tir/src/builder.rs`:
- Around line 12397-12405: The default-method branch in builder.rs is writing a
redundant MemberResolution::InterfaceMethod into self.resolutions that is always
overwritten later by InterfaceDefaultMethod. Remove the insert in the
default-method loop around the bound check, and keep InterfaceMethod only in the
required-method loop where no later overwrite occurs; use the existing
iface_loc/method_name and func_loc resolution paths to keep the intended
behavior clear.
In
`@baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/catch_throw.baml`:
- Line 10: The semantic tokens fixture for catch handling is missing coverage
for the catch binding identifier, so update the `catch_throw.baml` expectations
to include the bound name in the `MayFail(x) catch (e) {` case and the other
referenced catch blocks. Make sure the `semantic_tokens` test asserts token
classification for the catch variable itself, not just the `catch` keyword, so
the resolution-driven identifier behavior is verified through the existing
fixture.
In
`@baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/constructor_spread.baml`:
- Line 8: The semantic-tokens fixture for constructor spreads is missing the
spread operand token, so extend the expected token list in the semantic token
test around the MyClass constructor spread case to include the x in ...x. Update
the fixture used by the semantic_tokens test for constructor_spread.baml (and
the related cases in the same test block) so the spread operand is asserted and
regressions in operand classification are caught.
In
`@baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/dynamic_type_builder.baml`:
- Around line 21-27: The semantic-tokens fixture records `functions
[ExtractResume]` but never asserts that `ExtractResume` is classified as a
function reference, so add the missing token assertion in the
`dynamic_type_builder.baml` test around `test ReturnDynamicClassTest` using the
existing semantic token expectations for `functions[...]`. Make sure the
assertion explicitly covers the `ExtractResume` identifier so regressions in
function-reference classification inside `functions [...]` are caught, and apply
the same update to the matching fixture noted in the comment.
In
`@baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/field_alias.baml`:
- Around line 35-37: Update the semantic tokens fixture for field alias handling
so it asserts both alias endpoints, not just the `as` keyword. In
`field_alias.baml`, expand the expectations around the `name as display_name`
alias to include tokens for the source field (`name`) and the target alias
(`display_name`) alongside the existing `as` token, using the same semantic
token test pattern used elsewhere in these fixtures.
In
`@baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/index_access.baml`:
- Around line 45-46: The semantic tokens fixture is incorrectly marked as
diagnostic-free even though the preceding examples document validation errors.
Update the fixture around the diagnostics marker so it no longer uses
<no-diagnostics-expected> in the index_access.baml test content; either add the
expected diagnostics for the cases shown above or split the negative examples
into a separate fixture and keep this one truly diagnostic-free. Use the
existing semantic_tokens fixture structure and diagnostics comment block to
locate the change.
In
`@baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/ns_images_pipeline.baml`:
- Around line 40-41: The semantic token test only asserts the client property
key in ns_images_pipeline.baml and misses the actual client symbol reference, so
regressions in client-reference classification could go unnoticed. Update the
expected tokens in the semantic tokens test data to also assert the
AiGatewayImagenImageClient reference inside the function config, using the
existing semantic token fixture around the client/prompt entries to keep the
check tied to the relevant symbols.
In
`@baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/requires_clause.baml`:
- Around line 1-30: The semantic-tokens fixture for requires-chain coverage does
not currently exercise the transitive Aged member through Person requires Named,
Aged. Update the test content in requires_clause.baml to add a Person-typed or
self.age access that resolves age via the requires chain, so the fixture
actually validates the grandparent-interface flow instead of only the concrete
Employee field. Use the existing Person, Aged, and get_age symbols to place the
new access alongside the current example.
In `@baml_language/crates/baml_lsp2_actions/src/definition.rs`:
- Around line 426-436: The interface member resolution in definition.rs is
currently collapsing both MemberResolution::InterfaceMethod and
MemberResolution::InterfaceField to Definition::Interface, which makes
go-to-definition jump to the interface header instead of the accessed member.
Update the definition lookup in the branch handling
MemberResolution::InterfaceMethod and MemberResolution::InterfaceField to use
the member-specific identifier from the resolution and resolve the corresponding
interface method/field span, following the same pattern used by the Field and
method branches above. Ensure utils::definition_span is called with the member
declaration target rather than the interface declaration.
In `@baml_language/crates/baml_lsp2_actions/src/tokens/index.rs`:
- Around line 56-63: The token indexing logic in the scope filter is skipping
template-body scopes entirely, which leaves template interpolations like string
tokens unprocessed. Update the scope handling in the token indexer to not
unconditionally continue on scope.is_template_body, and instead let those bodies
be indexed with the same inference context as the enclosing ExprBody. Keep the
existing function/lambda/let scope checks in the token indexing flow, but ensure
template-body synthetic scopes are still traversed so resolution-driven
highlighting works.
- Around line 157-176: In tokens/index.rs, the fallback in the
path-classification logic currently marks unresolved leaf segments as Namespace,
which incorrectly masks typos as valid identifiers. Update the resolve/classify
flow around the `resolve_path_at` and `classify::classify_resolved` branch so
only non-leaf prefixes can fall back to `SemanticTokenType::Namespace`, while
the final unresolved segment stays neutral when `inference.resolution(expr_id)`
is missing and `local_root` does not apply.
In `@baml_language/crates/tools_semantic_tokens/src/analysis.rs`:
- Around line 202-207: `load_fixture` and `list_fixture_names` are swallowing
load failures by fabricating empty results, so update them to return errors
instead of defaulting to empty source or empty names. In `load_fixture`, use the
parsed file presence in `parsed.files` as a required condition and surface an
`InvalidData` error when no virtual file is available; in `list_fixture_names`,
stop flattening away `read_dir` failures and propagate the underlying I/O error.
Keep the fix localized to the `load_fixture` and `list_fixture_names` paths so
callers can distinguish “no data” from “failed to load.”
- Around line 225-243: The diff_count function is only comparing token type and
modifiers for matching position/length, so text-only snapshot drift can be
missed even when the committed semantic_tokens fixture would still change.
Update diff_count in analysis.rs to also compare the normalized committed lexeme
or raw snapshot text for each token key, and treat any text-only mismatch as a
diff alongside the existing type/modifier checks.
In `@baml_language/crates/tools_semantic_tokens/src/index.html`:
- Around line 440-445: The load-failure handler in the fixture fetch flow should
clear stale state instead of only rendering the error pane. In the catch path
around getJson in index.html, reset detail, disable the Accept control, and
clear the inspector/status state so the UI does not keep reflecting the previous
fixture after a failed /api/fixture request.
- Around line 456-469: The accept button flow in the click handler for acceptEl
leaves the control disabled after a failed /api/accept request, so the user
cannot retry without reselecting a fixture. Update the async click handler to
restore the enabled state in a finally block after the getJson("/api/accept"),
refreshFixtures(), and select(selected) sequence, and base that restore on the
current selection state (for example the same condition used before disabling).
- Around line 296-312: Stop interpolating fixture names directly into HTML
attributes in renderList(). The current esc() usage in the fixture button markup
still leaves data-name and title vulnerable because quotes are not escaped.
Update the row-building logic in renderList() to set these attributes through
DOM APIs or use an attribute-safe escape for f.name before inserting it, while
keeping the Scratchpad row consistent.
- Around line 414-427: The renderScratch flow applies any finished /api/tokens
response even if it was started for an older scratchText value, so update it to
ignore stale results. Add a request version/sequence check inside renderScratch
around the getJson("/api/tokens") call and only assign scratchTokens and render
when the response still matches the latest request for the current scratchText,
using the existing renderScratch, scratchTokens, and selected/SCRATCH logic to
locate the fix.
In `@baml_language/crates/tools_semantic_tokens/src/main.rs`:
- Around line 87-95: The port selection logic in bind can overflow when adding
the scan offset to a high preferred port. Update the loop in bind to clamp the
upper bound at u16::MAX before probing ports, so the SocketAddr construction
never wraps or panics; keep the rest of the TcpListener::bind retry behavior
unchanged and preserve the existing error path when no port is available.
In `@baml_language/crates/tools_semantic_tokens/src/server.rs`:
- Around line 183-196: resolve_fixture currently trusts Path::is_file, which
follows symlinks and can let fixture names escape fixtures_dir; update the
validation in resolve_fixture to reject symlinked entries before returning the
path. Use the existing resolve_fixture helper and its dir.join(name) result to
check the resolved candidate itself is a regular file owned inside the fixture
tree, and return an error for any symlink or non-regular path before
/api/fixture or /api/accept can use it.
In `@baml_language/crates/tools_semantic_tokens/src/staleness.rs`:
- Around line 31-37: The staleness watcher in newest_source_mtime only tracks
this crate and baml_lsp2_actions, but semantic token behavior also depends on
baml_compiler2_tir, baml_compiler2_mir, and baml_compiler_syntax. Update the
roots list (and any related watcher logic in staleness.rs) so those compiler
crates are included in the mtime scan, ensuring edits there trigger the viewer
restart and keep semantic tokens fresh.
---
Outside diff comments:
In `@baml_language/crates/baml_compiler2_mir/src/lower.rs`:
- Around line 6111-6145: The interface-method value lowering in the path that
starts by checking self.locals for the first segment only handles plain locals,
so captured receivers like let f = x.eq bypass bound-method dispatch and get
treated as field/map chains. Update the logic in lower.rs around the
method-candidate resolution branch to recognize captured roots as valid
receivers, then route them through lower_path_receiver_to_local and
emit_bound_method_candidate_switch just like local roots, instead of falling
through to lower_multi_segment_path_as_field_chain.
---
Nitpick comments:
In `@baml_language/crates/tools_semantic_tokens/src/analysis.rs`:
- Around line 46-244: Add a focused #[cfg(test)] module in analysis.rs to unit
test the pure helpers instead of relying only on fixture flows. Cover
offset_to_line_col, utf16_offset, parse_expected, and diff_count with cases for
non-ASCII/UTF-16 positioning, escaped or quoted snapshot text lines, and
text-only token diffs. Use the existing helper symbols and Token shape to keep
the tests local and resilient to line changes.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: 3f4c5b6e-52a8-4a11-bea9-e5d65b8d4846
⛔ Files ignored due to path filters (1)
baml_language/Cargo.lockis excluded by!**/*.lock
📒 Files selected for processing (100)
baml_language/Cargo.tomlbaml_language/crates/baml_cli/src/paint.rsbaml_language/crates/baml_compiler2_mir/src/lower.rsbaml_language/crates/baml_compiler2_tir/src/builder.rsbaml_language/crates/baml_compiler2_tir/src/inference.rsbaml_language/crates/baml_compiler_syntax/src/syntax_kind.rsbaml_language/crates/baml_lsp2_actions/Cargo.tomlbaml_language/crates/baml_lsp2_actions/src/definition.rsbaml_language/crates/baml_lsp2_actions/src/lib.rsbaml_language/crates/baml_lsp2_actions/src/tokens.rsbaml_language/crates/baml_lsp2_actions/src/tokens/GAPS.mdbaml_language/crates/baml_lsp2_actions/src/tokens/classify.rsbaml_language/crates/baml_lsp2_actions/src/tokens/index.rsbaml_language/crates/baml_lsp2_actions_tests/src/runner.rsbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/array_pattern.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/attributes.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/backtick_string.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/block_attributes.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/break_continue.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/byte_strings.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/cancel_token.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/catch_all_keyword.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/catch_throw.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/class.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/client.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/config_dictionary.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/const_binding.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/const_let_else_defer.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/constructor_spread.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/deep_method_call.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/dynamic_type_builder.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/enum_decls.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/enum_member_access.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/exhaustive.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/extends_chain.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/field_alias.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/for_loops.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/function.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/generators.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/generic_field_chain.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/generic_function_call.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/header_comments.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/http_config.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/if_else.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/if_expression.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/if_let_chain.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/implements_for.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/index_access.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/interface_impl.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/interfaces_inferred_generic_type_args.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/interfaces_iter_core.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/interfaces_sort_comparable.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/is_operator.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/iterator.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/jinja_control_prompt.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/json_map_literal.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/json_parse_stringify_intrinsics.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/lambda_advanced.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/literal_types.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/llm_function.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/map_literal_keys.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/map_type_and_methods.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/match_expr.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/match_literal_types.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/method_explicit_type_args.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/multi_segment_path.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/ns_agent_clients.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/ns_ansi.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/ns_game_clients.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/ns_images_pipeline.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/ns_sentiment.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/patterns_new.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/remap_role.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/requires_clause.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/retry_policy_valid_retry.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/self_qualified_call.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/signature_variety.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/stream_llm_inferred_typeargs.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/task_group.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/template_string.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/template_string_calls.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/test_block.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/test_expr_throwing_body.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/testset.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/testset_vibes_nested.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/throws_clause.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/top_level_binding.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/type_alias.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/type_aliases_jinja.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/watch_let.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/while_loop.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/with_keyword.bamlbaml_language/crates/bex_project/src/bex_lsp/multi_project/request.rsbaml_language/crates/tools_semantic_tokens/Cargo.tomlbaml_language/crates/tools_semantic_tokens/README.mdbaml_language/crates/tools_semantic_tokens/src/analysis.rsbaml_language/crates/tools_semantic_tokens/src/index.htmlbaml_language/crates/tools_semantic_tokens/src/main.rsbaml_language/crates/tools_semantic_tokens/src/server.rsbaml_language/crates/tools_semantic_tokens/src/staleness.rs
…oolean literals Move the semantic-tokens classifier toward the rust-analyzer architecture: - scope_resolution_index: a per-ScopeId salsa-cached resolution index (RA's body-granularity memoization) — editing one scope only invalidates that scope's index, not the whole file. build() merges them for a full document; resolve_token_class resolves a single name on demand (RA's Semantics::resolve model), walking the scope chain. - semantic_tokens_in_range: viewport highlighting (RA's highlight_range) — a range-gated walk that skips out-of-range subtrees and resolves on demand, so only the scopes the viewport touches are ever indexed. Proven equal to the full result filtered to the range across every sub-range (new range_tokens_test). - boolean semantic-token type: true/false -> `boolean` (was keyword). The inert KW_TRUE/FALSE/NULL/AS/TYPE token kinds + is_ident_token/is_keyword updates are laid for the principled parser-token remap (deferred: it ripples into hover/formatter/CST snapshots — see tokens/DESIGN.md). - tokens/DESIGN.md + GAPS.md: target RA architecture, prerequisite parser/AST changes, and remaining phases (LSP range/delta wiring, flat traversal, parser-token remap, typed-AST accessors).
…ing) Advertise and implement the range request: convert the LSP range to byte offsets, call semantic_tokens_in_range (on-demand, only resolves the scopes the viewport touches), and delta-encode via a shared encoder factored out of the full handler. Range query proven equal to full-filtered-to-range.
…okens) Advertise Delta + cache the last-sent tokens per file under a monotonic result_id. The delta handler diffs the new token array against the cached one (common prefix/suffix trim at token granularity; offsets scaled by 5 since the LSP stream is 5 u32 per token) and returns just the edits, falling back to the full set on a result_id miss. The full handler now emits a result_id. Diff covered by unit tests (identical/middle/append/truncate).
Replace the recursive Walk::node dispatch with rust-analyzer's flat traverse shape: one preorder_with_tokens() loop (Walk::run) that range-gates each subtree, hands subtree-spanning nodes (strings, comments, type exprs, object literals, generators) to a wholesale handler then skip_subtree, and classifies every other token from its parent kind via the free fn classify_token (rust-analyzer's IdentClass::classify_token model). The stateful cases (type-alias type-kw/name, object-field key, generic-param name) read off typed ast accessors (ObjectField::key, GenericParam::name) instead of positional sibling scanning. Behavior-preserving: all 440 tests pass and the 78 semantic_tokens fixtures are byte-identical.
Make the `as` cast/projection/field-link keyword and the `type` alias / associated-type keyword real tokens (KW_AS / KW_TYPE) instead of bare WORDs, so the semantic-tokens classifier reads them by kind rather than by text. Updates every token-kind consumer (the principled cleanup the remap enables): - ast.rs: TypeAliasDef::name no longer skips a leading `type` WORD (it's KW_TYPE now); the five `(WORD && text=="as")` checks read KW_AS. - baml_fmt: add TypeKw / As keyword tokens; TypeAliasDecl + AssociatedTypeDecl `keyword` and InterfaceFieldLink + AssociatedProjectionType `as_token` fields read them (drops the "type is just a WORD" workaround). - lsp classifier: `as` classifies via is_keyword (the text-match arm is gone); classify_type_decl_word detects KW_TYPE for the decl-vs-binding distinction. - 28 parser CST snapshots regenerated — purely mechanical WORD -> KW_AS/KW_TYPE (the meta-type value `type` correctly stays WORD). Validated: compiler2_ast 73/73, lsp 440/440, baml_tests 2661/2665 (the 4 generic_arg_soundness failures are pre-existing WIP red tests for a known TIR generics gap, confirmed failing on base too).
Make the boolean and null literals real tokens instead of bare WORDs, so the semantic-tokens classifier reads them by kind (the last text-match in the classifier's value-literal path is now backed by a real token). Updates the token-kind consumers the remap touches: - ast.rs: the block tail-expression accessor accepts KW_TRUE/FALSE/NULL (a trailing `false`/`true`/`null` is the block's value — without this a bool-returning function tripped E0029 "missing return value"). - baml_fmt: a KeywordLiteral token (spanning the three kinds) + a Literal::Keyword variant; Expression::from_cst routes the bare literal tokens to it (they were formatted as bare-WORD Paths before). - 39 parser CST snapshots regenerated — purely mechanical WORD -> KW_*. Validated: lsp 440/440, compiler2_ast 73/73, baml_tests 2661/2665 (the 4 generic_arg_soundness failures are pre-existing, fail on base).
Value-position literals are now KW_TRUE/KW_FALSE/KW_NULL (classified by kind) and type-position ones go through type_run, so the transitional WORD text match in token() is unreachable. The classifier no longer text-matches any literal.
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
baml_language/crates/baml_lsp2_actions_tests/src/range_tokens_test.rs (1)
39-49: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winExercise the new keyword remaps in this parity fixture.
This sample never emits
as,type,true,false, ornull, so the range/full equivalence check still skips several token kinds this PR reclassified. Adding one cast plus one boolean/null literal here would make the new range path much harder to regress.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@baml_language/crates/baml_lsp2_actions_tests/src/range_tokens_test.rs` around lines 39 - 49, The parity fixture in range_equals_filtered_full still does not exercise the newly remapped keywords, so update the sample used by check to include at least one cast and boolean/null literals. Use the range_tokens_test::range_equals_filtered_full test and the existing sample block to add expressions that emit as, type, true, false, or null so the range/full token comparison covers the reclassified token kinds.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@baml_language/crates/baml_compiler_syntax/src/ast.rs`:
- Around line 197-203: ObjectField::key() is only accepting SyntaxKind::WORD, so
it misses valid keys tokenized as KW_CLIENT. Update the key() accessor in ast.rs
to recognize KW_CLIENT alongside WORD when extracting the first non-trivia token
from self.syntax, so object fields named client are returned correctly and
downstream consumers can see them.
In `@baml_language/crates/baml_lsp2_actions/src/tokens/index.rs`:
- Around line 61-65: `resolve_token_class()` is caching on the raw
`scope_at_offset()` scope id, which can differ for sibling nested scopes that
still map to the same owning body. Update the memoization key to use the
inference owner scope returned by `scope_resolution_index()`/`scope_body(...)`
(the owning Function/Lambda/Let body) so identical ranges share one Salsa entry.
Keep the lookup flow in `resolve_token_class()` and `scope_resolution_index()`
aligned, and ensure any cache access in the range path uses the normalized owner
scope rather than the original scope id.
---
Nitpick comments:
In `@baml_language/crates/baml_lsp2_actions_tests/src/range_tokens_test.rs`:
- Around line 39-49: The parity fixture in range_equals_filtered_full still does
not exercise the newly remapped keywords, so update the sample used by check to
include at least one cast and boolean/null literals. Use the
range_tokens_test::range_equals_filtered_full test and the existing sample block
to add expressions that emit as, type, true, false, or null so the range/full
token comparison covers the reclassified token kinds.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: 3de4b39f-568d-4701-b741-8daacf8f9631
⛔ Files ignored due to path filters (61)
baml_language/crates/baml_tests/snapshots/broken_syntax/optional_parameter_defaults/baml_tests__broken_syntax__optional_parameter_defaults__02_parser__function_type_defaults.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/broken_syntax/pending_greaters_fix/baml_tests__broken_syntax__pending_greaters_fix__02_parser__extra_greater.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/broken_syntax/test_invalid_contexts/baml_tests__broken_syntax__test_invalid_contexts__02_parser__main.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/broken_syntax/type_annotation_errors/baml_tests__broken_syntax__type_annotation_errors__02_parser__type_malformed.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/compiles/config_dictionary/baml_tests__compiles__config_dictionary__02_parser__valid_dictionary.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/compiles/json_alias_basic/baml_tests__compiles__json_alias_basic__02_parser__json_alias_basic.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/compiles/json_map_literal/baml_tests__compiles__json_map_literal__02_parser__json_map_literal.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/compiles/lambda_advanced/baml_tests__compiles__lambda_advanced__02_parser__lambda_advanced.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/compiles/lexical_scoping/baml_tests__compiles__lexical_scoping__02_parser__lexical_scoping.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/compiles/literal_union_arithmetic/baml_tests__compiles__literal_union_arithmetic__02_parser__main.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/compiles/method_explicit_type_args/baml_tests__compiles__method_explicit_type_args__02_parser__method_explicit_type_args.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/compiles/namespaces_type_resolution/baml_tests__compiles__namespaces_type_resolution__02_parser__main.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/compiles/namespaces_type_resolution/baml_tests__compiles__namespaces_type_resolution__02_parser__ns_auth_auth.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/compiles/namespaces_type_resolution/baml_tests__compiles__namespaces_type_resolution__02_parser__ns_llm_models.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/compiles/optional_function_parameters/baml_tests__compiles__optional_function_parameters__02_parser__main.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/compiles/parser_expressions/baml_tests__compiles__parser_expressions__02_parser__precedence.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/compiles/parser_statements/baml_tests__compiles__parser_statements__02_parser__break_continue.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/compiles/patterns_new/baml_tests__compiles__patterns_new__02_parser__patterns_new.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/compiles/stream_crossfile/baml_tests__compiles__stream_crossfile__02_parser__file_a.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/compiles/test_expr_name_concat/baml_tests__compiles__test_expr_name_concat__02_parser__main.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/compiles/test_raw_string_name/baml_tests__compiles__test_raw_string_name__02_parser__main.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/compiles/testset_basic/baml_tests__compiles__testset_basic__02_parser__main.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/compiles/testset_dynamic/baml_tests__compiles__testset_dynamic__02_parser__main.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/compiles/testset_nested/baml_tests__compiles__testset_nested__02_parser__main.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/compiles/type_annotation/baml_tests__compiles__type_annotation__02_parser__type_alias_interaction.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/compiles/type_annotation/baml_tests__compiles__type_annotation__02_parser__type_positions.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/compiles/type_builder_test/baml_tests__compiles__type_builder_test__02_parser__test.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/attr_disambiguation/baml_tests__diagnostic_errors__attr_disambiguation__02_parser__non_field.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/backtick_strict_types/baml_tests__diagnostic_errors__backtick_strict_types__02_parser__main.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/basic_types/baml_tests__diagnostic_errors__basic_types__02_parser__types.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/builtin_io/baml_tests__diagnostic_errors__builtin_io__02_parser__io_ops.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/builtin_io/baml_tests__diagnostic_errors__builtin_io__02_parser__shell_ops.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/catch_return_type_mismatch/baml_tests__diagnostic_errors__catch_return_type_mismatch__02_parser__catch_return_type_mismatch.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/duplicate_class_span/baml_tests__diagnostic_errors__duplicate_class_span__02_parser__repro.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/format_checks/baml_tests__diagnostic_errors__format_checks__02_parser__binary_expr.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/format_checks/baml_tests__diagnostic_errors__format_checks__02_parser__config_decls.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/format_checks/baml_tests__diagnostic_errors__format_checks__02_parser__loop_stmts.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/format_checks/baml_tests__diagnostic_errors__format_checks__02_parser__match_exprs.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/format_checks/baml_tests__diagnostic_errors__format_checks__02_parser__other_exprs.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/format_checks/baml_tests__diagnostic_errors__format_checks__02_parser__type_alias_decls.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/function_type_throws/baml_tests__diagnostic_errors__function_type_throws__02_parser__negative.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/function_types/baml_tests__diagnostic_errors__function_types__02_parser__function_types.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/generics/baml_tests__diagnostic_errors__generics__02_parser__classes.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/headers_edge_cases/baml_tests__diagnostic_errors__headers_edge_cases__02_parser__edge_cases.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/match_exhaustiveness/baml_tests__diagnostic_errors__match_exhaustiveness__02_parser__match_exhaustiveness.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/null_handling/baml_tests__diagnostic_errors__null_handling__02_parser__null_handling.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/optional_parameter_defaults/baml_tests__diagnostic_errors__optional_parameter_defaults__02_parser__unsupported_contexts.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/parser_constructors/baml_tests__diagnostic_errors__parser_constructors__02_parser__array_constructor.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/parser_constructors/baml_tests__diagnostic_errors__parser_constructors__02_parser__map_constructor.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/stream_types/baml_tests__diagnostic_errors__stream_types__02_parser__complex.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/stream_types/baml_tests__diagnostic_errors__stream_types__02_parser__main.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/stream_types/baml_tests__diagnostic_errors__stream_types__02_parser__ns_foo_with_top_level_refs.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/test_expr_name_type_error/baml_tests__diagnostic_errors__test_expr_name_type_error__02_parser__main.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/test_expr_wrong_runner/baml_tests__diagnostic_errors__test_expr_wrong_runner__02_parser__main.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/test_with_runner_ambiguity/baml_tests__diagnostic_errors__test_with_runner_ambiguity__02_parser__main.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/type_aliases/baml_tests__diagnostic_errors__type_aliases__02_parser__type_aliases.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/type_aliases/baml_tests__diagnostic_errors__type_aliases__02_parser__type_aliases_json.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/unknown_type_error/baml_tests__diagnostic_errors__unknown_type_error__02_parser__unknown_type_error.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/void_return_type/baml_tests__diagnostic_errors__void_return_type__02_parser__function_type_throws.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/void_return_type/baml_tests__diagnostic_errors__void_return_type__02_parser__void_function_type.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/void_return_type/baml_tests__diagnostic_errors__void_return_type__02_parser__void_type_position_errors.snapis excluded by!**/*.snap
📒 Files selected for processing (23)
baml_language/crates/baml_cli/src/paint.rsbaml_language/crates/baml_compiler2_ast/src/lower_expr_body.rsbaml_language/crates/baml_compiler_parser/src/parser.rsbaml_language/crates/baml_compiler_syntax/src/ast.rsbaml_language/crates/baml_compiler_syntax/src/syntax_kind.rsbaml_language/crates/baml_fmt/src/ast/declarations.rsbaml_language/crates/baml_fmt/src/ast/expressions.rsbaml_language/crates/baml_fmt/src/ast/tokens.rsbaml_language/crates/baml_fmt/src/ast/types.rsbaml_language/crates/baml_lsp2_actions/src/tokens.rsbaml_language/crates/baml_lsp2_actions/src/tokens/DESIGN.mdbaml_language/crates/baml_lsp2_actions/src/tokens/index.rsbaml_language/crates/baml_lsp2_actions_tests/src/lib.rsbaml_language/crates/baml_lsp2_actions_tests/src/range_tokens_test.rsbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/break_continue.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/cancel_token.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/config_dictionary.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/interfaces_inferred_generic_type_args.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/interfaces_iter_core.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/interfaces_sort_comparable.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/watch_let.bamlbaml_language/crates/bex_project/src/bex_lsp/multi_project/mod.rsbaml_language/crates/bex_project/src/bex_lsp/multi_project/request.rs
✅ Files skipped from review due to trivial changes (1)
- baml_language/crates/baml_lsp2_actions/src/tokens/DESIGN.md
🚧 Files skipped from review as they are similar to previous changes (8)
- baml_language/crates/baml_cli/src/paint.rs
- baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/cancel_token.baml
- baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/break_continue.baml
- baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/config_dictionary.baml
- baml_language/crates/baml_compiler_syntax/src/syntax_kind.rs
- baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/watch_let.baml
- baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/interfaces_sort_comparable.baml
- baml_language/crates/baml_lsp2_actions/src/tokens.rs
- enum-variant (`Enum.Variant` patterns) and interface-field-link (`name as display_name`) resolve through the same package-resolution system as namespace/item lookup (resolve_enum_variant, resolve_field), verified against the type's actual variants/fields - builtin namespaces flagged defaultLibrary; primitives unified; named/ optional call args, bigint literals, and catch bindings classified - typing-robustness tests (never panic on incomplete/mid-edit input): curated + nested-scope always-run; full-fixture sweep #[ignore]d - parser: find_token_after_hashes returns None at EOF, fixing two index-out-of-bounds panics on incomplete raw strings
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@baml_language/crates/baml_lsp2_actions/src/completions.rs`:
- Around line 1319-1321: The `DefinitionSite::CatchBinding(_)` handling is still
incomplete in local-type consumers, so hover/type display can disagree with
completions. Update the `type_info.rs` logic that currently returns `"unknown"`
for `DefinitionSite::CatchBinding(_)` to resolve the catch binding type the same
way as `completions.rs` does, likely by reusing `find_binding_ty_for_local` or
the shared local-type lookup path used for `PatternBinding` and `CatchBinding`
in `find_binding_ty_for_local`/`DefinitionSite` matching.
In `@baml_language/crates/baml_lsp2_actions/src/tokens/classify.rs`:
- Around line 17-45: The builtin type allowlist in classify_primitive is
incomplete, so add the missing primitive names to PRIMITIVE_TYPES. Update the
PRIMITIVE_TYPES list in classify.rs to include void and type so
classify_primitive continues returning SemanticTokenType::Type with
ModifierSet::DEFAULT_LIBRARY for all builtin types.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: 81e11b9c-3521-4821-96bc-4475180189f8
📒 Files selected for processing (39)
baml_language/crates/baml_compiler2_hir/src/builder.rsbaml_language/crates/baml_compiler2_hir/src/semantic_index.rsbaml_language/crates/baml_compiler2_mir/src/lower.rsbaml_language/crates/baml_compiler2_tir/src/resolve.rsbaml_language/crates/baml_compiler_parser/src/parser.rsbaml_language/crates/baml_compiler_syntax/src/ast.rsbaml_language/crates/baml_lsp2_actions/src/completions.rsbaml_language/crates/baml_lsp2_actions/src/definition.rsbaml_language/crates/baml_lsp2_actions/src/describe.rsbaml_language/crates/baml_lsp2_actions/src/tokens.rsbaml_language/crates/baml_lsp2_actions/src/tokens/classify.rsbaml_language/crates/baml_lsp2_actions/src/tokens/index.rsbaml_language/crates/baml_lsp2_actions/src/type_info.rsbaml_language/crates/baml_lsp2_actions_tests/src/lib.rsbaml_language/crates/baml_lsp2_actions_tests/src/range_tokens_test.rsbaml_language/crates/baml_lsp2_actions_tests/src/typing_robustness_test.rsbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/bigint_literals.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/call_named_args.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/cancel_token.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/catch_all_keyword.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/catch_param.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/catch_throw.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/exhaustive.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/field_alias.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/interfaces_inferred_generic_type_args.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/interfaces_sort_comparable.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/iterator.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/json_parse_stringify_intrinsics.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/ns_ansi.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/ns_game_clients.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/patterns_new.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/stream_llm_inferred_typeargs.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/task_group.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/testset.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/testset_vibes_nested.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/with_keyword.bamlbaml_language/crates/bex_project/src/bex_lsp/multi_project/request.rsbaml_language/crates/tools_semantic_tokens/src/server.rsbaml_language/crates/tools_semantic_tokens/src/staleness.rs
✅ Files skipped from review due to trivial changes (1)
- baml_language/crates/baml_lsp2_actions_tests/src/lib.rs
🚧 Files skipped from review as they are similar to previous changes (25)
- baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/testset.baml
- baml_language/crates/baml_lsp2_actions_tests/src/range_tokens_test.rs
- baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/field_alias.baml
- baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/catch_throw.baml
- baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/json_parse_stringify_intrinsics.baml
- baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/testset_vibes_nested.baml
- baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/with_keyword.baml
- baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/catch_all_keyword.baml
- baml_language/crates/baml_lsp2_actions/src/definition.rs
- baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/iterator.baml
- baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/task_group.baml
- baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/stream_llm_inferred_typeargs.baml
- baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/ns_ansi.baml
- baml_language/crates/baml_lsp2_actions/src/tokens/index.rs
- baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/exhaustive.baml
- baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/ns_game_clients.baml
- baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/cancel_token.baml
- baml_language/crates/tools_semantic_tokens/src/server.rs
- baml_language/crates/bex_project/src/bex_lsp/multi_project/request.rs
- baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/interfaces_sort_comparable.baml
- baml_language/crates/baml_compiler_syntax/src/ast.rs
- baml_language/crates/tools_semantic_tokens/src/staleness.rs
- baml_language/crates/baml_compiler2_mir/src/lower.rs
- baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/patterns_new.baml
- baml_language/crates/baml_lsp2_actions/src/tokens.rs
- builder.rs (tir): drop redundant InterfaceMethod resolution that is always clobbered by InterfaceDefaultMethod in the same iteration - ast.rs: ObjectField::key() accepts KW_CLIENT (client-named object fields) - definition.rs + item_tree/builder (hir): interface member go-to-def jumps to the member, not the interface header (interface field/method span plumbing mirroring the class-field/enum-variant pattern) + regression tests - tokens/index.rs: normalize the range-path resolution cache key to the inference-owner scope so sibling block/template scopes share one entry - viewer (tools_semantic_tokens): surface fixture load failures instead of empty results; count text-only snapshot drift as a diff; escape fixture names in HTML attributes; discard stale /api/tokens responses; clear state on load failure; re-enable Accept after a failed save; clamp port scan to u16::MAX; reject symlinked fixtures; watch the compiler crates - requires_clause fixture: exercise the transitive requires-chain member Skipped 5 stale/incorrect comments (classifier emits no token for the cited positions, or the flagged sections are genuine auto-generated output).
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (1)
baml_language/crates/tools_semantic_tokens/src/index.html (1)
498-498: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winDon't overwrite the Accept state with
selected === SCRATCH.Line 498 clobbers the state that
select()/fixture rendering just computed. That can re-enable Accept after the load-failure path at Lines 463-472, and after a successful accept it can turn the button back on even when the refreshed fixture has no diffs.Suggested fix
} finally { - acceptEl.disabled = selected === SCRATCH; + acceptEl.disabled = + selected === SCRATCH || !detail || detail.diff_count === 0; }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@baml_language/crates/tools_semantic_tokens/src/index.html` at line 498, The Accept button state in the fixture rendering flow is being overwritten by the `selected === SCRATCH` assignment, which clobbers the state already computed by `select()` and the load-failure handling. Update the logic around `select()`, the fixture rendering path, and the `acceptEl` state update so `acceptEl.disabled` reflects the real computed acceptability instead of being reset based on `SCRATCH`; keep the existing failure and post-accept refresh behavior intact.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@baml_language/crates/baml_lsp2_actions/src/definition_at_tests.rs`:
- Around line 459-461: The definition-at tests are too permissive because
snippets like "-> fullname" and "-> encode" can match both the interface
declaration and other locations such as alias entries or impl methods. Update
the assertions in definition_at_tests to verify a unique target location/snippet
for each case, using the existing test helpers and descriptions around the
fullname and encode checks, so the tests only pass when navigation lands on the
interface declaration itself and not the impl site.
---
Duplicate comments:
In `@baml_language/crates/tools_semantic_tokens/src/index.html`:
- Line 498: The Accept button state in the fixture rendering flow is being
overwritten by the `selected === SCRATCH` assignment, which clobbers the state
already computed by `select()` and the load-failure handling. Update the logic
around `select()`, the fixture rendering path, and the `acceptEl` state update
so `acceptEl.disabled` reflects the real computed acceptability instead of being
reset based on `SCRATCH`; keep the existing failure and post-accept refresh
behavior intact.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: b12c1d8f-fe97-4f01-bc94-68cbb9e92ef1
📒 Files selected for processing (14)
baml_language/crates/baml_compiler2_hir/src/builder.rsbaml_language/crates/baml_compiler2_hir/src/item_tree.rsbaml_language/crates/baml_compiler2_tir/src/builder.rsbaml_language/crates/baml_compiler2_tir/src/inference.rsbaml_language/crates/baml_compiler_syntax/src/ast.rsbaml_language/crates/baml_lsp2_actions/src/definition.rsbaml_language/crates/baml_lsp2_actions/src/definition_at_tests.rsbaml_language/crates/baml_lsp2_actions/src/tokens/index.rsbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/requires_clause.bamlbaml_language/crates/tools_semantic_tokens/src/analysis.rsbaml_language/crates/tools_semantic_tokens/src/index.htmlbaml_language/crates/tools_semantic_tokens/src/main.rsbaml_language/crates/tools_semantic_tokens/src/server.rsbaml_language/crates/tools_semantic_tokens/src/staleness.rs
💤 Files with no reviewable changes (1)
- baml_language/crates/baml_compiler2_tir/src/builder.rs
🚧 Files skipped from review as they are similar to previous changes (9)
- baml_language/crates/baml_lsp2_actions/src/definition.rs
- baml_language/crates/tools_semantic_tokens/src/main.rs
- baml_language/crates/tools_semantic_tokens/src/staleness.rs
- baml_language/crates/baml_compiler2_tir/src/inference.rs
- baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/requires_clause.baml
- baml_language/crates/baml_lsp2_actions/src/tokens/index.rs
- baml_language/crates/tools_semantic_tokens/src/server.rs
- baml_language/crates/tools_semantic_tokens/src/analysis.rs
- baml_language/crates/baml_compiler_syntax/src/ast.rs
DESIGN.md and GAPS.md were working notes for the semantic-tokens rework. validate-markdown only allows README.md or whitelisted long-term docs, so drop them rather than whitelist throwaway notes.
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@baml_language/crates/baml_lsp2_actions/src/type_info.rs`:
- Around line 708-717: The binding-type lookup in type_info.rs is starting from
the enclosing function scope via infer_scope_types, which can return a
same-index PatId from the wrong body before the nested scope is checked. Update
the hover/type resolution logic in the binding-type lookup path to prefer
find_binding_ty_in_scopes(db, index, scope_id, pat_id) as the primary source and
avoid using func_scope_id-first probing for body-local pattern/catch bindings.
Keep the existing display_local_binding_ty formatting, but ensure the lookup
follows the current scope chain so closures and nested blocks resolve the
correct binding type.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: 2fa702f1-64ea-4318-a5c2-102218f8b5a5
📒 Files selected for processing (23)
baml_language/Cargo.tomlbaml_language/crates/baml_compiler2_ast/src/lower_expr_body.rsbaml_language/crates/baml_compiler2_hir/src/builder.rsbaml_language/crates/baml_compiler2_mir/src/lower.rsbaml_language/crates/baml_compiler2_tir/src/builder.rsbaml_language/crates/baml_compiler_parser/src/parser.rsbaml_language/crates/baml_compiler_syntax/src/ast.rsbaml_language/crates/baml_compiler_syntax/src/syntax_kind.rsbaml_language/crates/baml_fmt/src/ast/expressions.rsbaml_language/crates/baml_lsp2_actions/src/definition_at_tests.rsbaml_language/crates/baml_lsp2_actions/src/tokens.rsbaml_language/crates/baml_lsp2_actions/src/tokens/classify.rsbaml_language/crates/baml_lsp2_actions/src/type_info.rsbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/interfaces_iter_core.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/interfaces_sort_comparable.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/json_map_literal.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/lambda_advanced.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/method_explicit_type_args.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/ns_game_clients.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/test_expr_throwing_body.bamlbaml_language/crates/baml_lsp2_actions_tests/test_files/syntax/hover/catch_binding.bamlbaml_language/crates/tools_semantic_tokens/Cargo.tomlbaml_language/stow.toml
✅ Files skipped from review due to trivial changes (4)
- baml_language/crates/baml_lsp2_actions_tests/test_files/syntax/hover/catch_binding.baml
- baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/test_expr_throwing_body.baml
- baml_language/stow.toml
- baml_language/crates/baml_compiler2_ast/src/lower_expr_body.rs
🚧 Files skipped from review as they are similar to previous changes (16)
- baml_language/crates/tools_semantic_tokens/Cargo.toml
- baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/json_map_literal.baml
- baml_language/crates/baml_compiler2_tir/src/builder.rs
- baml_language/crates/baml_compiler2_hir/src/builder.rs
- baml_language/crates/baml_lsp2_actions/src/definition_at_tests.rs
- baml_language/crates/baml_compiler_syntax/src/syntax_kind.rs
- baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/method_explicit_type_args.baml
- baml_language/crates/baml_lsp2_actions/src/tokens/classify.rs
- baml_language/crates/baml_fmt/src/ast/expressions.rs
- baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/lambda_advanced.baml
- baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/interfaces_sort_comparable.baml
- baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/ns_game_clients.baml
- baml_language/crates/baml_compiler_syntax/src/ast.rs
- baml_language/crates/baml_compiler_parser/src/parser.rs
- baml_language/crates/baml_lsp2_actions/src/tokens.rs
- baml_language/crates/baml_compiler2_mir/src/lower.rs
…chain CodeRabbit: PatId is body-local, so probing the enclosing function scope's inference first could pick up a same-index binding from another body for pattern/catch bindings inside closures or nested blocks, giving the wrong hover type. Use only the collision-safe find_binding_ty_in_scopes walk from the current scope chain.
Both sides had independently added MemberResolution variants for interface members accessed on interface-typed values; canary's InterfaceVirtualMethod/ InterfaceVirtualField (from the #3983 interfaces TIR rework) subsume this branch's InterfaceMethod/InterfaceField, so the branch variants are dropped and consumers renamed (InterfaceDefaultMethod -> InterfaceConcreteMethod). Kept from this branch: go-to-definition navigation for virtual interface members (canary returned None), extended to cover virtual calls to default methods, which canary's variant also records. Fixed a latent branch bug the merge exposed: MIR catch lowering queried captured-ness and recorded binding locals under DefinitionSite::PatternBinding, but this branch registers catch-clause bindings as DefinitionSite::CatchBinding, which silently dropped the make_cell strategy for captured catch bindings (caught by canary's new lexical_scoping bytecode/exec tests). Test fallout re-baselined after review: semantic-token fixtures gained 'throws never' on interface methods (canary's new explicit-throws rule) and block_attributes now expects canary's E0014 duplicate-attribute diagnostics; three canary-added parser snapshots re-lex true/null as KW_TRUE/KW_NULL (this branch's keyword re-lexing, same as the 40 snapshots it already updated).
Binary size checks passed✅ 7 passed
Generated by |
meefs
pushed a commit
to meefs/baml
that referenced
this pull request
Jul 13, 2026
… first (BoundaryML#4002) Clean revert of a05ec5a (BoundaryML#3867, "resolution-first semantic tokens rework"), coordinated with @codeshaunted. ## Why BoundaryML#3867 merged ~30 minutes after BoundaryML#4000 (the LSP mutex/latency fix: position-encoding negotiation, connection-scoped sessions, bounded ingress) opened, and the two rework the same LSP boundary. Landing order matters: BoundaryML#4000 is the infrastructure BoundaryML#3867's feature should sit on. Integrating BoundaryML#3867 inside BoundaryML#4000's merge would mean rewriting its encoder and cache inside a merge commit — worse for review and for feature ownership. Reverting and re-landing on top is cleaner for everyone. The revert also removes a live encoding issue from canary until the re-land: `encode_semantic_tokens` currently emits fixed-encoding positions (`LineIndex`) and **byte** lengths, which miscounts any non-ASCII identifier under the LSP-default UTF-16. ## Re-land guidance (agreed with Avery) The classification engine (resolution index, CST walk, `tools_semantic_tokens` visualizer) re-lands byte-for-byte. Only the LSP boundary adapts: 1. Encode positions **and lengths** via the negotiated `PositionCodec` (available at every handler after BoundaryML#4000) instead of `LineIndex` + byte lengths. 2. Move `semantic_tokens_cache` into connection-scoped session state — sessions are per-connection with per-connection encodings after BoundaryML#4000, so a process-global cache would cross-serve wrongly-encoded tokens and trample delta baselines. The atomic result-id sequence can stay global. 3. Optional: `returns(ref)` instead of `returns(clone)` on the tracked query to avoid an O(tokens) clone per request; `handle_request_with_cancellation` for the new handlers. ## Sequence 1. This revert merges. 2. BoundaryML#4000 folds in canary (now conflict-free) and merges. 3. BoundaryML#3867 re-lands adapted — we're happy to prep the integration branch. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01VEMxti5WQ8bCgJUuX9AQvo <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added support for highlighting `dynamic`, `with`, and `throws` syntax. - Improved semantic highlighting for names, types, members, strings, numbers, and multi-part paths. - **Bug Fixes** - Updated catch-clause bindings to provide more consistent navigation and local-variable handling. - Adjusted parsing and formatting for contextual keywords such as `as` and `type`. - Corrected hover information for catch bindings. - **Breaking Changes** - Semantic tokens no longer include modifiers or range/delta responses. - Boolean and null values are no longer represented as literal semantic tokens. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
meefs
pushed a commit
to meefs/baml
that referenced
this pull request
Jul 13, 2026
…oundaryML#3867) (BoundaryML#4005) Stacked on BoundaryML#4002 (retargets to canary when it merges). Re-applies the one change in BoundaryML#3867 that was a standalone bug fix rather than part of the semantic-tokens feature, so the revert doesn't reintroduce it — addressing CodeRabbit's Critical finding on BoundaryML#4002. `find_token_after_hashes` could return the one-past-the-end index after a hash run at EOF, and `parse_raw_string` indexed `tokens` with it directly — a panic on incomplete input like a bare `##` at end of file, present on canary long before BoundaryML#3867. Under the new LSP core (BoundaryML#4000) a parser panic on the source path additionally poisons the project into a terminal broken state, so this class matters more now. Both hunks are byte-for-byte @codeshaunted's fix from BoundaryML#3867 (credited as co-author). New here: the regression test the original lacked — verified to panic (`index out of bounds: the len is 15 but the index is 15`) with the guards removed, and covering bare `#`/`##` at top level, inside expression bodies, and after `let`. Validation: `cargo test -p baml_compiler_parser` 140/140 (139 pre-existing + the new test), clippy `-D warnings` clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01VEMxti5WQ8bCgJUuX9AQvo <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved handling of incomplete raw-string syntax at end-of-file. * Prevented unexpected parser crashes when inputs end with standalone hash characters. * Malformed inputs (including truncated bare-hash sequences) now return parse errors instead of failing. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Avery Townsend <[email protected]> Co-authored-by: Claude Fable 5 <[email protected]>
pull Bot
pushed a commit
to justinlietz93/baml
that referenced
this pull request
Jul 14, 2026
…oc fixes (BoundaryML#4007) Stacked on BoundaryML#4000 (retargets to canary when it merges). Batches the four small items deferred from the BoundaryML#4000 review cycle so the approved, CI-green main PR stayed untouched. 1. **Per-file `PositionCodec` in `references` and `workspace/symbol`** — CodeRabbit's one valid Major on BoundaryML#4000: codec construction scans the entire target file to build its line table, and both handlers built one per *result*; a symbol with N usages in one file did N identical scans. Now one codec per distinct file via a lazy map inside each handler (the same pattern `document_symbol` already used). 2. **Memoization contract tests** (ported from the alternate implementation's test suite): `file_annotations` and `semantic_tokens` execute exactly once cold, zero times warm, and once again after a source edit — asserted by counting salsa `WillExecute` events, so future implementation rewrites (including the BoundaryML#3867 re-land) keep the tests meaningful. This also formally answers the stale "bypasses Salsa memoization" review claim on BoundaryML#3969 by pinning the invariant. 3. **`TEST_INSTRUCTIONS.md`**: the "Verify all tests pass" section now includes `cargo test --lib` (previously the documented flow skipped the workspace unit tests entirely — where all of BoundaryML#4000's new tests live), and two case-sensitive `UPDATE_EXPECT` references fixed (lowercase `update_expect=1` is a silent no-op for expect-test). Validation: `bex_project` 45/45, `baml_lsp2_actions_tests` 377/377 (incl. the 2 new memoization tests), clippy `-D warnings` clean, wasm check clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01VEMxti5WQ8bCgJUuX9AQvo <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Performance** * Improved responsiveness for “Find References” and workspace symbol searches, especially when results span multiple locations in the same file. * **Bug Fixes** * Strengthened language-server behavior verification to ensure cached results are reused correctly and refreshed after source changes. * **Documentation** * Clarified testing guidance for Rust changes and language-server snapshot updates. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Claude Fable 5 <[email protected]>
meefs
pushed a commit
to meefs/baml
that referenced
this pull request
Jul 14, 2026
…L#3867) on BoundaryML#4000 (BoundaryML#4011) Re-lands BoundaryML#3867 (reverted in BoundaryML#4002 to let BoundaryML#4000 land first) as a revert-of-revert on top of BoundaryML#4000, following the re-land guidance agreed in BoundaryML#4002. ## Byte-for-byte re-land The classification engine is unchanged from what BoundaryML#3867 shipped: the span-keyed resolution index, the CST classification walk, interface-member resolution, the contextual-keyword re-lexing (`as`/`type`/`true`/`false`/`null`), all 78 semantic-token fixtures, and the `tools_semantic_tokens` viewer. ## Adapted to BoundaryML#4000 (LSP boundary only) - **Negotiated encoding**: `encode_semantic_tokens` now encodes through the connection's `PositionCodec` — `delta_start`/`length` are negotiated code units (UTF-16 default, surrogate pairs handled), and multiline tokens split into per-line segments. This also fixes the byte-length miscount for non-ASCII identifiers that BoundaryML#3867 originally shipped (called out in BoundaryML#4002). - **Request idiom**: `semanticTokens/full`, `full/delta`, and `range` use `encoding_for_request` + `read_for_request` — bounded gate with ambient cancellation observed at entry / post-gate / completion — instead of `try_lock_db`. - **Connection-scoped delta cache**: `semantic_tokens_cache` is now per-connection (fresh in `connection_scoped_lsp_session`, next to the encoding negotiation), so concurrent clients can't cross-serve wrongly-encoded tokens or trample each other's `previous_result_id`. The `result_id` sequence stays process-global. - **`returns(ref)`**: the tracked `semantic_tokens` query keeps BoundaryML#4000's `returns(ref)` + `salsa::Update`, avoiding an O(tokens) clone per request. **Not re-landed**: BoundaryML#4002's incidental `pnpm-lock.yaml` refresh (fiddle-proxy entries for an untracked package) — unrelated to BoundaryML#3867 on both sides; the diff is strictly `baml_language/`-scoped. ## New tests - Negotiated-encoding token encoding: UTF-16 vs UTF-8 lengths, astral-plane surrogate pairs, multiline segment splitting (`request.rs`). - Connection-scoped token-cache freshness asserted in the session-scoping test (`mod.rs`). ## Validation - Full suite: 7486/7486 (nextest) + `baml_src baml_test` 2057/2057 standalone. - `bex_project` 52/52 incl. the new encoding + session tests. - clippy clean (workspace, all targets), `cargo check --target wasm32-unknown-unknown -p bex_project` clean. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added richer, type-aware semantic highlighting, including booleans, escape sequences, declarations, properties, methods, enum members, and library symbols. - Added range-based and delta semantic-token updates for faster editor refreshes. - Added a local semantic-token viewer with fixture comparison, scratchpad editing, token inspection, and snapshot acceptance. - Improved navigation to interface fields and methods, including aliased members. - **Bug Fixes** - Improved type information and completion support for destructuring and catch bindings. - Corrected contextual keyword and literal parsing, formatting, and highlighting. - Improved hover results for caught values. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
13 tasks done
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reworks the BAML semantic-tokens classifier to be resolution-driven, mirroring rust-analyzer: walk the lossless CST for token positions + syntactic tokens (keywords/operators/punctuation), and resolve identifier meaning through the HIR/inference instead of guessing from the CST.
What's here
tokens/index.rs— span-keyed resolution index over every inference-bearing scope (function / lambda / let) via the new uniformtir::scope_bodylookup, so identifiers insidespawn/block/closure bodies classify with their own scope's inference.resolve_interface_membernow recordsMemberResolution::InterfaceMethod/InterfaceField. So.as<I>.method(), interfaceSelfmethods, and chained interface calls classify through the normal path; typos stay neutral. MIR-safe (new variants →Noneitem-refs, fall through to existing interface dispatch; 323 runtime interface tests pass).as/with/iskeywords, generators, generic type args + params (typeParameter), associated types (decl + binding), self member accesses (no longernamespace), methods after.as<Interface>, string escapes (escapeSequence), byte strings, qualified type paths,true/false/null, object/map keys vs values.tools_semantic_tokens— standalone web viewer for the classifier (no VSCode needed).Tests
78 semantic-token fixtures; 439 LSP + 323 runtime-interface + 172 tir/mir tests pass; clippy-clean.
Next steps / gaps
See
crates/baml_lsp2_actions/src/tokens/GAPS.md— the priority is scaling like rust-analyzer (on-demand per-token resolution +semanticTokens/rangeviewport + delta), plus a couple remaining feature items (optional params at call site, abooleanliteral token type).Summary by CodeRabbit
New Features
Bug Fixes