Repository navigation
test(baml_tests): add native coverage for Rust VM tests - #4234
Closed
codeshaunted wants to merge 2 commits into
Closed
codeshaunted wants to merge 2 commits into
codeshaunted wants to merge 2 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
⏭️ Performance benchmarks were skippedPerf benchmarks (CodSpeed) are opt-in on pull requests — they no longer run on every push. They always run automatically after merge to To run them on this PR, do any of the following, then push a commit (or re-run CI):
|
Contributor
📝 WalkthroughWalkthroughThe PR adds BAML test coverage for ErrorContext rendering, reflection dispatch and signatures, union and interface type handling, typed JSON serialization, string parsing, and escaped/raw string literal semantics, while removing corresponding Rust integration tests. ChangesLanguage behavior regression tests
Estimated code review effort: 4 (Complex) | ~45 minutes Possibly related PRs
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@baml_language/crates/baml_tests/baml_src/ns_errorcontext/errorcontext.baml`:
- Around line 516-520: Update errorcontext_budget_rendering_check to assert that
wide_hex contains exactly two truncation markers by requiring
wide_hex.split(marker).length() == 3, while preserving the existing deep_hex and
wide_hex marker checks.
In
`@baml_language/crates/baml_tests/baml_src/ns_parse_companions/parse_companions.baml`:
- Around line 67-68: Update the function directives in ParseCompanionsTestClient
usage by removing the colons from client and prompt, matching the syntax used by
the sibling ParseCompanionsParseAliasPayload function so the BAML parser accepts
them.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 5e3ee9b6-4dfb-4ebc-8aec-421eeaf03e3c
⛔ Files ignored due to path filters (9)
baml_language/crates/baml_tests/snapshots/baml_src/_root.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/baml_src/errorcontext.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/baml_src/generic_union_returns.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/baml_src/interface_type_alias_iteration.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/baml_src/json_typed_union.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/baml_src/narrow_bind.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/baml_src/parse_companions.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/baml_src/reflect_call_any.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/baml_src/string_literals.snapis excluded by!**/*.snap
📒 Files selected for processing (8)
baml_language/crates/baml_tests/baml_src/ns_errorcontext/errorcontext.bamlbaml_language/crates/baml_tests/baml_src/ns_generic_union_returns/generic_union_returns.bamlbaml_language/crates/baml_tests/baml_src/ns_interface_type_alias_iteration/interface_type_alias_iteration.bamlbaml_language/crates/baml_tests/baml_src/ns_json_typed_union/json_typed_union.bamlbaml_language/crates/baml_tests/baml_src/ns_narrow_bind/narrow_bind.bamlbaml_language/crates/baml_tests/baml_src/ns_parse_companions/parse_companions.bamlbaml_language/crates/baml_tests/baml_src/ns_reflect_call_any/reflect_call_any.bamlbaml_language/crates/baml_tests/baml_src/ns_string_literals/string_literals.baml
Comment on lines
+516
to
+520
| function errorcontext_budget_rendering_check() -> bool { | ||
| let deep_hex = errorcontext_render_deep().to_utf8().to_hex(); | ||
| let wide_hex = errorcontext_render_wide().to_utf8().to_hex(); | ||
| let marker = "e280a6"; | ||
| deep_hex.includes(marker) && wide_hex.includes(marker) && wide_hex.split(marker).length() == 2 |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Assert two truncation markers correctly.
Line 520 accepts exactly one marker: two occurrences produce three split parts. This lets a single-marker render pass while rejecting the intended two-marker output.
Proposed fix
- deep_hex.includes(marker) && wide_hex.includes(marker) && wide_hex.split(marker).length() == 2
+ deep_hex.includes(marker) && wide_hex.includes(marker) && wide_hex.split(marker).length() == 3📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| function errorcontext_budget_rendering_check() -> bool { | |
| let deep_hex = errorcontext_render_deep().to_utf8().to_hex(); | |
| let wide_hex = errorcontext_render_wide().to_utf8().to_hex(); | |
| let marker = "e280a6"; | |
| deep_hex.includes(marker) && wide_hex.includes(marker) && wide_hex.split(marker).length() == 2 | |
| function errorcontext_budget_rendering_check() -> bool { | |
| let deep_hex = errorcontext_render_deep().to_utf8().to_hex(); | |
| let wide_hex = errorcontext_render_wide().to_utf8().to_hex(); | |
| let marker = "e280a6"; | |
| deep_hex.includes(marker) && wide_hex.includes(marker) && wide_hex.split(marker).length() == 3 |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@baml_language/crates/baml_tests/baml_src/ns_errorcontext/errorcontext.baml`
around lines 516 - 520, Update errorcontext_budget_rendering_check to assert
that wide_hex contains exactly two truncation markers by requiring
wide_hex.split(marker).length() == 3, while preserving the existing deep_hex and
wide_hex marker checks.
Binary size checks passed✅ 7 passed
Generated by |
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@baml_language/crates/baml_tests/src/string_literals.rs`:
- Around line 10-15: Update the string-literal error assertions in the test
cases around the cases array to verify the diagnostic kind is UnexpectedEof and
its message is “unexpected end of file” for each unterminated-string fixture,
rather than accepting any error; retain the existing labels for failure context.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 8172495c-efb7-4a2a-8cc8-dbbf29cb0d81
📒 Files selected for processing (8)
baml_language/crates/baml_tests/src/string_literals.rsbaml_language/crates/baml_tests/tests/errorcontext.rsbaml_language/crates/baml_tests/tests/generic_union_returns.rsbaml_language/crates/baml_tests/tests/interface_type_alias_iteration.rsbaml_language/crates/baml_tests/tests/json_typed_union.rsbaml_language/crates/baml_tests/tests/narrow_bind.rsbaml_language/crates/baml_tests/tests/parse_companions.rsbaml_language/crates/baml_tests/tests/reflect_call_any.rs
💤 Files with no reviewable changes (7)
- baml_language/crates/baml_tests/tests/interface_type_alias_iteration.rs
- baml_language/crates/baml_tests/tests/json_typed_union.rs
- baml_language/crates/baml_tests/tests/narrow_bind.rs
- baml_language/crates/baml_tests/tests/reflect_call_any.rs
- baml_language/crates/baml_tests/tests/generic_union_returns.rs
- baml_language/crates/baml_tests/tests/parse_companions.rs
- baml_language/crates/baml_tests/tests/errorcontext.rs
Comment on lines
10
to
15
| let cases = [ | ||
| // Unterminated string — no closing quote at all | ||
| ( | ||
| r#"function f() -> string { "hello }"#, | ||
| "unterminated string", | ||
| ), | ||
| // Backslash at EOF — backslash escapes nothing, string never closes | ||
| (r#"function f() -> string { "hello\"#, "backslash at EOF"), |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Assert the expected diagnostic, not merely any error.
label is only printed on failure; any unrelated parse or semantic error makes these regressions pass. Assert the expected UnexpectedEof/"unexpected end of file" diagnostic for each fixture.
Proposed fix
let cases = [
(
r#"function f() -> string { "hello }"#,
- "unterminated string",
+ "unterminated string",
+ "unexpected end of file",
),
- (r#"function f() -> string { "hello\"#, "backslash at EOF"),
+ (
+ r#"function f() -> string { "hello\"#,
+ "backslash at EOF",
+ "unexpected end of file",
+ ),
];
-for (source, label) in &cases {
+for (source, label, expected_message) in &cases {
let db = setup_test_db(source);
let diagnostics = collect_diagnostics(&db);
- let has_error = diagnostics
+ let has_expected_error = diagnostics
.iter()
- .any(|d| matches!(d.severity, Severity::Error));
+ .any(|d| {
+ matches!(d.severity, Severity::Error)
+ && d.message.to_string().contains(expected_message)
+ });
assert!(
- has_error,
+ has_expected_error,📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| let cases = [ | |
| // Unterminated string — no closing quote at all | |
| ( | |
| r#"function f() -> string { "hello }"#, | |
| "unterminated string", | |
| ), | |
| // Backslash at EOF — backslash escapes nothing, string never closes | |
| (r#"function f() -> string { "hello\"#, "backslash at EOF"), | |
| let cases = [ | |
| ( | |
| r#"function f() -> string { "hello }"#, | |
| "unterminated string", | |
| "unexpected end of file", | |
| ), | |
| ( | |
| r#"function f() -> string { "hello\"#, | |
| "backslash at EOF", | |
| "unexpected end of file", | |
| ), | |
| ]; |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@baml_language/crates/baml_tests/src/string_literals.rs` around lines 10 - 15,
Update the string-literal error assertions in the test cases around the cases
array to verify the diagnostic kind is UnexpectedEof and its message is
“unexpected end of file” for each unterminated-string fixture, rather than
accepting any error; retain the existing labels for failure context.
4 tasks done
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Scope note
The forced-GC cleanup case remains Rust-only. Native top-level tests run in parallel, and this case was not reliable when forced collection overlapped the wider corpus.
Testing
Summary by CodeRabbit
$parseJSON vs plain text, string literal escaping/raw strings, and generic union/nullable handling.