Repository navigation
feat(literals): hex, octal, and binary int/bigint literals with underscore separators - #4065
Merged
Merged
Conversation
…score separators Add 0x/0o/0b integer and bigint literals plus _ digit separators in all numeric literals, following Rust's semantics and rustc's validation architecture: - The lexer deliberately over-accepts (0b/0o eat any decimal digits, prefixes also match uppercase, a bare 0x is one token) so malformed literals stay single tokens with good spans. - baml_base::num_lit is the single validator/parser for all literal token text; underscores are stripped there, and float text is normalized during AST lowering so downstream f64 parsing is unchanged. - New E0152 covers uppercase prefixes (with the fixed spelling), no digits, invalid digits (one diagnostic per digit, rustc-style), and integer literals above i64::MAX, which previously lowered silently to 0 via unwrap_or(0). The i63 int range check stays in TIR (E0150). - diags are threaded through lower_type_expr so type-level literals (0x1 | 0x2) validate too; the syntax literal accessors now return the raw token instead of parsed values. - Config and test-arg duck-typing contexts parse the new forms silently and still fall back to string. One deliberate divergence from Rust: a float exponent must start with a digit (1e_1 is not a float) because logos miscompiles a leading _* loop in the exponent, breaking backtracking for plain 1e10. Tests port rustc's numeric-literal corpus: lexer and num_lit unit tests, a diagnostic_errors/invalid_numeric_literals project, and run-pass fixtures in ns_ints/ns_bigints/ns_floats (hex bitwise ops, 0xFF.to_string() munch boundary, int.max_value() in hex, 2^80-1 bigint).
|
The latest updates on your projects. Learn more about Vercel for GitHub.
1 Skipped Deployment
|
⏭️ Performance benchmarks were skippedPerf benchmarks (CodSpeed) are opt-in on pull requests — they no longer run on every push. They always run automatically after merge to To run them on this PR, do any of the following, then push a commit (or re-run CI):
|
Contributor
|
Important Review skippedReview was skipped due to path filters ⛔ Files ignored due to path filters (1)
CodeRabbit blocks several paths by default. You can override this behavior by explicitly including those paths in the path filters. For example, including ⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughNumeric literal handling now supports base prefixes and underscore separators across lexing, shared parsing, AST lowering, type lowering, diagnostics, documentation, and regression tests. Invalid literals produce structured ChangesNumeric literal support
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant Source
participant Lexer
participant ASTLowering
participant Diagnostics
Source->>Lexer: tokenize numeric literal
Lexer->>ASTLowering: preserve literal token text
ASTLowering->>ASTLowering: parse and normalize numeric value
ASTLowering->>Diagnostics: emit E0152 for invalid literal
Possibly related PRs
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
baml_language/crates/baml_base/src/num_lit.rs (1)
133-134: 🎯 Functional Correctness | 🔵 TrivialRun
cargo test --libfrombaml_language/The repository root has no
Cargo.toml; usebaml_language/Cargo.toml(or--manifest-path) so the unit tests actually run.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@baml_language/crates/baml_base/src/num_lit.rs` around lines 133 - 134, Update the test workflow for the tests module in num_lit.rs to run cargo test --lib from the baml_language manifest, using baml_language/Cargo.toml or --manifest-path instead of the repository root.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@baml_language/crates/baml_base/src/num_lit.rs`:
- Around line 133-134: Update the test workflow for the tests module in
num_lit.rs to run cargo test --lib from the baml_language manifest, using
baml_language/Cargo.toml or --manifest-path instead of the repository root.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: 4c4ec820-b9d1-43ca-9004-837563fff1d1
⛔ Files ignored due to path filters (15)
baml_language/crates/baml_tests/snapshots/baml_src/bigints.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/baml_src/floats.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/baml_src/ints.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/compiles/catch_arm_return/baml_tests__compiles__catch_arm_return__01_lexer__catch_arm_return.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/compiles/numeric_literal_method_call/baml_tests__compiles__numeric_literal_method_call__01_lexer__numeric_literal_method_call.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/invalid_numeric_literals/baml_tests__diagnostic_errors__invalid_numeric_literals__01_lexer__negative.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/invalid_numeric_literals/baml_tests__diagnostic_errors__invalid_numeric_literals__01_lexer__positive.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/invalid_numeric_literals/baml_tests__diagnostic_errors__invalid_numeric_literals__02_parser__negative.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/invalid_numeric_literals/baml_tests__diagnostic_errors__invalid_numeric_literals__02_parser__positive.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/invalid_numeric_literals/baml_tests__diagnostic_errors__invalid_numeric_literals__03_hir.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/invalid_numeric_literals/baml_tests__diagnostic_errors__invalid_numeric_literals__04_tir.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/invalid_numeric_literals/baml_tests__diagnostic_errors__invalid_numeric_literals__05_diagnostics.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/invalid_numeric_literals/baml_tests__diagnostic_errors__invalid_numeric_literals__10_formatter__negative.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/invalid_numeric_literals/baml_tests__diagnostic_errors__invalid_numeric_literals__10_formatter__positive.snapis excluded by!**/*.snapbaml_language/crates/baml_tests/snapshots/diagnostic_errors/json_static_method_arity/baml_tests__diagnostic_errors__json_static_method_arity__01_lexer__json_static_method_arity.snapis excluded by!**/*.snap
📒 Files selected for processing (19)
baml_language/crates/baml_base/src/lib.rsbaml_language/crates/baml_base/src/num_lit.rsbaml_language/crates/baml_builtins2/baml_std/baml/bigint.bamlbaml_language/crates/baml_builtins2/baml_std/baml/int.bamlbaml_language/crates/baml_compiler2_ast/src/lib.rsbaml_language/crates/baml_compiler2_ast/src/lower_config_item.rsbaml_language/crates/baml_compiler2_ast/src/lower_cst.rsbaml_language/crates/baml_compiler2_ast/src/lower_expr_body.rsbaml_language/crates/baml_compiler2_ast/src/lower_type_expr.rsbaml_language/crates/baml_compiler2_ast/src/lowering_diagnostic.rsbaml_language/crates/baml_compiler_diagnostics/src/diagnostic.rsbaml_language/crates/baml_compiler_lexer/src/tokens.rsbaml_language/crates/baml_compiler_syntax/src/ast.rsbaml_language/crates/baml_compiler_syntax/src/tests.rsbaml_language/crates/baml_tests/baml_src/ns_bigints/bigints.bamlbaml_language/crates/baml_tests/baml_src/ns_floats/floats.bamlbaml_language/crates/baml_tests/baml_src/ns_ints/ints.bamlbaml_language/crates/baml_tests/projects/diagnostic_errors/invalid_numeric_literals/negative.bamlbaml_language/crates/baml_tests/projects/diagnostic_errors/invalid_numeric_literals/positive.baml
Binary size checks passed✅ 7 passed
Generated by |
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds `0xFF`, `0o755`, `0b1010` integer and bigint (`0xFFn`) literals, plus `_` digit separators in all numeric literals (`1_000_000`, `1_000.5`, `1e1_0`, `1_000n`), following Rust's semantics.
How
Studied rustc's implementation and ported its architecture (no code copied — ours is logos-based):
Bug fixed along the way
`let x = 99999999999999999999` previously lowered silently to 0 (`parse::().unwrap_or(0)`). It now errors with E0152 "integer literal is too large". The i63 `int` range check is unchanged (TIR E0150), so the two layers compose: > 2^62-1 → E0150, > i64::MAX → E0152.
Deliberate divergence from Rust
A float exponent must start with a digit: `1e_1` is not a float (`1e1_` is fine). logos miscompiles a leading `_*` loop in the exponent pattern and breaks backtracking for plain `1e10`/`1e`. Documented on the `FloatLiteral` regex.
Tests
Ported rustc's numeric-literal test corpus:
Three pre-existing lexer snapshots changed benignly: string interiors like `"05_diagnostics"` now tokenize as `05_` + `diagnostics` instead of `05` + `_diagnostics`; parser reassembly is identical (no parser/diagnostics snapshots moved).
Summary by CodeRabbit
New Features
Documentation
Tests