PROT-5778: support identifiers that begin with digits - #60
Open
MichaelGHSeg wants to merge 1 commit into
Open
Conversation
The lexer dispatched any digit-leading token to lexNumber, which
consumed the digit run and returned a Number without checking what
followed. An identifier like "1_x_coffee_buyer" therefore lexed as
Number(1) followed by Ident(_x_coffee_buyer), which then failed to
parse ("Unexpected token in statement").
lexNumber now hands off to lexIdent when the token is followed by
identifier characters, so the whole thing lexes as a single Ident.
The handoff is guarded on the token containing a digit: the dispatcher
also routes a bare sign here, so "-foo" keeps its previous behavior.
JSON property names are arbitrary customer data rather than
programmer-chosen symbols, so leading digits are legitimate. This
mirrors the equivalent fix in the Go implementation
(segmentio/fql#86) -- both now lex "friends.1.last" identically.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
lexNumberdispatched on the first character, so any digit-leading token was consumed as a number without checking what followed. An identifier like1_x_coffee_buyerlexed asNumber(1)+Ident(_x_coffee_buyer)and then failed to parse.It now hands off to
lexIdentwhen the token is followed by identifier characters, so the whole thing lexes as oneIdent.JSON property names are arbitrary customer data, not programmer-chosen symbols, so leading digits are legitimate input.
Context
1_x_coffee_buyer).@segment/fql-ts, consumed bypackages/appandpackages/gateway-api, so the app would still have rejected these expressions even with the other two merged.segmentio/fql-wasmwraps the Go implementation and picks the fix up via a version bump, so it needs no code change.Note on the guard
The handoff is guarded on the token containing a digit. Unlike the Go dispatcher, this one routes a bare
-/+tolexNumberwithout requiring a digit to follow, so-fooreaches this path; the guard keeps it behaving as before rather than silently becoming an identifier.Test plan
Lexer passes digit-first ident fixturesblock:1val,123audience,1_x_coffee_buyer,1-x-coffee-buyer, plus dotted-path formsNumber(123,123 456)-4,+5,+5.4,-3.2,0.4,10 10yarn lintcleanfriends.1.lastnow lexes identically to the Go implementation