Skip to content

PROT-5778: support identifiers that begin with digits - #60

Open
MichaelGHSeg wants to merge 1 commit into
masterfrom
mgh/prot-5778-digit-first-idents
Open

PROT-5778: support identifiers that begin with digits#60
MichaelGHSeg wants to merge 1 commit into
masterfrom
mgh/prot-5778-digit-first-idents

Conversation

@MichaelGHSeg

@MichaelGHSeg MichaelGHSeg commented Sep 9, 2026

Copy link
Copy Markdown

Summary

lexNumber dispatched on the first character, so any digit-leading token was consumed as a number without checking what followed. An identifier like 1_x_coffee_buyer lexed as Number(1) + Ident(_x_coffee_buyer) and then failed to parse.

It now hands off to lexIdent when the token is followed by identifier characters, so the whole thing lexes as one Ident.

properties.1_x_coffee_buyer = true

before:  ident(properties) dot number(1) ident(_x_coffee_buyer) operator(=) ident(true)
         -> ast error: "Unexpected token in statement"

after:   ident(properties) dot ident(1_x_coffee_buyer) operator(=) ident(true)
         -> parses

JSON property names are arbitrary customer data, not programmer-chosen symbols, so leading digits are legitimate input.

Context

  • Jira: PROT-5778
  • Customer impact: The Knot Worldwide / Nestlé could not build destination filter triggers on audience keys beginning with a number (e.g. 1_x_coffee_buyer).
  • This is the third of three places the same bug had to be fixed, since FQL has three implementations:
    • Go lexer — segmentio/fql#86
    • Subscription-editor regex — segmentio/app#26243
    • this PR@segment/fql-ts, consumed by packages/app and packages/gateway-api, so the app would still have rejected these expressions even with the other two merged.
  • segmentio/fql-wasm wraps the Go implementation and picks the fix up via a version bump, so it needs no code change.

Note on the guard

The handoff is guarded on the token containing a digit. Unlike the Go dispatcher, this one routes a bare -/+ to lexNumber without requiring a digit to follow, so -foo reaches this path; the guard keeps it behaving as before rather than silently becoming an identifier.

Test plan

  • New Lexer passes digit-first ident fixtures block: 1val, 123audience, 1_x_coffee_buyer, 1-x-coffee-buyer, plus dotted-path forms
  • Asserts purely numeric tokens are still Number (123, 123 456)
  • Existing Number fixtures unchanged and passing, including -4, +5, +5.4, -3.2, 0.4, 10 10
  • Full suite green (69 tests, 7 suites); yarn lint clean
  • Verified friends.1.last now lexes identically to the Go implementation

The lexer dispatched any digit-leading token to lexNumber, which
consumed the digit run and returned a Number without checking what
followed. An identifier like "1_x_coffee_buyer" therefore lexed as
Number(1) followed by Ident(_x_coffee_buyer), which then failed to
parse ("Unexpected token in statement").

lexNumber now hands off to lexIdent when the token is followed by
identifier characters, so the whole thing lexes as a single Ident.
The handoff is guarded on the token containing a digit: the dispatcher
also routes a bare sign here, so "-foo" keeps its previous behavior.

JSON property names are arbitrary customer data rather than
programmer-chosen symbols, so leading digits are legitimate. This
mirrors the equivalent fix in the Go implementation
(segmentio/fql#86) -- both now lex "friends.1.last" identically.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant