Skip to content

fix(parser): replace string placeholder mutation with token-preserving JSON parser in MqlParser #1513

Open

Description

@ZhuchkaTriplesix

Summary

In lib/features/mongodb/mql_parser.dart (lines 610–672):

_normalizeJsObject() transforms relaxed JavaScript object literal syntax (e.g. unquoted keys { age: 25 }, ObjectId(...), ISODate(...)) into strict JSON for dart:convert's json.decode.

To protect string literals from regex manipulation, it temporarily replaces them with placeholder strings:

final placeholder = '@@@STR_${strings.length}@@@';
strings.add(json.encode(currentStr.toString()));
tokenized.write(placeholder);
...
// Later restores them:
processed = processed.replaceAll('@@@STR_$i@@@', strings[i]);

This introduces data corruption hazards when user queries contain @@@STR_ or complex JavaScript structures.

Architectural Solution: Specialized Micro-Scanner Functions

Replace naive placeholder mutation with a set of modular, micro-optimized scanner functions:

  1. static int _scanJsString(String input, int start, StringBuffer out)
    • Reads single- or double-quoted strings, handles escape sequences (\\', \\"), and writes valid JSON string escaping directly to out.
  2. static ({Object? value, int end})? _scanBsonConstructor(String input, int start)
    • Micro-scanner for MongoDB helper constructors:
      • ObjectId("..." )
      • ISODate("..." )
      • NumberLong(...) / NumberInt(...)
      • RegExp(...)
  3. static int _scanUnquotedKey(String input, int start, StringBuffer out)
    • Scans unquoted JavaScript keys ([a-zA-Z0-9_$]+) followed by a colon and wraps them as "key": directly in the output stream without global regexes.
  4. static int _skipWhitespace(String input, int start)
    • Fast pointer advancement using codeUnitAt.

This turns the parser into a reliable, single-pass forward stream transformer with zero risk of marker collision.

Acceptance Criteria

  • JavaScript object literals are parsed via modular micro-scanner functions.
  • No synthetic sentinel strings (such as @@@STR_) or global replaceAll calls are used.
  • User queries containing unusual characters or literals like @@@STR_ parse without corruption.
  • Unit tests added covering BSON constructors and quote edge cases in test/features/mongodb/mql_parser_test.dart.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P3Low priority / Polish & EnhancementsbackendBackend database driver execution and queriesbugSomething isn't workingparserTheme parser epic label: parser

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions