Repository navigation
Measure CometRegex whitelist coverage on representative regex patterns #5812
Description
What is the problem the feature request solves?
The initial CometRegex whitelist introduced by PR #5415 is deliberately conservative. It rejects commonly used constructs such as ., ^, $, \d, \w, and \s.
We currently do not know what fraction of representative real-world rlike patterns is admitted for automatic native execution. TPC-DS does not contain useful rlike coverage, so it cannot answer this question.
Without an admission-rate measurement, it is difficult to prioritize follow-up compatibility work or estimate how much real-world workload coverage the whitelist provides.
Describe the potential solution
Identify one or more representative regex corpora, such as public SQL workloads, open-source application queries, or safely shareable production-derived patterns.
Run every literal pattern through CometRegex and report:
- the corpus source and collection method;
- total, admitted, and rejected pattern counts;
- the admission percentage;
- duplicate-handling methodology;
- rejection counts grouped by unsupported construct or analyzer rule;
- useful pattern-length and complexity distributions.
Use the results to prioritize follow-up work such as support for leading ^ or safe normalization of selected Java regex constructs.
The whitelist should not be weakened solely to increase the reported admission percentage.
Additional context
Follow-up from #5415 and #5351.
This tracks the representative admission-rate measurement requested during review of #5415: #5415 (review)
The original issue notes that the TPC query corpus does not contain rlike usage: #5351
Activity
sam-1112 commentedon Nov 11, 2026
take
sam-1112 commentedon Nov 11, 2026
I have preliminary exploratory results for this issue and would like feedback on the corpus scope before opening a PR.
The tooling reads commit-pinned public sources and measures admission with the unchanged
CometRegex.supportLevel()(RegexFlavor.RLike). Python call-site patterns come from AST string values after receiver review, with no SQL unescaping. Spark and Hive fixture literals are decoded with the Spark and Hive APIs and checked withjava.util.regex. Occurrences are source locations; unique patterns are exact decoded strings, with no regex canonicalization. The whitelist and runtime behavior are unchanged.The only qualified application sample is Gentropy, one genomics repository. Its patterns mostly check alleles, identifiers, or dates:
Four rejected Gentropy occurrences (three patterns) contain both
^and$. In analysis-only syntax controls, removing only the leading^still leaves all four rejected. The other rejected occurrence contains\d. This describes the sample; it does not propose a whitelist change.Other sources investigated:
python_mozetl, Fink broker, and Fink filters: no raw predicate keywords in the scoped packages.has_pattern()implementation usesregexp_extract, so those examples do not measure RLike coverage. Two intended RLIKE predicates have fixed-looking pattern text in SQL that also interpolates an unchecked column expression, so they are classified asUNKNOWNand excluded from the admission denominator.Spark SQL and Hive fixtures were measured on separate denominators and are not pooled with the application rate. The two Hive fixture groups contain the same 19 eligible unique decoded patterns. The reports include exclusions, rejection-feature analysis, length and structure distributions, and reproduction instructions.
This sample is too small and too narrowly selected to describe representative Spark workload coverage. The small number of checked-in literal patterns is a finding of this bounded survey; it does not show how often Spark applications use regex.
A PR would add the dev-only tooling and aggregate reports under
dev/regex-coverage/. It would not vendor upstream SQL, change production code, or close this issue.Would that exploratory PR be useful now, or should I broaden the application sample first? The extractor measures direct
Column.rlikeliterals, verifiedfunctions.litpatterns, and reviewed rule constructors. Notebooks, genericF.expr, and runtime-supplied patterns stay outside the measured set. Are there particular redistributable public Spark workloads, with checked-in RLIKE literals outside the repositories above, that you would recommend?