Use proper result set when evaluating parameters - #782
Conversation
|
| } | ||
|
|
||
| for (let i = 0; i < this.lookupStages.length; i++) { | ||
| // Within a stage, we can resolve lookups concurrently. |
There was a problem hiding this comment.
With the implementation from this PR, joins are no longer processed concurrently. It's possible to add that back with minor added complexity, but:
- this is only relevant for complex sync streams
- we already evaluate other users / queriers concurrently, to the point where bucket storage is likely the bottleneck and not JS
So I don't think this is necessarily something worth doing, but I can change this here / in a follow-up PR if needed.
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 68a4c2dc8b
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: af866d1014
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
1 participant
When querying buckets for Sync Streams, we generally try to resolve parameters independently to form a cartesian product in the end. This is correct for most streams, but goes wrong when a parameter index has more than one column. For example, in
SELECT a.* FROM a, b WHERE a.c1 = b.c1 AND a.c2 = b.c2 AND b.u = auth.user_id(), the two parameters areb.c1andb.c2. If we encounter multiple rows ofbthrough a lookup result, we can't assume those to be independent parameters though! We can only pair parameters that originate from the same row.This is currently implemented by tracking provenance for each parameter value back to the lookup this originally came from. When we build the cartesian product in the end, we ignore values with incompatible provenance from different rows. Unfortunately, tracking provenance is both kind of expensive and very tricky to get right.
Semantically, evaluating bucket parameters involves:
This replaces the previous querier logic with an actual result set implementation: We start out with a unit set of one row without columns, then go through added lookups that are cross-joined (table-valued functions) or inner-joined (parameter lookups). If we end up with an empty intermediate result set at any point, we know there won't be any buckets and bail out early.
Intersection parameters require special consideration now, but can be implemented by going through the result set and deleting rows where the columns don't match.
AI use: The approach is manual, most tests and some implementation details are generated with Claude Code.