From 14ad05938fc06db3435d6a6cb2ca6d275e64e1ea Mon Sep 17 00:00:00 2001 From: TimelordUK Date: Sat, 12 Sep 2026 21:08:41 +0100 Subject: [PATCH 1/2] docs(parity): log P46-P48, found writing the countries examples Three divergences, all found asking countries.csv ordinary questions rather than by widening the corpus. Two are silent wrong answers. P46 - a column-to-column comparison in WHERE returns zero rows. Not one operator or one type: `WHERE region = region` matches nothing. The AST is correct, so it is the WHERE evaluator, and the same comparison projected in the SELECT list is right. Same "zero rows reads as no data" disguise as P30. P47 - MIN/MAX compare by type variant before value. DataValue derives PartialOrd, so every Float outranks every Integer for MAX (and the reverse for MIN) whatever the numbers are: MAX(area) over countries.csv answers 34.2 instead of Russia's 17098242. compare_datavalues already does this correctly, which is why ORDER BY and the batched window path agree with DuckDB. Also recorded: SQL_CLI_BATCH_WINDOW=0 changes the answer, because the row-wise window path shares the bare comparison. P48 - a comparison with a NULL operand projects false instead of NULL. The P18/P19 three-valued rule at a site the 2026-08-22 fix did not reach. Found while building P46's anchor case: the control that was meant to AGREE came back DIFFER for an unrelated reason. Corpus grows by seven cases (six DIFFER, one AGREE anchor) across tiers 01, 02 and 10; the contract holds at 188 cases, 157 AGREE. Verified against the pinned DuckDB 1.5.5. Also updated: - P40's "what works today" understated itself. CROSS JOIN a slot table and take one SPLIT_PART per slot, and a bounded explode works today with the generator running once - it sidesteps both stacked defects rather than working around them. Recorded with the SQL and its two honest costs. - P45 gains the CROSS JOIN spelling, which fails with a different message. - NEXT in the fix-order table is now P46-P48; the previous NEXT is "then". - A fourth discovery lesson: writing examples is discovery. Examples are written to be interesting rather than exhaustive, which is how they wander outside the tested envelope. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01GfDkQm9s234uEDhdX6rvmf --- docs/SQL_PARITY.md | 214 +++++++++++++++++- tests/comparison/corpus/01_select.toml | 22 ++ tests/comparison/corpus/02_where.toml | 22 ++ .../comparison/corpus/10_aggregate_nulls.toml | 34 +++ 4 files changed, 285 insertions(+), 7 deletions(-) diff --git a/docs/SQL_PARITY.md b/docs/SQL_PARITY.md index 1b46e35d..f646ae7f 100644 --- a/docs/SQL_PARITY.md +++ b/docs/SQL_PARITY.md @@ -65,9 +65,9 @@ session apiece to fix properly, so **discovery is paused and the effort moves to picking them off**. Widen the corpus again when the open list is short, or opportunistically when a fix needs a case that doesn't exist yet. -Corpus coverage today: tiers 01–10, **181 cases** (156 AGREE / 10 DIFFER / -12 GAP / 1 OURS_ONLY / 2 BOTH_ERR as of 2026-09-06, after [P41](#p41) added four -MODE cases). The largest single movement so far remains the 2026-09-05 +Corpus coverage today: tiers 01–10, **188 cases** (157 AGREE / 16 DIFFER / +12 GAP / 1 OURS_ONLY / 2 BOTH_ERR as of 2026-09-12, after [P46](#p46)–[P48](#p48) +added seven — six DIFFER and one AGREE anchor). The largest single movement so far remains the 2026-09-05 NULL-ordering slice, which closed [P13](#p13) stage 2 and [P17](#p17) together — eleven cases in one change. **Tier 10 (aggregate & NULL edges) is still deliberately partial** — it holds the P14, P18–P20 and P41 cases and their @@ -91,7 +91,8 @@ Suggested fix order, by silent blast radius: | ~~9c~~ | ~~[P17](#p17) + [P13](#p13) stage 2 — NULL ordering~~ | ✅ **Fixed 2026-09-05** — 141 → **152 AGREE**, eleven cases in one change. Both halves were the same comparator's NULL rule, so they were taken as one slice. The two sorts that disagreed with each other now *share* one function (`compare_for_order_by`), which is the part that stops the divergence recurring; the window site turned out to be sorting NULL as the **maximum** via a derived `PartialOrd`, not merely following a different rule | | ~~9d~~ | ~~[P37](#p37) window in `WHERE` returns 0 rows~~ | ✅ **Fixed 2026-09-05** — corpus count unchanged, and that is the finding: the case is `OURS_ONLY` before *and* after, so the harness cannot see this fix or a future regression of it (first entry of that kind — the regression test is a Rust module). The filed root cause was wrong: `ExpressionLifter` *does* lift from `WHERE`. The real defect was one arm in the WHERE evaluator answering FALSE for any bare value used as a predicate — `WHERE true` returned zero rows too. It did **not** close [P15](#p15), which needs the opposite change | | ~~9e~~ | ~~[P41](#p41) `MODE` tie-break is random per run~~ | ✅ **Fixed 2026-09-06** — 152 → **156 AGREE** (four new cases). Small, as predicted, but not where it was filed: the named `ModeState` was a *shadowed* implementation and fixing it moved nothing. Reference does specify a rule and it is **first-occurrence**, not the "smallest value wins" this row proposed. Unblocked both example files, now FORMAL. Spun off [P42](#p42), [P43](#p43), [R12](ENGINE_REFACTORING.md#r12) | -| **NEXT** | [P14](#p14), [P20](#p20), [P23](#p23) | Smaller, self-contained, decisions already taken. Was row 9b | +| **NEXT** | [P46](#p46) column-vs-column `WHERE` returns 0 rows, [P47](#p47) `MIN`/`MAX` ranked by type, [P48](#p48) NULL comparison projects `false` | All three found 2026-09-12 writing one examples file, all three silent. P46 is the [P30](#p30) disguise again — zero rows reads as "no data" — and P47 hands back a number wrong by a factor of half a million from an ordinary `GROUP BY`. Both are cheap: P47's correct comparator is already in the tree, and P48 is the [P18](#p18)/[P19](#p19) rule at one more site. P46 and P48 sit in neighbouring code, so take them together | +| then | [P14](#p14), [P20](#p20), [P23](#p23) | Smaller, self-contained, decisions already taken. Was row 9b, then NEXT until the 2026-09-12 findings displaced it | | 9f | [P42](#p42) `MODE` is numeric-only | Companion to [R12](ENGINE_REFACTORING.md#r12), and cheap if taken with it: the shadowed implementation already handles non-numerics and preserves type, so the fix is largely to stop the live path throwing away what it knows. Also buys the corpus its clearest tie-break case | | 10 | [P22](#p22), [P25](#p25), [P26](#p26), [P15](#p15), [P32](#p32), [P38](#p38) | Hard errors — visible, so less urgent than any of the above | | 10b | [P39](#p39) `x/0` errors, voiding the whole statement | Hard error like row 10, but the only one whose blast radius is the *query* rather than the cell. Settle the four inconsistent call sites as one decision; it currently has no live probe (see the entry) | @@ -118,6 +119,16 @@ the fix; the entry describes the symptom that was looked for, not necessarily th defect. The same session's sweep for other consumers of the same route turned up [P31](#p31), a live zero-rows bug in the `--limit` flag that nobody had reported. +**A fourth, from P46–P48 (2026-09-12): writing examples is discovery.** Those +three came out of one afternoon writing `examples/countries.sql` — not from the +corpus, not from a fix, but from asking a real dataset ordinary questions +("which borders cross a continent", "largest country per region"). Two were +silent wrong answers in shapes the corpus did not cover, and the corpus grew by +seven cases as a result. The examples directory is cheap discovery surface and +worth mining deliberately when the open list runs short: unlike the corpus, it +is written to be *interesting* rather than to be exhaustive, which is exactly how +it wanders outside the tested envelope. + **A fourth lesson, from closing P41 (2026-09-06): the entry named a fix site, and the fix site was dead code.** `ModeState` in `src/sql/aggregates/mod.rs` is exactly what you find by grepping for MODE, and it is shadowed — a second, newer @@ -1762,8 +1773,33 @@ out of date. emits `value,index`, i.e. `WITH ORDINALITY` for free, which is worth keeping. - **What works today:** `SPLIT_PART(path,'/',n)` — scalar, resolves columns - normally, fine for fixed positions. There is no way to handle - arbitrary-depth paths. + normally, fine for fixed positions. + +- **A *bounded* explode also works today, and this entry undersold it + (added 2026-09-12).** `CROSS JOIN` a table of slot numbers onto the rows and + take one part per slot: + + ```sql + WITH slots AS (SELECT value AS n FROM RANGE(1, 20)), + c AS (SELECT * FROM #countries) + SELECT cca3, SPLIT_PART(borders, ',', n) AS neighbour + FROM c + CROSS JOIN slots + WHERE borders <> '' AND SPLIT_PART(borders, ',', n) <> ''; + ``` + + That turns 250 country rows into 649 border pairs. Nothing here needs a + per-row generator call — the generator runs **once**, for the slot numbers, + and the *scalar* `SPLIT_PART` does the per-row work, so it sidesteps both + defects above rather than working around them. `examples/countries.sql` uses + it twice (`borders`, then `languages`) and it is the form to reach for until + piece 3 lands. + + Two costs, which are why this is a workaround and not the feature: the bound + is hand-picked (20 slots for a real maximum of 16 — too few silently truncates, + too many just costs a scan), and the source is scanned once per slot. But + "there is no way to handle arbitrary-depth paths" overstated the gap: any + depth you are willing to bound is reachable today. - **Found:** 2026-09-04, from real use rather than the corpus — a TeamCity API response piped into `READ_JSON('-', …)`, wrapped in a CTE to extract an @@ -1969,6 +2005,7 @@ out of date. |---|---|---| | `SELECT * FROM range(1,3)` | 1,2,3 ✅ | ✅ | | `SELECT * FROM t JOIN range(1,3) ON t.id = value` | **Parse error: Expected ON keyword after JOIN table** 🚫 | joins | + | `SELECT * FROM t CROSS JOIN range(1,3)` | **Parse error: Unexpected '(' after end of statement** 🚫 | joins | | `WITH a AS (SELECT * FROM range(1,3)), b AS (...) SELECT * FROM a JOIN b ON a.value = b.value` | ✅ | ✅ | - **Found:** 2026-09-12, designing the system tables @@ -1980,7 +2017,12 @@ out of date. inconvenience. A family of system tables is specifically meant to be joined to one another, so this becomes the common case rather than an edge. - **Workaround, and it is a decent one:** wrap each side in a CTE, which parses - and executes correctly today. `SYSTEM_TABLES.md` documents that form. + and executes correctly today. `SYSTEM_TABLES.md` documents that form, and + `examples/countries.sql` leans on it — the bounded explode in + [P40](#p40) is `CROSS JOIN RANGE(1, 20)` with the generator wrapped in a CTE + for exactly this reason. Note the `CROSS JOIN` spelling fails with a *different* + message (the parser stops at the `(` and reports trailing tokens), which is + the same grammar hole seen from the other side. - **Decision:** fix in the parser — the joined source should accept the same productions as the `FROM` source, which is the general shape of the bug: two places that should parse "a table source" and only one of them knows the full @@ -1989,6 +2031,164 @@ out of date. --- +### P46 — A column-to-column comparison in `WHERE` returns zero rows +- **Status:** 🔴 OPEN — silent, and it returns the emptiest possible wrong answer +- **Corpus:** `02_where.toml :: where_col_lt_col`, `where_col_eq_col_string` + (both `expect = "DIFFER"`); the contrast anchor is + `01_select.toml :: select_col_vs_col_projected`, which AGREEs. +- **Observed:** if both sides of a comparison are columns, the predicate never + matches. Not one operator, not one type — the shape: + + | Query | Ours | DuckDB 1.5.5 | + |---|---|---| + | `WHERE id < partner_id` (`null_edges.csv`) | **0 rows** 🚫 | ids 1, 4, 7, 9 | + | `WHERE currency = trading_currency` (`instruments.csv`) | **0 rows** 🚫 | 57 of 200 | + | `WHERE bid_price > ask_price` (`instruments.csv`) | **0 rows** 🚫 | 108 of 200 | + | `WHERE region = region` — *the same column on both sides* | **0 rows** 🚫 | every row | + | `SELECT id < partner_id AS lt` — same comparison, projected | ✅ correct | ✅ | + + The `region = region` row is the tell: no data, no type, no NULL and no + coercion is involved. Any `Column op Column` predicate is false. +- **Not the parser.** `--query-plan` shows the WHERE clause as + `BinaryOp { left: Column(region), op: "=", right: Column(subregion) }` — the + AST is exactly right, so this lives in the WHERE evaluator. And the same + comparison evaluated in the `SELECT` list is correct, which localises it + further: comparison works, *this site's* comparison does not. +- **Found:** 2026-09-12, writing `examples/countries.sql`. "Which land borders + cross a continent boundary" is `WHERE region <> nb_region` after joining the + border pairs back to the countries, and it returned nothing. The shape is + ordinary in joined or staged data: bid against ask, claimed against settled, + this row's region against its neighbour's. +- **Why it bites:** zero rows reads as *no such data*, the same disguise that + made [P30](#p30) dangerous — and the natural next move is to loosen the query + rather than to doubt the engine. It was only caught here because the answer was + known in advance (Russia borders China, so the result cannot be empty). +- **Workaround, and it is the one `examples/countries.sql` uses:** do the + comparison in the `SELECT` list, where it works, and filter on *that* column + downstream — `CASE WHEN a = b THEN 'internal' ELSE 'cross' END AS frontier` + into a CTE or `#temp`, then `WHERE frontier = 'cross'`. Arithmetic serves the + same purpose when the columns are numeric: project + `COUNT('*') - SUM(flag) AS remainder`, then `WHERE remainder = 0`. +- **Decision:** fix. While in there, check the other predicate sites for the + same hole — `JOIN ... ON` and `HAVING` both compare operands of their own — + and re-probe from a second angle first, per the [P28](#p28) lesson: this entry + was written from `WHERE` alone. +- **Related:** [P30](#p30) (same zero-rows disguise), [P7](#p7) (a + multi-condition join evaluating two-column operands by position — possibly the + same root, worth checking together), [P48](#p48) (the NULL half of the same + comparison story, found while pinning this one). + +--- + +### P47 — `MIN`/`MAX` compare by type variant before value, so one float outranks every integer +- **Status:** 🔴 OPEN — silent, and wrong by orders of magnitude +- **Corpus:** `10_aggregate_nulls.toml :: max_mixed_int_float`, + `min_mixed_int_float`, `max_mixed_int_float_from_csv` (all `expect = "DIFFER"`). +- **Observed:** in a column holding both `Integer` and `Float` values, `MAX` + returns the largest *float* and `MIN` the smallest *integer*, whatever the + actual numbers are: + + | Query | Ours | DuckDB 1.5.5 | + |---|---|---| + | `MAX(v)` over `1, 2.5, 10` | **2.5** 🚫 | 10 | + | `MIN(v)` over `5, 2.5, 10` | **5** 🚫 | 2.5 | + | `MAX(area)` over `countries.csv` | **34.2** 🚫 | 17098242 | + | `MAX(CAST(area AS FLOAT))` | 17098242 ✅ | ✅ | + | `ORDER BY area DESC LIMIT 1` | Russia ✅ | ✅ | + | `MAX(v) OVER (PARTITION BY g)` | 10 ✅ | 10 | + | the same window with `SQL_CLI_BATCH_WINDOW=0` | **2.5** 🚫 | 10 | + + The third row is the one to feel: the largest country in the world, reported + as a 34 km² scattering of Pacific islets, by a `GROUP BY` that looks entirely + ordinary. +- **Root cause, and it is small.** `DataValue` derives `PartialOrd` + (`src/data/datatable.rs:204`), so a comparison between two *different* variants + is decided by the variant order in the enum declaration — + `String < InternedString < Integer < Float < Boolean < DateTime < Vector < Null` + — before any value is looked at. `MinMaxState` then compares with bare `<`/`>`: + `src/sql/aggregate_functions/mod.rs:439` in the current registry, + `src/sql/aggregates/mod.rs:180` in the older one. So every `Float` beats every + `Integer` for `MAX`, and every `Integer` beats every `Float` for `MIN`. +- **The correct comparator is already in the tree.** `compare_datavalues` + (`src/data/datavalue_compare.rs`) has explicit `Integer`↔`Float` arms that + compare `*i as f64` against the float — which is why `ORDER BY` and the batched + window path get this right while the aggregate does not. Same shape as + [P17](#p17): two sites that should share one rule, and only one of them knows + it. +- **A performance flag changes the answer.** The row-wise window path + (`src/sql/window_functions/aggregates.rs:198,259`) carries the same bare + comparison, so `SQL_CLI_BATCH_WINDOW=0` — documented in `CLAUDE.md` as an + opt-out for a *speed* optimisation — silently switches window `MAX` from right + to wrong. Whatever else the fix does, it should leave the two window paths + agreeing. +- **How a plain CSV ends up with a mixed column** (this is why the bug is + reachable without literals): type inference samples early rows. In + `countries.csv` only three of 250 areas carry decimals — Monaco 2.02, US Minor + Outlying Islands 34.2, Vatican City 0.44 — and they sit at rows 140, 233 and + 237, so the column was typed `Integer` and those three stayed `Float`. +- **Wider than numbers.** The same variant order puts every `Boolean` above + every `Float` and every `String` below every number, so `MIN`/`MAX` over any + mixed-type column ranks by type first. NULLs are not affected — the + accumulator skips them explicitly before comparing. +- **Found:** 2026-09-12, writing `examples/countries.sql` — a per-region + `MAX(area)` gave Europe 2.02 (Monaco) and the Americas 34.2, which is the kind + of wrong that only stands out because the right answer happens to be famous. +- **Decision:** fix by routing both `MinMaxState` implementations through + `compare_datavalues`, and sweep the other bare `DataValue` comparisons while + there — `src/data/arithmetic_evaluator.rs:1272,1311` and the two window sites + above are the same code shape. **Confirm which implementation is live before + changing it** ([P41](#p41)'s lesson: the named one was a shadow, and fixing it + moved nothing); there are two registries plus a third MIN/MAX path in the + evaluator. +- **Workaround until then:** `CAST(col AS FLOAT)` inside the aggregate, which is + what `examples/countries.sql` does, with the reason in a comment so it does not + read as noise. +- **Related:** [P17](#p17) (a derived `PartialOrd` at the window ORDER BY site — + same root class), [P41](#p41)/[R12](ENGINE_REFACTORING.md#r12) (shadowed + aggregate implementations), [P42](#p42) (`MODE`'s type handling). + +--- + +### P48 — A comparison with a NULL operand projects `false` instead of NULL +- **Status:** 🔴 OPEN — silent, and it disagrees with our own `WHERE` evaluator +- **Corpus:** `01_select.toml :: select_col_vs_col_null_operand` + (`expect = "DIFFER"`). +- **Observed:** a projected comparison coerces a NULL operand to a definite + answer: + + | Query (`null_edges.csv`, row 3, where `partner_id` is NULL) | Ours | DuckDB 1.5.5 | + |---|---|---| + | `SELECT id < partner_id AS lt` | **false** 🚫 | NULL | + | `SELECT id = partner_id AS eq` | **false** 🚫 | NULL | + | `CASE WHEN id < partner_id THEN 'y' ELSE 'n' END` | 'n' ✅ | 'n' ✅ | + | `WHERE id < partner_id` (for contrast) | row excluded ✅ | excluded ✅ | + + The `CASE` row is correct in both engines and for the same reason — UNKNOWN + takes the `ELSE` branch — so the defect is specifically the *value* a + comparison yields when it is projected rather than branched on. +- **This is the [P18](#p18)/[P19](#p19) rule at a site that fix did not reach.** + That change (2026-08-22) gave the WHERE evaluator three-valued logic; a + projected comparison goes through the arithmetic evaluator instead, which still + returns a definite `false`. So the same predicate means two different things + depending on where it is written — and a flag projected in one statement then + filtered in the next (the [P46](#p46) workaround, a pattern the examples now + use deliberately) quietly turns "unknown" into "no". +- **Found:** 2026-09-12, and worth a method note: this case was added as the + *anchor* for [P46](#p46) — the control that was supposed to AGREE and so prove + the comparison evaluator itself was fine. It came back DIFFER for an unrelated + reason. A case added to localise one bug found a second; the anchor was then + narrowed with `WHERE partner_id IS NOT NULL` so it can still do its original + job, and the unguarded form kept as this entry's case. +- **Decision:** fix — the comparison arms of the arithmetic evaluator should + yield NULL when either operand is NULL, matching the WHERE evaluator rather + than diverging from it. Small, and it wants doing alongside [P46](#p46), which + is in neighbouring code. +- **Related:** [P18](#p18), [P19](#p19), [P20](#p20) (`||` treating NULL as an + empty string — the same coercion instinct in the string operators), + [P46](#p46). + +--- + ## Deferred / won't fix (intentional) ### D1 — Recursive CTEs (`WITH RECURSIVE`) diff --git a/tests/comparison/corpus/01_select.toml b/tests/comparison/corpus/01_select.toml index 52c2343c..0f8c01b5 100644 --- a/tests/comparison/corpus/01_select.toml +++ b/tests/comparison/corpus/01_select.toml @@ -61,3 +61,25 @@ sql = "SELECT symbol, price FROM trades ORDER BY price DESC LIMIT 5" id = "order_by_expr" data = "trades.csv" sql = "SELECT symbol, price, volume FROM trades ORDER BY price * volume DESC" + +# --- P46 contrast: the comparison that fails in WHERE works in projection --- + +[[case]] +id = "select_col_vs_col_projected" +data = "null_edges.csv" +sql = "SELECT id, partner_id, id < partner_id AS lt FROM null_edges WHERE partner_id IS NOT NULL ORDER BY id" +# The anchor for P46. Column-vs-column is evaluated correctly HERE, which is +# what localises that bug to the WHERE evaluator rather than to comparison in +# general. If this case ever starts failing, P46 is bigger than filed. +# The NULL rows are excluded on purpose - they are P48's business, not P46's, +# and they are pinned separately as select_col_vs_col_null_operand below. + +[[case]] +id = "select_col_vs_col_null_operand" +data = "null_edges.csv" +sql = "SELECT id, partner_id, id < partner_id AS lt FROM null_edges ORDER BY id" +expect = "DIFFER" +# P48, found 2026-09-12 while building the P46 anchor above. Same query without +# the IS NOT NULL guard: for a NULL operand we project false, DuckDB projects +# NULL. The three-valued-logic fix of P18/P19 landed in the WHERE evaluator; +# this is the same rule missing at the projection site. diff --git a/tests/comparison/corpus/02_where.toml b/tests/comparison/corpus/02_where.toml index 3e003492..d563f828 100644 --- a/tests/comparison/corpus/02_where.toml +++ b/tests/comparison/corpus/02_where.toml @@ -178,3 +178,25 @@ expect = "GAP" # the same UNKNOWN-under-NOT rule as P19. So when the parse gap is closed, this # doubles as coverage that NOT LIKE inherits the fixed NULL semantics; the # evaluator side is already implemented and returns UNKNOWN for a NULL operand. + +# --- P46: a column-to-column comparison in WHERE returns zero rows --- + +[[case]] +id = "where_col_lt_col" +data = "null_edges.csv" +sql = "SELECT id, partner_id FROM null_edges WHERE id < partner_id ORDER BY id" +expect = "DIFFER" +# P46, found 2026-09-12 writing examples/countries.sql. Both operands are +# columns, and that alone is enough: we return NO rows, DuckDB returns +# 1,4,7,9 (the NULL-partner rows are correctly excluded by both). +# The same comparison in the SELECT list is right - see +# 01_select.toml :: select_col_vs_col_projected. + +[[case]] +id = "where_col_eq_col_string" +data = "instruments.csv" +sql = "SELECT instrument_id, currency, trading_currency FROM instruments WHERE currency = trading_currency ORDER BY instrument_id LIMIT 5" +expect = "DIFFER" +# P46 again, on strings and on equality, to pin that the defect is the +# column-vs-column *shape* rather than one operator or one type. DuckDB matches +# 57 of the 200 rows; we match none. diff --git a/tests/comparison/corpus/10_aggregate_nulls.toml b/tests/comparison/corpus/10_aggregate_nulls.toml index 7cec14c0..a730e98d 100644 --- a/tests/comparison/corpus/10_aggregate_nulls.toml +++ b/tests/comparison/corpus/10_aggregate_nulls.toml @@ -205,3 +205,37 @@ sql = "SELECT team, MODE(score) AS m FROM null_edges GROUP BY team ORDER BY team # Per-group state: each group must tally independently, and the all-NULL group # ('delta') must still produce its row with NULL. Catches a fix applied to the # ungrouped path only. + +# --- P47: MIN/MAX compare by type variant before value --- + +[[case]] +id = "max_mixed_int_float" +data = "null_edges.csv" +sql = "WITH t AS (SELECT 1 AS v UNION ALL SELECT 2.5 AS v UNION ALL SELECT 10 AS v) SELECT MAX(v) AS mx FROM t" +expect = "DIFFER" +# P47, found 2026-09-12 writing examples/countries.sql. One column holding both +# Integer and Float values: MAX returns 2.5 because DataValue's derived +# PartialOrd ranks the Float variant above Integer regardless of the numbers. +# DuckDB returns 10. The table is irrelevant to the query - the literal CTE is +# the smallest shape that produces a mixed-type column deterministically. + +[[case]] +id = "min_mixed_int_float" +data = "null_edges.csv" +sql = "WITH t AS (SELECT 5 AS v UNION ALL SELECT 2.5 AS v UNION ALL SELECT 10 AS v) SELECT MIN(v) AS mn FROM t" +expect = "DIFFER" +# The mirror of the case above, and it needs the true minimum to BE the float: +# we return 5 (an Integer beats any Float for MIN), DuckDB returns 2.5. With +# the values of the MAX case this would pass for the wrong reason. + +[[case]] +id = "max_mixed_int_float_from_csv" +data = "countries.csv" +sql = "SELECT MAX(area) AS mx FROM countries" +expect = "DIFFER" +# P47 as it is actually met. Three of the 250 areas carry decimals (Monaco +# 2.02, US Minor Outlying Islands 34.2, Vatican City 0.44) and they appear late +# in the file, so CSV type inference leaves the column mixed Integer/Float. +# MAX(area) is then 34.2 instead of 17098242 - the largest country in the world +# by a factor of half a million. Keep this alongside the literal-CTE cases: a +# fix that only handles literals would leave this one wrong. From ab7e60c96cf33833b297494415e88f93bd4e281b Mon Sep 17 00:00:00 2001 From: TimelordUK Date: Sat, 12 Sep 2026 21:08:41 +0100 Subject: [PATCH 2/2] docs(examples): generalise the countries border explosion The Russia query hand-typed its fourteen neighbour codes into a SPLIT list. Unpack borders for every country instead: CROSS JOIN a table of slot numbers onto the rows and pull the nth code out with SPLIT_PART, which turns 250 countries into 649 border pairs and needs no per-row generator call. Same trick reused for the languages column. Nine sections on top of that, staged through #border_pairs and #edges: - most neighbours (China 16, Russia 14) - the land borders that leave their own continent - doubly landlocked - falls out as exactly Liechtenstein and Uzbekistan - border symmetry: forward and reverse keys, LEFT JOIN, IS NULL. Finds the one hole in the data - Sri Lanka lists IND, India does not list LKA - haversine from latlng with RADIANS/SIN/COS/ACOS, no geo functions needed, plus CONVERT for miles. New Zealand 18764 km from London - widest and tightest land borders, deduped to one direction per pair - currency unions, language reach, island nations with the flag emoji - ROW_NUMBER/SUM OVER with a REPEAT bar chart Two workarounds carry their reason in a comment and cite the entry: the region comparison is done in the SELECT list because of P46, and the area aggregate is CAST to FLOAT because of P47. Runs clean end to end in ~2.7s; the examples harness passes. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01GfDkQm9s234uEDhdX6rvmf --- examples/countries.sql | 327 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 325 insertions(+), 2 deletions(-) diff --git a/examples/countries.sql b/examples/countries.sql index bddb8a9b..fd28b66f 100644 --- a/examples/countries.sql +++ b/examples/countries.sql @@ -1,4 +1,6 @@ +-- stage all countries in a tmp table + WITH countries AS ( SELECT * @@ -10,6 +12,8 @@ INTO #countries FROM countries; go +-- select single country - note the borders + SELECT "name.common" AS name, region, @@ -25,6 +29,8 @@ FROM #countries WHERE "name.common" = 'Russia'; go +-- for Russia inflate borders join back to countries + WITH all AS ( SELECT * @@ -51,11 +57,328 @@ INNER JOIN codes ON all.cca3 = codes.code ORDER BY name ASC; GO +-- counts within each region + SELECT region, + landlocked, COUNT('*') AS n FROM #countries -GROUP BY region -ORDER BY n desc, region ASC; +GROUP BY region,landlocked +ORDER BY region, landlocked, n desc; +GO + + +-- --------------------------------------------------------------------------- +-- borders is a comma-packed list, so unpack it for EVERY country at once: +-- CROSS JOIN a slot number onto each row and pull the nth item out with +-- SPLIT_PART. 20 slots is enough for China, the record holder with 16. +-- This is the generic version of the hand-typed SPLIT list above. +-- --------------------------------------------------------------------------- + +WITH + slots AS ( + SELECT value AS n + FROM RANGE(1, 20) + ), + c AS ( + SELECT * + FROM #countries + ) +SELECT + cca3 AS iso, + "name.common" AS name, + region, + landlocked, + CAST(SPLIT_PART(latlng, ',', 1) AS FLOAT) AS lat, + CAST(SPLIT_PART(latlng, ',', 2) AS FLOAT) AS lng, + SPLIT_PART(borders, ',', n) AS neighbour_iso +INTO #border_pairs +FROM c +CROSS JOIN slots +WHERE borders <> '' + AND SPLIT_PART(borders, ',', n) <> ''; +GO + +-- who has the most neighbours? + +SELECT + name, + iso, + region, + COUNT('*') AS neighbours +FROM #border_pairs +GROUP BY name, iso, region +ORDER BY neighbours DESC, name ASC +LIMIT 15; +GO + +-- --------------------------------------------------------------------------- +-- join the pairs back to #countries to pick up the neighbour's attributes. +-- NOTE: the engine cannot compare two columns in a WHERE clause yet (P46 in +-- docs/SQL_PARITY.md - it returns zero rows), so the region comparison is done +-- here in the SELECT list, where it works, and filtered in the next statement. +-- --------------------------------------------------------------------------- + +WITH + bp AS ( + SELECT * + FROM #border_pairs + ), + nb AS ( + SELECT + cca3, + "name.common" AS nb_name, + region AS nb_region, + CAST(landlocked AS INT) AS nb_landlocked, + CAST(SPLIT_PART(latlng, ',', 1) AS FLOAT) AS nb_lat, + CAST(SPLIT_PART(latlng, ',', 2) AS FLOAT) AS nb_lng + FROM #countries + ) +SELECT + bp.name, + bp.region, + bp.lat, + bp.lng, + CAST(bp.landlocked AS INT) AS landlocked, + nb.nb_name, + nb.nb_region, + nb.nb_landlocked, + nb.nb_lat, + nb.nb_lng, + CASE WHEN bp.region = nb.nb_region THEN 'internal' ELSE 'cross-region' END AS frontier, + CASE WHEN bp.name < nb.nb_name THEN 'keep' ELSE 'mirror' END AS pair_dedupe +INTO #edges +FROM bp +INNER JOIN nb ON bp.neighbour_iso = nb.cca3; +GO + +-- the land borders that leave their own continent + +SELECT + name, + region, + nb_name, + nb_region +FROM #edges +WHERE frontier = 'cross-region' +ORDER BY region ASC, name ASC; +GO + +-- --------------------------------------------------------------------------- +-- doubly landlocked: landlocked, and every neighbour is landlocked too. +-- there are exactly two in the world. +-- --------------------------------------------------------------------------- + +SELECT + name, + COUNT('*') AS neighbours, + SUM(nb_landlocked) AS landlocked_neighbours, + COUNT('*') - SUM(nb_landlocked) AS coastal_neighbours +INTO #landlocked_summary +FROM #edges +WHERE landlocked = 1 +GROUP BY name; +GO + +SELECT + name, + neighbours, + landlocked_neighbours +FROM #landlocked_summary +WHERE coastal_neighbours = 0 +ORDER BY name ASC; +GO + +-- --------------------------------------------------------------------------- +-- data quality: borders ought to be symmetric. Build a forward key and a +-- reverse key for every pair, then LEFT JOIN the reverse key back onto the +-- forward keys - a miss is a one-sided border in the source data. +-- --------------------------------------------------------------------------- + +WITH + a AS ( + SELECT + iso, + neighbour_iso, + iso || '>' || neighbour_iso AS fwd, + neighbour_iso || '>' || iso AS rev + FROM #border_pairs + ), + b AS ( + SELECT iso || '>' || neighbour_iso AS fwd + FROM #border_pairs + ) +SELECT + a.iso AS claims, + a.neighbour_iso AS but_not_listed_by, + a.fwd AS one_sided_border +FROM a +LEFT JOIN b ON a.rev = b.fwd +WHERE b.fwd IS NULL +ORDER BY a.iso ASC; GO +-- --------------------------------------------------------------------------- +-- no geo functions required - haversine straight off latlng with +-- RADIANS/SIN/COS/ACOS, then CONVERT for the imperial column. +-- --------------------------------------------------------------------------- + +WITH + c AS ( + SELECT + "name.common" AS name, + region, + CAST(SPLIT_PART(latlng, ',', 1) AS FLOAT) AS lat, + CAST(SPLIT_PART(latlng, ',', 2) AS FLOAT) AS lng + FROM #countries + WHERE latlng <> '' + ) +SELECT + name, + region, + lat, + lng, + ROUND(6371 * ACOS( + SIN(RADIANS(51.5)) * SIN(RADIANS(lat)) + + COS(RADIANS(51.5)) * COS(RADIANS(lat)) * COS(RADIANS(lng - (-0.13))) + ), 0) AS km_from_london, + ROUND(CONVERT(6371 * ACOS( + SIN(RADIANS(51.5)) * SIN(RADIANS(lat)) + + COS(RADIANS(51.5)) * COS(RADIANS(lat)) * COS(RADIANS(lng - (-0.13))) + ), 'km', 'miles'), 0) AS miles_from_london +FROM c +ORDER BY km_from_london DESC +LIMIT 10; +GO + +-- the widest land borders: neighbours whose centroids are furthest apart. +-- every border appears twice in #edges (A>B and B>A), so pair_dedupe keeps +-- one direction only. + +SELECT + name, + nb_name, + region, + nb_region, + ROUND(6371 * ACOS( + SIN(RADIANS(lat)) * SIN(RADIANS(nb_lat)) + + COS(RADIANS(lat)) * COS(RADIANS(nb_lat)) * COS(RADIANS(nb_lng - lng)) + ), 0) AS centroid_km +FROM #edges +WHERE pair_dedupe = 'keep' +ORDER BY centroid_km DESC +LIMIT 10; +GO + +-- and the tightest ones - microstates and enclaves + +SELECT + name, + nb_name, + ROUND(6371 * ACOS( + SIN(RADIANS(lat)) * SIN(RADIANS(nb_lat)) + + COS(RADIANS(lat)) * COS(RADIANS(nb_lat)) * COS(RADIANS(nb_lng - lng)) + ), 0) AS centroid_km +FROM #edges +WHERE pair_dedupe = 'keep' +ORDER BY centroid_km ASC +LIMIT 10; +GO + +-- --------------------------------------------------------------------------- +-- currency unions - one currency code, many countries +-- --------------------------------------------------------------------------- + +SELECT + currencies, + COUNT('*') AS members, + STRING_AGG("name.common", ',') AS countries +FROM #countries +WHERE currencies <> '' +GROUP BY currencies +ORDER BY members DESC, currencies ASC +LIMIT 10; +GO + +-- --------------------------------------------------------------------------- +-- languages is another packed list - same slot trick, then count the reach +-- --------------------------------------------------------------------------- + +WITH + slots AS ( + SELECT value AS n + FROM RANGE(1, 6) + ), + c AS ( + SELECT * + FROM #countries + ) +SELECT + SPLIT_PART(languages, ',', n) AS language, + "name.common" AS name, + region +INTO #languages +FROM c +CROSS JOIN slots +WHERE languages <> '' + AND SPLIT_PART(languages, ',', n) <> ''; +GO + +SELECT + language, + COUNT('*') AS countries, + COUNT(DISTINCT region) AS regions +FROM #languages +GROUP BY language +ORDER BY countries DESC, language ASC +LIMIT 15; +GO + +-- --------------------------------------------------------------------------- +-- island nations: no land borders, and not landlocked either +-- --------------------------------------------------------------------------- + +SELECT + flag, + "name.common" AS name, + region, + subregion, + capital, + RENDER_NUMBER(area, 'compact') AS area_km2 +FROM #countries +WHERE borders = '' + AND landlocked = 0 + AND area > 10000 +ORDER BY area DESC +LIMIT 20; +GO + +-- --------------------------------------------------------------------------- +-- window functions plus a text bar chart. CAST(area AS FLOAT) is deliberate: +-- area mixes integers and floats, and raw MIN/MAX rank by type before value +-- (P47 in docs/SQL_PARITY.md - MAX(area) answers 34.2, not Russia). +-- --------------------------------------------------------------------------- + +WITH + c AS ( + SELECT + "name.common" AS name, + region, + CAST(area AS FLOAT) AS area + FROM #countries + WHERE area > 0 + ) +SELECT + ROW_NUMBER() OVER (ORDER BY area DESC) AS rank_world, + ROW_NUMBER() OVER (PARTITION BY region ORDER BY area DESC) AS rank_in_region, + name, + region, + RENDER_NUMBER(area, 'compact') AS area_km2, + ROUND(100.0 * area / SUM(area) OVER (), 2) AS pct_of_world, + ROUND(100.0 * area / SUM(area) OVER (PARTITION BY region), 2) AS pct_of_region, + REPEAT('#', CAST(area / 500000 AS INT)) AS bar +FROM c +ORDER BY area DESC +LIMIT 20; +GO