Check compatibility against a real Redis, twice over

.github/workflows/ci.yml

@@ -0,0 +1,48 @@
+# The harness gets checked here. The comparison itself does not run on a shared
+# runner yet, because it wants a Redis built from the release tarball and that
+# build is slower than everything else in this file put together. It runs on the
+# boxes, next to the benchmarks, against the same binaries.
+#
+# What is worth checking here is the part that can be wrong without anyone
+# noticing: the corpus files parse, the register refuses an entry with no reason
+# on it, and the RESP reader keeps two different spellings of nothing apart.
+name: ci
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ci-${{ github.ref }}
+  cancel-in-progress: true
+
+env:
+  CARGO_TERM_COLOR: always
+  CARGO_INCREMENTAL: 0
+  RUSTFLAGS: -D warnings
+
+jobs:
+  check:
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - uses: actions/[email protected]
+      - uses: dtolnay/rust-toolchain@stable
+        with:
+          components: rustfmt, clippy
+      - uses: Swatinem/[email protected]
+      - run: cargo fmt --all --check
+      - run: cargo clippy --workspace --all-targets
+      - run: cargo test --workspace
+
+  shell:
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    steps:
+      - uses: actions/[email protected]
+      - name: shellcheck
+        run: shellcheck -s sh suite/*.sh

.gitignore

@@ -0,0 +1,4 @@
+/target
+# The vendored Redis tree. suite/fetch.sh puts it here and it is their code at a
+# pinned version, not ours, so it does not belong in our history.
+/redis

Cargo.toml

@@ -0,0 +1,25 @@
+[workspace]
+resolver = "3"
+members = ["crates/yo-compat"]
+
+[workspace.package]
+version = "0.1.0"
+edition = "2024"
+rust-version = "1.94"
+license = "MIT OR Apache-2.0"
+repository = "https://github.com/tamnd/yo-compat"
+homepage = "https://yo.tamnd.dev"
+authors = ["tamnd <[email protected]>"]
+
+[workspace.lints.rust]
+missing_docs = "warn"
+
+[workspace.lints.clippy]
+alloc_instead_of_core = "warn"
+
+# Nothing here is on a measured path. This binary talks to two servers over
+# loopback and compares what they say, so the build wants to be quick and the
+# panics want to be readable.
+[profile.release]
+opt-level = 2
+debug = 1

README.md

@@ -1,2 +1,46 @@
 # yo-compat
-Redis compatibility suite for yo: the command scoreboard, the differential harness, the RDB corpus, the client library matrix and the divergence register
+
+Redis compatibility for [yo](https://github.com/tamnd/yo), checked against a real Redis rather than against the documentation.
+
+There are two halves here and they catch different things.
+
+## The real Redis test suite
+
+`suite/fetch.sh` downloads the Redis release tarball at a pinned version and drops its `tests/` directory and its `runtest` script into `redis/` unmodified. Not a rewrite of the suite, not a subset we typed out ourselves, and not a description of what we think it checks. It is their tests, run against our server.
+
+```
+suite/fetch.sh
+suite/runtest.sh
+```
+
+That starts `yodb serve` and points the suite at it with `runtest --host --port`, which is the suite's external mode. With no arguments it runs the files listed in `suite/passing.txt`, and `--all` runs everything under `unit/`.
+
+Every failure is real. A test that fails because we have not implemented the command yet is still a compatibility failure, it is just one with a date on it. Nothing is patched out to make the number look better, and moving a file into `suite/passing.txt` is a decision somebody makes on purpose in a diff.
+
+## The differential harness
+
+`yocompat` starts a real Redis and starts yodb, sends both of them the same commands, and reports every reply that is not byte for byte the same.
+
+```
+cargo build --release
+./target/release/yocompat --redis /opt/yo-bench/bin/redis-server-8.10.1
+./target/release/yocompat --resp3
+```
+
+This catches what their suite cannot. Their tests assert what Redis's own developers thought was worth asserting, which leaves out everything they take for granted: that a missing key comes back as `$-1` and not `_`, that `INCRBYFLOAT` replies with a bulk string and not a double, that `GETRANGE` on a missing key is an empty bulk string and not a null, that the first word of an error is `WRONGTYPE` and not `ERR`. Clients branch on all of it.
+
+The corpus files under `corpus/` are lists of commands with no expected answers in them, because the expected answer is whatever Redis says. `--resp3` sends `HELLO 3` first and runs the same corpus again, which is where the reply types that change between protocol versions show up.
+
+Divergences are sorted worst first in the report. Accepting a command Redis refuses is a correctness bug that will corrupt data for a client that relied on the refusal. Refusing it with different wording is a bug too, but a smaller one.
+
+## The register
+
+`divergences.toml` is the list of differences we have decided to live with, each with a reason written next to it. Anything in it is counted and not reported. Anything else fails the run.
+
+The file is deliberately awkward to add to. A divergence with no reason on it does not load at all, which means saying "we behave differently here and this is why" has to be a diff somebody wrote and somebody else read.
+
+## What is not here yet
+
+The command scoreboard, which is the other thing this repository is supposed to hold: a generated table of every Redis command against what yo does with it. Right now the honest version of that table is short, and it belongs here rather than in a README that can go stale quietly.
+
+Also the RDB corpus and the client library matrix. Both wait on yo having something to load and something for a real client to talk to.

corpus/errors.txt

@@ -0,0 +1,70 @@
+# Being wrong the same way they are.
+#
+# Clients branch on error text. Not on an error type, because RESP does not have
+# one, on the first word and sometimes on the whole string. A server that
+# refuses the right commands with the wrong words breaks a client library in a
+# way that looks like a bug in the library, so the wording is part of the
+# protocol whether anyone likes it or not.
+
+# Wrong number of arguments, at both ends.
+GET
+GET one two
+SET
+SET onlykey
+SET key value extra
+APPEND
+APPEND key
+STRLEN
+STRLEN a b
+INCR
+INCR a b
+INCRBY key
+DECRBY key
+INCRBYFLOAT key
+GETRANGE key
+GETRANGE key 0
+GETRANGE key 0 1 2
+SETRANGE key
+SETRANGE key 0
+GETSET key
+GETDEL
+GETDEL a b
+MSET
+MSET onlykey
+MSET a 1 b
+MSETNX
+MSETNX a 1 b
+MGET
+SETEX key
+SETEX key 100
+PSETEX key 100
+SETNX key
+LCS key
+LCS a b c
+
+# Arguments that are the wrong shape rather than the wrong number.
+SETRANGE key notanoffset value
+SETRANGE key -1 value
+GETRANGE key notastart 10
+GETRANGE key 0 notanend
+SETEX key notaseconds value
+PSETEX key notamillis value
+LCS a b NOTANOPTION
+LCS a b IDX MINMATCHLEN notanumber
+LCS a b LEN IDX
+
+# A command nobody has.
+NOTACOMMAND
+NOTACOMMAND with args
+GETT key
+
+# Subcommands that do not exist under commands that do.
+CONFIG NOTASUBCOMMAND
+CONFIG
+COMMAND NOTASUBCOMMAND
+
+# Casing is not supposed to matter anywhere.
+set lower value
+GeT lower
+INCRBYFLOAT lower 1
+sTrLeN lower

corpus/expire.txt

@@ -0,0 +1,71 @@
+# Expiry, without ever asking what the remaining time is.
+#
+# TTL is not in here on purpose. A corpus that asks two servers how many
+# milliseconds are left on a key is a corpus that fails on a slow afternoon, and
+# a flaky compatibility test gets ignored, which is worse than not having it.
+# What is checked instead is the part that is deterministic: which option
+# combinations are accepted, what each one replies, whether the value is still
+# there afterwards, and which ones are refused.
+
+SET vanish value EX 100
+GET vanish
+SET vanish value PX 100000
+GET vanish
+SET vanish value EXAT 33229822800
+GET vanish
+SET vanish value PXAT 33229822800000
+GET vanish
+SET vanish value KEEPTTL
+GET vanish
+
+SETEX seconds 100 value
+GET seconds
+PSETEX millis 100000 value
+GET millis
+
+# A zero or negative expiry is refused rather than treated as delete now, and
+# the key it was aimed at is left alone.
+SET keep original
+SET keep replaced EX 0
+GET keep
+SET keep replaced EX -1
+GET keep
+SETEX zero 0 value
+GET zero
+SETEX negative -1 value
+PSETEX zeroms 0 value
+SETEX huge 9999999999999999 value
+
+# The options that cannot be used together.
+SET both value EX 100 PX 100000
+SET both value EX 100 KEEPTTL
+SET both value NX XX
+SET both value EX
+SET both value EX notanumber
+SET both value PERSIST
+
+# GETEX is the one that reads and changes the expiry in the same breath, and
+# with no options at all it changes nothing.
+SET readme value EX 100
+GETEX readme
+GET readme
+GETEX readme PERSIST
+GET readme
+GETEX readme EX 200
+GETEX readme PX 200000
+GETEX readme EXAT 33229822800
+GETEX readme PXAT 33229822800000
+GETEX missing
+GETEX missing PERSIST
+GETEX readme EX 100 PERSIST
+GETEX readme EX notanumber
+GETEX readme NOTANOPTION
+
+# An expiry that has already passed is a delete, and the reply says so from
+# whichever side you look at it.
+SET gone value EXAT 1
+GET gone
+SET gone2 value PXAT 1
+GET gone2
+GETEX readme EXAT 1
+GET readme

corpus/numbers.txt

@@ -0,0 +1,85 @@
+# The counters, and what they consider a number.
+#
+# Redis is stricter here than people expect and the strictness is the
+# compatible part. A value with a leading space is not a number. A value with a
+# trailing newline is not a number. An integer stored as "010" is not the same
+# as one stored as "10" once you increment it. Every one of those is a place an
+# implementation written from the documentation gets it wrong.
+
+INCR counter
+INCR counter
+GET counter
+INCRBY counter 10
+INCRBY counter -20
+GET counter
+DECR counter
+DECRBY counter 5
+GET counter
+
+# Starting from a string that happens to look like a number.
+SET fromstring 100
+INCR fromstring
+GET fromstring
+SET negative -100
+INCR negative
+GET negative
+
+# Leading zeroes are stored as typed and are not a number to increment.
+SET padded 010
+INCR padded
+GET padded
+
+# Whitespace either side is not a number, in either direction.
+SET leading " 11"
+INCR leading
+SET trailing "11 "
+INCR trailing
+SET newline "11\n"
+INCR newline
+
+# Empty is not zero.
+SET blank ""
+INCR blank
+
+# Not numbers at all.
+SET word hello
+INCR word
+INCRBY word 1
+DECR word
+INCRBYFLOAT word 1.0
+
+# The ends of the range. One past the top is an error and not a wrap.
+SET big 9223372036854775807
+INCR big
+INCRBY big 1
+SET small -9223372036854775808
+DECR small
+DECRBY small 1
+SET big2 9223372036854775806
+INCR big2
+GET big2
+
+# The increment itself has to be a number, and has to fit.
+INCRBY counter notanumber
+INCRBY counter 9223372036854775808
+INCRBY counter 1.5
+
+# INCRBYFLOAT is a different command with different rules. It replies with a
+# bulk string, it trims trailing zeroes, and it takes exponent notation.
+SET float 10.5
+INCRBYFLOAT float 0.1
+INCRBYFLOAT float -5
+INCRBYFLOAT float 5.0e3
+GET float
+SET whole 10
+INCRBYFLOAT whole 0.1
+INCRBYFLOAT whole -0.1
+SET intish 3.0
+INCRBYFLOAT intish 1.000000000000000005
+INCRBYFLOAT newfloat 1.5
+GET newfloat
+INCRBYFLOAT float notafloat
+INCRBYFLOAT float nan
+INCRBYFLOAT float inf
+SET hexish 0x10
+INCRBYFLOAT hexish 1