M8: full Redis compatibility, modules included #9

Full Redis compatibility, including the modules. The claim is 100 percent, so the gate is the real Redis test suite and a public triage file for anything that does not pass, rather than a percentage nobody can check.

Scope

• JSON.*, the full RedisJSON surface
• FT.*, listed per command, including FT.HYBRID from Redis 8.4. Forty one commands are shipped. The sixteen index commands are verified over 136 raw socket comparisons against 8.10.1 on both protocols, which is FT.CREATE, FT.ALTER, the four drop spellings, the alias family, FT.INFO and FT._LIST along with their five underscore forms. FT.EXPLAIN and FT.EXPLAINCLI come with the whole query language behind them, verified across all three dialects at zero differences over more than 570000 generated queries, the fixed escape, Unicode, tag escape and random tag corpora, and fifty one batch files, with one registered divergence in D-62. The document tokenizer is in as well, so a value read as a document and a value read as a query now land on the same terms, verified over 7000 generated values for the terms and 600 for the places at no differences, with one registered divergence in D-63. The inverted index is in too, a term dictionary with a blocked posting list per term carrying the frequency, the field mask and the places, with the block size, the frequency rule and the place numbering all measured against 8.10.1 rather than guessed. The numeric index is in beside it, a sorted run of values a range is cut out of, with the nine edge cases measured against the reference rather than reasoned about, both spellings of zero, the two infinities, a reversed range and the three kinds of value that never enter the index at all. The document table is in too, which is what turns a number in a posting list back into a key and what carries the weighted length and the largest frequency a scorer asks each document for. All nine scorers are in and measured, the default BM25STD agreeing with 8.10.1 to the last digit of a double, which needed the single precision k1 constant the other side folds. The tag index is the third of the per field indexes and is in, where a value is matched whole rather than broken into words, with the whole byte model measured over seven probes, including the truncation at the first NUL, the six bytes the trim treats as space and the rule that a value whose fold comes out empty is indexed as the bytes it arrived as. The routine that reads a document is in on top of all of it, so a key and its fields go in one end and a document with a number, a score, a payload, a weighted length, a largest frequency, a term in every list it belongs to, its numbers and its tag values comes out the other, with two rules nobody would guess measured rather than assumed, that a stem occurrence is worth one whatever the field weight is and that a numeric field which will not parse loses the whole document instead of being skipped. The end to end check indexes two documents and reproduces the two scores 8.10.1 answered to the last bit of a double. The fan out over it is in as well, so a key goes to every index whose prefix and source cover it and comes out of every index when it goes, with five more probes settling when an index picks a key up, including that any change rewrites the whole document, that a key of the wrong type under the prefix is passed over rather than counted as a failure, and that an index follows a key by name across every database. The failure counters are real with it, so FT.INFO now reports num_docs, max_doc_id, num_terms, num_records, records_per_doc_avg, hash_indexing_failures and both Index Errors blocks off what the index actually holds. The fan out is wired into the keyspace now, so a hash command on a running server reaches every index that follows the key, and FT.CREATE reads the keys that were already there, with the signal coming out of the hash group being what the indexes have to be told rather than whether the command was a write. Three more rules came out of measuring that, none of them guessable and all of them visible in FT.INFO, that HDEL of the last field is counted as an indexing failure and spends no document number, that a deadline taking the last field spends one and is counted as nothing, and that HSETEX with a deadline that has already passed is two pieces of news from one command so the number moves twice and the value never reaches the index. FT.CREATE is refused anywhere but on database zero as well, and the initial scan reads only that database while the following afterwards is server wide. The keyspace group reaches the indexes now as well, so a key a DEL, UNLINK, EXPIRE, RENAME, COPY, RESTORE or SORT STORE touched is read again or erased the same way the reference does it, with the whole model measured over 31 side by side cases rather than reasoned about. Two rules came out of that, that a rename inside a followed prefix moves the document across keeping the number it had and is the one write on a followed key that spends no number, and that the keyspace group reads the key back on database zero whatever database the command ran on where a hash command reads the database it ran on. The one place this parts company with the reference is a rename onto a key that already held a document, where the reference strands the old one under its own number and reports a document count one too many, which is registered as D-64 rather than copied. The walk over an index is in now as well, so a parsed query goes in one end and every document that answers it comes out the other in number order, each one carrying the shape of how it answered, which covers a word, a union, an intersection, a negation, an optional, a bare star, a numeric range, a tag with its values and all five expansions, where a geo filter and a vector query answer nothing rather than answering wrongly. Four scoring rules came out of measuring 8.10.1 and none of them is guessable, that a union adds up what every branch was worth where an expansion counts only the first term in byte order the document holds, that a bare star is a term whose idf is one so it scores as BM25 with the rarity taken out, that a prefix, a suffix or an infix shorter than two bytes stands for nothing at all without an error while a pattern and a fuzzy word are not held to that, and that a tag value written as several words is one value with a space in it which scores like a term that answered once. A phrase and a slop are in on top of that, which is one rule and not two: each word is given one of the places it was found at and the words are close enough when the last of those places is no further from the first than the slop plus one less than the number of words, in order the places may repeat so a phrase of one word written twice answers a document holding it once, and in any order they may not all stand on the same place. Who takes part in that check is measured too, where one tag value, a number, a negation and an optional stay out of it and a union takes part with no places of its own, which is why a tag with one value is built as that value rather than as a union of one. A hundred and eleven queries over a twelve document corpus are generated from the real server under the second dialect, keys and scores together, and every one of them agrees here to the last few bits of a double. FT.SEARCH is in now as well, so a query goes in one end and the documents that answer it come out the other with the fields of the keys they came from, which runs in two halves: the query is parsed, walked, scored, sorted and cut to the window under the registry lock, and the keys are read afterwards with that lock let go of. Fifteen option keywords are read and six are refused on purpose under D-65. Everything about it was measured rather than guessed, including that the index name resolves before the argument list is read and the argument list is read before the query, that RETURN 0 reads like NOCONTENT on the wire but is not the same flag since a later RETURN puts the fields back, that a backwards FILTER is only refused on a field the schema holds as a number, and that a document whose key is on another database is dropped from the reply and taken off the total. The scoring model for a numeric range was the one nobody would guess, where a range leaf is a match with no word in it so the four older scorers give it what a wildcard gets and the three standard ones give it nothing, visible in TFIDF scoring alpha @n:[1 2] at exactly twice what it scores alpha and BM25STD scoring the two the same, and a bare star beside a FILTER is dropped rather than intersected for the same reason. The wire comparison against 8.10.1 is at 32 rows of 1572 differing, eighteen of them BM25STD.TANH in the last bit where the reference's libm is not correctly rounded and this is, registered as D-66, and the rest the six refused options and the IDS node an INKEYS puts in FT.EXPLAIN's tree. FT.AGGREGATE is in now as well, the query, LOAD, the search options it shares with FT.SEARCH, ADDSCORES, WITHSORTKEYS, the reply shape on both protocols and the count at the front of it, with the six pipeline steps refused as unknown arguments under D-67 until they are written. The grammar rule behind it was measured rather than guessed: once a pipeline step has been read the search side words stop being taken, so LOAD 1 @t VERBATIM is refused at position 4 where LIMIT 0 1 VERBATIM is a perfectly good query. LOAD counts words and not fields, a name that follows AS wins, the named loads come first deduped by output name and LOAD star appends the rest of the key's fields after them. The count at the front has four cases and only one of them is the number of documents that answered, and the odd one out is BM25STD.NORM with ADDSCORES, which has to see the whole answer before any of it can be written, so it reports the real total whatever the window is and hands the rows back in descending document number. Two fixes to already shipped commands fell out of the measuring, that LIMIT 1 0 is now refused on all three commands and that FT.EXPLAIN's read and drop keyword list no longer leaks into FT.AGGREGATE. 320 of 320 scorer cases, 252 of 252 window cases and 83 of 83 grammar and error cases agree with 8.10.1 exactly on both protocols. GROUPBY with REDUCE is in now as well, all twelve reducers, COUNT, COUNT_DISTINCT, COUNT_DISTINCTISH, SUM, MIN, MAX, AVG, STDDEV, QUANTILE, TOLIST, FIRST_VALUE and RANDOM_SAMPLE, living as pure folds in a new yo-search module with no wire and no index behind them. Every rule in them was measured against 8.10.1 and none is the obvious guess, that a document not holding the property is out of the fold entirely, that a value which is not a number is out of the numeric fold and still in COUNT_DISTINCT and TOLIST, that a fold with no number in it answers nan for SUM, AVG and QUANTILE, inf for MIN, -inf for MAX and 0 for STDDEV, that the deviation divides by one less than the count, that the quantile takes the value at one below the fraction of the count rounded up and never interpolates, and that FIRST_VALUE orders by the field rather than the value so a NUMERIC field puts 9 before 10 where an untyped loaded value puts 10 before 9. The parser follows the reference's own order of checks, where the argument count is read and then the arguments are read before the reducer name is ever looked up, which is why REDUCE NOPE complains about the missing count and REDUCE NOPE 5 @n complains about the four arguments that are not there. A reducer with no AS gets the generated name a real server builds out of the function and every argument with its @ stripped, joined by commas and lowercased. A group step closes the door on the base row the way a LOAD closes it on the search words, and a grouping step folds every document that answered whatever the window is, so the count at the front is the number of groups on both protocols and the LIMIT applies to the groups afterwards. 68 argument shapes, 38 value shapes and 8 RESP3 shapes agree with the reference, with four new divergences out of the measuring: D-68 the group order, which is not reproducible on the reference between two processes or across a restart and is first seen order here, D-69 COUNT_DISTINCTISH counting exactly rather than estimating, D-70 a FIRST_VALUE whose ordering key is missing, and D-71 an arity that segfaults 8.10.1, FT.AGGREGATE i * GROUPBY 0 REDUCE QUANTILE 1 @n, reproduced twice. D-67 shrinks from six refused steps to five. APPLY and FILTER are in now as well, along with the little language they are written in, which is a new yo-search module with no index and no wire in it. The language was measured against 8.10.1 rather than reasoned about and nearly all of the work is in the corners: a number literal carries its own sign so 1+2 is two numbers in a row and not a sum, there is no minus that takes one operand, the not binds looser than a comparison and tighter than the and, a comparison cannot be chained, two values of different sorts are ordered by their sort with nothing at the bottom and a list above it, two numbers that cannot be ordered come out equal rather than neither so 0/0 == 1 is one, text that will not read as a number beside a number is an error for the four orderings and never for the two equalities, and both of the two word operators stop as soon as their left hand side settles the answer. All 31 functions are there with the arities and the error lines a real server answers, including the type errors that name the argument by its real position and end in the index nought a real server always prints, the strictness of substr's two number arguments against the coercion every arithmetic function does, and timefmt checking its second argument under the name the pair is kept under rather than the name it was written with. The date functions walk the C library's calendar down to the three ISO week fields and to a shape naming none of the date reading against a time with nothing in it, which is the day before the first of January 1900. Four rules around the language came out of the same measuring, that a property not on the row is an error naming the property where a property holding nothing is a value, that a reducer answer keeps the type the property had, that a step which cannot work a row out stops the pipeline where it stands and warns, and that a failure before any row of the reply was written makes the whole command answer the error instead. 218 cases on both protocols agree with the reference apart from two registered divergences, D-72 the order the documents are walked in, which is keyspace scan order on a real server because it indexes on a background scan and document number here because it indexes on the write, and D-73 matched_terms, which answers nothing until the executor carries the terms behind a match through to the pipeline for SUMMARIZE and HIGHLIGHT as well. D-67 shrinks again from five refused steps to three. SORTBY with MAX is in now as well, which orders the rows by as many properties as it was given with a direction for each and cuts them down to a number, and the count in front of the key list counts words rather than properties so a direction eats one of them. A word in that window that is neither a property nor a direction is refused, a direction with nothing in front of it is dropped, the whole list is read before any of it is looked up, and a second SORTBY is refused unless a GROUPBY stands between the two. SORTBY reads its properties onto the row the way a LOAD does whether or not the schema called them sortable, a row that does not hold one of them goes last whichever way that key runs, a sortable text field read through the sort comes back folded where a LOAD of it comes back as it was written, and it is the one step a LOAD star does not open up. The find behind the work is that LIMIT is not a window on the answer at all but a step of the pipeline standing where the client wrote it, sharing one step with the SORTBY in front of it, reaching back past an APPLY or a FILTER and stopping at a GROUPBY, which is measured five ways and fixed four LIMIT ordering differences the APPLY and FILTER suite had been carrying. WITHSORTKEYS answers a real sort key now rather than a null, the value the first sort key found on the row after a hash for a number and after a dollar for text, written to seventeen significant digits where the row holds twelve. 105 SORTBY and MAX shapes leave seven differences on both protocols and every one of them is the group order of D-68 or the walk order of D-72, and D-72 gains a line saying a SORTBY takes that difference away for every row it can tell apart. D-67 shrinks once more from three refused steps to two, WITHCURSOR and EXPLAINSCORE. WITHCURSOR is in now as well, along with FT.CURSOR READ, DEL and GC, so an answer is handed back a chunk at a time as a two element reply, the ordinary reply for the chunk and the number to read the rest of it under, with the default chunk of 1000 rows, the default idle time of 300000 ms and the limit of 128 cursors an index all measured against 8.10.1 rather than read off the documentation. The number at the front of a chunk turned out to be neither the total nor the chunk size but a fill simulation, what the pipeline pulled while the chunk was filling less what a FILTER threw away during it, taken at the first row written on RESP2 and at the end of the chunk on RESP3, one document at a time without a LOAD and a whole fill at a time with one, held back by the window only when the window has an offset because LIMIT 0 n is a cap on what gets written rather than a step of the pipeline, while an answer that had to exist in full before any of it could be written says the real total on the first chunk and nought after that. The rule came out of about a thousand measured chunk sequences over a grid of filter expressions, offsets, windows, chunk widths and both protocols, and the implementation matches it on all 850 shapes the grid walks, with 114 handpicked cursor and error shapes agreeing with the reference apart from three that differ only by the walk order of D-72. FT.INFO cursor_stats is live with it rather than nought. Two divergences came out of the work, D-74 that a cursor holds the rows the query made rather than a paused pipeline, and D-75 that a FILTER underneath a window with an offset can report a different count per chunk than the reference does, which comes down to where the reference puts its two loaders and the wire does not show that. A bug with nothing to do with cursors fell out of needing an index bigger than a handful of documents for the first time, that FT.AGGREGATE with a SORTBY that has at least one key and no width of its own hands back ten rows where yo was handing back every row, which no corpus in use up to now was big enough to show. D-67 is down to EXPLAINSCORE alone, SORTBY and WITHSORTKEYS on a search are in now as well, along with the sortable value store behind them, so a field the schema calls SORTABLE has its value copied into the document table as the sort will compare it, folded for text and parsed for a number, and is sorted and windowed inside the walk without ever going back to the key, while a field the schema does not call sortable is sorted just as well the other way round, the whole answer handed back in document number order and the value read off the key once the registry lock has been let go of, because reading a key under that lock would deadlock against a writer that takes the keyspace first. Both roads use the same comparison so both give the same answer and a tie goes to the document written first either way. The reply field order came out of the same measuring and is fixed with it: a real server does not answer a key in that key's own field order but keeps one ordered list of names over the whole window, appending a name the first time a row carries one and writing each row in that list's order leaving out what the row has nothing under, which is why the same key comes back one way at LIMIT 0 7 and another way at LIMIT 1 2 and why a sort puts its own field in front of everything, since the sort registers its name before any row is read. A sort key is a hash and seventeen significant digits for a number, a dollar and the bytes for text and a null for a row with no value, and WITHSORTKEYS with no SORTBY is taken and answers a null on every row. 120 cases over eight corpora on both protocols leave eighteen differences on each and every one of them was already registered, five the reference scoring off postings it has not collected yet, six the order of rows that compare equal, two FT.EXPLAIN reading and dropping SUMMARIZE and HIGHLIGHT, one a pipeline step answering its own error early and the rest walk order. D-65 is down to four refused options, D-76, D-77 and D-78 are new and the register is at 78. SUMMARIZE and HIGHLIGHT are in now as well, so a field the query matched comes back as fragments, one per match, each the match with as much of the text either side as the budget allows, and a field it did not comes back as its own front cut at a word, with both budgets counted in real bytes against six bytes per unit of LEN so a field of short words keeps more of them than a field of long ones under the same LEN. Three of the rules took measuring to find and none is in the documentation. A fragment covering more than one match pays a unit of LEN for every word between the outermost two and the two sides split what is left, but a stop word in that gap is free, so ten of the between two matches leave the fragment exactly as wide as no words at all would. FRAGS is applied before any context is worked out, so a fragment runs into the fragment next to it rather than into the next match, and dropping one with FRAGS 1 lets the one that is left run straight over the match the dropped one was built around. And a field with no match in it is left whole rather than cut down when a HIGHLIGHT covers the same field, which is odd and is what a real server does. Three probe tables settle it against 8.10.1, 99 cases over a twelve document corpus covering every LEN from nought to fourteen, every FRAGS from nought to six, separators, FIELDS lists, TAGS, repeated clauses, RETURN, NOCONTENT, ten queries and nineteen error shapes, a sweep of 360 rows over stop word gaps of nought to ten words at every LEN, and 23 cases pinning down which fields each clause touches. All of them agree on every value, and the four that still differ at all differ only in the order rows and fields come back in, which the control case with no clause shows too. FT.EXPLAIN reads the whole clause and drops it the way a real server does now rather than refusing it. D-80 and D-81 are new, the first that a prefix marks every word it stood for here where the reference marks whichever one its expansion wrote last, the second that the fields of a row come back in schema order, and D-65 is down to the two options still refused, GEOFILTER and EXPLAINSCORE. GEOFILTER and EXPLAINSCORE are next, and the terms this work carries through to the reply are the same machinery D-73 matched_terms and EXPLAINSCORE both wait on. GEOFILTER is in now as well, along with the same circle written inside the query, over a real geo index that had been reading a GEO field, checking it and throwing it away. The index is the numeric one again with the point folded into the number, since a fifty two bit interleave is something an f64 holds exactly, so a circle is nine score ranges out of the numeric walk with the real distance checked on everything they turn up, which is what both a sorted set and a search index do on the other side, so the geohash arithmetic moved down a crate and the two callers now share one copy of it rather than agreeing by luck about where a boundary falls. Two real bugs at the far edges of the projection came out of measuring it, that a point at longitude 180 or at the top latitude scales to one bucket past its axis so the score carries a fifty third or a fifty fourth bit and decode was refusing those scores, which had left GEOPOS answering nothing for a place on the date line, and that no ordinary box cover can reach such a score from anywhere except that exact spot while the reference finds those places from ordinary centres, so a point on the top edge of an axis now lands in the last bucket of that axis on the way in and on the way out both. The rest is measured rather than reasoned about, that the field is cut at the first space or comma so 12,34 and 12 34 are the same place and a leading space means there is no longitude at all, that a value which is not a point loses the whole document the way a bad number does, that the query form reports the unit before the point and the point before the radius, and that the option form reads a NaN and refuses an infinity, which is the other way round from every other number on the command. Fifty one cases over eleven documents, both sides of the date line, both poles, the aggregate, the explain and every error line, all agree with the reference. Two of them only started agreeing once the probe waited for the reference to finish its background scan after FT.CREATE, because anything written while that runs is indexed twice over there, which is where the long standing num_records gap came from as well, so there is nothing to fix on that one. EXPLAINSCORE is in now as well, which prints the arithmetic behind a score as a tree beside the number, and every line of it was read off 8.10.1 rather than worked out from the formula, because the printed shape is not the shape of the arithmetic: the three scorers that divide by how far the words landed apart print their leaves undivided and do the division once at the top, so TFIDF says words TFIDF 9.00 / norm 3 / slop 1 rather than three lots of a third. The walk hands back a shape rather than a flat list to carry it, a hit being the document, what answered it and how far apart the words were, and a found being a term, an intersection, a union, a wildcard or one of the three blanks a filter, a negation and an optional come to, which is what keeps the number and the account of it from drifting apart since the scorer and the explainer read the same thing. The slop is a closed formula and it is measured, the square root of the sum over adjacent pairs of children of the smallest gap between their places squared, truncated to a whole number, and one when there is nothing to measure, and only three scorers divide by it so the walk only works it out when one of those asked. Two shapes came out of the comparison rather than out of the manual, that a union with one branch left is that branch, which is why fox explains as a bare leaf since the parser gives every word a union to hold the stem behind it and nothing is ever written under +fox where the stem and the word are the same, while a union that really does stand for two words keeps its wrapper even when only one of them answered, and that a range is guessed at the whole index when an intersection sorts its branches rather than at how many documents are in it, which is why @n:[1 5] fox explains the word first though four documents are in the range and seven hold the word. The stem fix rides along, that a document holding zeta running runs is three tokens long with a largest frequency of two and the two is the stem behind the last two words. 185 cases over 20 queries and 9 scorers agree line for line with 8.10.1 on the same twelve document corpus and the one that differs differs in the order a hash hands its fields back, which is D-81, and a second sweep of 35 queries over 4 scorers agrees except for the last bit of a few scores, which is D-66. D-67 is gone, since FT.AGGREGATE reads EXPLAINSCORE and drops it the way a real server does. D-82 and D-83 are new, which word an expansion names and a geo filter explaining as one leaf rather than as a union of cells, and D-65 is down to the node an INKEYS list puts in the reference's tree. FT.TAGVALS and the three FT.DICT commands are in now as well, which is a tag field's values read straight back out of its index and the word lists FT.SPELLCHECK will read later. The values come back as they are stored, folded and trimmed on an ordinary field and left alone on a CASESENSITIVE one, sorted by their bytes either way, which puts a capital in front of a small letter, and the name asked about is the attribute and never the identifier so a field declared AS gg is asked about by gg. Looking up the index counts as a use of it on the two roads that refuse the field as well as on the one that answers. A dictionary is neither a key nor an index, so TYPE says none for the name and KEYS never lists it, there is no command to make one because the first word added does that, an empty term is dropped without being counted, nothing folds case so a list can hold alpha and ALPHA at once, and a name nobody ever added to dumps empty rather than complaining, which is the one place in the group where a missing name is forgiven. The lists ride on the registry because they have the same lifetime the indexes do, one table for the server, reachable from any database, and emptied when the keyspace is. Twenty eight cases against 8.10.1 agree exactly on both protocols, where the two list replies are sets on RESP3. The four FT.SUG commands are in now as well, which is the first slice of the family a client can see with TYPE, since a suggestion dictionary is the one thing the search module puts in the keyspace: TYPE answers trietype0 for one, EXISTS sees it, KEYS lists it, EXPIRE works on it and a key holding anything else answers WRONGTYPE on all four. What a lookup answers is not the score the client stored but that score divided by exp of twice the edit distance and again by the square root of one plus the length gap, worked out in single precision, where the term in that gap is counted in runes and the prefix the client sent is counted in bytes, so a two byte accented prefix against a five rune term divides by the square root of three rather than of four. The exact match rule was the find: a term the module calls exact scores two to the thirty first so it sorts in front of everything, but only the front half of the word is ever compared, because the module hands a rune count to a comparison that counts bytes and a rune is two bytes wide, which is why abcdX scores the same as abcde while aXcde does not and why a term spelled with U+01E9 counts as one spelled with U+00E9 when the two land on the odd rune out. MAX gets two goes at its number, a whole number read the strict way and range checked so 0 and -1 are out of bounds, then a double cut towards zero where anything under one is a conversion that failed rather than a range that did, so 0 is out of bounds and 0.0, 00 and -0 are not. An add of an empty term stores nothing and still makes the key, and it is FT.SUGDEL that clears an empty dictionary away even when the term it was asked for was not there. A hundred and sixteen cases against 8.10.1 agree byte for byte on both protocols, covering the scores, the payloads, the keyspace side, the arity boundary and all seven error sentences, with D-84 and D-85 new for the order ties come back in and for a MAX too large to honour, which takes a real server down and is answered here FT.SYNUPDATE and FT.SYNDUMP are in as well, and the model behind them was the surprise: a synonym group is not a comparison made at query time, it is a term of its own, the group id with a tilde in front, written into the index next to every word of the group, so a query for a word in a group turns into a union of the word and that term and a group of a hundred words costs one posting list rather than a hundred lookups. The group term is worth what the word is worth, the field weight and all, and it counts toward the document's largest frequency but not toward its length, which is half the treatment a stem gets. The lookup is on the word and never on the stem, so a group holding running does nothing for a document that only said runs, and a stopword is dropped before the lookup so a group holding the never leaves anything behind. A synonym update without SKIPINITIALSCAN reads the whole index out of the keyspace again and renumbers every document, and it does that even for an index that was created with SKIPINITIALSCAN, so the two flags have nothing to do with each other. Terms fold on the way in with the query fold and group ids do not, so ECOLE dumps as ecole and an emoji dumps whole while G1 stays G1. Twenty nine cases against 8.10.1 agree on both protocols on everything but the dump order, covering the searches, the explains, the errors, the alias, max_doc_id across a rescan and the TFIDF and BM25 breakdowns, with D-86 new for one false positive above U+FFFF and D-87 for the order a dump comes back in. FT.SPELLCHECK is in, and its model is not what anybody would have designed: the score a suggestion carries is how common the word is with nothing about the edit distance in it, the check that decides a word is already spelled right is blind to the field the query named while the search for candidates honours it, the query is parsed with the stopword list off because a real server checks then and suggests thin for it, and a tag value is asked about with the field it named dropped so @g:{helo} is answered out of the text rather than out of the tag values. Ties in the reply are not in alphabetical order, they are in the order a trie over the suggestions reads them out, a node's own word before its children and children ordered by the best score anywhere under them, which is why hellp answers hello, help, hell, helm, hallo. One hundred and twenty three cases against 8.10.1 agree on both protocols, with D-88 new for the handful of words a real server's hand built automaton lets through at one edit more than the distance asked for. Two things fell out of the same work: the fuzzy edit distance counted bytes where a real server counts characters, so %ecole% now reaches école, and PARAMS counted its words in the wrong order on FT.SEARCH and FT.AGGREGATE as much as here. FT.CONFIG is in with all sixty nine settings, their defaults, their descriptions and their bounds read off a real server, at zero differences over a hundred and forty three cases on both protocols. Nothing about it globs, the single word star is the only way to ask for all of them, SET checks the name then whether the setting can move then the value and only then that there were extra words, and that last one is the status EXCESSARGS after the value has already been written. A number is read strictly the way Redis reads an integer and then loosely the way C reads a double, so 0x10 is sixteen and 010 is ten at the same time, and which of the two readers found a negative changes the sentence it gets back. The settings are not in the keyspace, so a flush leaves them where they were while the dictionaries beside them go. D-89 is new for the gap between remembering a setting and honouring it, which is being closed one setting at a time. FT.PROFILE is in as well, which runs a search or an aggregation and hands back the ordinary reply with the tree the query turned into and the list of steps the rows went through bolted onto it, verified over thirty nine cases on each protocol against 8.10.1 covering every iterator kind, LIMITED, the search steps, the pipeline steps and every error line, with one registered divergence in D-90 for the times that are not taken and the two shape differences that go with them. Vectors are stored now, so a KNN and a VECTOR_RANGE both answer instead of matching nothing. A field keeps every document's vector at full precision in one flat run of floats and both questions are a pass over it, which is exact, so an HNSW field gets perfect recall and pays a linear pass for it, and that is D-91 until the graph lands. The blob is read at the parse where the schema is, because the same bytes are a different vector in a field of half precision and a field of single, and half, single, double and bfloat16 are all read. A KNN picks the k nearest and then answers them in document order, which is measured, and what the clause in front of it matched is carried through rather than thrown away, because a vector clause narrows an answer and does not rescore it. Four measured error lines come with it for a negative radius and for the three ways a count can be wrong, including the bound of 288230376151711744 that the query text is compared against as a double while a parameter is compared against it as an integer. Eighty five cases against 8.10.1, eighty three identical, and the two left are the runtime options in a KNN bracket. A vector query answers its distance now as well, first in the row, under __v_score unless an AS inside the bracket or a YIELD_DISTANCE_AS after it renamed it, and a range clause shows one only when the query asks. A RETURN matches it on the name the row would carry it under rather than the field a value would have come from, which is why naming it answers it, renaming it answers nothing at all and giving its name to another field answers the distance under that name. A SORTBY can sort on it either way round. The numbers are collected off the parsed tree and worked out again from the store rather than carried out of the walk, so a query with two vector clauses in it answers two distances, outermost first. Moving the sort property lookup to after the parse fixed a precedence bug with it, because a real server reads the whole query before it looks any property up. Fifty comparison cases against 8.10.1 on both protocols at no differences, plus two old bugs fixed on the way, a SUMMARIZE behind a KNN having no words to mark and the summarizer trimming the distance itself. D-92 is new for an old WITHSORTKEYS sigil difference on a non sortable numeric field that turned up in the comparison and has nothing to do with vectors. FT.AGGREGATE answers the same distance as a property of the base row, first, with no LOAD needed at all, and every step of the pipeline can name it, so a LOAD, an APPLY, a FILTER, a GROUPBY, the pipeline SORTBY and a REDUCE argument all take it. Getting it there meant reading the argument list twice, once to check every word and once more with the query already parsed and the yields already on the row, which is what a real server does and which fixed the three phase error precedence we had backwards, the words first, then the query, then the pipeline. An aggregation also keeps the row order a nearest neighbour clause makes where a search sorts it away, ties going to the document written first, and a range clause leaves the rows in document order. A hundred and eighteen cases against 8.10.1 on both protocols, a hundred and sixteen identical, and the two left are the random cursor id in a WITHCURSOR reply. FT.PROFILE prints a vector query properly now as well. A vector step says how it was answered, STANDARD_KNN with nothing in front of it, HYBRID_ADHOC_BF when a clause narrowed it down first and RANGE_QUERY for a range that yields, and a range that yields is a METRIC SORTED BY ID - VECTOR DISTANCE iterator where one that does not is an ID-LIST-SORTED. What a clause narrowed itself down with hangs under the step as a single Child iterator with its own term, count and estimate, where before we printed an empty list and threw the child away, and a Metrics Applier result processor sits behind the index whenever the query yields a distance. Sixty four profile cases against 8.10.1 on both protocols with every time blanked out, sixty two identical, and the two left are the numeric block splitting already registered in D-90. Dropping an index takes its documents with it now, which it did not before: FT.DROPINDEX i DD took the word and left the keys alone, and FT.DROP deletes them by default with KEEPDOCS to say not to. What goes is what the index actually read, off its own document table, so a string under the same prefix or a hash a FILTER threw out stays, and the keys go from the database the drop ran on rather than the one the index was made on. Thirty two cases against 8.10.1 covering all four spellings crossed with every word, all identical, plus the second index over the same prefix and the drop sent from another database. The runtime options a vector clause carries are checked now rather than kept whatever anybody wrote, against the index behind the field and the kind of clause together, which is what decides whether an option exists at all and which of seven errors it gets: EF_RUNTIME only on a graph and only on a clause counting neighbours, EPSILON on a graph or a Vamana and then refused on a clause counting neighbours with a line of its own, a policy and a batch size only where there is something under the clause to run over. The bracket and the attributes hung off it are one list for duplicates, the whole bracket is read before any of it is checked, and the options are then checked one at a time in the order they were written. 396 cases against 8.10.1 across all three index kinds and both clause kinds, all identical, and the 148 case comparison this started from went from 102 differing to none. FT.HYBRID is in now, which takes a text query and a vector query over one index and folds the two rankings into one. The SEARCH section carries its own query, its own scorer and its own name for the score it hands out, and the scorer name is looked up only once the whole section has been read, which is what decides which of two errors a SCORER VSIM gets. The VSIM section is three ordered stages, a neighbour or a range clause, then a filter, then a name for its score, and each of them closes the ones in front of it. Each clause names its own runtime option, K and EF_RUNTIME on one and RADIUS and EPSILON on the other, and checks its own values with a line of its own rather than borrowing the pipeline's. The filter inside a VSIM is query text rather than a pipeline expression, so it is parsed on its own and hung off the vector clause, which keeps every error offset pointing into what the client wrote. The window cuts both branches and a neighbour count only narrows its own further, so the vector cut is the smaller of the two. RRF and linear combining are both in, with the constant, the weights and the window all measured. The whole aggregation pipeline runs on top, and a timeout past the foreground limit comes back with the warning a real server adds when it caps one. The query parser learned the second dialect's numeric comparisons along the way, because the reference reads @n>2 as a query and the VSIM filter hands it one: a comparison is the range it stands for, the six of them cover the usual signs, the field has to be numeric or it is refused before the value is read, and the first dialect has no reading for any of it. A 63 query by 2 dialect comparison of that went from 108 differing to one, an unbalanced paren offset the bracket form already diverges on. Sixty seven cases against 8.10.1 at no differences, and a wider sixty one case run on both protocols with one left, which is the aggregation group order already registered. EXPLAINSCORE and WITHCURSOR are both in now and D-93 is closed. The working goes in front of every row holding the merged number, the line it came from and a tree naming the rank the document took in each branch with the text scorer's own explanation nested inside, and it is refused beside a GROUPBY. WITHCURSOR is a cursor per branch rather than one over the merged answer, answering the two numbers under SEARCH and VSIM with no rows at all, each of them reading back that branch's own rows with that branch's own yield in front of them and every step dropped, and an index has to have room for both or it opens neither. That is 42 more cases on both protocols at no differences. The five deprecated document commands are in as well, FT.ADD, FT.SAFEADD, FT.GET, FT.MGET and FT.DEL, which are the original document interface and are all still working on 8.10.1. FT.ADD writes an ordinary hash and lets the ordinary indexing notification pick it up, and the score, the language and the payload go into that hash under __score, __language and __payload and each one that goes in also switches the matching field on the index definition, which is what makes a plain HSET writing __score do nothing on a fresh index and something on an index one FT.ADD has already touched. 132 cases on both protocols at no differences. An index FILTER is applied on the write path too, so a prefix says which keys to go and read and the filter says which of the values that come back are kept, with the whole model measured off 8.10.1 first: a field the key does not hold is false of every comparison and exists is the only way to see it, inverting a comparison that came back false is true while inverting the read itself is not, a function handed a field that is not there refuses the document, and values are compared as the bytes under the key whatever the schema calls them. That is 60 more cases at no differences, and the create refuses a filter that will not read with the same SEARCH_EXPR line an FT.ADD ... IF answers. D-94 registers the one thing that cannot match, which is that a filter naming no field at all segfaults the reference on the first write under the prefix. _FT.DEBUG is in as well, which is eight of the sixty two subcommands a real server registers and they are the eight that read a structure back rather than driving the collector or the background scan: the term dictionary, one term's posting list, one tag field's values, one numeric field's documents, the document table's row for one key, and the three questions about document numbers. The whole family was measured rather than read, because HELP on 8.10.1 answers a bare list of names with no usage text against any of them, and the probes settled the arity of each subcommand, the seven error lines of which six carry no code word at all, the two protocol shapes, the flag bits, the obfuscated field numbering and the strict integer read. Three of the findings would not have been guessed: DOCINFO looks the key up before it validates its keyword, the obfuscated field number is the field's place in the whole schema rather than its place among the sortables, and a GEO field answers DUMP_NUMIDX because a point is a number underneath. 200 cases on both protocols with four differences left, all of them the HELP list. D-95, D-96 and D-97 register the three things that cannot match, which are the startup gate the reference hides the family behind, the range tree nesting DUMP_NUMIDX reports, and that HELP names the eight that answer rather than all sixty two.
• BF.*, the full RedisBloom bloom surface plus BF.DEBUG, byte compatible with the C module and verified over 1143 raw socket comparisons against 8.10.1 with no differences
• CF.*, CMS.*, TOPK.* and TDIGEST.*, all four shipped, verified over 1714, 2039, 2449 and 2540 raw socket comparisons against 8.10.1 with no differences
• TS.*, 21 of 21 shipped. The whole family is in and verified over 22019 raw socket comparisons against 8.10.1 on both protocols, error sentences and check orderings included. The last three are the 8.10 additions, the two joined reads TS.NRANGE and TS.NREVRANGE and the tail read TS.READ
• Vector sets
• Scripting and functions on mlua, Lua 5.1. The interpreter is in and every command runs, which is EVAL, EVALSHA, EVAL_RO, EVALSHA_RO, FCALL and FCALL_RO, along with SCRIPT LOAD, KILL and DEBUG and the whole FUNCTION subcommand set. It is Lua 5.1 built from source inside our own build, one state per thread so two clients running scripts never wait for each other, with the value conversion measured both ways against 8.10.1 on both protocols rather than read off the documentation, including the order the six named table fields are looked for in and what a field of the wrong type does. The error handling is Redis's own, so a script gets its line number, its script name and its raised table back the way a real server hands them over, which needed a raw C trampoline in front of each library function because a raised Lua value cannot cross a Rust frame and Lua 5.1 drops the caller's frame on a tail call into a Lua function. The sandbox is in too: an empty environment per run, the redis table behind an empty proxy, wrappers on the three base functions that walk round a metatable, and a pass after every run that puts the global table back the way it was found. Verified over 289 raw socket comparisons across four probe suites, with everything that differs registered in D-98 through D-108. All four libraries a real server links into its Lua are in. bit is Mike Pall's LuaBitOp, with the magic number rounding in tobit rather than a truncation, verified over 1968 bodies at one difference which was the count of names on the global table. cjson is Mark Pulford's 2.1.0 ported from the C in the Redis source rather than probed from the outside, which is what got the parse errors right down to the character offset, and it carries the three behaviours nobody guesses: an empty table is an object, a number is fourteen significant digits, and null is a value of its own rather than nil. It is verified over 2990 bodies in five rounds at no differences outside tostring on an address. struct is Roberto Ierusalimschy's, ported from the same directory, and it carries three quirks the C has and nobody would write on purpose: a missing pack argument says got nil where unpack says got no value, a short string is blamed on the argument after the one it came from, and size ignores an option it does not know while pack refuses the same one. It is verified over 1092 bodies in three rounds at no differences, including 420 randomly generated formats packed and unpacked again, with one registered divergence in D-107. cmsgpack is Salvatore Sanfilippo's lua-cmsgpack 0.4.0, ported from the same directory, and it keeps the two mistakes the C has been making for years because a script has been told them all along, which are the complaint about argument zero when pack is called with nothing and the length of the input reported as the limit when a bad offset is refused. It is verified over 250 bodies in four rounds at no differences, covering every pack shape, every unpack path, the offsets, the members and every failure. Wiring it in also closed a hole in the guard that sits in front of every library table, where a read from Rust rather than from Lua saw a whole library as empty, so cjson.encode(bit) now complains about a function the way a real server does instead of answering an empty object. pairs(_G) is 36, which is what a real server answers, and the one thing left that does not match is the order a script walks a table this server builds in its own prelude, which is D-108. The library surface is in as well. A library is a script loaded once and called by name forever after, and the registry here holds data and never a Lua value, so each thread compiles a library the first time it runs one of its functions and the digest of the code beside the compiled callbacks is what tells a thread its copy has been replaced under it. A real server keeps a whole second Lua state for functions and this keeps a second set of globals inside the same state instead, which gets the same 35 names a real server reports, keeps KEYS from an earlier EVAL out of reach and is stricter about writes in exactly the way a real server is. The shebang parser, the three case rules across the two dictionaries, the check order in FUNCTION LOAD, the fourteen sentences redis.register_function turns arguments away with and the order FCALL checks a name, a key count and the read only flag in were all measured against 8.10.1 first. Verified over 110 raw socket comparisons in two suites at seven differences, all of them accounted for, with the FUNCTION LIST order registered as D-109 and a library's own variables being per thread rather than per process as D-110. FUNCTION DUMP and FUNCTION RESTORE are in too, which is the pair that moves a set of libraries between servers. The payload is an RDB payload and not something invented here, so the framing sits in yo-kv beside the rest of the RDB and the command file reads none of it, and a payload written here loads into 8.10.1 and one written there loads here, compressed libraries included. A restore compiles every library in the payload into an empty registry before it takes any of them on, so a payload that is going to be refused leaves the server exactly as it was. That work also fixed a payload with no checksum being refused everywhere, which is what a server with checksumming turned off writes. Verified over 45 more raw socket comparisons at one difference, the version byte in the footer, registered along with the missing compression as D-111.
• Transactions, which is MULTI, EXEC, DISCARD, WATCH and UNWATCH. All five are in and verified against 8.10.1 over sixty six scripted cases plus raw byte comparison of the four EXEC reply shapes on both protocols. The watches live beside the key on the server, each row carrying a stamp and whether the key was there when it was watched, which is what reproduces the rule that deleting a key that was never there does not fail a watch on it. Three differences are registered as D-112, D-113 and D-114.
• Pub/sub, which is SUBSCRIBE, UNSUBSCRIBE, PSUBSCRIBE, PUNSUBSCRIBE, SSUBSCRIBE, SUNSUBSCRIBE, PUBLISH, SPUBLISH and PUBSUB. All nine are in and verified against 8.10.1 over fifteen scripted RESP2 cases and five RESP3 cases at no differences, covering the three unsubscribe shapes, the RESP2 subscribe gate, the introspection subcommands and the help text. The three namespaces are kept apart the way Redis keeps them apart, so a channel, a pattern and a shard channel can share a name and hear none of each other, and the count on a subscribe reply is channels plus patterns for the first two families and shard channels alone for the third. Getting a message from the thread that published it to the thread that owns the subscriber goes through a mailbox per thread, held next to the registry rather than in Local, because Local is cache line aligned precisely so no other thread writes into it. A publish renders nothing and allocates the body once behind an Arc however many subscribers it reaches, and delivery happens in the flush the thread was already about to do, so a subscriber costs nothing until there is one. The RESP2 gate sits where server.c puts it, after the no_multi and OOM refusals and before the MULTI queue, which is what makes EXEC sent by a subscriber an EXECABORT while a command handed over by EXEC is never asked. Two divergences are registered, D-115 for the millisecond a message can wait on a sleeping thread until there is a poller wakeup, and D-116 for the order PUBSUB CHANNELS answers in
• Keyspace notifications, which is notify-keyspace-events and the six channels it publishes on. The machinery, the setting, all six collection classes and the four subkey channels are in, so a client subscribed to __keyspace@0__:mykey or __keyevent@0__:del hears about every write those groups make, and a client subscribed to __subkeyspaceitem@0__:mykey\nfield hears about the one field it cares about. Four hundred and eighty seven cases were run against 8.10.1 over a raw socket under twenty one settings of the flag, comparing every channel and payload in the order they arrived and the replies alongside them, and they agree on all of them. Most of the work in a class is deciding when to stay quiet, because the reply and the notification answer different questions: LTRIM says so even when nothing moved, SMOVE onto the set the member came from says nothing, ZADD on a member already at that score is not a write, HDEL k a b answering one has not said which of the two fields went, HGETEX ... PERSIST replies the value whether or not there was a deadline to take off, and anything that took the last of a collection says the key went with it. A body says what it did through a thread local the funnel arms before the command and drains after, so a call site reads one word and tests a bit in it and does not touch the server, the pub/sub lock or the key, which is what makes a server with the setting off pay nothing across several hundred call sites, and the questions a call site has to ask to know what to report sit behind the same check. EXEC and a script re-enter the funnel per command, so their notifications come out between the commands rather than all at the end, which is the reference's order, and a blocking pop served late publishes from the waiter path. The stream class is the one that breaks the pattern: nothing there is followed by a del, because a stream with no entries left is still a key, and XACKDEL says deleted about an entry that was already gone from the log and fires nothing for it. The hash class is the one that carries field names, on four channels that sit outside A because they are extra copies of events already in it, with a length prefixed comma joined field list and two rules about a key or an event name that would not read back apart. Keys that reached their deadline and keys an eviction took say so too, which is the first pair of events here that no command asked for, so they travel out of yo-kv on a hook rather than out of a call site, and neither of them drags a del along behind it. That pair is also what turned up the fact that the active expire cycle had never run in the deployed server, since the sweep hung off a maintenance slice the serve loop does not use, so a server that wrote under a deadline and never read back held every key until somebody looked. A key coming into being goes out the same way and for the same reason, since whether the name was free is no more part of a write's answer than a deadline passing is, and the one place that knows is the single function underneath every group that puts a record in the map. Eighty two more cases covering every way a key can be created agree with the reference, including the position, which is in front of the write that caused it. Getting that right meant fixing three places where a key arriving and a value changing had been treated as the same thing: RENAME, COPY and RESTORE now take the whole destination record away rather than writing over its body, and SORT ... STORE now keeps the record where it stands rather than taking it away. A write that throws away the whole of what was under a name says so as well, and says the kind changed when it did, which is a line between replacing a value and reaching into one rather than a rule about writes: SET, MSET, GETSET and every store form are on one side of it and APPEND, SETRANGE, INCR, RPUSH and SADD are on the other, the same way round as the reference. RENAME and COPY say that pair last where everything else says it first, on both sides and for the same reason, which is that they take the destination away and put another key in its place, so nothing is left under the name to notice it going. Six commands that had been writing in silence now say what they did, and the rule they share is a dirty guard rather than the fact that a write ran: SETBIT and BITFIELD say setbit only when a bit came out different or the value had to get longer, so SETBIT k 1 0 on a bit that was already zero says nothing while SETBIT k 1000 0 on a short value says it, and neither half of that is readable from the reply, since SETBIT answers the old bit and a value that was created or padded reads back as a zero bit either way. PFADD says pfadd when a register moved or the key was created and PFMERGE says it on the destination whatever it merged, including a merge with no sources at all, both under PFADD's name and on the string class because a sketch is stored as a string. GEOADD says zadd on the sorted set class when it added or moved a member, which is the ZADD underneath saying what it did, GEOSEARCHSTORE says geosearchstore and both writable GEORADIUS forms say georadiusstore with GEORADIUSBYMEMBER folded in, and a store that found nothing takes the destination away and says del. The cross database pair was wrong until this went in as well, since MOVE k 1 and COPY a b DB 1 were putting the news of a new key on the database the connection was on rather than the one the key landed on, which nothing caught because the birth notice comes from underneath where there is no database number to be had. Reads that found nothing now say so as well, on the class Redis spells m and keeps out of A. A real server fires that one from inside lookupKey, which covers every command at once because every read there goes through the same function, and there is no such funnel here, so the question is asked in front of the command instead, out of a table of which of a command's arguments are keys it is going to read. The ordinary case in that table is a read only command in a group whose values live in the keyspace, and around it sit the forms whose keys are behind a count, the store families' sources rather than their destinations, the two stream reads whose keys hide after STREAMS with XREAD looking at each of them twice, and the dozen writes that read something before they change it, which is GETDEL, GETEX, GETSET, COPY, DELEX, SORT, PFMERGE, MIGRATE, the stream acknowledgements and SET and BITFIELD in the one shape each where they read. Three rules came off 8.10.1 over a raw socket and each has a test behind it. A command that failed while it was still reading its own arguments never reached a lookup and says nothing, so the misses reported in front of it are taken back once the error says which half it was. A command that failed on what it found keeps what it had already said, so SINTER nk s with a string at s says the miss and then answers WRONGTYPE. And a read over several keys stops where the command stops looking, so SINTER s nk says nothing at all, while MGET l nk says it because MGET answers a nil for the list and carries on. The keys a SORT pattern names are the one set nothing could ask about in front of the command, since they are built out of the elements while the sort is running, so those travel out on the same hook the storage layer already uses for the events no command reports. About a hundred and eighty cases across four settings of the flag agree with the reference, and the four hundred and eighty seven case suite still shows only the one known ordering difference. The module reads say it too, and for the same reason from one level up, since the module API's own key opener goes through the lookup that fires it, so a read only module command reads the keys its table row names the way a core one does. Four things are not that and all four were measured rather than reasoned about: TS.INFO and CF.COMPACT say nothing though both are flagged read only, the search group says nothing at all including FT.GET and FT.MGET against an index that exists with the two suggestion dictionary reads the exception inside it, JSON.DEBUG keeps its key behind the subcommand, and TDIGEST.MERGE and CMS.MERGE read a destination and then a list of sources and stop at the first source that is not there, differing at the destination because the t-digest reads its own and the sketch writes to its own. The retraction that takes a miss back off a command which failed on its own arguments does not apply to them, since theirs is the other order and the key is opened before the rest of the line is read. A hundred and forty one module cases agree with the reference. keyspace_hits and keyspace_misses in INFO stats are in as well, off the same question and the same list of names, because a real server counts them in lookupKey beside the miss it fires there and skips both for the same reason, which is a lookup on the way to a write. They are not counted from the walk in front of the command, though: the walk costs a stripe lock and a map probe per key, the notification is off on nearly every server and the counters are always on, so the counting happens where the lookup already is and the dispatcher arms it once per command with the one fact the storage layer cannot know. What that leaves is a shape difference rather than a rule difference, since a real server reaches its lookup once per key it was sent and this one reaches its own as many times as the command needs, so ZRANGE asks for the window and then walks it and GEOSEARCH asks whether the key is there and then where the centre is and then reads it. A guard turns the counting off for the rest of a command once the lookup that counts has happened, in nine places, each next to the lookup it is about, and the half of it that covers a key on the way to a write is Redis's own LOOKUP_WRITE flag under another name. A hundred and eighty five cases across every group agree with 8.10.1 in both the hit shape and the miss shape, with XREAD the one exception at D-121, since a real server looks each stream up twice and counts both. The measurement needed a subtraction of its own, which is D-122: a real server tells its modules about every write whatever the flag says, one of the bundled ones opens the event key in database zero when it hears, and those lookups land in the same two counters, so each case was run in database nine twice to take the module's share back out. CONFIG RESETSTAT puts both back, which costs holding each stripe for as long as it takes to write two zeroes, because they sit on the stripes beside expired_keys rather than on a line every thread would have to own to write to. Hash fields that reach a deadline of their own say hexpired now, and that closed D-117. The reap names every field it took, the four subkey channels carry the list, and a hash left with no fields says del after it the way the last HDEL does. The active half of it is a second cycle with a list of its own, because a field deadline lives inside the hash body rather than in the record, so the marked index the key sweep draws from cannot see one and is spoken for by the volatile eviction policies anyway. The keyspace keeps the names of the hashes that have ever taken a field deadline instead, a name goes on once and comes off when the key is gone or is no longer a hash, and the cycle walks that list with a cursor for at most one pass a tick, so a hash nobody reads loses its fields within a tick of the deadline the way a real server's does and a server that has never used the HEXPIRE family pays one comparison for the whole thing. All twenty five hash field expiry cases agree with 8.10.1 now where twenty of them differed, and INFO stats gained expired_subkeys and expired_subkeys_active alongside, with CONFIG RESETSTAT clearing the expiry and eviction totals the way resetServerStats does. What is left is the one place SMOVE puts the news of a new key in a different order, which is D-118, and the replies a module read gives at an empty key when an argument behind it does not parse, which is D-120.
• Replication with PSYNC and partial resync. The master half is in and verified. A real redis-server 8.10.1 pointed at yo with replicaof attaches, loads the dataset, follows the command stream and ends up with the same keyspace, checked over 78 named cases covering every rewrite and random workloads of 4000 commands over four databases comparing values, deadlines, per field hash deadlines, stream contents, group state and pending lists with delivery counts, at nothing different. REPLCONF, PSYNC and SYNC are shipped, the propagation shapes were sniffed off a real 8.10.1 master rather than taken from the documentation, expiry and eviction cross as deletions through the keyspace event hook, and partial resync is verified by breaking the link and watching the replica log a successful partial resynchronization. WAIT counts real replicas, ROLE and INFO replication report them, and a replica carries the S flag in CLIENT LIST. The replica half is in too. REPLICAOF, SLAVEOF and a --replicaof flag spelled the way redis-server spells it point a server at a master, and the link dials it, walks the handshake, takes the snapshot and applies the command stream forever through the ordinary dispatcher, so notifications fire and the search indexes are kept up on the replica the same as anywhere else. --masterauth, --masteruser and --replica-read-only go with it, CONFIG reads and writes all three, and DEBUG CHANGE-REPL-ID is in. Verified against a real 8.10.1 master four ways: a single hop with matching digests and offsets, three consecutive partial resyncs giving sync_full:0 sync_partial_ok:3 sync_partial_err:0, a four node chain of a real master and two yo servers and a real replica landing on one digest and one offset across all of it, and a promotion in the middle of that chain that kept both sub-replicas online with no resync at all. FAILOVER is in now, which is the whole of this box. It pauses writes with the same pause CLIENT PAUSE arms, waits for a replica to acknowledge every byte ever written, then becomes a replica of that one and tells it to become the master in the same breath by putting a fourth word on the PSYNC it sends, so the old master gets a CONTINUE and no snapshot and the pause lifts, and from a client the whole thing is one slow write. TO, TIMEOUT, FORCE and ABORT are the reference's including its parsing quirks, the five refusals are the reference's sentences in the reference's order, master_failover_state reports all three words and a PSYNC or SYNC from anybody else while one is going on is NOMASTERLINK. Verified against real Redis 8.10.1 four ways: the argument surface at 38 of 39 identical with the one difference being the COMMAND DOCS arguments list which is D-13, a yo to yo handover settling instantly at a shared offset with no snapshot and the old history kept as master_replid2, a real Redis master handing its job to a yo replica and then following it at a matching digest, and a yo master handing its job to a real Redis replica which logged the failover request and accepted a partial resync of zero bytes at sync_full:0 and sync_partial_ok:1. The failure paths were exercised with the replica stopped, so waiting-for-sync in INFO, REPLICAOF and a second FAILOVER both refused with the reference's sentences, CLIENT UNPAUSE answering OK without lifting the pause, a held write coming back after the timeout with everything put back, and FAILOVER ABORT freeing a held write in half a second.
• Cluster with 16384 slots and CLUSTER MIGRATION. The foundation is in, merged as Cluster mode: the slots, the table and the redirections #520, which is the slot table, the whole CLUSTER container and the redirections. Every key maps to one of the 16384 slots by the reference's own recipe, which is CRC16 over the part between the first brace and the next one when there is one and over the whole key when there is not, checked against a running server on the cases that make people trip, an empty key and an empty tag and an unclosed brace. The container answers thirty one subcommands, so that is the reports NODES, INFO, SLOTS, SHARDS, LINKS, MYID, MYSHARDID, SLOT-STATS, COUNTKEYSINSLOT, GETKEYSINSLOT, KEYSLOT, REPLICAS, SLAVES and COUNT-FAILURE-REPORTS, the slot editing ADDSLOTS, ADDSLOTSRANGE, DELSLOTS, DELSLOTSRANGE, SETSLOT and FLUSHSLOTS, the state changing RESET, BUMPEPOCH, SET-CONFIG-EPOCH, SAVECONFIG and FORGET, and the ones that answer their real refusal because the bus is not in yet, MEET, REPLICATE, FAILOVER, MIGRATION and SYNCSLOTS. The routing gate follows getNodeByQuery in the reference's own order, so an unbound first slot is reported before a cross slot is, and it sits in the dispatch chain after the ACL check and before the memory one, which is where the reference has it. That gives MOVED for a key this node does not own, ASK for a key that has already gone from a slot on its way out, TRYAGAIN for a command whose keys are split across a slot half way through a move, CROSSSLOT for a command touching two slots, CLUSTERDOWN when a slot in the range is not covered, ASKING for one command's life the way CLIENT REPLY SKIP does, and READONLY and READWRITE on the connection. A transaction is refused when it is queued rather than when it runs, so EXEC answers EXECABORT, which is what the reference does. The node table is kept on disk in the reference's nodes.conf format including the aux fields, which are deliberately left off what CLUSTER NODES puts on the wire because a client that learned to parse them would break the next time somebody added one. Both of the reference's two state delays are modelled, the two seconds a node covered at boot waits and the five seconds a node that was uncovered and then healed waits, which was measured off a real server's own log rather than read out of the source. Measured against a cluster enabled redis-server 8.10.1 built from source over 174 cases at 173 identical, the one left being the empty subcommands array on COMMAND INFO which is D-114 and is not about cluster, plus nine unit tests for the redirections since a single node cannot be made to produce a MOVED on its own. Two divergences came out of it, D-149 for MEET and MIGRATION IMPORT checking every argument in full and then refusing because the bus is not in, and D-150 for the per slot key counts walking the whole database once rather than keeping a counter per slot. The cluster bus is in now too, merged as The cluster bus #522, which is the binary protocol nodes talk to each other over on the client port plus ten thousand. The wire format is 8.10.1's own, a 2256 byte header carrying the signature and the length and the protocol version and the sender id and its slot bitmap and its epochs and its replication offset and its flags and its state, then either a gossip section of 104 byte entries or a typed body, with all eleven message types in. The extension block on a ping carries the shard id and the internal secret, padded the way getAlignedPingExtSize pads them, and the secret converges on the lexicographically smallest one anybody has seen. A listener thread takes the bus port and a cron thread on a hundred millisecond tick dials out to nodes with no link, pings the node that has been quiet longest, sends MEET to a node still carrying the meet flag, marks a node that has not answered for the node timeout as possibly failed and promotes that to failed once a majority of masters agree. Gossip about a node nobody knows creates it with the id and the flags that were gossiped rather than as a handshake node, which is how the third node in a cluster is discovered without anybody meeting it directly. Verified live on a four node cluster of one real redis-server 8.10.1 and three yo nodes, where MEET works in both directions, the third node is learned purely through gossip, all four agree on CLUSTER NODES and CLUSTER SLOTS and cluster_state and cluster_slots_assigned and cluster_known_nodes and cluster_size, slots assigned on one node show up on every other, MOVED points at the right node from every node, a PUBLISH on a yo node reaches a subscriber on the real redis node, CLUSTER REPLICATE makes a yo node a replica the reference lists correctly, and the table and the replication link both survive a restart of every yo node. Four things came out of getting that to agree. CLUSTER SLOTS walks slots ascending grouping consecutive slots with the same owner rather than walking nodes, which is what clusterCommandSlots does and what a client caching the reply by position depends on, and it filters replicas the way isReplicaAvailable does so one whose replication offset is still zero is left out. The node table on disk is no longer read back as live state, so the link state field is skipped, the two timestamps come back as now, and a replica's config epoch loads as zero, all three matching clusterLoadConfig, without which a restarted node believed it was already linked to everybody and never dialled out. A node starts with both timestamps at zero, which is why the myself line reads 0 0 on a real server, and the handshake timeout is measured against the creation time instead. A master sends a PING down a replication link that has been quiet for ten seconds, without which a replica's offset never moves and it stays out of everybody's CLUSTER SLOTS. Three more divergences came out of it, D-151 for the bus running on threads of its own rather than the shared poller D-115 wants, D-152 for a replica not standing for election when its master fails, and D-153 for the replication timing settings not being in the config table, and D-149 is rewritten to say what MIGRATION IMPORT is really waiting on. Moving a slot between two nodes works now, merged as Moving a slot between two nodes #524. A slot migration is four steps and the third one never worked, because MIGRATE was sending a plain RESTORE and the node being sent the keys does not own the slot yet, so every key came straight back as a MOVED pointing at the node doing the sending. The reference solves that with a second spelling, RESTORE-ASKING, which is the same command with an ASKING built into it, and it switches to that spelling on one condition and nothing else, whether the sending server is a cluster node, so a MIGRATE between two servers that are not in a cluster still sends the ordinary word and still works against a server too old to have heard of the other one. The routing gate reads the command's own asking flag rather than knowing the name, which is how the reference spells it too. The fourth step was wrong in a way that only showed up on a cluster of more than two nodes, since SETSLOT NODE moved the owner and saved the file and stopped, so the node that had just taken the slot was making a claim on it with the same config epoch it started with and every other node quietly kept the old owner. That arm is rewritten to match clusterCommandSetSlot, so the target has to be a master, the slot is not handed to somebody else while this node still holds keys for it, a master that has just given away its last slot follows whoever took it, and closing an import takes a config epoch above everybody else's and tells the whole cluster right then rather than waiting for the next ping. CLUSTER BUMPEPOCH is the same code underneath and now answers STILL when there is nothing left to outrank, which is what a real server does on a second call. Verified on the four node mixed cluster rather than by reading it, moving slot 6001 in all three directions, yo to yo, yo to reference and reference to yo, fifteen checks each and all four nodes agreeing on the new owner afterwards, with the three refusals word for word what a real server answers and a new 22 case differential of the SETSLOT and BUMPEPOCH arms identical on both. What is left on this row is the other way of moving a slot, which is atomic slot migration in D-149, and the failover vote in D-152. The first of those turned out to be blocking rather than optional: redis-cli 8.10's own resharding tool picks atomic slot migration whenever every node in the cluster reports a version at 8.4.0 or above, and there is no flag and no fallback to make it pick the path that now works, so an operator running the shipped tool against a cluster of these still cannot move a slot. The way into that path is in now, merged as The internal connection #525. Atomic slot migration does not run over the bus, it runs over an ordinary client connection that one node opens to another, so the first thing a node needs is a way to prove it is a node, which is AUTH with the name internal connection and the forty character secret the bus already gossips until every node agrees on one. CLUSTER SYNCSLOTS is what that opens. It is refused to anybody who cannot produce the secret and the connection is dropped along with the refusal, which is worth copying, because nothing behind that command is defended against being driven out of order and the gate is the whole defence. Every arm of it is checked in full and answered in the reference's own sentences, including the slot ranges being sorted and joined up where they touch before they are checked, so that a slot named twice is caught and a pair of ranges that meet is one range, CONF handling node-id, slot-info, capa and asm-task and carrying on past an option it did not understand to still answer OK at the end, RDBCHANNEL having a real answer on a node with nothing running, and the arms that expect no reply answering with none. DEBUG MARK-INTERNAL-CLIENT and DEBUG INTERNAL_SECRET came with it, since they are how any of this is driven by a test without standing up a second server. Checked over a new 48 case differential against 8.10.1 covering every arm, every arity and both sides of the gate, at no differences. The giving up side of that path is in now, merged as Send a slot range as a snapshot #529. A node that has logged in as the cluster asks for a slot range with CLUSTER SYNCSLOTS SYNC, gets a task created for it and is told to open a second connection, and when it opens that one with CLUSTER SYNCSLOTS RDBCHANNEL it is sent a snapshot of exactly those slots. The snapshot is a stream of ordinary commands rather than an RDB file, which is the only way a sixteen thousandth of the keyspace can be sent at all, so it is the libraries, a SELECT for every database that holds anything, a count of the keys and the deadlines of each slot in front of the first key of it, the keys themselves, and an end marker. A string goes as the SET that would set it with its deadline behind it, which is what the reference does so the far side can take a large value apart as it arrives rather than having to hold all of it before any of it is usable, and everything else goes as a RESTORE with an absolute deadline. It is taken with every write on the server held off, which is what this has instead of a fork, so what the far side gets is one moment of the dataset rather than a smear across several, and that freeze already existed for a full resync and has been pulled out so both callers use the one piece of machinery. CLUSTER MIGRATION STATUS and CANCEL report and cancel the real task now instead of answering empty and nought, with the reference's twelve field map, its state words, its error sentences and its bound on how many finished tasks are kept, and a connection that was carrying a migration going away fails the task, because half a slot range on the far side is the one outcome nobody may be left with. Checked with a new differential that plays the destination side of the handshake against a real 8.10.1 and against this in turn and compares the two snapshots command by command, at every line matching, with the only differences left being the two DUMP payload encodings already registered as D-111. The 48 case differential is still at no differences and the 174 case cluster one is still at 173, and thirteen new tests cover the snapshot shape, the string rule, the per slot counts, a key outside the range asked for, the handshake in order and every refusal on both halves of it. The writes that land behind the snapshot are in now too, merged as Stream the changes a moving slot picks up during a migration #532. The snapshot is one instant and everything written after it still has to reach the node taking the slots over, so every command that propagates goes past the migration on its way to the replicas, already rendered in the form a replica would see, and is kept or dropped on the slot its key is in. A command with no keys is dropped, a command whose keys land outside the moving ranges is dropped, and a command whose keys land in two different slots cancels the migration, which is what a real server does with it as well. The stream is switched on inside the same freeze the snapshot is read under, so the two meet exactly and nothing is sent twice or missed. CLUSTER SYNCSLOTS ACK does something now as well, recording how far the destination has got, refusing to go backwards, and moving the task to handoff-prep once the destination is within a megabyte of what has been sent. That came with a second differential that plays the destination side of the handshake, throws the snapshot away and compares the stream of changes command by command against a real 8.10.1, at ten commands and no differences, and it found a replication difference that had nothing to do with migration: SPOP on a set with one member left was crossing as DEL where a real server always sends SREM with the member it took and lets the far side notice it is holding an empty set. The four settings a migration runs under are in the config table now as well, merged as Put the four slot migration settings in the config table #534, which is the lag a handoff is allowed to start at, how long writes may stay paused, how long the far side gets to drain and how many finished tasks are kept, with the reference's defaults, units, ranges, refusals and its rule that two of the four are hidden from a pattern and found only by their own name. That turned up two bugs in CONFIG GET that had nothing to do with migration: a name was matched with regard to case where a real server matches without, so CONFIG GET MAXMEMORY answered nothing at all, and the name a setting is answered under is the one the client spelled when they spelled it out and its own name when they gave a pattern, which is the shape of the reference's code rather than a decision it made. The handoff is in now too, merged as Stop the writes and end the stream when a moving slot changes hands #535. Once the destination says it is caught up the source stops taking writes, waits out the ones that were already running, sends CLUSTER SYNCSLOTS STREAM-EOF and sits in stream-eof waiting for the far side to claim the slots. Reads keep working and a write is held rather than refused, so a client that writes during a handoff sees a pause and not an error, and a far side that never takes the slots is given up on after the write pause timeout with the reference's own sentence, which starts the writes again and leaves the slots and the keys where they were. The reference gets there by waiting for a socket buffer to drain because it is single threaded, and here the pause goes on first and the freeze is used as a barrier, because a write feeds the migration stream while it still holds its stripe and so coming out of the freeze means every write in flight has already been sent. That came with three more differentials against a real 8.10.1, one for the pause, one for the timeout and one for the state a failure is blamed on, all identical line for line, and they found two more compatibility bugs that had nothing to do with the handoff: a failure sentence was naming the state the task had just moved to where a real server names the state it was in when it went wrong, and rdb_channel_state always said none where a real server says completed once the snapshot has gone out. The last step is in as well, merged as Finish a slot migration when the far side claims the slots #536. When the far side claims the slots over the bus with a higher epoch this node believes the claim, finishes the task, lets the writes go and drops the keys behind the slots it does not own any more, writing one TRIMSLOTS naming the ranges so that its own replicas drop the same keys rather than going on answering for data that has moved. The task has to be moving exactly the slots that were claimed, which is the reference's rule and is stricter than it looks, because a claim covering half of what a migration is moving is not that migration finishing but something else happening to the cluster while it ran, and the migration is given up on rather than reported as done. How the drop reaches whoever is following this node has three shapes and they are not a choice, they are three different situations: a migration finishing sends one TRIMSLOTS and fires a del event per key because an operator asked for it and is watching, a slot that changed hands some other way sends a deletion per key and fires nothing because nobody asked for those keys to go, and a TRIMSLOTS arriving from a master fires the events and propagates nothing because the command that asked for the trim is already on the stream. A node that gave away its last slot and is now following the node that took them keeps its keys, since it is about to be sent the whole dataset by its new master and throwing them away first would only mean copying them straight back. TRIMSLOTS itself is new, a top level command rather than a CLUSTER subcommand because that is where the reference puts it, and the refusal in the middle of it is the one that matters: a master will not empty a slot it is still serving whatever it is told, so a TRIMSLOTS that arrives late or names the wrong range cannot take live data with it. That came with a new differential covering eighteen ways of getting the command wrong plus the success path, at 35 same and 0 different, and it turned up one timing difference that has nothing to do with the trim, which is that a slot hole opened and closed between two of the reference's own timer ticks is a hole it never sees where this looks again on the spot and so takes the five second rejoin delay for a hole that lasted a millisecond, registered as D-154. The taking side is in now as well, so CLUSTER MIGRATION IMPORT on a node names the slots it wants, works out from its own view of the cluster which node owns them, refuses the same five ways the reference refuses in the same order, books a task, answers with its id straight away and then goes and gets the slots on a thread of its own. It opens the same two connections the reference opens and logs both of them in as the cluster, asks for the slots on the first and the snapshot on the second, applies the snapshot while the changes made during it pile up unread behind it, works through the pile, follows the stream, says how far it has got about once a second, and when the source says the stream has ended it claims the slots with an epoch bump and a broadcast that tells the rest of the cluster where they went, which the old owner hears and acts on the same way it would for any slot that moved. Writing it turned up two real bugs: a migration was being failed by its own success, because the source held on to the snapshot connection after the snapshot had been read and the destination closing its end looked like a channel dropping under a live move, and an import that failed partway left the keys it had already applied sitting in slots this node does not own for the next attempt to fill on top of. That came with a new differential covering the error surface at 28 of 29 identical to 8.10.1, the one difference being COMMAND INFO for a subcommand answering nil, which is D-114 and is nothing to do with migration. Moves have been driven both ways between two of these nodes and both ways between one of these and the real 8.10.1 in the same cluster, with every type and a key whose deadline was still counting down on the other side, and with all four nodes agreeing on the new owner afterwards. Ninety megabytes across thirty thousand keys moved in 0.58s with a nought millisecond write pause while a writer landed increments on the source the whole time, every counter arriving at exactly the value the source had acknowledged across 19884 increments with nothing lost or repeated across the handoff, and a move cancelled once 2248 keys had already landed left the destination with none of them and the source with all thirty thousand and still the owner. D-149 is now only about a single large collection crossing in one piece where a real server splits anything above 512 items, which no client can tell apart afterwards, and D-155 is new and says a migration that failed here keeps the reason it failed rather than the reason it was replaced. The two ways of moving a slot cannot be used on the same slot at once now either, so CLUSTER SETSLOT for a slot an atomic migration is moving is refused with the reference's own sentence naming the slot and the cancel that gets out of it, which matters because the two ways write the same three fields from two directions and the migration would otherwise finish by claiming a slot the operator had already given to somebody else, checked against a real 8.10.1 with a task running on both servers in the same state. The reference's other gate there is for a slot with a trim job still outstanding, which cannot happen here because a trim runs on the spot rather than being queued as a job, and that is also why its write redirect and its rdb and aof skips have no equivalent. The drain timeout is wired up as well, which was the last of the four migration settings that sat in the config table and was read by nothing. It catches a move that is not broken but never ends, where the far side works through everything that piled up behind the snapshot and says so and then stays further behind the stream than a handoff is allowed to start at because the source is taking writes faster than the destination can apply them, and the deadline is the longer of the setting and twice however long the far side took to get through the snapshot and the pile, since a side that needed a minute for the first part is not one to give ten seconds to for the rest. The span starts where the backlog ended rather than where the move did, which was worth checking on real nodes: with eight writers pushing four kilobyte values at the source faster than the destination could apply them the task sat in send-stream for twelve seconds with the setting at one millisecond and was not touched, and completed on its own the moment the writers stopped with all 23831 keys across and the source trimmed to nothing, while a move with nothing in its way and the same setting finishes in twenty four milliseconds and never arms the deadline at all. The failover vote is in now as well, merged as Let a replica stand for election when its master fails #542, which is the part of a cluster that has to work with nobody watching. A replica of a master the cluster has given up on waits half a second plus a random half second plus a second for every replica of the same master that holds more data than it does, bumps the epoch, asks every node for a vote, and takes its master's slots under that epoch once half the masters that serve a slot plus one have said yes. A master here answers one of those requests with the reference's checks in the reference's order, so it votes only if it serves a slot, only once per epoch, only for a real replica of a master it agrees is failed, not twice about the same master inside two node timeouts, and never for slots that have already moved somewhere under a newer epoch. There is no such thing as a no vote on this protocol, so a master that disapproves and a master that has crashed look the same from where the replica is standing, which is deliberate. The epoch a node last voted in goes in the config file where the reference puts it, so a node that voted, restarted and was asked again in the same epoch does not vote twice, and that was the one field in that file still being written as a constant nought. The flag that says vote for me even though my master is up is honoured, so a real server's replica doing a manual failover in a mixed cluster gets a vote from here. Testing it on real nodes rather than only in unit tests turned up a real bug in failure detection that had been there since the bus went in: a node with no link never gets a ping sent to it and the clock a node is measured against only starts when one goes out, so an address that stopped answering was dialled again every tenth of a second for ever and never marked as failing, which is why the first live run sat at fail question mark for sixty five seconds. A failed dial now starts the clock the way the reference does and an outstanding ping survives a reconnect rather than resetting it. Verified in both directions on real nodes. A yo master killed with a yo replica behind it and two masters watching, one of them the real 8.10.1, was replaced eighteen seconds later with all three slot ranges moved under a new epoch and every node agreeing at sixteen thousand three hundred and eighty four slots assigned, and the old master came back as a replica of the node that had replaced it. The other way round, on a cluster of two yo masters and one real master with a real 8.10.1 replica of one of the yo masters, killing that yo master promoted the real replica eighteen seconds later with all of its keys, and since the quorum was two out of three masters with the dead one out, the vote from the yo node is the one that made it. The manual failover an operator asks for is in as well, merged as Hand a shard over on purpose with CLUSTER FAILOVER #544, which is CLUSTER FAILOVER in all three forms. The plain form sends an MFSTART packet, the master holds its clients still for ten seconds and says so on every packet it sends afterwards, which turns the offset in the header into the last one there will ever be, and the replica stands only once its own offset reaches it. FORCE skips the asking and TAKEOVER skips the election too, bumping the epoch on its own authority. The election underneath is the same one as before with the delay and the rank dropped and a force ack flag on the request, and the pause is lifted rather than waited out on the timeout, a takeover, a win, a demotion and CLUSTER REPLICATE. Verified live in all four combinations with a writer running: a yo replica over a yo master in 0.27 seconds with 16407 of 16407 writes kept, a real 8.10.1 replica over a yo master in 0.07 seconds with 16121 of 16121 kept, a yo replica over a real 8.10.1 master in 0.23 seconds with 16690 of 16690 kept, and both TAKEOVER and FORCE against a real master with the real node stepping down behind them. The error surface matches over eight cases at 8 same 0 different. D-152 is deleted rather than narrowed. That is every part of this row, so it is ticked.
• ACL. The key finder it is built on is in. That is the key spec model Redis added in 7.0 on every one of the 429 command rows, a resolver copied line for line from getKeysUsingKeySpecs, and the per key flags that say whether a caller needs read or write over each key, which is not derivable from the command flags and had to come off the reference one command at a time. COMMAND INFO reports the specs, COMMAND GETKEYSANDFLAGS is in beside COMMAND GETKEYS, and the pair was measured against 8.10.1 over 1818 cases with the key spec field identical on every command both servers have. The ACL itself is in on top of it. A user is a name, a switch, a list of SHA-256 password hashes and one or more selectors, and a selector is a command bitmap, a list of key patterns and a list of channel patterns, which is the reference's model down to the reserved bit that records whether a command added tomorrow would be allowed. The rule language is ACLSetUser and ACLSetSelector written out, on and off, nopass and resetpass, a password or a hash added and removed again, allkeys and resetkeys and the %R and %W prefixed forms, allchannels and resetchannels, allcommands and nocommands, plus and minus over a command or a category, a selector in brackets, clearselectors and reset, with every refusal carrying the reference's own sentence wrapped the way acl.c wraps it. All thirteen subcommands ship, SETUSER, GETUSER, DELUSER, LIST, USERS, WHOAMI, CAT, GENPASS, DRYRUN, LOAD, SAVE, LOG and HELP, and a SETUSER is staged on a copy so a rule that fails halfway leaves nothing behind. The gate sits where processCommand puts it, after the transaction refusal and before the memory limit, and costs one relaxed load on a server whose users can all do everything. requirepass is now one rule on the default user rather than a password of its own, GENPASS takes its bytes from the operating system rather than from the engine's seeded generator, and passwords are stored hashed, which closed D-127. Categories are exact even with one table row a container, because the categories a container's subcommands hold beyond the container's own are written down beside the table, so -@admin takes CONFIG GET away and leaves CONFIG HELP alone and +@connection reaches CLIENT SETNAME without reaching CLIENT KILL. Measured against a real 8.10.1 over 1653 cases, 89 rule sets each followed by ACL LIST, ACL GETUSER and 15 ACL DRYRUN probes plus 30 straight commands, at 15 differences, all of which are ACL CAT listing a category's members where the reference names a container's subcommands one at a time, and none of which is a decision about a user. The one divergence is D-137, a rule naming a subcommand that does not exist. ACL LOG, ACL LOAD and ACL SAVE are in now too, which finishes the row. The log is a ring buffer of refusals, newest first, at most acllog-max-len of them, and every refusal the gate makes writes a row, as does a failed AUTH with the name that was tried rather than the name the connection is on. A row is the count, the reason, the context, the object, the user, the age in seconds, the whole CLIENT INFO line of the connection, the entry id and the two timestamps, and two refusals merge into one row with a count of two when the reason, the context, the object and the user all match and they are within sixty seconds of each other, with the scan that looks for a match reading at most the first ten rows the way the reference does. LOG RESET empties the list and leaves the ids where they were. LOAD and SAVE want a file, which yodb takes as --aclfile, and a server that was not given one answers both with the reference's paragraph about ACL SETUSER and CONFIG REWRITE. SAVE writes one user a line in name order to a temporary file, fsyncs it, renames it and fsyncs the directory, and LOAD applies the whole file to a table staged off to the side so a file with a mistake anywhere in it changes nothing at all and every complaint comes back in one sentence. acl-pubsub-default came with them, since a selector reads it at the moment it is made, and so did the five counters at the end of INFO stats, acl_access_denied_auth, acl_access_denied_cmd, acl_access_denied_key, acl_access_denied_channel and acl_access_denied_tls_cert, which are bumped before the length is looked at so a server with acllog-max-len 0 still counts what it refuses. Measured over a hundred and eighty four cases against a real 8.10.1 covering the log empty and after refusals of each kind, grouping and its ten row scan, RESET, the count argument and how it clamps, refusals inside MULTI, a failed AUTH by password and by HELLO, acllog-max-len at 0, 1, 3 and 128, twenty four whole ACL files including every error sentence the loader has, the save and the bytes it wrote, a file that is not there, both settings through CONFIG GET and CONFIG SET, and the five counters as deltas, at no differences. One divergence came out of it and it is about CONFIG rather than the ACL, D-138, a setting named twice in one CONFIG SET taken here and refused there.
• The six commands Redis 8.10 added: SUNIONCARD, SDIFFCARD, LMOVEM, BLMOVEM, HIMPORT and BACKUP, all shipped and all verified against 8.10.1 with a raw socket
• The connection and server administration commands, which is the CLIENT container, AUTH, DEBUG, MEMORY, SAVE, BGSAVE, BGREWRITEAOF, MONITOR, ROLE, FAILOVER and REPLICAOF. The CLIENT container is done apart from TRACKING, so ID, GETNAME, SETNAME, SETINFO, INFO, LIST, KILL, PAUSE, UNPAUSE, REPLY, NO-EVICT, NO-TOUCH and HELP, all measured field by field and sentence by sentence against 8.10.1 over a raw socket, with the forty fields of the report in Redis's order and its own numbers registered as D-123. Every connection publishes a row the other threads can read, which is what LIST and KILL are built on, and PAUSE is one word on the server that every command reads once. MONITOR is done too, the whole feed measured against 8.10.1 over seventy cases at no differences, which is the line format and the sdscatrepr quoting, the database a command is reported on, what reaches the feed and what never does across every container subcommand, the transaction order, the script order with its own calls carrying lua, the password redaction, the replica refusal a monitor gets when it touches a key, the pause exemption, the O flag letter and two monitors getting byte identical lines, with the pipelining order registered as D-124. SAVE, BGSAVE, BGREWRITEAOF, LASTSAVE and ROLE are done as well, built on the snapshot writer yo-kv already had, with the file written to a temporary name and synced and renamed the way the reference does it so a reader never sees half a file, measured against a redis-server 8.10.1 built from source with a writable directory over sixty one cases at one difference which is the pre-existing empty tips on COMMAND INFO, and with a real redis-server started on the dump.rdb yo writes answering all fourteen probes identically and redis-check-rdb reading the file and saying it looks OK, with the background save that is not in the background registered as D-125 and the rewrite that writes nothing as D-126. AUTH is done and requirepass with it, one password belonging to a user called default, a NOAUTH gate the whole command table passes through, the four no_auth commands getting past it, HELLO taking the AUTH option with its three complaints in the order the reference gives them, RESET putting a connection back to needing the password and a --requirepass flag on the server, measured at no differences over fifty seven cases against a redis-server 8.10.1 built from source, and the password is stored as a SHA-256 now that there is an ACL, which is what closed D-127. DEBUG is in as far as the subcommands that are about this server go, which is sixteen of them: HELP, PROTOCOL, ERROR, LOG, SLEEP, POPULATE, SET-ACTIVE-EXPIRE, PAUSE-CRON, DICT-RESIZING, SET-SKIP-CHECKSUM-VALIDATION, QUICKLIST-PACKED-THRESHOLD, RELOAD, OBJECT, SDSLEN, LISTPACK and QUICKLIST, with all thirteen PROTOCOL types answering the exact bytes 8.10.1 answers on both protocols and three of the four knobs gating something real rather than being remembered, measured at 194 of 196 cases identical against a redis-server 8.10.1 built from source, the two left being COMMAND DOCS having no doc_flags field which is D-130, and with the threshold nothing reads registered as D-128 and SLEEP holding one shard thread rather than the server as D-129. RELOAD is the one that is not about a knob, it writes the whole dataset to the dump file and reads it straight back, so it is the first real caller of the whole file RDB reader and the thing most of the Redis test suite leans on to check that a value survives a trip through the file. It takes MERGE, NOFLUSH and NOSAVE, refuses any other word, keeps the dataset intact when the file will not parse and leaves a half loaded database when the file goes wrong halfway, which is what the reference does, and it was measured against a redis-server 8.10.1 built from source over eleven cases and a fifty three command battery covering every type, the encoding of each, hash field deadlines, stream groups and pending entries, three databases and a search index, at no differences. Three divergences came out of it, MERGE being accepted and doing nothing along with NOFLUSH on its own not taking the server down the way 8.10.1 does in D-131, a reload not being one instant of the keyspace above one thread in D-132, and a refusal rather than a silent drop when a key has no RDB form in D-133. The four that only look are in now as well. None of them change anything and none of them count as a use of the key they are about, which is what a suite reading idle times across a run depends on. OBJECT answers the reference's line with the encoding, the serialized length, the LRU clock and the idle seconds all exact, plus the five quicklist fields on a list that broke into nodes, where the serialized length is the RDB body with the framing taken off and the LRU field is the clock now less the idle seconds wrapped into twenty four bits, both worked out against a running 8.10.1 rather than read off the source. SDSLEN answers the two lengths exactly. LISTPACK and QUICKLIST write the structure to the server's own output and answer the sentence saying so, each taking the representation it is named after and refusing the other whatever type the value is. Measured over 37 cases against a redis-server 8.10.1 built from source, covering all four over ten values, both error sentences on each, the wrong argument counts and the optional level argument, with everything matching once two things are set aside. Those are the at pointer, which has no robj here to point at and is D-134, and the four allocator numbers on the SDSLEN line, which are jemalloc questions that do not exist here and are D-135. Three cases are left after that, the serialized length of a list and of a stream, which is the RDB shape difference D-111 already covers seen through a second command. DIGEST and DIGEST-VALUE are in too, which makes eighteen subcommands. They are the pair the Redis suite leans on hardest, since checking that a replica caught up or that a reload came back is one number held against another, so the recipe is copied to the byte: xor for anything unordered and mix for anything ordered, where mix is SHA1 of the digest exclusive ored with SHA1 of the bytes, the object type folded in first as four big endian bytes, and the words !!expire!! and !!hexpire!! folded in for a deadline without the deadline itself so the number does not move as the clock runs. A score goes in the way fpconv_dtoa writes it rather than the way ZSCORE replies it, which differ for a whole number too big to be exact. Measured against a running 8.10.1 over 33 wire cases and then over a randomised 123 key dataset covering every type, hash field deadlines, key deadlines, several databases and an empty server, at no differences, with scores checked separately over 21 shapes including both infinities and the largest double. The one difference is a negative zero score in a listpack encoded sorted set, which is the reference losing a sign that yo keeps and is registered as D-136. MEMORY is in now, all six subcommands. USAGE adds up three things, the record in the arena which holds the header and the name and, for a string, the value itself, the body hanging off the record for a type that has one, and one key's share of the index, which is the same three a real server adds up under the names sds key, object and dict entry, so the shape of the answer matches even though none of the three numbers does. It is exact rather than sampled, because every collection already keeps its own byte count for maxmemory, so SAMPLES is parsed and checked and then changes nothing, which is D-6. A value that has been written out to the file is counted as the record alone, because that is all of it that is in memory, and it is deliberately not read back in to answer, since asking what a key costs should not be the thing that makes it cost that. STATS answers the same 36 fields in the same order plus one db row per database that is holding a key, so an empty server answers 36 pairs and a two database server 38, and the fields that are zero are zero either because nothing is behind them yet, like replication and cluster and the append only file, or because of a real design property, since a deadline lives in the record rather than an expires table and the index is never caught half rehashed. The four allocator fields are mapped onto the arena, which is the thing that actually fragments here, so allocator-fragmentation.bytes is the dead records that have not been compacted away yet. DOCTOR reports five of the eight complaints the reference can make, the peak against the total, the two fragmentation ones, the client buffer one and the script count one, with the thresholds and the wording taken from the reference, and the three left out need the process resident size or a replica, so writing them would mean writing a complaint that can never fire. The server also takes a real memory reading as it finishes building rather than guessing at one, which is startup.allocated and what keys.bytes-per-key divides by, and keeps a high water mark for peak.allocated. Measured against a redis-server 8.10.1 built from source over forty cases at no differences, covering every error sentence for a missing subcommand and an unknown one and a wrong arity and a bad SAMPLES, the HELP lines word for word, USAGE over all six types and a missing key, the STATS field names and their order, the db rows and the two fields inside one, DOCTOR on an empty server, and the COMMAND INFO row and both GETKEYS answers. That last part closes the four COMMAND GETKEYS differences that were left. REPLICAOF and SLAVEOF are in with the replica half, and so is DEBUG CHANGE-REPL-ID, which takes a fresh replication id and throws the old one away rather than keeping it as the second one the way a promotion does. FAILOVER is in as well, the three step handover with the reference's parsing, the reference's five refusals in the reference's order and master_failover_state in INFO, verified against real Redis 8.10.1 in both directions and on its failure paths, with D-148 registered for REPLICAOF not dropping this server's own replicas the way the reference does. What is left on this row is LOADAOF, once there is an append only file for it to be about.
• The divergence register, backed by divergences.toml

Exit gate

• 100 percent of the Redis test suite minus a published triage file
• yo runs as a replica of real Redis, and as a master to real Redis, both verified. The master direction is a real redis-server 8.10.1 pointed at yo with replicaof, which attaches, loads the dataset and follows the stream to the same keyspace. The replica direction is yo pointed at a real 8.10.1 with REPLICAOF, which ends on the same DEBUG DIGEST and the same offset, survives the link being broken with three partial resyncs in a row, and passes the master's replication id through unchanged to its own sub-replicas. A four node chain of a real master, two yo servers and a real Redis replica ends on one digest and one offset across all four.
• RDB round trip byte identical across the whole corpus, in both directions. The writing side is most of the way there. Strings over twenty bytes are LZF compressed on the way out now, with liblzf's own search rather than a better one, because LZF is not canonical and a payload compressed differently is a payload with different bytes in it, and the compressor is checked against the C one over 4000 inputs through a process that keeps its match table and 1500 through a fresh process each, at no differences. The shapes are right now as well. A list goes out as the quicklist type 18 that 8.10.1 writes, which is a count of nodes and then a container byte and a blob each, and it costs nothing to build because the packed band here is already the listpack a node payload wants and a chunk is already the entry region of one. That is where the bytes went too: a list of five thousand twenty byte values was four point two times the size a real server wrote it in and is now the same length to within a rounding error. A sorted set goes out highest score first, which is what a reader building a skiplist wants and is part of the format rather than a preference. A hash carrying field deadlines goes out under the listpackex type with the soonest deadline in front of it while it is small and the metadata type once it is not, and the small one is a copy of the blob rather than a re-encode. Making that copy safe meant laying the widened band out the way Redis lays it out, soonest deadline first with the fields that have none behind them, which also makes a reap stop at the first row that is still alive and makes HGETALL walk a hash in the order a real server walks it. Of seventeen shaped DUMP cases covering every type, thirteen now match to the byte apart from the ten byte footer. The four left are a big set and two big hashes, where the member order differs at the same length, and a stream, which goes out as the older type. Four hundred random programs of HSET, HPEXPIREAT, HPERSIST and HDEL agree on every field, value, deadline, encoding and the order HGETALL answers in. Four hundred random programs of pushes, pops, sets, inserts and trims agree on every element of every list. Cross RESTORE is clean both ways for every shape, redis-check-rdb reads a file written here, a real 8.10.1 started on that file loads every key with matching encodings and field deadlines, and this server loads an 8.10.1 file the same way. D-111 is now only the two footer version bytes. D-157 is where the nodes fall in a long list, D-158 is the member order in a big set or hash, and D-159 is the eight byte bound in front of a listpackex payload, which is exact here and a stale lower bound on the reference. The reading side is in now as well, a whole file reader that walks a real dump from the header to the checksum and hands back one key at a time with its database, its deadline and its body, and it has its first caller in DEBUG RELOAD, which writes the file and reads it back on the spot. That trip was measured against 8.10.1 over eleven cases at no differences, a search index was proven to come back with its documents, and redis-check-rdb reads a 1200 key file yo wrote and says it looks OK. yodb restore is in as well, which is G16 and the other end of yodb dump --format rdb. It reads a Redis RDB file, builds every value in it and reports what it holds without starting anything or writing anything, and yodb serve --restore builds the dataset out of one before the port opens, so a person with a dump off a real Redis can find out whether it loads here and then serve it. That was measured by building every type in both encodings on a redis-server 8.10.1 built from source along with hash field deadlines, a stream group, a pending entry and three databases, telling the reference to SAVE, starting a fresh yodb on the file it wrote, and asking both servers the same 54 command battery including OBJECT ENCODING on every key, at no differences. INFO persistence gained rdb_last_load_keys_loaded and rdb_last_load_keys_expired with it, since there is a load to report now. What is left on this row is the member order in a big set or a big hash, which is D-158, the stream type, and then PSYNC and D-48.
• The client library matrix green across all 13 clients
• 260 or more commands across 16 groups, every one with a storage plan recorded in commands.toml
• Prediction P-9 answered. FT.HYBRID RRF fusion is expected to match Redis 8.8's result ordering on 99 percent or more of the corpus

Notes

RedisGraph is not supported, on purpose, and it is in the divergence register with the reason.

Every divergence has to be a row in divergences.toml before it can ship. The failure mode this avoids is a compatibility claim that is true in aggregate and false in the specific case someone hits.

Spec: 16-implementation-plan.md M8, 12-resp3-and-compat.md.