From 32bc3e3bc65aae4b0e0e591c8b533ac53b7ca469 Mon Sep 17 00:00:00 2001 From: Joe Rivera Date: Fri, 16 Oct 2026 23:58:05 -0500 Subject: [PATCH 01/17] state_exec: grammar spec, conformance cases and parse corpus (grammar P0) Declare the state_exec s-expression grammar in docs/STATE_EXEC_GRAMMAR.md (W3C EBNF, Unicode source rules, literal tables, and a table of every intentional change from the hand-written parser, tagged by phase P2-P4). The parser itself is unchanged; later phases build it from this grammar. state_exec_grammar_test tags each case with the phase it holds from: landed cases must hold, pending cases must still differ, the doc's EBNF must be closed, and every production needs a case. state_exec_corpus_test pins how the 597 DSL scripts the repo carries parse today (state_exec_corpus/, regenerated with gen_state_exec_corpus.py and CVC_UPDATE_GOLDENS=1). Also correct parser.h (no quasiquote) and the developer guide (false and #f read as nil, bytes literals exist). --- docs/STATE_EXEC_DEVELOPER_GUIDE.md | 8 +- docs/STATE_EXEC_GRAMMAR.md | 306 +++ inc/cvc/state/state_exec/parser.h | 6 +- src/cvc/tests/CMakeLists.txt | 20 + src/cvc/tests/gen_state_exec_corpus.py | 268 ++ .../tests/state_exec_corpus/.gitattributes | 2 + src/cvc/tests/state_exec_corpus/expected.txt | 1366 +++++++++ src/cvc/tests/state_exec_corpus/scripts.txt | 2437 +++++++++++++++++ src/cvc/tests/state_exec_corpus_test.cpp | 102 + src/cvc/tests/state_exec_grammar_dump.h | 240 ++ src/cvc/tests/state_exec_grammar_test.cpp | 284 ++ 11 files changed, 5035 insertions(+), 4 deletions(-) create mode 100644 docs/STATE_EXEC_GRAMMAR.md create mode 100644 src/cvc/tests/gen_state_exec_corpus.py create mode 100644 src/cvc/tests/state_exec_corpus/.gitattributes create mode 100644 src/cvc/tests/state_exec_corpus/expected.txt create mode 100644 src/cvc/tests/state_exec_corpus/scripts.txt create mode 100644 src/cvc/tests/state_exec_corpus_test.cpp create mode 100644 src/cvc/tests/state_exec_grammar_dump.h create mode 100644 src/cvc/tests/state_exec_grammar_test.cpp diff --git a/docs/STATE_EXEC_DEVELOPER_GUIDE.md b/docs/STATE_EXEC_DEVELOPER_GUIDE.md index 048aa8e28..1b3bb5002 100644 --- a/docs/STATE_EXEC_DEVELOPER_GUIDE.md +++ b/docs/STATE_EXEC_DEVELOPER_GUIDE.md @@ -81,6 +81,9 @@ auto results = sched.run(); ## 2. The DSL Language +The full syntax (literals, escapes, Unicode rules) is specified in +[STATE_EXEC_GRAMMAR.md](STATE_EXEC_GRAMMAR.md). + ### Types | Type | Literal examples | C++ variant tag | @@ -88,9 +91,10 @@ auto results = sched.run(); | Integer | `42`, `-7` | `int64_t` | | Float | `3.14`, `1e-5` | `double` | | String | `"hello"` | `std::string` | -| Boolean | `true`, `false` | `bool` | +| Bytes | `b"\x00\xff"` | `bytes_value` | +| Boolean | `true`, `#t` (`false` and `#f` read as nil) | `bool` | | Symbol | `x`, `my-func` | `symbol` | -| Nil | `nil` | `std::monostate` | +| Nil | `nil`, `false`, `#f` | `std::monostate` | | List | `(1 2 3)` | `list_ptr` | ### Special Forms diff --git a/docs/STATE_EXEC_GRAMMAR.md b/docs/STATE_EXEC_GRAMMAR.md new file mode 100644 index 000000000..c979d0abb --- /dev/null +++ b/docs/STATE_EXEC_GRAMMAR.md @@ -0,0 +1,306 @@ +# state_exec Grammar + +This is the formal grammar of the state_exec s-expression language: the text that +`cvc::state_exec::parse()` and `parse_all()` read (`inc/cvc/state/state_exec/parser.h`). Ariadne +`.ari` documents embed it in `init`, `on`, `bind` and `*_when` fields. + +**Status: target grammar.** The parser in `src/cvc/state/state_exec/parser.cpp` is still the +hand-written one. It moves onto this grammar in phases (P2 to P4, see +[Changes from the pre-grammar parser](#changes-from-the-pre-grammar-parser)). Until then it differs +from this document exactly where that table says. `state_exec_grammar_test` checks both sides: +cases for landed phases must hold, and cases for pending phases must still differ. + +From P2 on, the parser is built from this declaration. Each production below becomes a +[PEGTL](https://github.com/taocpp/PEGTL) rule with the same name in +`src/cvc/state/state_exec/grammar.h`, and `state_exec_grammar_test` checks that the names match. + +## Notation + +The grammar uses the [W3C EBNF notation](https://www.w3.org/TR/xml/#sec-notation): `::=` defines +a production, `|` separates alternatives, `?` `*` `+` are optional, zero-or-more and one-or-more, +`A - B` matches what `A` matches unless `B` also matches it, `'…'` and `"…"` are literal text, +`[…]` is a character class, and `#xN` is the Unicode code point N. + +The input is the sequence of Unicode scalar values decoded from UTF-8. Bytes that are not +well-formed UTF-8 match nothing, so they are a parse error. + +Choices are ordered and matching is greedy, as in a PEG. The prose sections below state the +few places where that matters. + +## Grammar + +```ebnf +/* Whole inputs. parse_all() reads a program; parse() reads a single_datum. */ +program ::= bom? atmosphere* (datum atmosphere*)* +single_datum ::= bom? atmosphere* (datum atmosphere*)? + +/* Data */ +datum ::= list | quoted | text_literal | bytes_literal | atom +list ::= '(' atmosphere* (datum atmosphere*)* ')' +quoted ::= "'" atmosphere* datum + +/* Source characters and atmosphere */ +bom ::= #xFEFF +scalar_value ::= [#x0-#xD7FF] | [#xE000-#x10FFFF] +source_char ::= scalar_value - (control | bidi_control | #xFEFF) +control ::= [#x0-#x8] | [#xE-#x1F] | [#x7F-#x84] | [#x86-#x9F] +bidi_control ::= #x61C | [#x202A-#x202E] | [#x2066-#x2069] +atmosphere ::= whitespace | line_comment +whitespace ::= [#x9-#xD] | #x20 | #x85 | #x200E | #x200F | #x2028 | #x2029 +other_space ::= #xA0 | #x1680 | [#x2000-#x200A] | #x202F | #x205F | #x3000 +line_comment ::= ';' (source_char - line_end)* +line_end ::= #xA | #xD + +/* Atoms: the longest run of constituents, then classified (see "Atoms") */ +atom ::= boolean | nil | number | symbol +token ::= constituent+ +constituent ::= source_char - (whitespace | other_space | delimiter | reserved) +delimiter ::= #x22 | #x27 | #x28 | #x29 | #x3B /* " ' ( ) ; */ +reserved ::= #x2C | #x5B | #x5C | #x5D | #x60 | #x7B | #x7C | #x7D /* , [ \ ] ` { | } */ +boolean ::= '#t' | 'true' +nil ::= '#f' | 'false' | 'nil' +number ::= integer | decimal_float | float_special +integer ::= sign? (dec_digits | radix_integer) +radix_integer ::= ('0x' | '0X' | '#x' | '#X') hex_digits + | ('0o' | '0O' | '#o' | '#O') oct_digits + | ('0b' | '0B' | '#b' | '#B') bin_digits +sign ::= '+' | '-' +dec_digits ::= [0-9] ('_'? [0-9])* +hex_digits ::= hex ('_'? hex)* +oct_digits ::= [0-7] ('_'? [0-7])* +bin_digits ::= [0-1] ('_'? [0-1])* +hex ::= [0-9A-Fa-f] +decimal_float ::= sign? ( dec_digits '.' dec_digits? exponent? + | '.' dec_digits exponent? + | dec_digits exponent ) +exponent ::= [eE] sign? dec_digits +float_special ::= '+inf.0' | '-inf.0' | '+nan.0' | '-nan.0' +radix_prefixed ::= sign? [0#] [xXoObB] constituent* +symbol ::= token - (boolean | nil | number | radix_prefixed) + +/* Text strings */ +text_literal ::= text_string | raw_text +text_string ::= '"' (string_char | text_escape)* '"' +string_char ::= source_char - (#x22 | #x5C) +text_escape ::= '\' (simple_escape | octal_escape | hex_escape | unicode_escape) +simple_escape ::= [abfnrtv] | #x22 | #x5C +octal_escape ::= [0-7] ([0-7] [0-7]?)? +hex_escape ::= 'x' hex hex +unicode_escape ::= 'u' hex hex hex hex + | 'u{' hex (hex (hex (hex (hex hex?)?)?)?)? '}' +raw_text ::= 'r' raw_body +raw_body ::= '"' raw_char* '"' + | hash_fence '"' raw_char* '"' hash_fence +hash_fence ::= '#'+ +raw_char ::= source_char + +/* Byte strings */ +bytes_literal ::= byte_string | raw_bytes +byte_string ::= 'b"' (string_char | byte_escape)* '"' +byte_escape ::= '\' (simple_escape | octal_escape | hex_escape) +raw_bytes ::= 'br' raw_body +``` + +## Source text + +- **Encoding.** Source is UTF-8. A malformed byte sequence, an overlong form or an encoded + surrogate is an error. One leading U+FEFF (a byte-order mark) is skipped; U+FEFF anywhere else + is an error. +- **Forbidden characters.** Outside of escapes, these may not appear anywhere, including strings + and comments: + - C0 controls other than TAB, LF, VT, FF and CR; + - DEL; + - C1 controls other than NEL; + - the bidirectional embedding, override and isolate controls U+202A–U+202E and + U+2066–U+2069, plus U+061C. This is the "Trojan Source" defence (CVE-2021-42574): these + characters can make source text display in a different order from the order it parses in. + Write them as `\u{202E}` and so on when a string needs one. +- **Whitespace** is Unicode Pattern_White_Space: TAB, LF, VT, FF, CR, space, NEL, U+200E, + U+200F, U+2028 and U+2029. The other Unicode spaces (`other_space`: NBSP, U+3000 and so on) are + allowed inside strings and comments, but between tokens they are an error rather than a silent + separator, because they look like a space and are not one. +- **Comments** run from `;` to the end of the line. LF and CR both end a line. +- **Positions.** `parse_error::line()` and `col()` are 1-based, and point at the character where + the error was found, such as the non-hex character in `"\x4"`. A new line starts after LF, + after CR, and once after CRLF. Columns count code points, not bytes. +- **Nesting.** At most 256 lists and quotes may be nested. Deeper input is an error, not a stack + overflow. The limit is sized for the 64 KiB default stack of a wasm build; real scripts nest far + less. + +## Data + +| Production | Value (`value_t` alternative) | +|---|---| +| `list` | `list_ptr` of the elements; `()` is an empty list | +| `quoted` | the two-element list `(quote datum)` | +| `text_string`, `raw_text` | `std::string` holding UTF-8 text | +| `byte_string`, `raw_bytes` | `bytes_value` | +| `boolean` | `true` | +| `nil` | nil (`std::monostate`). `#f` and `false` read as nil, not as `bool false` | +| `integer` | `int64_t` | +| `decimal_float`, `float_special` | `double` | +| `symbol` | `symbol` | + +`parse_all()` returns one value per top-level datum. `parse()` returns the one datum. It returns +nil for input that holds only atmosphere, and anything after the datum is an error. + +## Atoms + +An atom is the longest run of constituents (`token`). The whole token is then classified, in this +order: + +1. `#t` and `true` are `boolean`. `#f`, `false` and `nil` are `nil`. +2. A token that matches `number` exactly is a number. +3. A token that matches `radix_prefixed` but is not a valid `integer` is an error, for example + `0x1G`, `0b102`, `#xyz` or `0x`. A typo in a hex literal is reported, not read as a symbol. +4. Every other token is a `symbol`. That includes `+`, `-`, `...`, `->`, `1+`, `#foo`, `inf` and + `nan`. + +**Symbols** may use any character except whitespace, `other_space`, the delimiters `" ' ( ) ;`, +the reserved characters, controls and bidi controls. So `λ`, `→`, `café`, `日本語` and emoji +(including ZWJ sequences and variation selectors) are all symbol characters. Symbols are not +normalized, so a composed and a decomposed `é` are different symbols. That matches the raw-bytes +policy for strings and keys in UNICODE_SUPPORT_ROADMAP.md (SE-4). The grammar does not check +confusable or default-ignorable characters (U+200B and the like) in symbols. + +**Reserved characters** `` , [ \ ] ` { | } `` are errors outside strings and comments. They are +kept free for quasiquote (`` ` ``, `,`), dict and vector literals (`{}` and `[]`; a dict literal is +sketched in roadmap/STATE_BINARY_STREAMING.md) and quoted symbols (`|…|`). + +**Numbers.** + +| Form | Examples | Value | +|---|---|---| +| decimal integer | `42`, `-7`, `+5`, `007`, `1_000_000` | `int64_t` | +| hex, octal, binary integer | `0x1F`, `#x1f`, `-0x10`, `0o17`, `#o17`, `0b1011`, `#b1011`, `0xFFFF_FFFF` | `int64_t` | +| float | `1.5`, `1.`, `.5`, `1e3`, `-2.5E-3`, `6.022_140e23` | `double` | +| special float | `+inf.0`, `-inf.0`, `+nan.0`, `-nan.0` | `double` | + +- `_` may separate digits, one at a time, never at the start or end of a digit run. +- The sign goes before any radix prefix: `-0x10`, not `0x-10`. +- An integer must fit in `int64_t`, from -9223372036854775808 to 9223372036854775807. Overflow is + an error, not a silent conversion to `double`. +- Floats are read locale-independently and correctly rounded. A float too large for a `double` + (`1e999`) is an error. One too small rounds to zero or to a subnormal. +- There are no hex floats, rationals or complex numbers. + +## Strings + +A **text string** (`"…"`) is UTF-8 text. Any source character may appear in it except `"` and +`\`, including raw line breaks. A string can span lines. Escapes: + +| Escape | Meaning in a text string | Meaning in a byte string | +|---|---|---| +| `\n` `\t` `\r` | LF, TAB, CR | the same bytes | +| `\a` `\b` `\f` `\v` | BEL, BS, FF, VT | the same bytes | +| `\"` `\\` | `"` and `\` | the same bytes | +| `\N`, `\NN`, `\NNN` (octal, longest match) | code point 0–0o777; `\0` is NUL | byte 0–0o377; larger is an error | +| `\xNN` (exactly two hex digits) | code point U+0000–U+00FF | byte 0x00–0xFF | +| `\uXXXX` (exactly four) or `\u{X…}` (one to six) | that code point | not allowed | + +- Any other character after `\` is an error. +- A `\u` escape that names a surrogate (U+D800–U+DFFF) or a value above U+10FFFF is an error. +- Escaped code points are stored UTF-8 encoded. So `"\xe9"`, `"é"` and `"\351"` are all the + one-character string `é`. + +A **byte string** (`b"…"`) is a `bytes_value`. A non-ASCII source character contributes its +UTF-8 encoding, so `b"é"` is the two bytes C3 A9. + +A **raw string** (`r"…"`) or **raw byte string** (`br"…"`) processes no escapes: every character +between the quotes is taken as written, backslashes included. To put `"` in a raw string, fence +it with one or more `#`: `r#"say "hi""#`. The string ends at the first `"` followed by as many `#` +as opened it, so `r##"a"#b"##` is `a"#b`. + +**Prefixes.** `b`, `r` and `br` start a literal only at the start of a datum and only when +touching the quote, or the `#` fence of a raw string. `ab"x"` is the symbol `ab` followed by the +string `x`, and `r#foo` is a symbol. Once a datum starts with `b"`, `r"`, `br"` or a prefix, +hashes and `"`, it is a string literal, and leaving it unterminated is an error. + +## Changes from the pre-grammar parser + +Phase P2 replaces the hand-written lexer with the grammar, including the number syntax. P3 adds +the Unicode source rules and strict escapes. P4 adds octal escapes, the extra C escapes and raw +strings. Every row is a case in `state_exec_grammar_test`. + +| Input | Pre-grammar parser | Grammar | Phase | +|---|---|---|---| +| `a)` (a stray `)`) | the symbols `a` and `)` | error | P2 | +| `[a]`, `a,b`, `` `a ``, `{}`, `a\|b`, `a\b` | odd one-character or glued symbols | error (reserved) | P2 | +| `parse("1 2")` | `1`; the rest is ignored | error | P2 | +| `'` at end of input | `(quote nil)` | error | P2 | +| 257 nested lists | parsed | error | P2 | +| `; c` CR `1` | comment runs to LF, so no datum | `1` | P2 | +| `0x1F` | `31.0`, a double (via `strtod`) | `31` | P2 | +| `#x1F`, `0o17`, `0b101`, `1_000` | symbols | integers | P2 | +| `0xZZ`, `#bar`, `0x` | symbols | error | P2 | +| `+5` | `5.0`, a double | `5` | P2 | +| `9223372036854775808` | `9.2233720368547758e18`, a double | error | P2 | +| `inf`, `nan`, `-INF` | doubles | symbols | P2 | +| `+inf.0`, `-inf.0`, `+nan.0` | symbols | doubles | P2 | +| `1e999` | `+inf` | error | P2 | +| `1.5` with a `,`-decimal C locale set by the host | symbol (from reading `strtod`; not tested) | `1.5` | P2 | +| bytes that are not UTF-8 | kept in the symbol or string | error | P3 | +| U+FEFF at the start | a symbol before the first datum | skipped | P3 | +| NBSP between `a` and `b` | one symbol, `a b` with an NBSP | error | P3 | +| NEL or U+2028 between `a` and `b` | one symbol | two symbols | P3 | +| a raw bidi control or C0 control, in a string or comment | kept | error | P3 | +| `"\q"` | `\q`, backslash kept | error | P3 | +| `"\uD800"`, `"\u{110000}"` | U+FFFD | error | P3 | +| error columns | bytes | code points | P3 | +| `"\101"`, `"\012"` | `\101`; NUL then `12` | `A`; LF | P4 | +| `b"\377"`, `b"\400"` | `\377` kept; `\400` kept | the byte FF; error | P4 | +| `"\a\b\f\v"` | backslashes kept | BEL BS FF VT | P4 | +| `r"a\nb"`, `br"…"` | the symbol `r` or `br`, then a string | raw string, raw bytes | P4 | +| `r#"say "hi""#` | a symbol `r#`, strings and symbols | raw string `say "hi"` | P4 | + +Printing is not part of the grammar. `to_string` round-trip (escaping strings, shortest floats, +`+inf.0`) is phase P5. + +## Not in the grammar + +These were considered and left out. Each would need its own decision: + +- block comments (`#| … |#`) and datum comments (`#;`); +- quasiquote, unquote and unquote-splicing (the reserved `` ` `` and `,`); +- quoted symbols (`|…|`), dict literals (`{…}`) and vector literals (`[…]`); +- hex byte literals such as `#x"de ad be ef"`; +- `\N{NAME}` escapes, which need the Unicode name tables; +- line continuations inside strings; +- hex floats; +- NFC normalization of symbols or strings. + +## Tests and corpus + +- `src/cvc/tests/state_exec_grammar_test.cpp` checks the grammar itself: + - every EBNF production above is defined once, and every one except `program` and + `single_datum` is used; + - every production has at least one accept or reject case; + - each case holds once its phase has landed, and still differs while its phase is pending. + `kLandedPhase` in the test says which phases have landed. +- `src/cvc/tests/state_exec_corpus_test.cpp` parses the corpus: every DSL script the repo carries, + drawn from the `.ari` files, the docs' `lisp` blocks, the C++ test sources and the pycvc tests. + It compares each parse with `state_exec_corpus/expected.txt`, so a parser change cannot alter + how an existing script reads without that file changing in the same PR. + +To regenerate the corpus after adding scripts, run the extractor (it needs PyYAML), then rewrite +the expected parses: + +```sh +python src/cvc/tests/gen_state_exec_corpus.py +CVC_UPDATE_GOLDENS=1 build/bin/state_exec_corpus_test +``` + +The corpus held 597 scripts when it was made, and all of them parse under the pre-grammar +parser. Of the grammar's intentional changes, only the reserved `{` `}` touch any of them: a dict +literal sketched in roadmap/STATE_BINARY_STREAMING.md, which no parser has implemented. The 177 +s-expression scripts in txwtf's cfos tests, checked for the txwtf.cfos-to-state_exec port, touch +none. + +The expected parses use the test's own dump format (`state_exec_grammar_dump.h`), not +`to_string`. In that format: +- integers print in decimal; +- whole doubles print as `N.0`, other doubles in the shortest form that reads back exactly, and + the specials as `+inf.0`, `-inf.0` and `+nan.0`; +- strings print escaped, bytes as `b"…"`, and nil as `nil`; +- a symbol is wrapped in `|…|` when its bare text would read as something else; +- a parse error prints as `!error line:col`. diff --git a/inc/cvc/state/state_exec/parser.h b/inc/cvc/state/state_exec/parser.h index 2d3267648..8a96aec50 100644 --- a/inc/cvc/state/state_exec/parser.h +++ b/inc/cvc/state/state_exec/parser.h @@ -3,8 +3,10 @@ * @brief S-expression parser for the state_exec DSL. * * Converts source text into a tree of value_t nodes. Supports atoms - * (integers, floats, strings, symbols), lists, quote/quasiquote/unquote, - * and line comments. Reports parse errors with line/column information. + * (integers, floats, strings, byte strings, symbols), lists, quote ('x), and + * line comments. Reports parse errors with line/column information. The + * syntax is specified in docs/STATE_EXEC_GRAMMAR.md, which also lists where + * this parser still differs from it. */ #ifndef CVC_STATE_EXEC_PARSER_H #define CVC_STATE_EXEC_PARSER_H diff --git a/src/cvc/tests/CMakeLists.txt b/src/cvc/tests/CMakeLists.txt index 7517e5ce9..00cb4e9dd 100644 --- a/src/cvc/tests/CMakeLists.txt +++ b/src/cvc/tests/CMakeLists.txt @@ -65,6 +65,8 @@ add_executable(state_exec_evaluator_test state_exec_evaluator_test.cpp) add_executable(state_exec_stackless_evaluator_test state_exec_stackless_evaluator_test.cpp) add_executable(state_exec_codec_test state_exec_codec_test.cpp) add_executable(state_exec_unicode_test state_exec_unicode_test.cpp) +add_executable(state_exec_grammar_test state_exec_grammar_test.cpp) +add_executable(state_exec_corpus_test state_exec_corpus_test.cpp) add_executable(state_exec_async_test state_exec_async_test.cpp) add_executable(async_lane_test async_lane_test.cpp) add_executable(state_exec_scheduler_test state_exec_scheduler_test.cpp) @@ -326,6 +328,8 @@ list(APPEND TEST_TARGETS state_exec_stackless_evaluator_test state_exec_codec_test state_exec_unicode_test + state_exec_grammar_test + state_exec_corpus_test state_exec_async_test async_lane_test state_exec_scheduler_test @@ -1235,6 +1239,18 @@ target_link_libraries(state_exec_unicode_test GTest::gtest GTest::gtest_main ) +target_link_libraries(state_exec_grammar_test + PRIVATE + cvc + GTest::gtest + GTest::gtest_main +) +target_link_libraries(state_exec_corpus_test + PRIVATE + cvc + GTest::gtest + GTest::gtest_main +) target_link_libraries(state_exec_async_test PRIVATE @@ -1558,6 +1574,8 @@ target_compile_features(state_exec_evaluator_test PRIVATE cxx_std_20) target_compile_features(state_exec_stackless_evaluator_test PRIVATE cxx_std_20) target_compile_features(state_exec_codec_test PRIVATE cxx_std_20) target_compile_features(state_exec_unicode_test PRIVATE cxx_std_20) +target_compile_features(state_exec_grammar_test PRIVATE cxx_std_20) +target_compile_features(state_exec_corpus_test PRIVATE cxx_std_20) target_compile_features(state_exec_async_test PRIVATE cxx_std_20) target_compile_features(async_lane_test PRIVATE cxx_std_20) target_compile_features(state_exec_scheduler_test PRIVATE cxx_std_20) @@ -1966,6 +1984,8 @@ cvc_discover_tests(state_exec_evaluator_test) cvc_discover_tests(state_exec_stackless_evaluator_test) cvc_discover_tests(state_exec_codec_test) cvc_discover_tests(state_exec_unicode_test) +cvc_discover_tests(state_exec_grammar_test) +cvc_discover_tests(state_exec_corpus_test) cvc_discover_tests(state_exec_async_test) cvc_discover_tests(async_lane_test) cvc_discover_tests(state_exec_scheduler_test) diff --git a/src/cvc/tests/gen_state_exec_corpus.py b/src/cvc/tests/gen_state_exec_corpus.py new file mode 100644 index 000000000..ca7d54d8c --- /dev/null +++ b/src/cvc/tests/gen_state_exec_corpus.py @@ -0,0 +1,268 @@ +#!/usr/bin/env python3 +"""Regenerate the state_exec parse corpus (state_exec_corpus/scripts.txt). + +The corpus is a frozen snapshot of the DSL source text the repo already carries, so the +grammar-driven parser (docs/STATE_EXEC_GRAMMAR.md) can be checked against how the +pre-grammar parser read every one of them. It collects: + +* every YAML scalar in a tracked ``*.ari`` file whose text starts with ``(`` (``init``, + ``on``, ``bind``, ``*_when`` and friends), +* every ``lisp``/``scheme`` fenced block in a tracked ``*.md`` file (unlabeled blocks are + skipped: the ones that start with ``(`` are ASCII diagrams), +* every C++ string literal (adjacent literals concatenated, escapes decoded) that starts + with ``(`` and ends with ``)`` in a tracked ``src/**/*.cpp``/``*.h`` file that mentions + ``state_exec`` or ``ariadne``. A one-line literal must start with ``(`` itself, which + skips messages like ``" (expected path:...)"``, and the parentheses must balance, which + skips prefixes a test completes at run time, +* every Python string constant of that shape in ``bindings/pycvc/test_pycvc_*.py``. + +Identical scripts are kept once, under the first source that has them. The expected parse +of each script is not computed here: ``state_exec_corpus_test`` writes +``state_exec_corpus/expected.txt`` when run with ``CVC_UPDATE_GOLDENS=1``. + +Usage (from anywhere in the checkout; needs PyYAML): + + python src/cvc/tests/gen_state_exec_corpus.py +""" + +from __future__ import annotations + +import ast +import re +import subprocess +import sys +from pathlib import Path + +import yaml + +ROOT = Path(__file__).resolve().parents[3] +OUT = Path(__file__).resolve().parent / "state_exec_corpus" / "scripts.txt" + +# A corpus entry is "@@@ script " + the script's lines; no script line may start with +# "@@@ ", which the writer checks. +MARK = "@@@ " + + +def tracked(*patterns: str) -> list[str]: + out = subprocess.run( + ["git", "ls-files", "--", *patterns], + cwd=ROOT, + capture_output=True, + text=True, + check=True, + ).stdout + return sorted(p for p in out.splitlines() if p) + + +def balanced(text: str) -> bool: + """True if the parentheses outside strings and ; comments balance.""" + depth, i, n = 0, 0, len(text) + while i < n: + c = text[i] + if c == ";": + while i < n and text[i] != "\n": + i += 1 + elif c == '"': + i += 1 + while i < n and text[i] != '"': + i += 2 if text[i] == "\\" else 1 + elif c == "(": + depth += 1 + elif c == ")": + depth -= 1 + if depth < 0: + return False + i += 1 + return depth == 0 + + +def looks_like_program(text: str) -> bool: + # Only for literals in C++ and Python sources, where a fragment (a prefix the test + # completes at run time) is common: those are skipped as unbalanced. + s = text.strip() + if "\n" not in s and not text.startswith("("): + return False + return len(s) >= 3 and s.startswith("(") and s.endswith(")") and balanced(s) + + +# -- .ari ------------------------------------------------------------------------------------ + + +def ari_scripts(path: str): + # BaseLoader keeps every scalar a string, so the key `on:` is not read as YAML 1.1 `true`. + doc = yaml.load((ROOT / path).read_text(encoding="utf-8"), Loader=yaml.BaseLoader) + + def walk(node, where): + if isinstance(node, dict): + for k, v in node.items(): + yield from walk(v, f"{where}.{k}" if where else str(k)) + elif isinstance(node, list): + for i, v in enumerate(node): + yield from walk(v, f"{where}[{i}]") + elif isinstance(node, str) and node.lstrip().startswith("("): + yield f"{path}:{where}", node + + yield from walk(doc, "") + + +# -- Markdown -------------------------------------------------------------------------------- + +FENCE = re.compile(r"^\s*(`{3,})\s*([\w+-]*)\s*$") + + +def md_scripts(path: str): + lines = (ROOT / path).read_text(encoding="utf-8").split("\n") + i = 0 + while i < len(lines): + m = FENCE.match(lines[i]) + if not m: + i += 1 + continue + ticks, lang = m.group(1), m.group(2) + j = i + 1 + while j < len(lines) and not lines[j].strip().startswith(ticks): + j += 1 + body = "\n".join(lines[i + 1 : j]) + if lang in ("lisp", "scheme"): + yield f"{path}:{i + 1}", body + i = j + 1 + + +# -- C++ ------------------------------------------------------------------------------------- + +CPP_TOKEN = re.compile( + r""" + (?P//[^\n]*|/\*.*?\*/) + | (?P(?:u8|u|U|L)?R"(?P[^()\\\s]{0,16})\((?P.*?)\)(?P=delim)") + | (?P(?:u8|u|U|L)?"(?P(?:[^"\\\n]|\\.)*)") + | (?P'(?:[^'\\\n]|\\.)*') + | (?P\s+) + | (?P.) + """, + re.DOTALL | re.VERBOSE, +) +CPP_ESC = re.compile( + r"\\(x[0-9A-Fa-f]+|u[0-9A-Fa-f]{4}|U[0-9A-Fa-f]{8}|[0-7]{1,3}|.)", re.DOTALL +) +SIMPLE = { + "n": "\n", + "t": "\t", + "r": "\r", + "a": "\a", + "b": "\b", + "f": "\f", + "v": "\v", + "\\": "\\", + '"': '"', + "'": "'", + "?": "?", +} + + +def cpp_unescape(body: str) -> str: + # Escapes become the bytes the compiler would put in a narrow (UTF-8) literal; the result is + # decoded as UTF-8 so a "\xC3\xA9" pair reads back as the character it encodes. + out = bytearray() + + def emit_cp(cp: int): + out.extend(chr(cp).encode("utf-8", "surrogatepass")) + + pos = 0 + for m in CPP_ESC.finditer(body): + out.extend(body[pos : m.start()].encode("utf-8")) + e = m.group(1) + if e[0] == "x": + out.append(int(e[1:], 16) & 0xFF) + elif e[0] in "uU": + emit_cp(int(e[1:], 16)) + elif e[0] in "01234567": + out.append(int(e, 8) & 0xFF) + else: + out.extend(SIMPLE.get(e, "\\" + e).encode("utf-8")) + pos = m.end() + out.extend(body[pos:].encode("utf-8")) + return out.decode("utf-8", "surrogateescape") + + +def cpp_scripts(path: str): + text = (ROOT / path).read_text(encoding="utf-8") + if "state_exec" not in text and "ariadne" not in text: + return + pending, start = None, 0 # concatenation of adjacent literals, and where it began + for m in CPP_TOKEN.finditer(text): + kind = m.lastgroup + if kind in ("raw", "str"): + piece = ( + m.group("rawbody") if kind == "raw" else cpp_unescape(m.group("body")) + ) + if pending is None: + pending, start = piece, m.start() + else: + pending += piece + elif kind in ("ws", "comment"): + continue + else: + if pending is not None and looks_like_program(pending): + yield f"{path}:{text.count(chr(10), 0, start) + 1}", pending + pending = None + if pending is not None and looks_like_program(pending): + yield f"{path}:{text.count(chr(10), 0, start) + 1}", pending + + +# -- Python ---------------------------------------------------------------------------------- + + +def py_scripts(path: str): + tree = ast.parse((ROOT / path).read_text(encoding="utf-8")) + for node in ast.walk(tree): + if ( + isinstance(node, ast.Constant) + and isinstance(node.value, str) + and looks_like_program(node.value) + ): + yield f"{path}:{node.lineno}", node.value + + +# -- main ------------------------------------------------------------------------------------ + + +def main() -> int: + sources = [] + sources += [s for p in tracked("*.ari") for s in ari_scripts(p)] + sources += [s for p in tracked("*.md") for s in md_scripts(p)] + sources += [s for p in tracked("src/*.cpp", "src/*.h") for s in cpp_scripts(p)] + sources += [ + s for p in tracked("bindings/pycvc/test_pycvc_*.py") for s in py_scripts(p) + ] + + seen: set[str] = set() + entries = [] + for where, script in sources: + script = script.replace("\r\n", "\n") + if script in seen: + continue + seen.add(script) + if any(line.startswith(MARK) for line in script.split("\n")): + print( + f"{where}: a line starts with {MARK!r}; cannot store it", + file=sys.stderr, + ) + return 1 + entries.append((where, script)) + + OUT.parent.mkdir(parents=True, exist_ok=True) + with OUT.open("w", encoding="utf-8", newline="\n", errors="surrogateescape") as f: + f.write( + "# state_exec parse corpus. Generated by src/cvc/tests/gen_state_exec_corpus.py;\n" + ) + f.write( + "# do not edit by hand. Each entry is a '@@@ script ' line, then the script.\n" + ) + for where, script in entries: + f.write(f"{MARK}script {where}\n{script}\n") + print(f"{len(entries)} scripts -> {OUT.relative_to(ROOT)}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/src/cvc/tests/state_exec_corpus/.gitattributes b/src/cvc/tests/state_exec_corpus/.gitattributes new file mode 100644 index 000000000..6b27b86eb --- /dev/null +++ b/src/cvc/tests/state_exec_corpus/.gitattributes @@ -0,0 +1,2 @@ +# Corpus files are compared byte for byte; keep them LF on every checkout. +* text eol=lf diff --git a/src/cvc/tests/state_exec_corpus/expected.txt b/src/cvc/tests/state_exec_corpus/expected.txt new file mode 100644 index 000000000..b198a8cff --- /dev/null +++ b/src/cvc/tests/state_exec_corpus/expected.txt @@ -0,0 +1,1366 @@ +# Expected parses of scripts.txt, in the grammar_dump format of +# state_exec_grammar_dump.h. Written by state_exec_corpus_test with +# CVC_UPDATE_GOLDENS=1; do not edit by hand. +@@@ script bindings/pycvc/demo.ari:init +(state-set "demo.agents" "128") +(state-set "demo.speed" "1.0") +(state-set "demo.wire" "false") +@@@ script bindings/pycvc/demo.ari:windows[0].children[5].bind +(if (> (int (state-get "demo.agents")) 256) "many" "few") +@@@ script src/cvc/ariadne/components/fe_controls.ari:units.fe_controls.children[9].on +(state-set "graphics.root.children.domain.fe.request" "mesh") +@@@ script src/cvc/ariadne/components/fe_controls.ari:units.fe_controls.children[9].disabled_when +(= (state-get "graphics.root.children.domain.fe.busy") "1") +@@@ script src/cvc/ariadne/components/fe_controls.ari:units.fe_controls.children[13].visible_when +(= (state-get "graphics.root.children.domain.fe.problem") "poisson") +@@@ script src/cvc/ariadne/components/fe_controls.ari:units.fe_controls.children[14].visible_when +(= (state-get "graphics.root.children.domain.fe.problem") "heat") +@@@ script src/cvc/ariadne/components/fe_controls.ari:units.fe_controls.children[17].on +(state-set "graphics.root.children.domain.fe.request" "solve") +@@@ script src/cvc/ariadne/components/fe_controls.ari:units.fe_controls.children[17].disabled_when +(or (= (state-get "graphics.root.children.domain.fe.busy") "1") (= (state-get "graphics.root.children.domain.fe.stats.tets") "0")) +@@@ script src/cvc/ariadne/components/fe_controls.ari:units.fe_controls.children[18].on +(state-set "graphics.root.children.domain.fe.request" "stop") +@@@ script src/cvc/ariadne/components/fe_controls.ari:units.fe_controls.children[18].enabled_when +(= (state-get "graphics.root.children.domain.fe.stoppable") "1") +@@@ script src/cvc/ariadne/components/mesh_tools.ari:units.mesh_tools.children[10].on +(state-set "graphics.root.children.mesh.mesh_ops.request" "smooth") +@@@ script src/cvc/ariadne/components/mesh_tools.ari:units.mesh_tools.children[10].disabled_when +(= (state-get "graphics.root.children.mesh.mesh_ops.busy") "1") +@@@ script src/cvc/ariadne/components/mesh_tools.ari:units.mesh_tools.children[14].on +(state-set "graphics.root.children.mesh.mesh_ops.request" "decimate") +@@@ script src/cvc/ariadne/components/mesh_tools.ari:units.mesh_tools.children[21].on +(state-set "graphics.root.children.mesh.mesh_ops.request" "repair") +@@@ script src/cvc/ariadne/components/mesh_tools.ari:units.mesh_tools.children[22].on +(state-set "graphics.root.children.mesh.mesh_ops.request" "orient") +@@@ script src/cvc/ariadne/components/mesh_tools.ari:units.mesh_tools.children[23].on +(state-set "graphics.root.children.mesh.mesh_ops.request" "reset") +@@@ script src/cvc/ariadne/components/mesh_tools.ari:units.mesh_tools.children[28].visible_when +(= (state-get "graphics.root.children.mesh.mesh_ops.color.mode") "geodesic") +@@@ script src/cvc/ariadne/components/mesh_tools.ari:units.mesh_tools.children[29].on +(state-set "graphics.root.children.mesh.mesh_ops.request" "curvature") +@@@ script src/cvc/ariadne/components/sim_transport.ari:units.sim_transport.children[3].enabled_when +(= (state-get "sim.paused") "1") +@@@ script src/cvcGL/examples/bunny_shadow.ari:init +(state-set "show_bunny" "true") +@@@ script src/cvcGL/examples/bunny_shadow.ari:windows[3].children[3].on +(begin (state-set "lighting.key_intensity" "1.9") (state-set "lighting.key_azimuth" "-38") (state-set "lighting.key_elevation" "52") (state-set "lighting.key_cone" "34")) +@@@ script src/cvcGL/examples/hello.ari:init +(state-set "demo.agents" "256") +@@@ script src/cvcGL/examples/hello.ari:windows[0].children[3].visible_when +(> (int (state-get "demo.agents")) 256) +@@@ script src/cvcGL/examples/hello.ari:windows[0].children[4].bind +(if (> (int (state-get "demo.agents")) 256) "heavy" "ok") +@@@ script src/cvcGL/examples/nav_compute.ari:init +(begin (state-set "nav.status" "idle") (state-set "nav.result" "-")) +@@@ script src/cvcGL/examples/nav_compute.ari:windows[0].children[3].on +(begin (state-set "nav.status" "running") (nav-step-async) (state-set "nav.result" (msg-recv "nav.done")) (state-set "nav.status" "done")) +@@@ script src/cvcGL/examples/nav_convoy.ari:init +(begin (state-set "show.agents" "1") (state-set "show.grid" "1") (state-set "sim.paused" "0") (state-set "sim.speed" "1.0")) +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:134 +(int 3.7) +(int "123") +(int #t) +(float 5) +(float "3.14") +(float #t) +(is-int 42) +(is-int 3.14) +(is-float 3.14) +(is-float 42) +(is-string "hi") +(is-string 42) +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:158 +(begin (defun fact (n) (if (<= n 1) 1 (* n (fact (- n 1))))) (fact 10)) +(begin (set result (list)) (for x (list 1 2 3 4 5) (set result (append result (list (* x x))))) result) +(state-set "shared.counter" "0") +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:190 +(set g (generator (lambda () (yield 1) (yield 2) (yield 3)))) +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:200 +(range 5) +(range 2 8) +(range 0 10 3) +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:208 +(set g (range 3)) +(next g) +(next g) +(next g) +(next g) +(gen-done? g) +(for x (range 5) (print x)) +(collect (range 5)) +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:231 +(set evens (generator (lambda () (for x (range 10) (if (= (% x 2) 0) (yield x) nil))))) +(collect evens) +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:241 +(set squares (generator (lambda () (for x (range 5) (yield (* x x)))))) +(collect squares) +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:250 +(set taken (list)) +(for x (range 1000) (if (>= (length taken) 4) (break nil) (append taken x))) +taken +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:261 +(set running-sum (generator (lambda () (let ((total 0)) (for x (range 1 6) (begin (set total (+ total x)) (yield total))))))) +(collect running-sum) +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:273 +(set counter (generator (lambda () (let ((n 0)) (while (< n 1000) (begin (yield n) (set n (+ n 1)))))))) +(for x counter (if (= x 5) (break x) nil)) +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:288 +(while t (break)) +(while t (break 42)) +(for x (list 10 20 30 40) (if (> x 25) (break x) nil)) +(set sum 0) +(for i (list 1 2 3) (for j (list 10 20 30) (if (= j 20) (break nil) (set sum (+ sum j))))) +sum +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:448 +(set child-pid (spawn "(begin (state-set \"child.done\" \"yes\") 42)" "child-worker")) +(set cpid (fork)) +(self-pid) +(self-uid) +(ps) +(inspect 2) +(pause 2) +(resume 2) +(kill 2) +(memory-usage) +(memory-limit) +(time-elapsed) +(time-limit) +(step-count) +(message-count) +(message-limit) +(cluster-id) +(node-id) +(scheduler-stats) +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:531 +(state-set "myapp.config.timeout" "30") +(state-get "myapp.config.timeout") +(state-exists "myapp.config.timeout") +(state-children "myapp.config") +(state-delete "myapp.old-data") +(state-data-set "myapp.blob" some-value) +(state-data-get "myapp.blob") +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:554 +(state-expire "myapp.session.abc" 60) +(state-expire-at "myapp.session.abc" "2026-01-01T00:00:00Z") +(state-has-expiry "myapp.session.abc") +(state-is-expired "myapp.session.abc") +(state-clear-expiry "myapp.session.abc") +(state-sweep-expired "myapp.session") +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:582 +(msg-send "myapp.events.updates" "hello from process 1") +(msg-send "myapp.events.data" "{\"key\": \"value\"}" "application/json") +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:595 +(begin (set i 0) (while (< i 10) (begin (state-set (str-concat "queue.item." (str i)) (str (* i i))) (msg-send "queue.notify" (str i)) (set i (+ i 1)))) "producer-done") +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:607 +(begin (set count 0) (while (< count 10) (if (state-exists (str-concat "queue.item." (str count))) (begin (set val (state-get (str-concat "queue.item." (str count)))) (state-set (str-concat "consumed." (str count)) val) (set count (+ count 1))) nil)) count) +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:705 +(set msg (msg-recv "events.data")) +(get-attr msg "payload") +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:723 +(if (> (msg-pending "events.data") 0) (set msg (msg-recv "events.data")) (print "no messages yet")) +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:735 +(sleep 0.5) +(while #t (begin (if (> (msg-pending "work.queue") 0) (begin (set job (msg-recv "work.queue")) (process-job job)) (sleep 0.1)))) +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:756 +(begin (set results (list)) (set done nil) (while (not done) (begin (set m (msg-recv "tasks.input")) (set payload (get-attr m "payload")) (if (= payload "stop") (set done #t) (set results (append results (list payload)))))) results) +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:1183 +(begin (set my-pid (self-pid)) (state-set (str-concat "__migrate_request." (str my-pid)) "target-node-B") my-pid) +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:1308 +(string.split "a,b,c" ",") +(string.join (list "a" "b" "c") ",") +(string.replace "hello" "l" "r") +(string.trim " hello ") +(string.upper "hello") +(string.lower "HELLO") +(string.starts-with "hello" "he") +(string.ends-with "hello" "lo") +(string.contains "hello" "ell") +(string.substring "hello" 1 3) +(string.char-at "hello" 0) +(string.length "hello") +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:1325 +(math.sqrt 16) +(math.abs -5) +(math.floor 3.7) +(math.ceil 3.2) +(math.round 3.5) +(math.pow 2 10) +(math.log 2.718) +(math.sin 0) +(math.cos 0) +(math.min 3 7 1 5) +(math.max 3 7 1 5) +(math.clamp 15 0 10) +(math.pi) +(math.e) +(math.random) +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:1345 +(collections.map (lambda (x) (* x x)) (list 1 2 3)) +(collections.filter (lambda (x) (> x 2)) (list 1 2 3 4)) +(collections.reduce (lambda (a b) (+ a b)) (list 1 2 3) 0) +(collections.zip (list 1 2) (list "a" "b")) +(collections.flatten (list (list 1 2) (list 3 4))) +(collections.sort (list 3 1 2)) +(collections.reverse (list 1 2 3)) +(collections.range 1 5) +(collections.unique (list 1 2 2 3 3)) +(collections.dict-keys (dict "a" 1 "b" 2)) +(collections.dict-values (dict "a" 1 "b" 2)) +(collections.dict-merge (dict "a" 1) (dict "b" 2)) +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:1445 +(set i 0) +(while (< i 5) (let ((i (+ i 1))) (print i))) +(set i 0) +(while (< i 5) (begin (print i) (set i (+ i 1)))) +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:1466 +(state-set "counter" (str 42)) +(set raw (state-get "counter")) +(set n (int raw)) +(if (> n 40) "big" "small") +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:1478 +(set g (range 3)) +(collect g) +(collect g) +(defun make-range () (range 3)) +(collect (make-range)) +(collect (make-range)) +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:1496 +(msg-recv "work.queue") +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:1526 +(set done nil) +(for i (range 10) (if done nil (for j (range 10) (if (= j 3) (begin (set done #t) (break nil)) nil)))) +@@@ script docs/STREAMING.md:107 +(set cam (stream-open (dict "id" "cam0" "codec" "rgba8" "w" 1280 "h" 720))) +(get-attr cam "ok") +(get-attr cam "token") +(get-attr cam "seq_channel") +(stream-info cam) +(stream-close cam) +@@@ script docs/STREAMING.md:162 +(gl-bind-stream "screen" (get-attr cam "token")) +@@@ script docs/STREAMING.md:499 +(set cam (stream-open (dict "id" "cam0" "codec" "rgba8" "w" 1280 "h" 720))) +(stream-capture-camera (get-attr cam "token") 0) +(gl-bind-stream "screen" (get-attr cam "token")) +@@@ script docs/roadmap/ARIADNE_CONFERENCE_ROADMAP.md:97 +(set room (room-join "conf.example.org:7000" "standup" (dict "name" "Joe"))) +(set cam (stream-open (dict "id" "self/cam" "codec" "rgba8" "w" 1280 "h" 720))) +(stream-capture-camera (get-attr cam "token") 0) +(room-publish room (dict "kind" "camera" "token" (get-attr cam "token"))) +(set mic (stream-open (dict "id" "self/mic" "codec" "f32" "rate" 48000 "ch" 1))) +(stream-capture-audio (get-attr mic "token") (dict "device" 0)) +(room-publish room (dict "kind" "audio" "token" (get-attr mic "token"))) +@@@ script docs/roadmap/CVCGL-UI-DSL-ROADMAP.md:4452 +(state-set "demo.n" 256) +(window "Controls" (slider-int "Agents" :bind "demo.n" :lo 1 :hi 512) (combo "Belief" :bind "demo.belief" :options (list "shared" "grouped" "private")) (button "Reset" :on "reset")) +(scene (geometry-node "mesh" :source "file://blocks.off" :visible-when (= (state-get "demo.show") "1"))) +@@@ script docs/roadmap/DSL-GEOMETRY-VOLUME-ROADMAP.md:91 +(vol-read "mydata" "/path/to/brain.rawiv") +(set dims (vol-dims "mydata")) +(set bbox (vol-bbox "mydata")) +(vol-normalize "mydata" "normalized") +(vol-scale "normalized" 2.0 "scaled") +(iso "scaled" 0.5 "surface" (dict "method" "fastcontouring")) +(sdf "surface" "sdf-vol" (dict "dim" (list 128 128 128))) +(tet "sdf-vol" 0.0 "tet-mesh") +(vol-write "sdf-vol" "/path/to/output.rawiv") +(geom-write "tet-mesh" "/path/to/output.off") +@@@ script docs/roadmap/DSL-GEOMETRY-VOLUME-ROADMAP.md:288 +(iso "brain-vol" 128.0 "brain-surf" (dict "method" "fastcontouring" "improve" #t "compute-normals" #t)) +(sdf "brain-surf" "brain-sdf" (dict "dim" (list 256 256 256) "flip-normals" nil)) +@@@ script docs/roadmap/DSL-GEOMETRY-VOLUME-ROADMAP.md:307 +(defun process-brain (input-file output-dir) (begin (vol-read "raw" input-file) (print (str "Loaded: " (vol-dims "raw"))) (vol-normalize "raw" "norm") (vol-clip "norm" 0.1 0.9 "clipped") (iso "clipped" 0.5 "surface" (dict "method" "fastcontouring" "improve" #t "compute-normals" #t)) (print (str "Surface: " (geom-num-tris "surface") " tris")) (tet "clipped" 0.5 "tet-mesh" (dict "improve" #t)) (print (str "Tet mesh: " (geom-num-tets "tet-mesh") " tets")) (geom-write "surface" (str output-dir "/brain-surface.off")) (geom-write "tet-mesh" (str output-dir "/brain-tet.off")) (dict "tris" (geom-num-tris "surface") "tets" (geom-num-tets "tet-mesh")))) +(process-brain "/data/brain.rawiv" "/output") +@@@ script docs/roadmap/DSL-GEOMETRY-VOLUME-ROADMAP.md:343 +(defun process-all (file-list output-dir) (for f file-list (begin (print (str "Processing: " f)) (process-brain f output-dir)))) +@@@ script docs/roadmap/DSL-GEOMETRY-VOLUME-ROADMAP.md:353 +(defun distribute-pipeline (files) (for f files (begin (state-set (str "jobs." f ".status") "pending") (state-set (str "jobs." f ".input") f) (msg-send "work-queue" f)))) +(defun worker () (while #t (set job (msg-recv "work-queue")) (state-set (str "jobs." job ".status") "running") (process-brain (state-get (str "jobs." job ".input")) "/output") (state-set (str "jobs." job ".status") "done"))) +@@@ script docs/roadmap/STATE_BINARY_STREAMING.md:220 +(define s (stream-open |{:kind| "video_raw" :codec "rgba8" :w 1920 :h 1080 |}|)) +(stream-info s) +(gl-bind-stream node s) +(msg-recv "stream.evt") +@@@ script docs/roadmap/STATE_EXEC_PORTING_PLAN.md:1101 +(state-set "sensors.cache.temperature" (read-sensor)) +(state-expire "sensors.cache.temperature" 60) +(if (state-is-expired "sensors.cache.temperature") (begin (state-sweep-expired "sensors.cache") (refresh-cache)) (state-get "sensors.cache.temperature")) +(state-expire-at "jobs.batch-42" "2025-06-15T00:00:00Z") +@@@ script docs/roadmap/STATE_EXEC_PORTING_PLAN.md:1355 +(let ((pid (fork))) (if (= pid 0) (begin (state-set "results.child" (compute))) (begin (state-set "results.parent" (str "spawned child " pid))))) +@@@ script docs/roadmap/STATE_EXEC_PORTING_PLAN.md:1383 +(let ((pid (fork))) (if (= pid -1) (print "fork failed: resource limit") |...|)) +@@@ script docs/roadmap/STATE_EXEC_PORTING_PLAN.md:1470 +(require "string") +(split "hello world" " ") +(stdlib.string.split "hello world" " ") +(require "math" (quote (sqrt abs floor ceil))) +(sqrt 144) +(stdlib-modules) +(stdlib-list "string") +@@@ script docs/roadmap/STATE_EXEC_PORTING_PLAN.md:1709 +(require "geometry") +(distance 0 0 3 4) +(require "util") +(repeatedly 5 (lambda (i) (print i))) +@@@ script docs/roadmap/STATE_EXEC_PORTING_PLAN.md:1763 +(let ((mesh (state-data-get "/scene/model/geometry"))) (if (data? mesh) (begin (print (data-type mesh)) (state-data-set "/output/result" mesh)))) +@@@ script docs/roadmap/STATE_EXEC_PORTING_PLAN.md:1776 +(state-data-set "/scene/model/transformed" transformed-mesh) +@@@ script docs/roadmap/STATE_EXEC_PORTING_PLAN.md:1781 +(data-type mesh) +(data? mesh) +(data? 42) +(state-data-type "/path") +@@@ script docs/roadmap/STATE_EXEC_PORTING_PLAN.md:1853 +(state-set "/scene/camera/x" 10) +(state-set "/scene/camera/y" 20) +(state-set "/scene/camera/z" 30) +(state-batch (state-set "/scene/camera/x" 10) (state-set "/scene/camera/y" 20) (state-set "/scene/camera/z" 30)) +@@@ script docs/roadmap/STATE_EXEC_PORTING_PLAN.md:1876 +(state-lock "/scene/camera" (let ((x (state-get "/scene/camera/x")) (y (state-get "/scene/camera/y"))) (state-set "/scene/camera/distance" (sqrt (+ (* x x) (* y y)))))) +@@@ script src/cvc-cli/cvc_cli_test.cpp:1093 +(+ 1 2 3) +@@@ script src/cvc-cli/cvc_cli_test.cpp:1099 +(* 7 6) +@@@ script src/cvc-cli/cvc_cli_test.cpp:1105 +(let ((x 10)) (+ x 5)) +@@@ script src/cvc-cli/cvc_cli_test.cpp:1111 +(let ((f (lambda (x y) (+ x y)))) (f 3 4)) +@@@ script src/cvc-cli/cvc_cli_test.cpp:1120 +(+ 100 200 300) +@@@ script src/cvc-cli/cvc_cli_test.cpp:1206 +(let ((square (lambda (x) (* x x)))) (square 9)) +@@@ script src/cvc-cli/cvc_cli_test.cpp:1211 +(car (list 10 20 30)) +@@@ script src/cvc-cli/cvc_cli_test.cpp:1216 +(if (> 5 3) 99 0) +@@@ script src/cvc-cli/cvc_cli_test.cpp:1221 +(let ((a 10) (b 20)) (+ a b)) +@@@ script src/cvc-cli/cvc_cli_test.cpp:1228 +(begin 1 2 3 4 5 6 7 8 9 10) +@@@ script src/cvc-cli/cvc_cli_test.cpp:1238 +(let ((pi 3.14159) (r 5)) (* pi r r)) +@@@ script src/cvc-cli/cvc_cli_test.cpp:1881 +(no children) +@@@ script src/cvc-cli/cvc_cli_test.cpp:1891 +(+ 1 (+ 2 (+ 3 (+ 4 (+ 5 6))))) +@@@ script src/cvc-cli/main.cpp:1554 +(no children) +@@@ script src/cvc/ariadne/ariadne.cpp:1256 +(no image) +@@@ script src/cvc/ariadne/ftxui_backend.cpp:89 +(empty) +@@@ script src/cvc/tests/ariadne_loader_test.cpp:535 +(state-set "paused" "true") +@@@ script src/cvc/tests/ariadne_loader_test.cpp:2256 +(> (int (state-get "n")) 5) +@@@ script src/cvc/tests/ariadne_loader_test.cpp:2276 +(state-exists "ready") +@@@ script src/cvc/tests/ariadne_loader_test.cpp:2277 +(state-exists "busy") +@@@ script src/cvc/tests/ariadne_loader_test.cpp:2301 +(list "p" "q") +@@@ script src/cvc/tests/ariadne_loader_test.cpp:2402 +(state-exists "show") +@@@ script src/cvc/tests/ariadne_loader_test.cpp:3688 +(int (state-get "n")) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:80 +(closable) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:356 +(state-set "paused" (if (= (state-get "paused") "true") "false" "true")) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:376 +(begin (state-set "agents" "64") (state-set "speed" "1.0")) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:400 +(begin (msg-recv "async.done") (state-set "async.result" "resumed")) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:428 +(state-set "async.value" (msg-recv "vchan")) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:486 +(begin (nav-step-async) (state-set "nav.result" (msg-recv "nav.done"))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:511 +(state-set "tick.flag" "1") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:547 +(begin (state-set "input.last_key" (get-attr event "key")) (state-set "input.kind" (get-attr event "kind"))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:565 +(state-set "input.last_key" (get-attr event "key")) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:583 +(begin (state-set "ptr.kind" (get-attr event "kind")) (state-set "ptr.btn" (get-attr event "button"))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:606 +(state-set "widget.clicked" "yes") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:728 +(begin (state-set "ev.btn" (get-attr event "button")) (state-set "ev.has_x" (if (>= (get-attr event "x") 0) "yes" "no"))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:780 +(host-bump) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:959 +(state-set "agents" "128") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:960 +(no error) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1033 +(while #t (+ 1 1)) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1077 +(state-set "n" "10") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1121 +(= (state-get "show") "1") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1150 +(int 2) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1351 +(state-exists "never") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1384 +(state-set "x" 1) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1399 +(state-set "mode" "on") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1401 +(= (state-get "mode") "on") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1414 +(while #t 1) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1431 +(msg-recv "x") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1431 +(await 1) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1470 +(state-set "flag" "0") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1472 +(state-get "flag") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1474 +(state-get "never_set") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1490 +(foobar 1 2) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1541 +(is-string (str-concat "a" "b")) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1543 +(is-list (append (list 1) (list 2))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1545 +(apply and (list #t)) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1547 +(begin (print "x") #t) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1565 +(= (quote (1 2 3)) (quote (1 2 3))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1567 +(= (quote (1 2)) (quote (9 9))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1569 +(let ((xs (list 1 2 3))) (> (length xs) 2)) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1587 +(begin (defclass B (init (self) (while 1 1))) (B)) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1589 +(eval (quote #t)) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1591 +(begin (defmacro m () #t) #t) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1606 +(begin (set d (list 0 0)) (set i 0) (while (< i 30) (begin (set d (list d d)) (set i (+ i 1)))) (state-data-set "bomb" d)) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1614 +(is-null (state-data-get "bomb")) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1626 +(state-set "n" "3") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1647 +(state-set "lock" "1") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1649 +(state-exists "lock") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1686 +(state-set "n" "7") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1688 +(+ (int (state-get "n")) 1) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1689 +(state-get "n") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1775 +(list "shared" "grouped" "private") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1799 +(state-set "hint" "dynamic") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1803 +(state-get "hint") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1847 +(state-set "n" "2") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1912 +(* 1e+19 1e+19) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1928 +(begin (defun f () #t) (f)) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1930 +(f) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:2287 +(begin (set r (msg-recv (http-get-async "http://ex/x"))) (state-set "r.status" (get-attr r "status")) (state-set "r.ok" (get-attr r "ok")) (state-set "r.error" (get-attr r "error")) (state-data-set "r.body" (get-attr r "body"))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:2325 +(begin (set r (msg-recv (http-get-async "http://ex/down"))) (state-set "r.ok" (get-attr r "ok")) (state-set "r.error" (get-attr r "error"))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:2353 +(state-set "r.status" (get-attr (msg-recv (http-get-async "http://ex/slow")) "status")) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:2383 +(begin (set r (http-get "http://ex/x")) (state-set "r.status" (get-attr r "status")) (state-data-set "r.body" (get-attr r "body"))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:2414 +(state-set "r.status" (get-attr (await (http-get-async "http://ex/x")) "status")) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:2449 +(begin (set r (fetch "mem://x")) (state-set "r.ok" (get-attr r "ok")) (state-data-set "r.body" (get-attr r "body"))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:2481 +(state-data-set "r.body" (get-attr (await (fetch-async "mem://y")) "body")) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:2519 +(begin (set r (http-request "POST" "http://ex/api" (dict "headers" (dict "Authorization" "Bearer tok-123") "form" (dict "grip" 1 "risk" 2)))) (state-set "r.status" (get-attr r "status"))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:2560 +(begin (set r (http-get "http://ex/x" (dict "query" (dict "q" "hello world" "n" 2)))) (state-set "r.status" (get-attr r "status"))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:2592 +(begin (set r (http-request "PUT" "http://ex/raw" (dict "body" "raw-payload"))) (state-set "r.status" (get-attr r "status"))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:2625 +(begin (set g (await (http-get-async "http://ex/src"))) (set r (await (http-request-async "PUT" "http://ex/dst" (dict "body" (get-attr g "body") "headers" (dict "Authorization" "Bearer tok"))))) (state-set "r.status" (get-attr r "status"))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:2688 +(begin (state-set "r.t" (str (pool-double 21))) (state-set "r.a" (str (await (pool-double-async 10))))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:2754 +(begin (set h (stream-open (dict "id" "cam0" "w" 4 "h" 2))) (state-set "t.ok" (get-attr h "ok")) (state-set "t.token" (get-attr h "token")) (state-set "t.seq" (get-attr h "seq_channel")) (state-set "t.live" (get-attr (stream-info h) "live")) (state-set "t.closed" (get-attr (stream-close h) "ok")) (state-set "t.live2" (get-attr (stream-info h) "live"))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:2785 +(begin (set h (stream-open (dict "id" "cam0" "w" 4 "h" 2))) (state-set "t.token" (get-attr h "token")) (state-set "t.seq" (get-attr h "seq_channel"))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:2814 +(begin (set h (stream-open (dict "id" "cam0" "w" 4 "h" 2))) (state-set "t.ok" (get-attr h "ok"))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:2843 +(begin (set a (stream-open (dict "id" "c0" "w" 4 "h" 2 "expected_subscribers" -1))) (state-set "t.a" (get-attr a "ok")) (set b (stream-open (dict "id" "c1" "w" 999999 "h" 2))) (state-set "t.b" (get-attr b "ok"))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:2869 +(begin (set h (stream-open (dict "id" "cam0" "w" 4 "h" 2))) (state-set "t.ok" (get-attr h "ok")) (state-set "t.err" (get-attr h "error"))) +@@@ script src/cvc/tests/async_lane_test.cpp:498 +(+ (lane-double 21) (await (lane-double-async 10))) +@@@ script src/cvc/tests/async_lane_test.cpp:525 +(while t (apply-built (msg-recv "app#built.tracks"))) +@@@ script src/cvc/tests/async_lane_test.cpp:526 +(while t (apply-plain (msg-recv "built"))) +@@@ script src/cvc/tests/async_lane_test.cpp:621 +(while t (begin (tick) (await 0))) +@@@ script src/cvc/tests/async_lane_test.cpp:652 +(while t (sleep (msg-recv (sim-launch)))) +@@@ script src/cvc/tests/state_exec_advanced_integration_test.cpp:125 +(begin (set big "") (set i 0) (while (< i 100) (begin (set big (str-concat big "0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789")) (set i (+ i 1)))) (state-set "data.large_string" big) (length big)) +@@@ script src/cvc/tests/state_exec_advanced_integration_test.cpp:151 +(begin (state-data-set "data.typed_obj" (list 1 2 3 4 5)) (set obj (state-data-get "data.typed_obj")) (is-null obj)) +@@@ script src/cvc/tests/state_exec_advanced_integration_test.cpp:168 +(begin (set i 0) (while (< i 50) (begin (state-set "data.counter" (str i)) (set i (+ i 1)))) (state-get "data.counter")) +@@@ script src/cvc/tests/state_exec_advanced_integration_test.cpp:803 +(begin (state-set "queue.item.0" "apple") (state-set "queue.item.1" "banana") (state-set "queue.item.2" "cherry") (state-set "queue.count" "3") (msg-send "queue.ready" "items-available") "producer-done") +@@@ script src/cvc/tests/state_exec_advanced_integration_test.cpp:813 +(begin (set result (list)) (set i 0) (while (< i 3) (begin (set key (str-concat "queue.item." (str i))) (set val (state-get key)) (if (!= val nil) (set i (+ i 1)) nil))) (state-set "consumed.total" (str i)) i) +@@@ script src/cvc/tests/state_exec_advanced_integration_test.cpp:843 +(begin (state-set "partial.0" (str (+ 1 2 3))) "map-0-done") +@@@ script src/cvc/tests/state_exec_advanced_integration_test.cpp:849 +(begin (state-set "partial.1" (str (+ 4 5 6))) "map-1-done") +@@@ script src/cvc/tests/state_exec_advanced_integration_test.cpp:855 +(begin (state-set "partial.2" (str (+ 7 8 9))) "map-2-done") +@@@ script src/cvc/tests/state_exec_advanced_integration_test.cpp:861 +(begin (set s0 (state-get "partial.0")) (set s1 (state-get "partial.1")) (set s2 (state-get "partial.2")) (state-set "result.total" (str (+ (if (= s0 nil) 0 6) (if (= s1 nil) 0 15) (if (= s2 nil) 0 24)))) 45) +@@@ script src/cvc/tests/state_exec_advanced_integration_test.cpp:928 +(begin (spawn "(* 10 10)" "w0") (spawn "(* 20 20)" "w1") (spawn "(* 30 30)" "w2") 3) +@@@ script src/cvc/tests/state_exec_advanced_integration_test.cpp:947 +(begin (set total 0) (set i 0) (while (< i 5) (begin (set total (+ total (* i i))) (state-set "progress.current" (str i)) (state-set "progress.total" (str total)) (set i (+ i 1)))) total) +@@@ script src/cvc/tests/state_exec_advanced_integration_test.cpp:977 +(begin (state-set "document.header.title" "My Report") (state-set "document.header.author" "Alice") (state-set "document.header.date" "2026-05-24") (msg-send "document.events" "header-complete") "header-done") +@@@ script src/cvc/tests/state_exec_advanced_integration_test.cpp:986 +(begin (state-set "document.body.section1" "Introduction text...") (state-set "document.body.section2" "Methods text...") (state-set "document.body.section3" "Results text...") (msg-send "document.events" "body-complete") "body-done") +@@@ script src/cvc/tests/state_exec_advanced_integration_test.cpp:1013 +(begin (defun fib (n) (if (<= n 1) n (+ (fib (- n 1)) (fib (- n 2))))) (set result (fib 12)) (state-set "computation.fib12.result" (str result)) (state-set "computation.fib12.status" "complete") (msg-send "computation.notifications" "fib12-done") result) +@@@ script src/cvc/tests/state_exec_advanced_integration_test.cpp:1094 +(begin (while #t 1)) +@@@ script src/cvc/tests/state_exec_advanced_integration_test.cpp:1099 +(+ 1 2) +@@@ script src/cvc/tests/state_exec_advanced_integration_test.cpp:1110 +(+ 3 4) +@@@ script src/cvc/tests/state_exec_advanced_integration_test.cpp:1111 +(+ 5 6) +@@@ script src/cvc/tests/state_exec_async_test.cpp:138 +(- 10 3) +@@@ script src/cvc/tests/state_exec_async_test.cpp:140 +(* 4 5) +@@@ script src/cvc/tests/state_exec_async_test.cpp:144 +(/ 10 4) +@@@ script src/cvc/tests/state_exec_async_test.cpp:147 +(+ (* 2 3) (- 10 4)) +@@@ script src/cvc/tests/state_exec_async_test.cpp:152 +(< 1 2) +@@@ script src/cvc/tests/state_exec_async_test.cpp:153 +(< 2 1) +@@@ script src/cvc/tests/state_exec_async_test.cpp:157 +(= 5 5) +@@@ script src/cvc/tests/state_exec_async_test.cpp:158 +(= 5 6) +@@@ script src/cvc/tests/state_exec_async_test.cpp:163 +(if t 1 2) +@@@ script src/cvc/tests/state_exec_async_test.cpp:165 +(if nil 1 2) +@@@ script src/cvc/tests/state_exec_async_test.cpp:167 +(begin 1 2 3) +@@@ script src/cvc/tests/state_exec_async_test.cpp:169 +(begin (set x 42) x) +@@@ script src/cvc/tests/state_exec_async_test.cpp:172 +(begin (set i 0) (set sum 0) (while (< i 5) (begin (set sum (+ sum i)) (set i (+ i 1)))) sum) +@@@ script src/cvc/tests/state_exec_async_test.cpp:179 +(begin (set sum 0) (for x (list 1 2 3 4 5) (set sum (+ sum x))) sum) +@@@ script src/cvc/tests/state_exec_async_test.cpp:188 +(begin (set sq (lambda (x) (* x x))) (sq 5)) +@@@ script src/cvc/tests/state_exec_async_test.cpp:192 +(begin (set make-adder (lambda (n) (lambda (x) (+ x n)))) (set add5 (make-adder 5)) (add5 10)) +@@@ script src/cvc/tests/state_exec_async_test.cpp:200 +(begin (defun fact (n) (if (< n 2) 1 (* n (fact (- n 1))))) (fact 10)) +@@@ script src/cvc/tests/state_exec_async_test.cpp:209 +(let ((x 10) (y 20)) (+ x y)) +@@@ script src/cvc/tests/state_exec_async_test.cpp:214 +(begin (set f (lambda () (begin (return 42) 99))) (f)) +@@@ script src/cvc/tests/state_exec_async_test.cpp:220 +(quote (1 2 3)) +@@@ script src/cvc/tests/state_exec_async_test.cpp:229 +(begin (defmacro double-it (x) (* x 2)) (double-it 21)) +@@@ script src/cvc/tests/state_exec_async_test.cpp:237 +(eval (quote (+ 1 2))) +@@@ script src/cvc/tests/state_exec_async_test.cpp:242 +(str-concat "hello" " " "world") +@@@ script src/cvc/tests/state_exec_async_test.cpp:248 +(car (list 1 2 3)) +@@@ script src/cvc/tests/state_exec_async_test.cpp:249 +(car (cdr (list 1 2 3))) +@@@ script src/cvc/tests/state_exec_async_test.cpp:252 +(length (list 1 2 3 4 5)) +@@@ script src/cvc/tests/state_exec_async_test.cpp:257 +(begin (defun fib (n) (if (< n 2) n (+ (fib (- n 1)) (fib (- n 2))))) (fib 15)) +@@@ script src/cvc/tests/state_exec_async_test.cpp:268 +(await (+ 1 2)) +@@@ script src/cvc/tests/state_exec_async_test.cpp:275 +(+ 10 20) +@@@ script src/cvc/tests/state_exec_async_test.cpp:395 +(+ 10 20 30) +@@@ script src/cvc/tests/state_exec_async_test.cpp:449 +(begin (set i 0) (set s 0) (while (< i 5) (begin (set s (+ s i)) (set i (+ i 1)))) s) +@@@ script src/cvc/tests/state_exec_async_test.cpp:469 +(begin (set x 10) (set y 20) (+ x y)) +@@@ script src/cvc/tests/state_exec_async_test.cpp:470 +(begin (defun fact (n) (if (< n 2) 1 (* n (fact (- n 1))))) (fact 10)) +@@@ script src/cvc/tests/state_exec_async_test.cpp:471 +(begin (set sum 0) (for x (list 1 2 3 4 5) (set sum (+ sum x))) sum) +@@@ script src/cvc/tests/state_exec_async_test.cpp:472 +(let ((a 5) (b 10)) (* a b)) +@@@ script src/cvc/tests/state_exec_async_test.cpp:498 +(+ 40 2) +@@@ script src/cvc/tests/state_exec_async_test.cpp:509 +(begin 1 2 3 4 5) +@@@ script src/cvc/tests/state_exec_async_test.cpp:537 +(begin 1 2 3 4 5 6 7 8) +@@@ script src/cvc/tests/state_exec_async_test.cpp:570 +(nap) +@@@ script src/cvc/tests/state_exec_async_test.cpp:645 +(begin (set r (await 42)) r) +@@@ script src/cvc/tests/state_exec_async_test.cpp:670 +(begin (set s 0) (set i 0) (while (< i 200) (begin (set s (+ s i)) (set i (+ i 1)))) s) +@@@ script src/cvc/tests/state_exec_async_test.cpp:738 +(+ 2 3) +@@@ script src/cvc/tests/state_exec_bench_test.cpp:67 +(begin (defun fib (n) (if (<= n 1) n (+ (fib (- n 1)) (fib (- n 2))))) (fib 20)) +@@@ script src/cvc/tests/state_exec_bench_test.cpp:90 +(begin (set sum 0) (set i 0) (while (< i 1000) (begin (set sum (+ sum i)) (set i (+ i 1)))) sum) +@@@ script src/cvc/tests/state_exec_bench_test.cpp:116 +(begin (defun fib (n) (if (<= n 1) n (+ (fib (- n 1)) (fib (- n 2))))) (fib 15)) +@@@ script src/cvc/tests/state_exec_bench_test.cpp:141 +(begin (set x 0) (set i 0) (while (< i 100) (begin (set x (+ x i)) (set i (+ i 1)))) x) +@@@ script src/cvc/tests/state_exec_bench_test.cpp:179 +(begin (set lst (list)) (set i 0) (while (< i 500) (begin (set lst (append lst (list i))) (set i (+ i 1)))) lst) +@@@ script src/cvc/tests/state_exec_bench_test.cpp:253 +(begin (defun even? (n) (if (= n 0) #t (odd? (- n 1)))) (defun odd? (n) (if (= n 0) nil (even? (- n 1)))) (even? 200)) +@@@ script src/cvc/tests/state_exec_codec_test.cpp:297 +(begin (set sum 0) (set i 0) (while (< i 100) (begin (set sum (+ sum i)) (set i (+ i 1)))) sum) +@@@ script src/cvc/tests/state_exec_codec_test.cpp:320 +(+ 1 2 3 4 5 6 7 8 9 10) +@@@ script src/cvc/tests/state_exec_codec_test.cpp:427 +(/ 10 3) +@@@ script src/cvc/tests/state_exec_codec_test.cpp:495 +(let ((x 10)) (let ((y 20)) (+ x y))) +@@@ script src/cvc/tests/state_exec_codec_test.cpp:527 +(cdr (list 1 2 3)) +@@@ script src/cvc/tests/state_exec_codec_test.cpp:530 +(length (list 1 2 3 4)) +@@@ script src/cvc/tests/state_exec_codec_test.cpp:535 +(begin (set d (dict)) (set-attr d "key" 42) (get-attr d "key")) +@@@ script src/cvc/tests/state_exec_codec_test.cpp:544 +(begin (defun fib (n) (if (< n 2) n (+ (fib (- n 1)) (fib (- n 2))))) (fib 10)) +@@@ script src/cvc/tests/state_exec_coordinator_test.cpp:469 +(begin (+ 1 2) (+ 3 4) (+ 5 6)) +@@@ script src/cvc/tests/state_exec_coordinator_test.cpp:653 +(begin (+ 1 2) (+ 3 4)) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:41 +(- 10 3 2) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:43 +(* 3 4) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:47 +(% 10 3) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:49 +(+ (* 3 4) (- 10 5)) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:59 +(= 42 42) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:60 +(= 1 2) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:69 +(if nil 1) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:73 +(begin) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:78 +(quote (+ 1 2)) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:90 +(begin (set sum 0) (for x (list 1 2 3 4) (set sum (+ sum x))) sum) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:99 +(begin (set add1 (lambda (x) (+ x 1))) (add1 10)) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:114 +(begin (set sum-all (lambda (&rest args) (apply + args))) (sum-all 1 2 3)) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:123 +(begin (defun double (x) (* x 2)) (double 21)) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:130 +(begin (defun fact (n) (if (<= n 1) 1 (* n (fact (- n 1))))) (fact 5)) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:149 +(begin (defun early (x) (if (< x 0) (return -1) x)) (early -5)) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:172 +(root) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:188 +(car (cdr (list 10 20 30))) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:189 +(car (cons 5 (list 10 20))) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:192 +(length (list 1 2 3)) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:197 +(begin (set obj (dict "x" 10 "y" 20)) (set-attr obj "x" 99) (get-attr obj "x")) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:215 +(begin (defun fib (n) (if (<= n 1) n (+ (fib (- n 1)) (fib (- n 2))))) (fib 10)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:49 +(begin (set m (lambda (self) 99)) (set superd (dict "m" m)) (set clsd (dict "__super__" superd)) (set obj (dict "__class__" clsd)) (super obj m)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:57 +(begin (set superd (dict "mk" list)) (set clsd (dict "__super__" superd)) (set obj (dict "__class__" clsd)) (length (super obj mk 4 5))) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:64 +(begin (set m (lambda (self a) (+ a 90))) (set superd (dict "m" m)) (set clsd (dict "__super__" superd)) (set obj (dict "__class__" clsd)) (super obj m 9)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:72 +(begin (set superd2 (dict "mk" list)) (set superd1 (dict "__super__" superd2)) (set clsd (dict "__super__" superd1)) (set obj (dict "__class__" clsd)) (length (super obj mk 1))) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:79 +(begin (set superd (dict "m" 5)) (set clsd (dict "__super__" superd)) (set obj (dict "__class__" clsd)) (super obj m)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:101 +(while t 1) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:120 +(+ 2 2) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:145 +(set a 1) +(set b 2) +(+ a b) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:150 +(1 2) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:151 +(no-such-fn 1) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:152 +(begin (set v 7) (v 1)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:153 +(if t) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:154 +(while t) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:155 +(for x (list 1)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:156 +(for 1 (list 1) 2) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:157 +(for x 5 2) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:158 +(set a) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:159 +(set 1 2) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:160 +(quote 1 2) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:161 +(lambda (x)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:162 +(lambda (1) 2) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:163 +(lambda 5 2) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:164 +(let ((a 1))) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:165 +(let 5 2) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:166 +(let ((a)) 1) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:167 +(let ((1 2)) 3) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:168 +(defun f (x)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:169 +(defun 5 (x) 1) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:170 +(defmacro m (x)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:171 +(defmacro 5 (x) 1) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:172 +(defmacro m (1) 2) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:173 +(eval) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:174 +(super 1) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:175 +(defclass) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:176 +(defclass 5) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:180 +(begin (defun f () (begin (return) 9)) (f)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:184 +(begin (set f (lambda (x) x)) (f 1 2)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:185 +(begin (set f (lambda (&rest) 1)) (f 5)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:189 +(begin (set f (lambda () 42)) (f)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:192 +(lambda nil 42) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:196 +(begin (defun f () (return 5)) (+ (f) 1)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:200 +(begin (defmacro pick (a b) b) (pick 1)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:204 +(begin (defmacro m2 (x) (set y x) (+ y 1)) (m2 5)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:208 +(get-attr d "name") +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:209 +(send d "speak") +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:210 +(send d "add" 1 2) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:211 +(send d "finalize") +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:215 +(begin (defclass Empty) (Empty 42)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:220 +(begin (defclass P1 (m (self) 7)) (defclass C1 (P1) (n (self) 8)) (send (C1) "n")) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:226 +(super d speak) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:227 +(super d "speak") +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:228 +(super d add 3 4) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:237 +(super (dict "k" 1) m) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:238 +(super (Animal "a") speak) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:239 +(super d fly) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:243 +(begin (defclass KX (m (self) 7) 5 (5 1)) (send (KX) "m")) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:249 +(begin (set Q (lambda (x) x)) (defclass CQ Q (n (self) 4)) (send (CQ) "n")) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:252 +(begin (set Q2 (lambda (x) x)) (defclass CR (Q2) (n (self) 5)) (send (CR) "n")) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:286 +(set a 4) +(set b 5) +(+ a b) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:295 +(begin (set i 0) (while (< i 20) (set i (+ i 1))) i) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:346 +(return) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:347 +(yield) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:353 +(defun f (1) 1) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:365 +(begin (set v 7) (v)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:369 +(begin (set f (lambda (&rest xs) (length xs))) (f 1 2 3)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:378 +(begin (set f (lambda (x) (set y (* x 2)) (+ y 1))) (f 4)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:381 +(let () 42) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:384 +(begin (defun f (&rest xs) (length xs)) (f 1 2)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:392 +(for x (range 0) x) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:414 +(begin (defclass P2 (who (self) "parent")) (defclass C2 (P2)) (send (C2) "who")) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:421 +(begin (defclass V (m (self &rest more) (length more))) (send (V) "m" 1 2)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:424 +(begin (defclass M (m (self) (set y 4) (+ y 2))) (send (M) "m")) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:428 +(begin (defclass K (m (self) 7) (5 1)) (send (K) "m")) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:441 +(defclass B (m (self 5) 1)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:473 +(-) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:474 +(*) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:475 +(/) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:476 +(- 2.5) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:477 +(- 5.5 1 0.5) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:478 +(/ 4) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:479 +(+ 0.5 (list 1 2)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:480 +(+ (list 1.5) 1) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:481 +(* (list 2 3)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:482 +(* 2.0 (list 3 4)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:483 +(% 7.5 2) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:487 +(% 1 0) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:488 +(% 1.0 0.0) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:489 +(/ 1 0) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:490 +(< nil 1) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:491 +(car) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:492 +(send 5) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:496 +(int (list 1)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:497 +(float (list 1)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:498 +(str-concat "a" 1) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:502 +(car (list)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:503 +(length (cdr (list))) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:504 +(car 5) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:505 +(nth (list 1) 5) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:506 +(nth (list 1) "x") +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:507 +(set-nth (list 1) 5 0) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:508 +(del-nth (list 1) 5) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:509 +(begin (set l (list 1 2 3)) (del-nth l 1)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:510 +(length (dict "a" 1)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:511 +(length t) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:515 +(slice (list 1)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:516 +(length (slice (list 1 2 3) -1 2)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:517 +(length (slice (list 1 2) 0 -1)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:518 +(length (slice (list 1 2) 5 6)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:519 +(length (slice (list 1 2) 0 9)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:520 +(length (slice (list 1 2) 1 1)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:521 +(length (slice (list 1 2 3) 1)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:525 +(dict "a") +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:526 +(get-attr (dict 1 "x") "1") +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:527 +(get-attr (dict) "k") +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:528 +(get-attr 5 "k") +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:529 +(get-attr (dict) 5) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:530 +(del-attr (dict) "k") +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:531 +(begin (set d (dict "k" 3)) (del-attr d "k")) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:535 +(send (dict "a" 1) "m") +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:536 +(begin (set o (dict "__class__" (dict "m" (lambda (s) 1)))) (send o "m")) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:539 +(begin (set o (dict "__class__" (dict "m" 5))) (send o "m")) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:541 +(begin (set o (dict "__class__" (dict "x" 1))) (send o "m")) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:546 +(begin (set superd (dict "mk" list)) (set clsd (dict "x" 1 "__super__" superd)) (set obj (dict "__class__" clsd)) (length (send obj "mk" 4 5))) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:555 +(apply 5 (list 1)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:556 +(append 5 1) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:559 +(print 1 "a" 2.5) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:562 +(generator 5) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:563 +(next 5) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:564 +(gen-done? 5) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:565 +(collect 5) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:566 +(range 1 2 3 4) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:567 +(range "a") +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:568 +(range "a" 2) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:569 +(range 1 "b") +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:570 +(range 1 5 "c") +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:571 +(range 1 5 0) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:576 +(collect (generator (lambda () (begin (yield 1) (yield 2))))) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:585 +(and) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:586 +(and 1 nil 2) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:587 +(and 1 2) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:588 +(or nil nil) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:589 +(or nil 7) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:590 +(not nil) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:629 +(set c 1) +(+ c 1) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:756 +(+ 1 1) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:770 +(* 2 3) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:771 +(* 2 4) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:772 +(* 2 5) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:781 +(begin (set i 0) (while (< i 50) (set i (+ i 1))) i) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:795 +(begin (set i 0) (while (< i 40) (set i (+ i 1)))) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:808 +(set-attr box "sig" __signal__) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:809 +(begin (set i 0) (while (< i 6) (set i (+ i 1))) (get-attr box "sig")) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:820 +(begin (set i 0) (while (< i 4) (set i (+ i 1))) i) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:857 +(begin (set i 0) (while (< i 6) (set i (+ i 1))) (get-attr box "path")) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:863 +(set-attr box "path" p) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:878 +(begin (set i 0) (while (< i 6) (set i (+ i 1))) (get-attr box "n")) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:883 +(set-attr box "n" 1) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:883 +(set-attr box "n" 2) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:927 +(begin (set i 0) (while (< i 8) (set i (+ i 1))) (get-attr box "path")) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:954 +(begin (set i 0) (while (< i 100) (set i (+ i 1)))) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:987 +(+ 20 22) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:95 +(spawn "42" "child") +@@@ script src/cvc/tests/state_exec_integration_test.cpp:152 +(self-uid) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:650 +(begin (state-set "mailbox.test" "ready") (msg-send "mailbox.test" "hello-world")) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:664 +(begin (msg-send "test.path" "msg1") (msg-send "test.path" "msg2") (msg-send "test.path" "msg3") (message-count)) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:687 +(begin (msg-send "test" "1") (msg-send "test" "2") (msg-send "test" "3") (msg-send "test" "4")) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:706 +(begin (msg-send "test" "hello world!!") (msg-send "test" "this exceeds the byte cap") "done") +@@@ script src/cvc/tests/state_exec_integration_test.cpp:725 +(begin (state-set "shared.item.0" "100") (state-set "shared.item.1" "200") (state-set "shared.item.2" "300") (state-set "shared.count" "3") "done") +@@@ script src/cvc/tests/state_exec_integration_test.cpp:737 +(begin (state-get "shared.count")) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:758 +(begin (state-set "parent.uid" (self-uid)) (set child-pid (spawn "(+ 1 2)" "child-worker")) child-pid) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:774 +(begin (set result (msg-send "events.log" "{\"level\":\"info\"}" "application/json")) (get-attr result "status")) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:802 +(begin (msg-send "console.stdout" "Hello from DSL!") (msg-send "console.stdout" "Computing...") (set result (* 6 7)) (msg-send "console.stdout" (str-concat "Result: " (str result))) (msg-send "console.stderr" "Warning: example only") result) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:832 +(begin (set before (msg-pending "q.test")) (msg-send "q.test" "a") (msg-send "q.test" "b") (set after (msg-pending "q.test")) (list before after)) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:855 +(begin (msg-send "drain.test" "first") (msg-send "drain.test" "second") (msg-send "drain.test" "third") "sent") +@@@ script src/cvc/tests/state_exec_integration_test.cpp:873 +(begin (set m1 (msg-recv "drain.test")) (set m2 (msg-recv "drain.test")) (set m3 (msg-recv "drain.test")) (list (get-attr m1 "status") (get-attr m2 "status") (get-attr m3 "status"))) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:959 +(+ 100 200) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1028 +(begin (defun fib (n) (if (<= n 1) n (+ (fib (- n 1)) (fib (- n 2))))) (fib 10)) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1036 +(begin (defun fact (n) (if (<= n 1) 1 (* n (fact (- n 1))))) (fact 8)) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1056 +(begin (set procs (ps)) (length procs)) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1071 +(begin (set info (inspect (self-pid))) (get-attr info "pid")) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1085 +(begin (set i 0) (set sum 0) (while (< i 10) (begin (set sum (+ sum i)) (set i (+ i 1)))) sum) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1101 +(begin (set total 0) (for x (list 10 20 30 40) (set total (+ total x))) total) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1114 +(begin (set double (lambda (x) (* x 2))) (double 21)) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1125 +(let ((x 10) (y 20)) (+ x y)) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1135 +(begin (set d (dict "name" "alice" "age" 30)) (get-attr d "name")) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1157 +(begin (set stats (scheduler-stats)) (get-attr stats "total")) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1212 +(begin (state-set "data.x" "42") (state-get "data.x")) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1353 +(begin (set wid (state-watch "events.x" (lambda (p v) nil))) (state-unwatch wid)) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1367 +(begin (state-watch "trigger.value" (lambda (path val) (state-set "got-event" "yes"))) (state-set "trigger.value" "hello") (set tries 0) (while (and (not (state-exists "got-event")) (< tries 100)) (set tries (+ tries 1))) (state-get "got-event")) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1388 +(begin (state-watch "notify.path" (lambda (path val) (state-set "received-path" path))) (state-set "notify.path" "data") (set tries 0) (while (and (not (state-exists "received-path")) (< tries 100)) (set tries (+ tries 1))) (state-get "received-path")) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1417 +(begin (state-set "consumer.log" "") (state-watch "queue.item" (lambda (path val) (state-set "consumer.log" (str-concat (state-get "consumer.log") (state-get path) ",")))) (state-set "queue.item" "apple") (state-set "queue.item" "banana") (state-set "queue.item" "cherry") (set tries 0) (while (and (not (= (state-get "consumer.log") "apple,banana,cherry,")) (< tries 300)) (set tries (+ tries 1))) (state-get "consumer.log")) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1451 +(begin (state-set "stats.ticks" "") (state-watch "sensor.reading" (lambda (path val) (state-set "stats.ticks" (str-concat (state-get "stats.ticks") "|")))) (state-set "sensor.reading" "72") (state-set "sensor.reading" "73") (state-set "sensor.reading" "71") (state-set "sensor.reading" "74") (state-set "sensor.reading" "70") (set tries 0) (while (and (not (= (state-get "stats.ticks") "|||||")) (< tries 500)) (set tries (+ tries 1))) (state-get "stats.ticks")) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1487 +(begin (state-watch "pipeline.input" (lambda (path val) (state-set "pipeline.stage1" (str-concat "processed:" (state-get path))))) (state-watch "pipeline.stage1" (lambda (path val) (state-set "pipeline.stage2" (str-concat "[" (state-get path) "]")))) (state-set "pipeline.input" "hello") (set tries 0) (while (and (not (state-exists "pipeline.stage2")) (< tries 500)) (set tries (+ tries 1))) (state-get "pipeline.stage2")) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1525 +(begin (state-set "sampler.log" "") (set wid (state-watch "data.stream" (lambda (path val) (begin (set current-val (state-get path)) (set current-log (state-get "sampler.log")) (state-set "sampler.log" (str-concat current-log current-val ",")) (set updated (state-get "sampler.log")) (if (= updated (str-concat current-val ",")) nil (state-unwatch wid)))))) (state-set "data.stream" "alpha") (state-set "data.stream" "beta") (state-set "data.stream" "gamma") (state-set "data.stream" "delta") (set tries 0) (while (and (= (state-get "sampler.log") "") (< tries 300)) (set tries (+ tries 1))) (set tries 0) (while (< tries 100) (set tries (+ tries 1))) (state-get "sampler.log")) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1577 +(begin (state-watch "jobs.request" (lambda (path val) (state-set "jobs.result" (str-concat "DONE:" (state-get path))))) (set i 0) (while (< i 500) (set i (+ i 1))) "worker-done") +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1590 +(begin (state-set "jobs.request" "task-42") (set tries 0) (while (and (not (state-exists "jobs.result")) (< tries 500)) (set tries (+ tries 1))) (state-get "jobs.result")) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1650 +(begin (let ((i 0)) (while (< i 5) (begin (state-set (str-concat "queue.item." (str i)) (str (* i i))) (set i (+ i 1))))) (state-set "queue.count" "5") "produce-done") +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1664 +(begin (set tries 0) (while (and (not (state-exists "queue.count")) (< tries 1000)) (set tries (+ tries 1))) (let ((total 0)) (for i (range 5) (set total (+ total 1))) total)) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1691 +(begin (for i (range 10) (state-set (str-concat "batch." (str i)) (str (* i 10)))) (state-set "batch.done" "true") 10) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1703 +(begin (set tries 0) (while (and (not (state-exists "batch.done")) (< tries 2000)) (set tries (+ tries 1))) (let ((sum 0)) (for i (range 10) (set sum (+ sum (* i 10)))) sum)) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1731 +(begin (for i (range 1 6) (state-set (str-concat "sq." (str i)) (str (* i i)))) (state-set "sq.ready" "5") "squares-done") +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1743 +(begin (set tries 0) (while (and (not (state-exists "sq.ready")) (< tries 2000)) (set tries (+ tries 1))) (let ((g (generator (lambda () (for i (range 1 6) (let ((val (state-get (str-concat "sq." (str i))))) (yield (* 2 1))))))) (count 0)) (for x g (set count (+ count 1))) count)) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1797 +(begin (for p (range 4) (let ((key (str-concat "data.producer." (str-concat (str p) ".done")))) (set tries 0) (while (and (not (state-exists key)) (< tries 5000)) (set tries (+ tries 1))))) (let ((total 0)) (for i (range 12) (set total (+ total 1))) total)) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1828 +(begin (let ((squares (generator (lambda () (for i (range 1 8) (yield (* i i))))))) (let ((total 0)) (for v squares (set total (+ total v))) total))) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1851 +(begin (let ((src (generator (lambda () (yield 10) (yield 20) (yield 30)))) (doubled (generator (lambda () (for x src (yield (* x 2))))))) (collect doubled))) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1894 +(begin (set tries 0) (while (and (not (state-exists "status.2")) (< tries 5000)) (set tries (+ tries 1))) (let ((results (list))) (for i (range 3) (let ((val (state-get (str-concat "status." (str i))))) (append results val))) (length results))) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1921 +(begin (let ((evens (generator (lambda () (for i (range 0 10) (if (= (% i 2) 0) (yield i))))))) (collect evens))) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1946 +(begin (let ((counter (generator (lambda () (let ((n 0)) (while (< n 1000) (begin (yield n) (set n (+ n 1)))))))) (taken (list))) (for x counter (if (>= (length taken) 4) (break nil) (append taken x))) taken)) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1976 +(begin (let ((keys (generator (lambda () (yield "name") (yield "age") (yield "role")))) (vals (generator (lambda () (yield "alice") (yield "30") (yield "engineer")))) (pairs (list))) (let ((k (next keys)) (v (next vals))) (while (not (is-null k)) (begin (append pairs (str-concat k "=" v)) (set k (next keys)) (set v (next vals))))) pairs)) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:2021 +(begin (set tries 0) (while (and (not (state-exists "worker.2.result")) (< tries 5000)) (set tries (+ tries 1))) (let ((total 0)) (for i (range 3) (set total (+ total 1))) total)) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:2041 +(begin (let ((running-sum (generator (lambda () (let ((acc 0)) (for x (list 10 20 30 40 50) (begin (set acc (+ acc x)) (yield acc)))))))) (collect running-sum))) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:2073 +(begin (set received (list)) (set count 0) (while (< count 9) (begin (set msg (msg-recv "work.queue")) (append received (get-attr msg "status")) (set count (+ count 1)))) (set ones (generator (lambda () (for item received (yield 1))))) (length (collect ones))) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:2117 +(begin (set payloads (list)) (set n 0) (while (< n 8) (begin (set msg (msg-recv "data.in")) (append payloads (get-attr msg "status")) (set n (+ n 1)))) (set doubler (generator (lambda () (for p payloads (yield (str-concat p p)))))) (length (collect doubler))) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:2161 +(begin (set msg (msg-recv "ch")) 42) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:2170 +(begin (msg-send "ch" "hello") 99) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:2376 +(msg-send "sig" "hi") +@@@ script src/cvc/tests/state_exec_integration_test.cpp:2382 +(msg-pending "sig") +@@@ script src/cvc/tests/state_exec_integration_test.cpp:2400 +(msg-recv "sig") +@@@ script src/cvc/tests/state_exec_integration_test.cpp:2409 +(msg-send "raw.chan" "x") +@@@ script src/cvc/tests/state_exec_intrinsics_test.cpp:1526 +(self-pid) +@@@ script src/cvc/tests/state_exec_intrinsics_test.cpp:1536 +(cluster-id) +@@@ script src/cvc/tests/state_exec_intrinsics_test.cpp:1541 +(state-set "dsl.key" "dsl-value") +@@@ script src/cvc/tests/state_exec_intrinsics_test.cpp:1542 +(state-get "dsl.key") +@@@ script src/cvc/tests/state_exec_intrinsics_test.cpp:1547 +(state-set "dsl.exists" "yes") +@@@ script src/cvc/tests/state_exec_intrinsics_test.cpp:1548 +(state-exists "dsl.exists") +@@@ script src/cvc/tests/state_exec_intrinsics_test.cpp:1553 +(memory-limit) +@@@ script src/cvc/tests/state_exec_intrinsics_test.cpp:1558 +(spawn "(+ 1 2)") +@@@ script src/cvc/tests/state_exec_intrinsics_test.cpp:1585 +(string.upper "hello") +@@@ script src/cvc/tests/state_exec_intrinsics_test.cpp:1590 +(math.sqrt 25) +@@@ script src/cvc/tests/state_exec_intrinsics_test.cpp:1595 +(math.pi) +@@@ script src/cvc/tests/state_exec_intrinsics_test.cpp:1600 +(string.length "hello world") +@@@ script src/cvc/tests/state_exec_intrinsics_test.cpp:1605 +(collections.range 3) +@@@ script src/cvc/tests/state_exec_multiprocess_test.cpp:188 +(begin (set v1 (state-get "shared.x")) (set v2 (state-get "shared.y")) (set v3 (state-get "shared.z")) (str-concat v1 ":" v2 ":" v3)) +@@@ script src/cvc/tests/state_exec_multiprocess_test.cpp:231 +(begin (state-set "shared.x" "10") (state-set "shared.y" "20") (state-set "shared.z" "30") "written") +@@@ script src/cvc/tests/state_exec_multiprocess_test.cpp:325 +(begin (set m (msg-recv "oob.chan")) (get-attr m "payload")) +@@@ script src/cvc/tests/state_exec_multiprocess_test.cpp:394 +(begin (msg-send "oob.chan" "hello-from-A") "sent") +@@@ script src/cvc/tests/state_exec_multiprocess_test.cpp:489 +(begin (set reader (generator (lambda () (for i (range 5) (yield (state-get (str-concat "items.v" (str i)))))))) (set count 0) (set concat "") (for val reader (begin (set count (+ count 1)) (set concat (str-concat concat val ":")))) (state-set "result.consumer" (str-concat (str count) "|" concat)) count) +@@@ script src/cvc/tests/state_exec_multiprocess_test.cpp:553 +(begin (set val-gen (generator (lambda () (for i (range 5) (yield (str-concat "v" (str i))))))) (for val val-gen (begin (set idx (str-concat "items." val)) (state-set idx val))) (state-set "items.ready" "true") "produced") +@@@ script src/cvc/tests/state_exec_multiprocess_test.cpp:655 +(begin (set count 0) (for p (range 3) (for i (range 3) (begin (set key (str-concat "producer." (str p) ".item." (str i))) (if (state-exists key) (set count (+ count 1)) 0)))) count) +@@@ script src/cvc/tests/state_exec_multiprocess_test.cpp:807 +(begin (set result "") (set i 0) (while (< i 4) (begin (set m (msg-recv "pipe.data")) (set result (str-concat result (get-attr m "payload") ",")) (set i (+ i 1)))) (state-set "pipeline.result" result) result) +@@@ script src/cvc/tests/state_exec_multiprocess_test.cpp:892 +(begin (set payload-gen (generator (lambda () (for tag (list "aa" "bb" "cc" "dd") (yield tag))))) (for payload payload-gen (msg-send "pipe.data" payload)) "all-sent") +@@@ script src/cvc/tests/state_exec_multiprocess_test.cpp:988 +(begin (state-set "grpc.x" "alpha") (state-set "grpc.y" "beta") "written") +@@@ script src/cvc/tests/state_exec_multiprocess_test.cpp:1012 +(begin (str-concat (state-get "grpc.x") ":" (state-get "grpc.y"))) +@@@ script src/cvc/tests/state_exec_multiprocess_test.cpp:1064 +(begin (set m (msg-recv "grpc.chan")) (get-attr m "payload")) +@@@ script src/cvc/tests/state_exec_multiprocess_test.cpp:1086 +(begin (msg-send "grpc.chan" "hello-grpc") "sent") +@@@ script src/cvc/tests/state_exec_multiprocess_test.cpp:1147 +(begin (set squares (generator (lambda () (for i (range 5) (yield (* i i)))))) (let ((i 0)) (for sq squares (begin (state-set (str-concat "data.v" (str i)) (str sq)) (set i (+ i 1))))) (state-set "data.ready" "true") "produced") +@@@ script src/cvc/tests/state_exec_multiprocess_test.cpp:1179 +(begin (set reader (generator (lambda () (for i (range 5) (yield (int (state-get (str-concat "data.v" (str i))))))))) (set sum 0) (for val reader (set sum (+ sum val))) sum) +@@@ script src/cvc/tests/state_exec_parser_test.cpp:83 +(if (> x 0) "pos" "neg") +@@@ script src/cvc/tests/state_exec_parser_test.cpp:97 +(a (b (c (d)))) +@@@ script src/cvc/tests/state_exec_parser_test.cpp:143 +(+ 1 2) +@@@ script src/cvc/tests/state_exec_parser_test.cpp:152 +(defun f (x) x) +(f 42) +@@@ script src/cvc/tests/state_exec_scheduler_test.cpp:177 +(- 100 50) +@@@ script src/cvc/tests/state_exec_scheduler_test.cpp:280 +(begin 4 5 6) +@@@ script src/cvc/tests/state_exec_scheduler_test.cpp:806 +(begin (defun fib (n) (if (<= n 1) n (+ (fib (- n 1)) (fib (- n 2))))) (fib 8)) +@@@ script src/cvc/tests/state_exec_scheduler_test.cpp:813 +(begin (defun fact (n) (if (<= n 1) 1 (* n (fact (- n 1))))) (fact 6)) +@@@ script src/cvc/tests/state_exec_scheduler_test.cpp:826 +(begin (let ((sum 0)) (for i (list 1 2 3 4 5) (set sum (+ sum i))) sum)) +@@@ script src/cvc/tests/state_exec_scheduler_test.cpp:838 +(begin (let ((i 0) (sum 0)) (while (< i 10) (begin (set sum (+ sum i)) (set i (+ i 1)))) sum)) +@@@ script src/cvc/tests/state_exec_scheduler_test.cpp:852 +(begin (let ((double (lambda (x) (* x 2)))) (double 21))) +@@@ script src/cvc/tests/state_exec_scheduler_test.cpp:1112 +(if #t 42 99) +@@@ script src/cvc/tests/state_exec_scheduler_test.cpp:1113 +(let ((x 10)) (* x x)) +@@@ script src/cvc/tests/state_exec_scheduler_test.cpp:1113 +(length "hello") +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:157 +(begin (set i 0) (while (< i 1000) (set i (+ i 1))) i) +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:220 +(let ((g (generator (lambda () (yield 10) (yield 20) (yield 30))))) (list (next g) (next g) (next g))) +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:236 +(let ((g (generator (lambda () (yield 1))))) (list (next g) (next g))) +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:248 +(let ((g (generator (lambda () (yield (* 2 3)) (yield (+ 10 5)))))) (+ (next g) (next g))) +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:259 +(let ((r (range 0 5))) (+ (next r) (next r) (next r))) +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:267 +(collect (range 5)) +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:274 +(let ((total 0)) (for x (range 1 6) (set total (+ total x))) total) +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:284 +(let ((accum (list))) (for x (generator (lambda () (yield "a") (yield "b") (yield "c"))) (append accum x)) (length accum)) +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:298 +(collect (generator (lambda () (let ((i 0)) (while (< i 4) (begin (yield i) (set i (+ i 1)))))))) +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:316 +(while t (break)) +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:322 +(while t (break 42)) +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:326 +(begin (set i 0) (while t (if (= i 5) (break i) (set i (+ i 1))))) +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:335 +(for x (list 1 2 3 4 5) (if (= x 3) (break x) nil)) +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:342 +(for x (range 1000) (if (= x 5) (break x) nil)) +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:349 +(begin (set sum 0) (for i (list 1 2 3) (for j (list 10 20 30) (if (= j 20) (break nil) (set sum (+ sum j))))) sum) +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:360 +(begin (set i 0) (while t (begin (set i (+ i 1)) (if (= i 3) (break i) nil)))) +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:370 +(for x (list 1 2 3 4 5) (let ((doubled (* x 2))) (if (> doubled 6) (break doubled) nil))) +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:382 +(if #t 1 2) +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:383 +(begin 1 2) +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:384 +(quote 1) +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:385 +(defclass C (init (self) 1)) +@@@ script src/cvc/tests/state_exec_types_test.cpp:72 +(1 2 3) +@@@ script src/cvc/tests/state_exec_types_test.cpp:83 +(1 ("a" "b")) +@@@ script src/cvcGL/test/cvcgl_media_audio.cpp:50 +(if (= (state-get "cap.done") "1") nil (begin (stream-capture-audio (get-attr (stream-open (dict "id" "mic" "codec" "f32" "rate" 48000 "ch" 1 "frames_per_chunk" 256)) "token") (dict "rate" 48000 "ch" 1 "frames" 256 "mix" (list (dict "tone" (dict "hz" 440)) (dict "tone" (dict "hz" 660))))) (state-set "cap.done" "1"))) +@@@ script src/cvcGL/test/cvcgl_media_capture.cpp:52 +(if (= (state-get "cap.done") "1") nil (begin (stream-capture-camera (get-attr (stream-open (dict "id" "cam" "w" 64 "h" 48)) "token") (dict "synthetic" #t "w" 64 "h" 48 "hz" 240)) (state-set "cap.done" "1"))) +@@@ script src/cvcGL/test/cvcgl_media_play.cpp:52 +(if (= (state-get "play.done") "1") nil (begin (stream-open (dict "id" "mic" "codec" "f32" "rate" 48000 "ch" 1 "frames_per_chunk" 256 "subscriber_depth" 8)) (stream-capture-audio "streams.mic" (dict "rate" 48000 "ch" 1 "frames" 256 "tone" (dict "hz" 440))) (stream-play-audio "streams.mic") (state-set "play.done" "1"))) +@@@ script src/cvcGL/test/cvcgl_stream_bind.cpp:63 +(begin (set h (gl-bind-stream "screen" "streams.cam0")) (state-set "ok" (get-attr h "ok")) (state-set "node" (get-attr h "node")) (state-set "token" (get-attr h "token")) (state-set "done" "1")) +@@@ script src/cvcGL/test/cvcgl_stream_bind.cpp:81 +(begin (set h (gl-bind-stream "screen" (dict "token" "demo.streams.cam0"))) (state-set "ok" (get-attr h "ok")) (state-set "done" "1")) +@@@ script src/cvcGL/test/cvcgl_stream_bind.cpp:96 +(begin (set h (gl-bind-stream "ghost" "streams.cam0")) (state-set "ok" (get-attr h "ok")) (state-set "err" (get-attr h "error")) (state-set "done" "1")) +@@@ script src/cvcGL/test/cvcgl_stream_bind.cpp:109 +(begin (set h (gl-bind-stream "screen" "streams.cam0")) (state-set "ok" (get-attr h "ok")) (state-set "err" (get-attr h "error")) (state-set "done" "1")) +@@@ script src/cvcGL/test/cvcgl_stream_bind.cpp:123 +(begin (set h (gl-bind-stream "screen")) (state-set "ok" (get-attr h "ok")) (state-set "err" (get-attr h "error")) (state-set "done" "1")) +@@@ script bindings/pycvc/test_pycvc_exec.py:49 +(state-set "dsl.out" "wrote-from-dsl") +@@@ script bindings/pycvc/test_pycvc_exec.py:123 +(capture #t) +@@@ script bindings/pycvc/test_pycvc_exec.py:147 +(capture (make-bytes)) +@@@ script bindings/pycvc/test_pycvc_exec.py:21 +(* 6 7) +@@@ script bindings/pycvc/test_pycvc_exec.py:22 +(begin (set x 5) x) +@@@ script bindings/pycvc/test_pycvc_exec.py:23 +(str-concat "ab" "cd") +@@@ script bindings/pycvc/test_pycvc_exec.py:25 +(> 1 2) +@@@ script bindings/pycvc/test_pycvc_exec.py:47 +(state-get "shared.x") +@@@ script bindings/pycvc/test_pycvc_exec.py:64 +(py-double 21) +@@@ script bindings/pycvc/test_pycvc_exec.py:65 +(py-add 40 2) +@@@ script bindings/pycvc/test_pycvc_exec.py:66 +(py-greet "cvc") +@@@ script bindings/pycvc/test_pycvc_exec.py:68 +(+ 1 (py-double 20)) +@@@ script bindings/pycvc/test_pycvc_exec.py:81 +(stash "py.k" "py.v") +@@@ script bindings/pycvc/test_pycvc_exec.py:86 +(fetch "py.k") +@@@ script bindings/pycvc/test_pycvc_exec.py:98 +(boom) +@@@ script bindings/pycvc/test_pycvc_exec.py:119 +(capture 5) +@@@ script bindings/pycvc/test_pycvc_exec.py:121 +(capture "text") +@@@ script bindings/pycvc/test_pycvc_exec_async.py:29 +(begin (adouble 21)) +@@@ script bindings/pycvc/test_pycvc_exec_async.py:62 +(begin (slow 7)) +@@@ script bindings/pycvc/test_pycvc_exec_async.py:78 +(begin (get-attr (boom 1) "__async_error__")) +@@@ script bindings/pycvc/test_pycvc_exec_async.py:91 +(begin (inc (triple 4))) +@@@ script bindings/pycvc/test_pycvc_exec_async.py:110 +(begin (spawn "(while t 1)") (state-set "r" (str (slow 7))) (state-get "r")) +@@@ script bindings/pycvc/test_pycvc_exec_async.py:82 +(begin (idn 5)) +@@@ script bindings/pycvc/test_pycvc_exec_coro.py:58 +(begin (spawn "(while t 1)") (state-set "r" (str (acount 6))) (state-get "r")) +@@@ script bindings/pycvc/test_pycvc_exec_coro.py:25 +(begin (adbl 21)) +@@@ script bindings/pycvc/test_pycvc_exec_coro.py:40 +(begin (acount 10)) +@@@ script bindings/pycvc/test_pycvc_exec_coro.py:96 +(begin (plus10 (inc (triple 4)))) +@@@ script bindings/pycvc/test_pycvc_exec_coro.py:128 +(begin (reenter 1)) +@@@ script bindings/pycvc/test_pycvc_integration.py:60 +(state-set "dsl.done" "yes") +@@@ script bindings/pycvc/test_pycvc_integration.py:58 +(+ 0 (verts)) diff --git a/src/cvc/tests/state_exec_corpus/scripts.txt b/src/cvc/tests/state_exec_corpus/scripts.txt new file mode 100644 index 000000000..cbdc30509 --- /dev/null +++ b/src/cvc/tests/state_exec_corpus/scripts.txt @@ -0,0 +1,2437 @@ +# state_exec parse corpus. Generated by src/cvc/tests/gen_state_exec_corpus.py; +# do not edit by hand. Each entry is a '@@@ script ' line, then the script. +@@@ script bindings/pycvc/demo.ari:init +(state-set "demo.agents" "128") +(state-set "demo.speed" "1.0") +(state-set "demo.wire" "false") + +@@@ script bindings/pycvc/demo.ari:windows[0].children[5].bind +(if (> (int (state-get "demo.agents")) 256) "many" "few") +@@@ script src/cvc/ariadne/components/fe_controls.ari:units.fe_controls.children[9].on +(state-set "graphics.root.children.domain.fe.request" "mesh") +@@@ script src/cvc/ariadne/components/fe_controls.ari:units.fe_controls.children[9].disabled_when +(= (state-get "graphics.root.children.domain.fe.busy") "1") +@@@ script src/cvc/ariadne/components/fe_controls.ari:units.fe_controls.children[13].visible_when +(= (state-get "graphics.root.children.domain.fe.problem") "poisson") +@@@ script src/cvc/ariadne/components/fe_controls.ari:units.fe_controls.children[14].visible_when +(= (state-get "graphics.root.children.domain.fe.problem") "heat") +@@@ script src/cvc/ariadne/components/fe_controls.ari:units.fe_controls.children[17].on +(state-set "graphics.root.children.domain.fe.request" "solve") +@@@ script src/cvc/ariadne/components/fe_controls.ari:units.fe_controls.children[17].disabled_when +(or (= (state-get "graphics.root.children.domain.fe.busy") "1") (= (state-get "graphics.root.children.domain.fe.stats.tets") "0")) +@@@ script src/cvc/ariadne/components/fe_controls.ari:units.fe_controls.children[18].on +(state-set "graphics.root.children.domain.fe.request" "stop") +@@@ script src/cvc/ariadne/components/fe_controls.ari:units.fe_controls.children[18].enabled_when +(= (state-get "graphics.root.children.domain.fe.stoppable") "1") +@@@ script src/cvc/ariadne/components/mesh_tools.ari:units.mesh_tools.children[10].on +(state-set "graphics.root.children.mesh.mesh_ops.request" "smooth") +@@@ script src/cvc/ariadne/components/mesh_tools.ari:units.mesh_tools.children[10].disabled_when +(= (state-get "graphics.root.children.mesh.mesh_ops.busy") "1") +@@@ script src/cvc/ariadne/components/mesh_tools.ari:units.mesh_tools.children[14].on +(state-set "graphics.root.children.mesh.mesh_ops.request" "decimate") +@@@ script src/cvc/ariadne/components/mesh_tools.ari:units.mesh_tools.children[21].on +(state-set "graphics.root.children.mesh.mesh_ops.request" "repair") +@@@ script src/cvc/ariadne/components/mesh_tools.ari:units.mesh_tools.children[22].on +(state-set "graphics.root.children.mesh.mesh_ops.request" "orient") +@@@ script src/cvc/ariadne/components/mesh_tools.ari:units.mesh_tools.children[23].on +(state-set "graphics.root.children.mesh.mesh_ops.request" "reset") +@@@ script src/cvc/ariadne/components/mesh_tools.ari:units.mesh_tools.children[28].visible_when +(= (state-get "graphics.root.children.mesh.mesh_ops.color.mode") "geodesic") +@@@ script src/cvc/ariadne/components/mesh_tools.ari:units.mesh_tools.children[29].on +(state-set "graphics.root.children.mesh.mesh_ops.request" "curvature") +@@@ script src/cvc/ariadne/components/sim_transport.ari:units.sim_transport.children[3].enabled_when +(= (state-get "sim.paused") "1") +@@@ script src/cvcGL/examples/bunny_shadow.ari:init +(state-set "show_bunny" "true") + +@@@ script src/cvcGL/examples/bunny_shadow.ari:windows[3].children[3].on +(begin + (state-set "lighting.key_intensity" "1.9") + (state-set "lighting.key_azimuth" "-38") + (state-set "lighting.key_elevation" "52") + (state-set "lighting.key_cone" "34")) + +@@@ script src/cvcGL/examples/hello.ari:init +(state-set "demo.agents" "256") + +@@@ script src/cvcGL/examples/hello.ari:windows[0].children[3].visible_when +(> (int (state-get "demo.agents")) 256) +@@@ script src/cvcGL/examples/hello.ari:windows[0].children[4].bind +(if (> (int (state-get "demo.agents")) 256) "heavy" "ok") +@@@ script src/cvcGL/examples/nav_compute.ari:init +(begin + (state-set "nav.status" "idle") + (state-set "nav.result" "-")) + +@@@ script src/cvcGL/examples/nav_compute.ari:windows[0].children[3].on +(begin + (state-set "nav.status" "running") + (nav-step-async) + (state-set "nav.result" (msg-recv "nav.done")) + (state-set "nav.status" "done")) + +@@@ script src/cvcGL/examples/nav_convoy.ari:init +(begin + (state-set "show.agents" "1") + (state-set "show.grid" "1") + (state-set "sim.paused" "0") + (state-set "sim.speed" "1.0")) + +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:134 +;; Convert to integer — truncates floats, parses strings +(int 3.7) ;; => 3 +(int "123") ;; => 123 +(int true) ;; => 1 + +;; Convert to float — widens integers, parses strings +(float 5) ;; => 5.0 +(float "3.14") ;; => 3.14 +(float true) ;; => 1.0 + +;; Type predicates +(is-int 42) ;; => true +(is-int 3.14) ;; => false +(is-float 3.14) ;; => true +(is-float 42) ;; => false +(is-string "hi") ;; => true +(is-string 42) ;; => false +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:158 +;; Factorial +(begin + (defun fact (n) + (if (<= n 1) 1 + (* n (fact (- n 1))))) + (fact 10)) +;; => 3628800 + +;; Map over a list using for-loop +(begin + (set result (list)) + (for x (list 1 2 3 4 5) + (set result (append result (list (* x x))))) + result) +;; => (1 4 9 16 25) + +;; Producer-consumer via state tree +;; Process 1 writes, Process 2 reads +(state-set "shared.counter" "0") +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:190 +;; A generator that yields 1, 2, 3 +(set g (generator (lambda () + (yield 1) + (yield 2) + (yield 3)))) +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:200 +(range 5) ;; yields 0, 1, 2, 3, 4 +(range 2 8) ;; yields 2, 3, 4, 5, 6, 7 +(range 0 10 3) ;; yields 0, 3, 6, 9 +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:208 +;; Pull one value at a time +(set g (range 3)) +(next g) ;; => 0 +(next g) ;; => 1 +(next g) ;; => 2 +(next g) ;; => nil (exhausted) + +;; Check if exhausted +(gen-done? g) ;; => true + +;; Iterate with for +(for x (range 5) + (print x)) ;; prints 0 1 2 3 4 + +;; Materialise all values into a list +(collect (range 5)) ;; => (0 1 2 3 4) +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:231 +(set evens (generator (lambda () + (for x (range 10) + (if (= (% x 2) 0) + (yield x) nil))))) +(collect evens) ;; => (0 2 4 6 8) +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:241 +(set squares (generator (lambda () + (for x (range 5) + (yield (* x x)))))) +(collect squares) ;; => (0 1 4 9 16) +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:250 +(set taken (list)) +(for x (range 1000) + (if (>= (length taken) 4) + (break nil) + (append taken x))) +taken ;; => (0 1 2 3) +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:261 +(set running-sum (generator (lambda () + (let ((total 0)) + (for x (range 1 6) + (begin + (set total (+ total x)) + (yield total))))))) +(collect running-sum) ;; => (1 3 6 10 15) +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:273 +(set counter (generator (lambda () + (let ((n 0)) + (while (< n 1000) + (begin (yield n) + (set n (+ n 1)))))))) +;; Consume lazily with break +(for x counter + (if (= x 5) (break x) nil)) ;; => 5 +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:288 +;; Exit with no value (loop evaluates to nil) +(while t (break)) + +;; Exit with a value +(while t (break 42)) ;; => 42 + +;; Find first match +(for x (list 10 20 30 40) + (if (> x 25) (break x) nil)) ;; => 30 + +;; Nested loops — break exits only the innermost +(set sum 0) +(for i (list 1 2 3) + (for j (list 10 20 30) + (if (= j 20) (break nil) ;; exits inner for + (set sum (+ sum j))))) +sum ;; => 30 (10 + 10 + 10) +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:448 +;; Spawn a child process +(set child-pid (spawn "(begin (state-set \"child.done\" \"yes\") 42)" "child-worker")) + +;; Fork the current process +(set cpid (fork)) + +;; Query own identity +(self-pid) ;; => 1 +(self-uid) ;; => "alice" + +;; List all processes +(ps) ;; => list of dicts [{pid: 1, name: "...", status: "running", ...}, ...] + +;; Inspect a specific process +(inspect 2) ;; => dict with full process_info + +;; Control other processes +(pause 2) +(resume 2) +(kill 2) + +;; Query resource usage +(memory-usage) (memory-limit) +(time-elapsed) (time-limit) +(step-count) +(message-count) (message-limit) + +;; System identity +(cluster-id) ;; => "cluster-1" +(node-id) ;; => "node-A" +(scheduler-stats) ;; => {total: N, running: N, ...} +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:531 +;; Write a value +(state-set "myapp.config.timeout" "30") + +;; Read it back +(state-get "myapp.config.timeout") ;; => "30" + +;; Check existence +(state-exists "myapp.config.timeout") ;; => true + +;; List children of a node +(state-children "myapp.config") ;; => ("timeout") + +;; Delete a node +(state-delete "myapp.old-data") + +;; Store/retrieve binary data objects +(state-data-set "myapp.blob" some-value) +(state-data-get "myapp.blob") +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:554 +;; Expire a node 60 seconds from now +(state-expire "myapp.session.abc" 60) + +;; Expire at an absolute ISO 8601 time +(state-expire-at "myapp.session.abc" "2026-01-01T00:00:00Z") + +;; Query expiry status +(state-has-expiry "myapp.session.abc") ;; => true +(state-is-expired "myapp.session.abc") ;; => false (not yet) + +;; Clear expiry +(state-clear-expiry "myapp.session.abc") + +;; Sweep all expired nodes under a path (returns count removed) +(state-sweep-expired "myapp.session") +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:582 +;; Send a plain-text message to a state tree path +(msg-send "myapp.events.updates" "hello from process 1") + +;; Send with an explicit content type +(msg-send "myapp.events.data" "{\"key\": \"value\"}" "application/json") + +;; msg-send returns a dict: {status: "delivered", path: "myapp.events.updates"} +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:595 +(begin + (set i 0) + (while (< i 10) + (begin + (state-set (str-concat "queue.item." (str i)) (str (* i i))) + (msg-send "queue.notify" (str i)) + (set i (+ i 1)))) + "producer-done") +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:607 +(begin + ;; Poll for items (cooperative check) + (set count 0) + (while (< count 10) + (if (state-exists (str-concat "queue.item." (str count))) + (begin + (set val (state-get (str-concat "queue.item." (str count)))) + (state-set (str-concat "consumed." (str count)) val) + (set count (+ count 1))) + nil)) + count) +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:705 +;; Block until a message arrives on "events.data" +(set msg (msg-recv "events.data")) + +;; msg is a dict — extract the payload +(get-attr msg "payload") +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:723 +;; Non-blocking check +(if (> (msg-pending "events.data") 0) + (set msg (msg-recv "events.data")) + (print "no messages yet")) +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:735 +;; Sleep for half a second +(sleep 0.5) + +;; Polling loop with sleep +(while true + (begin + (if (> (msg-pending "work.queue") 0) + (begin + (set job (msg-recv "work.queue")) + (process-job job)) + (sleep 0.1)))) +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:756 +(begin + (set results (list)) + (set done false) + (while (not done) + (begin + (set m (msg-recv "tasks.input")) + (set payload (get-attr m "payload")) + (if (= payload "stop") + (set done true) + (set results (append results (list payload)))))) + results) +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:1183 +;; A process writing a migration request for itself +(begin + (set my-pid (self-pid)) + (state-set + (str-concat "__migrate_request." (str my-pid)) + "target-node-B") + ;; The coordinator or a management process can poll for + ;; __migrate_request.* entries and call coord.migrate() + my-pid) +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:1308 +(string.split "a,b,c" ",") ;; => ("a" "b" "c") +(string.join (list "a" "b" "c") ",") ;; => "a,b,c" +(string.replace "hello" "l" "r") ;; => "herro" +(string.trim " hello ") ;; => "hello" +(string.upper "hello") ;; => "HELLO" +(string.lower "HELLO") ;; => "hello" +(string.starts-with "hello" "he") ;; => true +(string.ends-with "hello" "lo") ;; => true +(string.contains "hello" "ell") ;; => true +(string.substring "hello" 1 3) ;; => "el" +(string.char-at "hello" 0) ;; => "h" +(string.length "hello") ;; => 5 +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:1325 +(math.sqrt 16) ;; => 4.0 +(math.abs -5) ;; => 5 +(math.floor 3.7) ;; => 3.0 +(math.ceil 3.2) ;; => 4.0 +(math.round 3.5) ;; => 4.0 +(math.pow 2 10) ;; => 1024.0 +(math.log 2.718) ;; => ~1.0 +(math.sin 0) ;; => 0.0 +(math.cos 0) ;; => 1.0 +(math.min 3 7 1 5) ;; => 1 +(math.max 3 7 1 5) ;; => 7 +(math.clamp 15 0 10) ;; => 10 +(math.pi) ;; => 3.14159... +(math.e) ;; => 2.71828... +(math.random) ;; => random double in [0,1) +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:1345 +(collections.map (lambda (x) (* x x)) (list 1 2 3)) ;; => (1 4 9) +(collections.filter (lambda (x) (> x 2)) (list 1 2 3 4)) ;; => (3 4) +(collections.reduce (lambda (a b) (+ a b)) (list 1 2 3) 0) ;; => 6 +(collections.zip (list 1 2) (list "a" "b")) ;; => ((1 "a") (2 "b")) +(collections.flatten (list (list 1 2) (list 3 4))) ;; => (1 2 3 4) +(collections.sort (list 3 1 2)) ;; => (1 2 3) +(collections.reverse (list 1 2 3)) ;; => (3 2 1) +(collections.range 1 5) ;; => (1 2 3 4) +(collections.unique (list 1 2 2 3 3)) ;; => (1 2 3) +(collections.dict-keys (dict "a" 1 "b" 2)) ;; => ("a" "b") +(collections.dict-values (dict "a" 1 "b" 2)) ;; => (1 2) +(collections.dict-merge (dict "a" 1) (dict "b" 2)) ;; => {a: 1, b: 2} +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:1445 +;; WRONG — i is scoped to the let body; the outer i never changes +(set i 0) +(while (< i 5) + (let ((i (+ i 1))) + (print i))) ;; infinite loop! + +;; RIGHT — use set for loop-carried state +(set i 0) +(while (< i 5) + (begin + (print i) + (set i (+ i 1)))) +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:1466 +(state-set "counter" (str 42)) +(set raw (state-get "counter")) ;; raw is "42" (string) +(set n (int raw)) ;; n is 42 (integer) +(if (> n 40) "big" "small") ;; => "big" +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:1478 +(set g (range 3)) +(collect g) ;; => (0 1 2) +(collect g) ;; => () — already exhausted + +;; To reuse, wrap in a factory +(defun make-range () (range 3)) +(collect (make-range)) ;; => (0 1 2) +(collect (make-range)) ;; => (0 1 2) +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:1496 +;; DEADLOCK — nothing else is running to send a message +(msg-recv "work.queue") +@@@ script docs/STATE_EXEC_DEVELOPER_GUIDE.md:1526 +(set done false) +(for i (range 10) + (if done nil + (for j (range 10) + (if (= j 3) + (begin (set done true) (break nil)) + nil)))) +@@@ script docs/STREAMING.md:107 +; open a video stream scoped to THIS document; returns a handle dict +(set cam (stream-open (dict "id" "cam0" "codec" "rgba8" "w" 1280 "h" 720))) +(get-attr cam "ok") ; => #t +(get-attr cam "token") ; => the canonical token to hand a sink +(get-attr cam "seq_channel") ; => a channel that ticks the latest frame seq + +(stream-info cam) ; => { ok live id token seq_channel evt_channel published subscribers } +(stream-close cam) ; => { ok #t } (also closed at document teardown) +@@@ script docs/STREAMING.md:162 +(gl-bind-stream "screen" (get-attr cam "token")) ; => { ok #t node token } +@@@ script docs/STREAMING.md:499 +; init: or a Start button's on: program +(set cam (stream-open (dict "id" "cam0" "codec" "rgba8" "w" 1280 "h" 720))) +; (Phase 3b) begin capture from the default camera into the stream: +(stream-capture-camera (get-attr cam "token") 0) +; bind it onto the scene's "screen" node: +(gl-bind-stream "screen" (get-attr cam "token")) +@@@ script docs/roadmap/ARIADNE_CONFERENCE_ROADMAP.md:97 +; ── join: server endpoint + room id; returns a room handle; mode comes from the server ── +(set room (room-join "conf.example.org:7000" "standup" (dict "name" "Joe"))) + +; ── publish my camera + mic into the room ── +(set cam (stream-open (dict "id" "self/cam" "codec" "rgba8" "w" 1280 "h" 720))) +(stream-capture-camera (get-attr cam "token") 0) ; §4 capture verb +(room-publish room (dict "kind" "camera" "token" (get-attr cam "token"))) + +(set mic (stream-open (dict "id" "self/mic" "codec" "f32" "rate" 48000 "ch" 1))) +(stream-capture-audio (get-attr mic "token") (dict "device" 0)) +(room-publish room (dict "kind" "audio" "token" (get-attr mic "token"))) + +; ── UI: a self tile + one tile per remote participant (roster-driven) ── +; scene / widget tree uses `repeat` over (room-roster room); each tile binds a +; (room-subscribe room PID "camera") proxy token into a `stream_view`, and +; (stream-play-audio (room-subscribe room PID "audio")) plays their voice. + +; ── controls: pure-.ari actions, no C++ ── +; button "Mute" on: (room-set-flag room (dict "mic" (not (room-flag room "self" "mic")))) +; button "Camera" on: (room-set-flag room (dict "cam" ...)) +; button "Leave" on: (room-leave room) +@@@ script docs/roadmap/CVCGL-UI-DSL-ROADMAP.md:4452 +(state-set "demo.n" 256) ; plain state, set before it is referenced +(window "Controls" + (slider-int "Agents" :bind "demo.n" :lo 1 :hi 512) + (combo "Belief" :bind "demo.belief" :options (list "shared" "grouped" "private")) + (button "Reset" :on "reset")) +(scene + (geometry-node "mesh" :source "file://blocks.off" + :visible-when (= (state-get "demo.show") "1"))) +@@@ script docs/roadmap/DSL-GEOMETRY-VOLUME-ROADMAP.md:91 +;; Load a volume from disk into the state tree +(vol-read "mydata" "/path/to/brain.rawiv") + +;; Query properties +(set dims (vol-dims "mydata")) ;; => (256 256 256) +(set bbox (vol-bbox "mydata")) ;; => (0.0 0.0 0.0 1.0 1.0 1.0) + +;; Volume arithmetic (result stored at new path) +(vol-normalize "mydata" "normalized") +(vol-scale "normalized" 2.0 "scaled") + +;; Extract isosurface → geometry +(iso "scaled" 0.5 "surface" (dict "method" "fastcontouring")) + +;; Compute SDF from geometry → new volume +(sdf "surface" "sdf-vol" (dict "dim" (list 128 128 128))) + +;; Tetrahedralize +(tet "sdf-vol" 0.0 "tet-mesh") + +;; Write results +(vol-write "sdf-vol" "/path/to/output.rawiv") +(geom-write "tet-mesh" "/path/to/output.off") +@@@ script docs/roadmap/DSL-GEOMETRY-VOLUME-ROADMAP.md:288 +(iso "brain-vol" 128.0 "brain-surf" + (dict "method" "fastcontouring" + "improve" #t + "compute-normals" #t)) + +(sdf "brain-surf" "brain-sdf" + (dict "dim" (list 256 256 256) + "flip-normals" #f)) +@@@ script docs/roadmap/DSL-GEOMETRY-VOLUME-ROADMAP.md:307 +(defun process-brain (input-file output-dir) + (begin + ;; Load + (vol-read "raw" input-file) + (print (str "Loaded: " (vol-dims "raw"))) + + ;; Pre-process + (vol-normalize "raw" "norm") + (vol-clip "norm" 0.1 0.9 "clipped") + + ;; Extract isosurface + (iso "clipped" 0.5 "surface" + (dict "method" "fastcontouring" + "improve" #t + "compute-normals" #t)) + (print (str "Surface: " (geom-num-tris "surface") " tris")) + + ;; Tetrahedralize for FEM + (tet "clipped" 0.5 "tet-mesh" + (dict "improve" #t)) + (print (str "Tet mesh: " (geom-num-tets "tet-mesh") " tets")) + + ;; Write results + (geom-write "surface" (str output-dir "/brain-surface.off")) + (geom-write "tet-mesh" (str output-dir "/brain-tet.off")) + + ;; Return summary + (dict "tris" (geom-num-tris "surface") + "tets" (geom-num-tets "tet-mesh")))) + +(process-brain "/data/brain.rawiv" "/output") +@@@ script docs/roadmap/DSL-GEOMETRY-VOLUME-ROADMAP.md:343 +(defun process-all (file-list output-dir) + (for f file-list + (begin + (print (str "Processing: " f)) + (process-brain f output-dir)))) +@@@ script docs/roadmap/DSL-GEOMETRY-VOLUME-ROADMAP.md:353 +;; Coordinator splits work across nodes +(defun distribute-pipeline (files) + (for f files + (begin + (state-set (str "jobs." f ".status") "pending") + (state-set (str "jobs." f ".input") f) + (msg-send "work-queue" f)))) + +;; Worker process +(defun worker () + (while #t + (set job (msg-recv "work-queue")) + (state-set (str "jobs." job ".status") "running") + (process-brain + (state-get (str "jobs." job ".input")) + "/output") + (state-set (str "jobs." job ".status") "done"))) +@@@ script docs/roadmap/STATE_BINARY_STREAMING.md:220 +(define s (stream-open {:kind "video_raw" :codec "rgba8" :w 1920 :h 1080})) +(stream-info s) ; descriptor + stats dict from the state tree +(gl-bind-stream node s) ; HOST action wires the C++ consumer to a GeometryNode texture +(msg-recv "stream.evt") ; lifecycle events (small dicts via post_message) +@@@ script docs/roadmap/STATE_EXEC_PORTING_PLAN.md:1101 +;; Expire a cache entry after 60 seconds +(state-set "sensors.cache.temperature" (read-sensor)) +(state-expire "sensors.cache.temperature" 60) + +;; Check before reading +(if (state-is-expired "sensors.cache.temperature") + (begin + (state-sweep-expired "sensors.cache") + (refresh-cache)) + (state-get "sensors.cache.temperature")) + +;; Set absolute deadline +(state-expire-at "jobs.batch-42" "2025-06-15T00:00:00Z") +@@@ script docs/roadmap/STATE_EXEC_PORTING_PLAN.md:1355 +(let ((pid (fork))) + (if (= pid 0) + (begin + ;; child: do work + (state-set "results.child" (compute))) + (begin + ;; parent: pid is the child's PID + (state-set "results.parent" (str "spawned child " pid))))) +@@@ script docs/roadmap/STATE_EXEC_PORTING_PLAN.md:1383 +(let ((pid (fork))) + (if (= pid -1) + (print "fork failed: resource limit") + ...)) +@@@ script docs/roadmap/STATE_EXEC_PORTING_PLAN.md:1470 +;; Import a module into the local environment +(require "string") +(split "hello world" " ") ;; → ("hello" "world") + +;; Or call with fully qualified path +(stdlib.string.split "hello world" " ") + +;; Import specific symbols +(require "math" '(sqrt abs floor ceil)) +(sqrt 144) ;; → 12 + +;; List available modules +(stdlib-modules) ;; → ("string" "math" "io" "collections") + +;; List functions in a module +(stdlib-list "string") ;; → ("split" "join" "replace" ...) +@@@ script docs/roadmap/STATE_EXEC_PORTING_PLAN.md:1709 +(require "geometry") +(distance 0 0 3 4) ;; → 5.0 + +(require "util") +(repeatedly 5 (lambda (i) (print i))) ;; prints 0-4 +@@@ script docs/roadmap/STATE_EXEC_PORTING_PLAN.md:1763 + (let ((mesh (state-data-get "/scene/model/geometry"))) + (if (data? mesh) + (begin + (print (data-type mesh)) ;; e.g. "CVC::Geometry" + (state-data-set "/output/result" mesh)))) +@@@ script docs/roadmap/STATE_EXEC_PORTING_PLAN.md:1776 + (state-data-set "/scene/model/transformed" transformed-mesh) +@@@ script docs/roadmap/STATE_EXEC_PORTING_PLAN.md:1781 + (data-type mesh) ;; → "CVC::Geometry" or nil + (data? mesh) ;; → t + (data? 42) ;; → nil + (state-data-type "/path") ;; → type name of data() on that node +@@@ script docs/roadmap/STATE_EXEC_PORTING_PLAN.md:1853 +;; Without batching: handleStateChanged fires 3 times, possibly spawning +;; 3 handler threads. +(state-set "/scene/camera/x" 10) +(state-set "/scene/camera/y" 20) +(state-set "/scene/camera/z" 30) + +;; With batching: handleStateChanged fires once per unique child, after +;; all three writes complete. If the camera object batches internally, +;; this collapses to a single flush. +(state-batch + (state-set "/scene/camera/x" 10) + (state-set "/scene/camera/y" 20) + (state-set "/scene/camera/z" 30)) +@@@ script docs/roadmap/STATE_EXEC_PORTING_PLAN.md:1876 +(state-lock "/scene/camera" + (let ((x (state-get "/scene/camera/x")) + (y (state-get "/scene/camera/y"))) + ;; x and y are consistent — no other writer can interleave + (state-set "/scene/camera/distance" + (sqrt (+ (* x x) (* y y)))))) +@@@ script src/cvc-cli/cvc_cli_test.cpp:1093 +(+ 1 2 3) +@@@ script src/cvc-cli/cvc_cli_test.cpp:1099 +(* 7 6) +@@@ script src/cvc-cli/cvc_cli_test.cpp:1105 +(let ((x 10)) (+ x 5)) +@@@ script src/cvc-cli/cvc_cli_test.cpp:1111 +(let ((f (lambda (x y) (+ x y)))) (f 3 4)) +@@@ script src/cvc-cli/cvc_cli_test.cpp:1120 +(+ 100 200 300) + +@@@ script src/cvc-cli/cvc_cli_test.cpp:1206 +(let ((square (lambda (x) (* x x)))) (square 9)) +@@@ script src/cvc-cli/cvc_cli_test.cpp:1211 +(car (list 10 20 30)) +@@@ script src/cvc-cli/cvc_cli_test.cpp:1216 +(if (> 5 3) 99 0) +@@@ script src/cvc-cli/cvc_cli_test.cpp:1221 +(let ((a 10) (b 20)) (+ a b)) +@@@ script src/cvc-cli/cvc_cli_test.cpp:1228 +(begin 1 2 3 4 5 6 7 8 9 10) +@@@ script src/cvc-cli/cvc_cli_test.cpp:1238 +(let ((pi 3.14159) (r 5)) (* pi r r)) + +@@@ script src/cvc-cli/cvc_cli_test.cpp:1881 +(no children) +@@@ script src/cvc-cli/cvc_cli_test.cpp:1891 +(+ 1 (+ 2 (+ 3 (+ 4 (+ 5 6))))) +@@@ script src/cvc-cli/main.cpp:1554 +(no children) + +@@@ script src/cvc/ariadne/ariadne.cpp:1256 +(no image) +@@@ script src/cvc/ariadne/ftxui_backend.cpp:89 +(empty) +@@@ script src/cvc/tests/ariadne_loader_test.cpp:535 +(state-set "paused" "true") +@@@ script src/cvc/tests/ariadne_loader_test.cpp:2256 +(> (int (state-get "n")) 5) +@@@ script src/cvc/tests/ariadne_loader_test.cpp:2276 +(state-exists "ready") +@@@ script src/cvc/tests/ariadne_loader_test.cpp:2277 +(state-exists "busy") +@@@ script src/cvc/tests/ariadne_loader_test.cpp:2301 +(list "p" "q") +@@@ script src/cvc/tests/ariadne_loader_test.cpp:2402 +(state-exists "show") +@@@ script src/cvc/tests/ariadne_loader_test.cpp:3688 +(int (state-get "n")) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:80 +(closable) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:356 +(state-set "paused" (if (= (state-get "paused") "true") "false" "true")) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:376 +(begin (state-set "agents" "64") (state-set "speed" "1.0")) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:400 +(begin (msg-recv "async.done") (state-set "async.result" "resumed")) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:428 +(state-set "async.value" (msg-recv "vchan")) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:486 +(begin (nav-step-async) (state-set "nav.result" (msg-recv "nav.done"))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:511 +(state-set "tick.flag" "1") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:547 +(begin (state-set "input.last_key" (get-attr event "key")) (state-set "input.kind" (get-attr event "kind"))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:565 +(state-set "input.last_key" (get-attr event "key")) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:583 +(begin (state-set "ptr.kind" (get-attr event "kind")) (state-set "ptr.btn" (get-attr event "button"))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:606 +(state-set "widget.clicked" "yes") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:728 +(begin (state-set "ev.btn" (get-attr event "button")) (state-set "ev.has_x" (if (>= (get-attr event "x") 0) "yes" "no"))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:780 +(host-bump) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:959 +(state-set "agents" "128") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:960 +(no error) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1033 +(while true (+ 1 1)) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1077 +(state-set "n" "10") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1121 +(= (state-get "show") "1") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1150 +(int 2) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1351 +(state-exists "never") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1384 +(state-set "x" 1) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1399 +(state-set "mode" "on") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1401 +(= (state-get "mode") "on") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1414 +(while true 1) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1431 +(msg-recv "x") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1431 +(await 1) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1470 +(state-set "flag" "0") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1472 +(state-get "flag") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1474 +(state-get "never_set") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1490 +(foobar 1 2) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1541 +(is-string (str-concat "a" "b")) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1543 +(is-list (append (list 1) (list 2))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1545 +(apply and (list #t)) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1547 +(begin (print "x") #t) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1565 +(= (quote (1 2 3)) (quote (1 2 3))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1567 +(= (quote (1 2)) (quote (9 9))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1569 +(let ((xs (list 1 2 3))) (> (length xs) 2)) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1587 +(begin (defclass B (init (self) (while 1 1))) (B)) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1589 +(eval (quote #t)) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1591 +(begin (defmacro m () #t) #t) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1606 +(begin (set d (list 0 0)) (set i 0) (while (< i 30) (begin (set d (list d d)) (set i (+ i 1)))) (state-data-set "bomb" d)) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1614 +(is-null (state-data-get "bomb")) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1626 +(state-set "n" "3") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1647 +(state-set "lock" "1") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1649 +(state-exists "lock") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1686 +(state-set "n" "7") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1688 +(+ (int (state-get "n")) 1) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1689 +(state-get "n") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1775 +(list "shared" "grouped" "private") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1799 +(state-set "hint" "dynamic") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1803 +(state-get "hint") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1847 +(state-set "n" "2") +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1912 +(* 1e19 1e19) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1928 +(begin (defun f () #t) (f)) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:1930 +(f) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:2287 +(begin (set r (msg-recv (http-get-async "http://ex/x"))) (state-set "r.status" (get-attr r "status")) (state-set "r.ok" (get-attr r "ok")) (state-set "r.error" (get-attr r "error")) (state-data-set "r.body" (get-attr r "body"))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:2325 +(begin (set r (msg-recv (http-get-async "http://ex/down"))) (state-set "r.ok" (get-attr r "ok")) (state-set "r.error" (get-attr r "error"))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:2353 +(state-set "r.status" (get-attr (msg-recv (http-get-async "http://ex/slow")) "status")) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:2383 +(begin (set r (http-get "http://ex/x")) (state-set "r.status" (get-attr r "status")) (state-data-set "r.body" (get-attr r "body"))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:2414 +(state-set "r.status" (get-attr (await (http-get-async "http://ex/x")) "status")) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:2449 +(begin (set r (fetch "mem://x")) (state-set "r.ok" (get-attr r "ok")) (state-data-set "r.body" (get-attr r "body"))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:2481 +(state-data-set "r.body" (get-attr (await (fetch-async "mem://y")) "body")) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:2519 +(begin (set r (http-request "POST" "http://ex/api" (dict "headers" (dict "Authorization" "Bearer tok-123") "form" (dict "grip" 1 "risk" 2)))) (state-set "r.status" (get-attr r "status"))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:2560 +(begin (set r (http-get "http://ex/x" (dict "query" (dict "q" "hello world" "n" 2)))) (state-set "r.status" (get-attr r "status"))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:2592 +(begin (set r (http-request "PUT" "http://ex/raw" (dict "body" "raw-payload"))) (state-set "r.status" (get-attr r "status"))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:2625 +(begin (set g (await (http-get-async "http://ex/src"))) (set r (await (http-request-async "PUT" "http://ex/dst" (dict "body" (get-attr g "body") "headers" (dict "Authorization" "Bearer tok"))))) (state-set "r.status" (get-attr r "status"))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:2688 +(begin (state-set "r.t" (str (pool-double 21))) (state-set "r.a" (str (await (pool-double-async 10))))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:2754 +(begin (set h (stream-open (dict "id" "cam0" "w" 4 "h" 2))) (state-set "t.ok" (get-attr h "ok")) (state-set "t.token" (get-attr h "token")) (state-set "t.seq" (get-attr h "seq_channel")) (state-set "t.live" (get-attr (stream-info h) "live")) (state-set "t.closed" (get-attr (stream-close h) "ok")) (state-set "t.live2" (get-attr (stream-info h) "live"))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:2785 +(begin (set h (stream-open (dict "id" "cam0" "w" 4 "h" 2))) (state-set "t.token" (get-attr h "token")) (state-set "t.seq" (get-attr h "seq_channel"))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:2814 +(begin (set h (stream-open (dict "id" "cam0" "w" 4 "h" 2))) (state-set "t.ok" (get-attr h "ok"))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:2843 +(begin (set a (stream-open (dict "id" "c0" "w" 4 "h" 2 "expected_subscribers" -1))) (state-set "t.a" (get-attr a "ok")) (set b (stream-open (dict "id" "c1" "w" 999999 "h" 2))) (state-set "t.b" (get-attr b "ok"))) +@@@ script src/cvc/tests/ariadne_runtime_test.cpp:2869 +(begin (set h (stream-open (dict "id" "cam0" "w" 4 "h" 2))) (state-set "t.ok" (get-attr h "ok")) (state-set "t.err" (get-attr h "error"))) +@@@ script src/cvc/tests/async_lane_test.cpp:498 +(+ (lane-double 21) (await (lane-double-async 10))) +@@@ script src/cvc/tests/async_lane_test.cpp:525 +(while t (apply-built (msg-recv "app#built.tracks"))) +@@@ script src/cvc/tests/async_lane_test.cpp:526 +(while t (apply-plain (msg-recv "built"))) +@@@ script src/cvc/tests/async_lane_test.cpp:621 +(while t (begin (tick) (await 0))) +@@@ script src/cvc/tests/async_lane_test.cpp:652 +(while t (sleep (msg-recv (sim-launch)))) +@@@ script src/cvc/tests/state_exec_advanced_integration_test.cpp:125 + + (begin + (set big "") + (set i 0) + (while (< i 100) + (begin + (set big (str-concat big "0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789")) + (set i (+ i 1)))) + (state-set "data.large_string" big) + (length big)) + +@@@ script src/cvc/tests/state_exec_advanced_integration_test.cpp:151 + + (begin + (state-data-set "data.typed_obj" (list 1 2 3 4 5)) + (set obj (state-data-get "data.typed_obj")) + (is-null obj)) + +@@@ script src/cvc/tests/state_exec_advanced_integration_test.cpp:168 + + (begin + (set i 0) + (while (< i 50) + (begin + (state-set "data.counter" (str i)) + (set i (+ i 1)))) + (state-get "data.counter")) + +@@@ script src/cvc/tests/state_exec_advanced_integration_test.cpp:803 + + (begin + (state-set "queue.item.0" "apple") + (state-set "queue.item.1" "banana") + (state-set "queue.item.2" "cherry") + (state-set "queue.count" "3") + (msg-send "queue.ready" "items-available") + "producer-done") + +@@@ script src/cvc/tests/state_exec_advanced_integration_test.cpp:813 + + (begin + (set result (list)) + (set i 0) + (while (< i 3) + (begin + (set key (str-concat "queue.item." (str i))) + (set val (state-get key)) + (if (!= val nil) + (set i (+ i 1)) + nil))) + (state-set "consumed.total" (str i)) + i) + +@@@ script src/cvc/tests/state_exec_advanced_integration_test.cpp:843 + + (begin + (state-set "partial.0" (str (+ 1 2 3))) + "map-0-done") + +@@@ script src/cvc/tests/state_exec_advanced_integration_test.cpp:849 + + (begin + (state-set "partial.1" (str (+ 4 5 6))) + "map-1-done") + +@@@ script src/cvc/tests/state_exec_advanced_integration_test.cpp:855 + + (begin + (state-set "partial.2" (str (+ 7 8 9))) + "map-2-done") + +@@@ script src/cvc/tests/state_exec_advanced_integration_test.cpp:861 + + (begin + (set s0 (state-get "partial.0")) + (set s1 (state-get "partial.1")) + (set s2 (state-get "partial.2")) + (state-set "result.total" + (str (+ (if (= s0 nil) 0 6) + (if (= s1 nil) 0 15) + (if (= s2 nil) 0 24)))) + 45) + +@@@ script src/cvc/tests/state_exec_advanced_integration_test.cpp:928 +(begin (spawn "(* 10 10)" "w0") (spawn "(* 20 20)" "w1") (spawn "(* 30 30)" "w2") 3) +@@@ script src/cvc/tests/state_exec_advanced_integration_test.cpp:947 + + (begin + (set total 0) + (set i 0) + (while (< i 5) + (begin + (set total (+ total (* i i))) + (state-set "progress.current" (str i)) + (state-set "progress.total" (str total)) + (set i (+ i 1)))) + total) + +@@@ script src/cvc/tests/state_exec_advanced_integration_test.cpp:977 + + (begin + (state-set "document.header.title" "My Report") + (state-set "document.header.author" "Alice") + (state-set "document.header.date" "2026-05-24") + (msg-send "document.events" "header-complete") + "header-done") + +@@@ script src/cvc/tests/state_exec_advanced_integration_test.cpp:986 + + (begin + (state-set "document.body.section1" "Introduction text...") + (state-set "document.body.section2" "Methods text...") + (state-set "document.body.section3" "Results text...") + (msg-send "document.events" "body-complete") + "body-done") + +@@@ script src/cvc/tests/state_exec_advanced_integration_test.cpp:1013 + + (begin + ;; Compute Fibonacci(12) = 144 + (defun fib (n) + (if (<= n 1) n + (+ (fib (- n 1)) (fib (- n 2))))) + (set result (fib 12)) + + ;; Write result to state tree + (state-set "computation.fib12.result" (str result)) + (state-set "computation.fib12.status" "complete") + + ;; Notify via messaging + (msg-send "computation.notifications" "fib12-done") + + result) + +@@@ script src/cvc/tests/state_exec_advanced_integration_test.cpp:1094 +(begin (while true 1)) +@@@ script src/cvc/tests/state_exec_advanced_integration_test.cpp:1099 +(+ 1 2) +@@@ script src/cvc/tests/state_exec_advanced_integration_test.cpp:1110 +(+ 3 4) +@@@ script src/cvc/tests/state_exec_advanced_integration_test.cpp:1111 +(+ 5 6) +@@@ script src/cvc/tests/state_exec_async_test.cpp:138 +(- 10 3) +@@@ script src/cvc/tests/state_exec_async_test.cpp:140 +(* 4 5) +@@@ script src/cvc/tests/state_exec_async_test.cpp:144 +(/ 10 4) +@@@ script src/cvc/tests/state_exec_async_test.cpp:147 +(+ (* 2 3) (- 10 4)) +@@@ script src/cvc/tests/state_exec_async_test.cpp:152 +(< 1 2) +@@@ script src/cvc/tests/state_exec_async_test.cpp:153 +(< 2 1) +@@@ script src/cvc/tests/state_exec_async_test.cpp:157 +(= 5 5) +@@@ script src/cvc/tests/state_exec_async_test.cpp:158 +(= 5 6) +@@@ script src/cvc/tests/state_exec_async_test.cpp:163 +(if t 1 2) +@@@ script src/cvc/tests/state_exec_async_test.cpp:165 +(if nil 1 2) +@@@ script src/cvc/tests/state_exec_async_test.cpp:167 +(begin 1 2 3) +@@@ script src/cvc/tests/state_exec_async_test.cpp:169 +(begin (set x 42) x) +@@@ script src/cvc/tests/state_exec_async_test.cpp:172 +(begin (set i 0) (set sum 0) (while (< i 5) (begin (set sum (+ sum i)) (set i (+ i 1)))) sum) +@@@ script src/cvc/tests/state_exec_async_test.cpp:179 +(begin (set sum 0) (for x (list 1 2 3 4 5) (set sum (+ sum x))) sum) +@@@ script src/cvc/tests/state_exec_async_test.cpp:188 +(begin (set sq (lambda (x) (* x x))) (sq 5)) +@@@ script src/cvc/tests/state_exec_async_test.cpp:192 +(begin (set make-adder (lambda (n) (lambda (x) (+ x n)))) (set add5 (make-adder 5)) (add5 10)) +@@@ script src/cvc/tests/state_exec_async_test.cpp:200 +(begin (defun fact (n) (if (< n 2) 1 (* n (fact (- n 1))))) (fact 10)) +@@@ script src/cvc/tests/state_exec_async_test.cpp:209 +(let ((x 10) (y 20)) (+ x y)) +@@@ script src/cvc/tests/state_exec_async_test.cpp:214 +(begin (set f (lambda () (begin (return 42) 99))) (f)) +@@@ script src/cvc/tests/state_exec_async_test.cpp:220 +(quote (1 2 3)) +@@@ script src/cvc/tests/state_exec_async_test.cpp:229 +(begin (defmacro double-it (x) (* x 2)) (double-it 21)) +@@@ script src/cvc/tests/state_exec_async_test.cpp:237 +(eval (quote (+ 1 2))) +@@@ script src/cvc/tests/state_exec_async_test.cpp:242 +(str-concat "hello" " " "world") +@@@ script src/cvc/tests/state_exec_async_test.cpp:248 +(car (list 1 2 3)) +@@@ script src/cvc/tests/state_exec_async_test.cpp:249 +(car (cdr (list 1 2 3))) +@@@ script src/cvc/tests/state_exec_async_test.cpp:252 +(length (list 1 2 3 4 5)) +@@@ script src/cvc/tests/state_exec_async_test.cpp:257 +(begin (defun fib (n) (if (< n 2) n (+ (fib (- n 1)) (fib (- n 2))))) (fib 15)) +@@@ script src/cvc/tests/state_exec_async_test.cpp:268 +(await (+ 1 2)) +@@@ script src/cvc/tests/state_exec_async_test.cpp:275 +(+ 10 20) +@@@ script src/cvc/tests/state_exec_async_test.cpp:395 +(+ 10 20 30) +@@@ script src/cvc/tests/state_exec_async_test.cpp:449 +(begin (set i 0) (set s 0) (while (< i 5) (begin (set s (+ s i)) (set i (+ i 1)))) s) +@@@ script src/cvc/tests/state_exec_async_test.cpp:469 +(begin (set x 10) (set y 20) (+ x y)) +@@@ script src/cvc/tests/state_exec_async_test.cpp:470 +(begin (defun fact (n) (if (< n 2) 1 (* n (fact (- n 1))))) (fact 10)) +@@@ script src/cvc/tests/state_exec_async_test.cpp:471 +(begin (set sum 0) (for x (list 1 2 3 4 5) (set sum (+ sum x))) sum) +@@@ script src/cvc/tests/state_exec_async_test.cpp:472 +(let ((a 5) (b 10)) (* a b)) +@@@ script src/cvc/tests/state_exec_async_test.cpp:498 +(+ 40 2) +@@@ script src/cvc/tests/state_exec_async_test.cpp:509 +(begin 1 2 3 4 5) +@@@ script src/cvc/tests/state_exec_async_test.cpp:537 +(begin 1 2 3 4 5 6 7 8) +@@@ script src/cvc/tests/state_exec_async_test.cpp:570 +(nap) +@@@ script src/cvc/tests/state_exec_async_test.cpp:645 +(begin (set r (await 42)) r) +@@@ script src/cvc/tests/state_exec_async_test.cpp:670 +(begin (set s 0) (set i 0) (while (< i 200) (begin (set s (+ s i)) (set i (+ i 1)))) s) +@@@ script src/cvc/tests/state_exec_async_test.cpp:738 +(+ 2 3) +@@@ script src/cvc/tests/state_exec_bench_test.cpp:67 +(begin (defun fib (n) (if (<= n 1) n (+ (fib (- n 1)) (fib (- n 2))))) (fib 20)) +@@@ script src/cvc/tests/state_exec_bench_test.cpp:90 +(begin (set sum 0) (set i 0) (while (< i 1000) (begin (set sum (+ sum i)) (set i (+ i 1)))) sum) +@@@ script src/cvc/tests/state_exec_bench_test.cpp:116 +(begin (defun fib (n) (if (<= n 1) n (+ (fib (- n 1)) (fib (- n 2))))) (fib 15)) +@@@ script src/cvc/tests/state_exec_bench_test.cpp:141 +(begin (set x 0) (set i 0) (while (< i 100) (begin (set x (+ x i)) (set i (+ i 1)))) x) +@@@ script src/cvc/tests/state_exec_bench_test.cpp:179 +(begin (set lst (list)) (set i 0) (while (< i 500) (begin (set lst (append lst (list i))) (set i (+ i 1)))) lst) +@@@ script src/cvc/tests/state_exec_bench_test.cpp:253 +(begin (defun even? (n) (if (= n 0) #t (odd? (- n 1)))) (defun odd? (n) (if (= n 0) #f (even? (- n 1)))) (even? 200)) +@@@ script src/cvc/tests/state_exec_codec_test.cpp:297 +(begin (set sum 0) (set i 0) (while (< i 100) (begin (set sum (+ sum i)) (set i (+ i 1)))) sum) +@@@ script src/cvc/tests/state_exec_codec_test.cpp:320 +(+ 1 2 3 4 5 6 7 8 9 10) +@@@ script src/cvc/tests/state_exec_codec_test.cpp:427 +(/ 10 3) +@@@ script src/cvc/tests/state_exec_codec_test.cpp:495 +(let ((x 10)) (let ((y 20)) (+ x y))) +@@@ script src/cvc/tests/state_exec_codec_test.cpp:527 +(cdr (list 1 2 3)) +@@@ script src/cvc/tests/state_exec_codec_test.cpp:530 +(length (list 1 2 3 4)) +@@@ script src/cvc/tests/state_exec_codec_test.cpp:535 +(begin (set d (dict)) (set-attr d "key" 42) (get-attr d "key")) +@@@ script src/cvc/tests/state_exec_codec_test.cpp:544 +(begin (defun fib (n) (if (< n 2) n (+ (fib (- n 1)) (fib (- n 2))))) (fib 10)) +@@@ script src/cvc/tests/state_exec_coordinator_test.cpp:469 +(begin (+ 1 2) (+ 3 4) (+ 5 6)) +@@@ script src/cvc/tests/state_exec_coordinator_test.cpp:653 +(begin (+ 1 2) (+ 3 4)) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:41 +(- 10 3 2) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:43 +(* 3 4) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:47 +(% 10 3) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:49 +(+ (* 3 4) (- 10 5)) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:59 +(= 42 42) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:60 +(= 1 2) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:69 +(if nil 1) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:73 +(begin) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:78 +(quote (+ 1 2)) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:90 +(begin (set sum 0) (for x (list 1 2 3 4) (set sum (+ sum x))) sum) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:99 +(begin (set add1 (lambda (x) (+ x 1))) (add1 10)) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:114 +(begin (set sum-all (lambda (&rest args) (apply + args))) (sum-all 1 2 3)) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:123 +(begin (defun double (x) (* x 2)) (double 21)) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:130 +(begin (defun fact (n) (if (<= n 1) 1 (* n (fact (- n 1))))) (fact 5)) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:149 +(begin (defun early (x) (if (< x 0) (return -1) x)) (early -5)) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:172 +(root) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:188 +(car (cdr (list 10 20 30))) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:189 +(car (cons 5 (list 10 20))) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:192 +(length (list 1 2 3)) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:197 +(begin (set obj (dict "x" 10 "y" 20)) (set-attr obj "x" 99) (get-attr obj "x")) +@@@ script src/cvc/tests/state_exec_evaluator_test.cpp:215 +(begin (defun fib (n) (if (<= n 1) n (+ (fib (- n 1)) (fib (- n 2))))) (fib 10)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:49 +(begin (set m (lambda (self) 99)) (set superd (dict "m" m)) (set clsd (dict "__super__" superd)) (set obj (dict "__class__" clsd)) (super obj m)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:57 +(begin (set superd (dict "mk" list)) (set clsd (dict "__super__" superd)) (set obj (dict "__class__" clsd)) (length (super obj mk 4 5))) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:64 +(begin (set m (lambda (self a) (+ a 90))) (set superd (dict "m" m)) (set clsd (dict "__super__" superd)) (set obj (dict "__class__" clsd)) (super obj m 9)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:72 +(begin (set superd2 (dict "mk" list)) (set superd1 (dict "__super__" superd2)) (set clsd (dict "__super__" superd1)) (set obj (dict "__class__" clsd)) (length (super obj mk 1))) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:79 +(begin (set superd (dict "m" 5)) (set clsd (dict "__super__" superd)) (set obj (dict "__class__" clsd)) (super obj m)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:101 +(while t 1) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:120 +(+ 2 2) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:145 +(set a 1) (set b 2) (+ a b) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:150 +(1 2) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:151 +(no-such-fn 1) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:152 +(begin (set v 7) (v 1)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:153 +(if t) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:154 +(while t) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:155 +(for x (list 1)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:156 +(for 1 (list 1) 2) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:157 +(for x 5 2) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:158 +(set a) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:159 +(set 1 2) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:160 +(quote 1 2) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:161 +(lambda (x)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:162 +(lambda (1) 2) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:163 +(lambda 5 2) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:164 +(let ((a 1))) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:165 +(let 5 2) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:166 +(let ((a)) 1) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:167 +(let ((1 2)) 3) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:168 +(defun f (x)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:169 +(defun 5 (x) 1) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:170 +(defmacro m (x)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:171 +(defmacro 5 (x) 1) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:172 +(defmacro m (1) 2) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:173 +(eval) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:174 +(super 1) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:175 +(defclass) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:176 +(defclass 5) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:180 +(begin (defun f () (begin (return) 9)) (f)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:184 +(begin (set f (lambda (x) x)) (f 1 2)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:185 +(begin (set f (lambda (&rest) 1)) (f 5)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:189 +(begin (set f (lambda () 42)) (f)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:192 +(lambda nil 42) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:196 +(begin (defun f () (return 5)) (+ (f) 1)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:200 +(begin (defmacro pick (a b) b) (pick 1)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:204 +(begin (defmacro m2 (x) (set y x) (+ y 1)) (m2 5)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:208 +(get-attr d "name") +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:209 +(send d "speak") +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:210 +(send d "add" 1 2) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:211 +(send d "finalize") +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:215 +(begin (defclass Empty) (Empty 42)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:220 +(begin (defclass P1 (m (self) 7)) (defclass C1 (P1) (n (self) 8)) (send (C1) "n")) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:226 +(super d speak) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:227 +(super d "speak") +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:228 +(super d add 3 4) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:237 +(super (dict "k" 1) m) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:238 +(super (Animal "a") speak) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:239 +(super d fly) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:243 +(begin (defclass KX (m (self) 7) 5 (5 1)) (send (KX) "m")) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:249 +(begin (set Q (lambda (x) x)) (defclass CQ Q (n (self) 4)) (send (CQ) "n")) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:252 +(begin (set Q2 (lambda (x) x)) (defclass CR (Q2) (n (self) 5)) (send (CR) "n")) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:286 +(set a 4) (set b 5) (+ a b) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:295 +(begin (set i 0) (while (< i 20) (set i (+ i 1))) i) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:346 +(return) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:347 +(yield) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:353 +(defun f (1) 1) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:365 +(begin (set v 7) (v)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:369 +(begin (set f (lambda (&rest xs) (length xs))) (f 1 2 3)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:378 +(begin (set f (lambda (x) (set y (* x 2)) (+ y 1))) (f 4)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:381 +(let () 42) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:384 +(begin (defun f (&rest xs) (length xs)) (f 1 2)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:392 +(for x (range 0) x) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:414 +(begin (defclass P2 (who (self) "parent")) (defclass C2 (P2)) (send (C2) "who")) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:421 +(begin (defclass V (m (self &rest more) (length more))) (send (V) "m" 1 2)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:424 +(begin (defclass M (m (self) (set y 4) (+ y 2))) (send (M) "m")) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:428 +(begin (defclass K (m (self) 7) (5 1)) (send (K) "m")) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:441 +(defclass B (m (self 5) 1)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:473 +(-) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:474 +(*) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:475 +(/) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:476 +(- 2.5) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:477 +(- 5.5 1 0.5) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:478 +(/ 4) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:479 +(+ 0.5 (list 1 2)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:480 +(+ (list 1.5) 1) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:481 +(* (list 2 3)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:482 +(* 2.0 (list 3 4)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:483 +(% 7.5 2) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:487 +(% 1 0) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:488 +(% 1.0 0.0) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:489 +(/ 1 0) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:490 +(< nil 1) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:491 +(car) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:492 +(send 5) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:496 +(int (list 1)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:497 +(float (list 1)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:498 +(str-concat "a" 1) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:502 +(car (list)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:503 +(length (cdr (list))) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:504 +(car 5) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:505 +(nth (list 1) 5) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:506 +(nth (list 1) "x") +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:507 +(set-nth (list 1) 5 0) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:508 +(del-nth (list 1) 5) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:509 +(begin (set l (list 1 2 3)) (del-nth l 1)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:510 +(length (dict "a" 1)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:511 +(length t) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:515 +(slice (list 1)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:516 +(length (slice (list 1 2 3) -1 2)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:517 +(length (slice (list 1 2) 0 -1)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:518 +(length (slice (list 1 2) 5 6)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:519 +(length (slice (list 1 2) 0 9)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:520 +(length (slice (list 1 2) 1 1)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:521 +(length (slice (list 1 2 3) 1)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:525 +(dict "a") +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:526 +(get-attr (dict 1 "x") "1") +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:527 +(get-attr (dict) "k") +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:528 +(get-attr 5 "k") +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:529 +(get-attr (dict) 5) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:530 +(del-attr (dict) "k") +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:531 +(begin (set d (dict "k" 3)) (del-attr d "k")) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:535 +(send (dict "a" 1) "m") +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:536 +(begin (set o (dict "__class__" (dict "m" (lambda (s) 1)))) (send o "m")) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:539 +(begin (set o (dict "__class__" (dict "m" 5))) (send o "m")) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:541 +(begin (set o (dict "__class__" (dict "x" 1))) (send o "m")) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:546 +(begin (set superd (dict "mk" list)) (set clsd (dict "x" 1 "__super__" superd)) (set obj (dict "__class__" clsd)) (length (send obj "mk" 4 5))) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:555 +(apply 5 (list 1)) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:556 +(append 5 1) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:559 +(print 1 "a" 2.5) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:562 +(generator 5) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:563 +(next 5) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:564 +(gen-done? 5) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:565 +(collect 5) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:566 +(range 1 2 3 4) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:567 +(range "a") +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:568 +(range "a" 2) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:569 +(range 1 "b") +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:570 +(range 1 5 "c") +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:571 +(range 1 5 0) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:576 +(collect (generator (lambda () (begin (yield 1) (yield 2))))) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:585 +(and) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:586 +(and 1 nil 2) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:587 +(and 1 2) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:588 +(or nil nil) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:589 +(or nil 7) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:590 +(not nil) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:629 +(set c 1) (+ c 1) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:756 +(+ 1 1) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:770 +(* 2 3) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:771 +(* 2 4) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:772 +(* 2 5) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:781 +(begin (set i 0) (while (< i 50) (set i (+ i 1))) i) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:795 +(begin (set i 0) (while (< i 40) (set i (+ i 1)))) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:808 +(set-attr box "sig" __signal__) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:809 +(begin (set i 0) (while (< i 6) (set i (+ i 1))) (get-attr box "sig")) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:820 +(begin (set i 0) (while (< i 4) (set i (+ i 1))) i) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:857 +(begin (set i 0) (while (< i 6) (set i (+ i 1))) (get-attr box "path")) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:863 +(set-attr box "path" p) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:878 +(begin (set i 0) (while (< i 6) (set i (+ i 1))) (get-attr box "n")) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:883 +(set-attr box "n" 1) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:883 +(set-attr box "n" 2) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:927 +(begin (set i 0) (while (< i 8) (set i (+ i 1))) (get-attr box "path")) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:954 +(begin (set i 0) (while (< i 100) (set i (+ i 1)))) +@@@ script src/cvc/tests/state_exec_gaps_test.cpp:987 +(+ 20 22) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:95 +(spawn "42" "child") +@@@ script src/cvc/tests/state_exec_integration_test.cpp:152 +(self-uid) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:650 + + (begin + (state-set "mailbox.test" "ready") + (msg-send "mailbox.test" "hello-world")) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:664 + + (begin + (msg-send "test.path" "msg1") + (msg-send "test.path" "msg2") + (msg-send "test.path" "msg3") + (message-count)) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:687 + + (begin + (msg-send "test" "1") + (msg-send "test" "2") + (msg-send "test" "3") + (msg-send "test" "4")) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:706 + + (begin + (msg-send "test" "hello world!!") + (msg-send "test" "this exceeds the byte cap") + "done") + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:725 + + (begin + (state-set "shared.item.0" "100") + (state-set "shared.item.1" "200") + (state-set "shared.item.2" "300") + (state-set "shared.count" "3") + "done") + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:737 + + (begin + (state-get "shared.count")) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:758 +(begin (state-set "parent.uid" (self-uid)) (set child-pid (spawn "(+ 1 2)" "child-worker")) child-pid) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:774 + + (begin + (set result (msg-send "events.log" "{\"level\":\"info\"}" "application/json")) + (get-attr result "status")) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:802 + + (begin + (msg-send "console.stdout" "Hello from DSL!") + (msg-send "console.stdout" "Computing...") + (set result (* 6 7)) + (msg-send "console.stdout" (str-concat "Result: " (str result))) + (msg-send "console.stderr" "Warning: example only") + result) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:832 + + (begin + (set before (msg-pending "q.test")) + (msg-send "q.test" "a") + (msg-send "q.test" "b") + (set after (msg-pending "q.test")) + (list before after)) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:855 + + (begin + (msg-send "drain.test" "first") + (msg-send "drain.test" "second") + (msg-send "drain.test" "third") + "sent") + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:873 + + (begin + (set m1 (msg-recv "drain.test")) + (set m2 (msg-recv "drain.test")) + (set m3 (msg-recv "drain.test")) + (list (get-attr m1 "status") + (get-attr m2 "status") + (get-attr m3 "status"))) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:959 +(+ 100 200) +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1028 + + (begin + (defun fib (n) + (if (<= n 1) n + (+ (fib (- n 1)) (fib (- n 2))))) + (fib 10)) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1036 + + (begin + (defun fact (n) + (if (<= n 1) 1 + (* n (fact (- n 1))))) + (fact 8)) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1056 + + (begin + (set procs (ps)) + (length procs)) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1071 + + (begin + (set info (inspect (self-pid))) + (get-attr info "pid")) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1085 + + (begin + (set i 0) + (set sum 0) + (while (< i 10) + (begin + (set sum (+ sum i)) + (set i (+ i 1)))) + sum) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1101 + + (begin + (set total 0) + (for x (list 10 20 30 40) + (set total (+ total x))) + total) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1114 + + (begin + (set double (lambda (x) (* x 2))) + (double 21)) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1125 + + (let ((x 10) (y 20)) + (+ x y)) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1135 + + (begin + (set d (dict "name" "alice" "age" 30)) + (get-attr d "name")) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1157 + + (begin + (set stats (scheduler-stats)) + (get-attr stats "total")) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1212 + + (begin + (state-set "data.x" "42") + (state-get "data.x")) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1353 + + (begin + (set wid (state-watch "events.x" (lambda (p v) nil))) + (state-unwatch wid)) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1367 + + (begin + (state-watch "trigger.value" + (lambda (path val) + (state-set "got-event" "yes"))) + ;; Trigger the watch + (state-set "trigger.value" "hello") + ;; Spin briefly — the handler fires at the next step boundary + (set tries 0) + (while (and (not (state-exists "got-event")) (< tries 100)) + (set tries (+ tries 1))) + (state-get "got-event")) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1388 + + (begin + (state-watch "notify.path" + (lambda (path val) + (state-set "received-path" path))) + (state-set "notify.path" "data") + (set tries 0) + (while (and (not (state-exists "received-path")) (< tries 100)) + (set tries (+ tries 1))) + (state-get "received-path")) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1417 + + (begin + ;; Consumer: watch for new items, append each to a log + (state-set "consumer.log" "") + (state-watch "queue.item" + (lambda (path val) + (state-set "consumer.log" + (str-concat (state-get "consumer.log") (state-get path) ",")))) + + ;; Produce 3 items — each triggers the handler reactively + (state-set "queue.item" "apple") + (state-set "queue.item" "banana") + (state-set "queue.item" "cherry") + + ;; Spin to let all handlers dispatch + (set tries 0) + (while (and (not (= (state-get "consumer.log") "apple,banana,cherry,")) + (< tries 300)) + (set tries (+ tries 1))) + + (state-get "consumer.log")) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1451 + + (begin + (state-set "stats.ticks" "") + (state-watch "sensor.reading" + (lambda (path val) + (state-set "stats.ticks" + (str-concat (state-get "stats.ticks") "|")))) + + ;; 5 sensor readings + (state-set "sensor.reading" "72") + (state-set "sensor.reading" "73") + (state-set "sensor.reading" "71") + (state-set "sensor.reading" "74") + (state-set "sensor.reading" "70") + + (set tries 0) + (while (and (not (= (state-get "stats.ticks") "|||||")) + (< tries 500)) + (set tries (+ tries 1))) + + (state-get "stats.ticks")) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1487 + + (begin + ;; Stage 1: prefix the input value + (state-watch "pipeline.input" + (lambda (path val) + (state-set "pipeline.stage1" + (str-concat "processed:" (state-get path))))) + + ;; Stage 2: wrap stage1 output + (state-watch "pipeline.stage1" + (lambda (path val) + (state-set "pipeline.stage2" + (str-concat "[" (state-get path) "]")))) + + ;; Kick off the pipeline + (state-set "pipeline.input" "hello") + + ;; Wait for full cascade + (set tries 0) + (while (and (not (state-exists "pipeline.stage2")) + (< tries 500)) + (set tries (+ tries 1))) + + ;; input="hello" -> stage1="processed:hello" -> stage2="[processed:hello]" + (state-get "pipeline.stage2")) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1525 + + (begin + (state-set "sampler.log" "") + + ;; Store the watch ID so the handler can unsubscribe + (set wid + (state-watch "data.stream" + (lambda (path val) + (begin + (set current-val (state-get path)) + (set current-log (state-get "sampler.log")) + (state-set "sampler.log" + (str-concat current-log current-val ",")) + ;; After seeing 2 items (log has 2 commas), unsubscribe + (set updated (state-get "sampler.log")) + (if (= updated (str-concat current-val ",")) + nil + (state-unwatch wid)))))) + + ;; Produce 4 events — only the first 2 should be processed + (state-set "data.stream" "alpha") + (state-set "data.stream" "beta") + (state-set "data.stream" "gamma") + (state-set "data.stream" "delta") + + ;; Wait for handler to finish + (set tries 0) + (while (and (= (state-get "sampler.log") "") + (< tries 300)) + (set tries (+ tries 1))) + + ;; Small extra wait so second event can complete + (set tries 0) + (while (< tries 100) (set tries (+ tries 1))) + + (state-get "sampler.log")) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1577 + + (begin + (state-watch "jobs.request" + (lambda (path val) + (state-set "jobs.result" + (str-concat "DONE:" (state-get path))))) + ;; Stay alive to process the request + (set i 0) + (while (< i 500) (set i (+ i 1))) + "worker-done") + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1590 + + (begin + (state-set "jobs.request" "task-42") + (set tries 0) + (while (and (not (state-exists "jobs.result")) + (< tries 500)) + (set tries (+ tries 1))) + (state-get "jobs.result")) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1650 + + (begin + (let ((i 0)) + (while (< i 5) + (begin + (state-set (str-concat "queue.item." (str i)) (str (* i i))) + (set i (+ i 1))))) + (state-set "queue.count" "5") + "produce-done") + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1664 + + (begin + ;; Busy-wait for count + (set tries 0) + (while (and (not (state-exists "queue.count")) (< tries 1000)) + (set tries (+ tries 1))) + ;; Read all items using a range generator + (let ((total 0)) + (for i (range 5) + (set total (+ total 1))) + total)) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1691 + + (begin + (for i (range 10) + (state-set (str-concat "batch." (str i)) (str (* i 10)))) + (state-set "batch.done" "true") + 10) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1703 + + (begin + ;; Wait for producer to signal completion + (set tries 0) + (while (and (not (state-exists "batch.done")) (< tries 2000)) + (set tries (+ tries 1))) + ;; Collect the sum using range generator + (let ((sum 0)) + (for i (range 10) + (set sum (+ sum (* i 10)))) + sum)) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1731 + + (begin + (for i (range 1 6) + (state-set (str-concat "sq." (str i)) (str (* i i)))) + (state-set "sq.ready" "5") + "squares-done") + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1743 + + (begin + ;; Wait for data + (set tries 0) + (while (and (not (state-exists "sq.ready")) (< tries 2000)) + (set tries (+ tries 1))) + ;; Generator that yields doubled square values + (let ((g (generator (lambda () + (for i (range 1 6) + (let ((val (state-get (str-concat "sq." (str i))))) + (yield (* 2 1))))))) + (count 0)) + (for x g + (set count (+ count 1))) + count)) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1797 + + (begin + ;; Wait for all 4 producers + (for p (range 4) + (let ((key (str-concat "data.producer." (str-concat (str p) ".done")))) + (set tries 0) + (while (and (not (state-exists key)) (< tries 5000)) + (set tries (+ tries 1))))) + ;; Count all 12 values using a range generator + (let ((total 0)) + (for i (range 12) + (set total (+ total 1))) + total)) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1828 + + (begin + ;; Source: range generator + (let ((squares (generator (lambda () + (for i (range 1 8) + (yield (* i i))))))) + ;; Consume and sum + (let ((total 0)) + (for v squares + (set total (+ total v))) + total))) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1851 + + (begin + (let ((src (generator (lambda () + (yield 10) + (yield 20) + (yield 30)))) + (doubled (generator (lambda () + (for x src + (yield (* x 2))))))) + (collect doubled))) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1894 + + (begin + ;; Wait for all 3 status keys + (set tries 0) + (while (and (not (state-exists "status.2")) (< tries 5000)) + (set tries (+ tries 1))) + ;; Generator-based scan + (let ((results (list))) + (for i (range 3) + (let ((val (state-get (str-concat "status." (str i))))) + (append results val))) + (length results))) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1921 + + (begin + (let ((evens (generator (lambda () + (for i (range 0 10) + (if (= (% i 2) 0) + (yield i))))))) + (collect evens))) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1946 + + (begin + (let ((counter (generator (lambda () + (let ((n 0)) + (while (< n 1000) + (begin (yield n) + (set n (+ n 1)))))))) + (taken (list))) + (for x counter + (if (>= (length taken) 4) + (break nil) + (append taken x))) + taken)) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:1976 + + (begin + (let ((keys (generator (lambda () + (yield "name") + (yield "age") + (yield "role")))) + (vals (generator (lambda () + (yield "alice") + (yield "30") + (yield "engineer")))) + (pairs (list))) + ;; Manual zip: advance both, build pairs + (let ((k (next keys)) + (v (next vals))) + (while (not (is-null k)) + (begin + (append pairs (str-concat k "=" v)) + (set k (next keys)) + (set v (next vals))))) + pairs)) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:2021 + + (begin + (set tries 0) + (while (and (not (state-exists "worker.2.result")) (< tries 5000)) + (set tries (+ tries 1))) + (let ((total 0)) + (for i (range 3) + (set total (+ total 1))) + total)) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:2041 + + (begin + (let ((running-sum (generator (lambda () + (let ((acc 0)) + (for x (list 10 20 30 40 50) + (begin + (set acc (+ acc x)) + (yield acc)))))))) + (collect running-sum))) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:2073 + + (begin + (set received (list)) + (set count 0) + (while (< count 9) + (begin + (set msg (msg-recv "work.queue")) + (append received (get-attr msg "status")) + (set count (+ count 1)))) + ;; Generator that maps each status to 1 + (set ones (generator (lambda () + (for item received + (yield 1))))) + ;; Collect into a list and take its length + (length (collect ones))) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:2117 + + (begin + (set payloads (list)) + (set n 0) + (while (< n 8) + (begin + (set msg (msg-recv "data.in")) + (append payloads (get-attr msg "status")) + (set n (+ n 1)))) + ;; Build a generator that doubles each collected payload string + (set doubler (generator (lambda () + (for p payloads + (yield (str-concat p p)))))) + ;; Collect the doubled strings and count them + (length (collect doubler))) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:2161 + + (begin + (set msg (msg-recv "ch")) + 42) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:2170 + + (begin + (msg-send "ch" "hello") + 99) + +@@@ script src/cvc/tests/state_exec_integration_test.cpp:2376 +(msg-send "sig" "hi") +@@@ script src/cvc/tests/state_exec_integration_test.cpp:2382 +(msg-pending "sig") +@@@ script src/cvc/tests/state_exec_integration_test.cpp:2400 +(msg-recv "sig") +@@@ script src/cvc/tests/state_exec_integration_test.cpp:2409 +(msg-send "raw.chan" "x") +@@@ script src/cvc/tests/state_exec_intrinsics_test.cpp:1526 +(self-pid) +@@@ script src/cvc/tests/state_exec_intrinsics_test.cpp:1536 +(cluster-id) +@@@ script src/cvc/tests/state_exec_intrinsics_test.cpp:1541 +(state-set "dsl.key" "dsl-value") +@@@ script src/cvc/tests/state_exec_intrinsics_test.cpp:1542 +(state-get "dsl.key") +@@@ script src/cvc/tests/state_exec_intrinsics_test.cpp:1547 +(state-set "dsl.exists" "yes") +@@@ script src/cvc/tests/state_exec_intrinsics_test.cpp:1548 +(state-exists "dsl.exists") +@@@ script src/cvc/tests/state_exec_intrinsics_test.cpp:1553 +(memory-limit) +@@@ script src/cvc/tests/state_exec_intrinsics_test.cpp:1558 +(spawn "(+ 1 2)") +@@@ script src/cvc/tests/state_exec_intrinsics_test.cpp:1585 +(string.upper "hello") +@@@ script src/cvc/tests/state_exec_intrinsics_test.cpp:1590 +(math.sqrt 25) +@@@ script src/cvc/tests/state_exec_intrinsics_test.cpp:1595 +(math.pi) +@@@ script src/cvc/tests/state_exec_intrinsics_test.cpp:1600 +(string.length "hello world") +@@@ script src/cvc/tests/state_exec_intrinsics_test.cpp:1605 +(collections.range 3) +@@@ script src/cvc/tests/state_exec_multiprocess_test.cpp:188 + + (begin + (set v1 (state-get "shared.x")) + (set v2 (state-get "shared.y")) + (set v3 (state-get "shared.z")) + (str-concat v1 ":" v2 ":" v3)) + +@@@ script src/cvc/tests/state_exec_multiprocess_test.cpp:231 + + (begin + (state-set "shared.x" "10") + (state-set "shared.y" "20") + (state-set "shared.z" "30") + "written") + +@@@ script src/cvc/tests/state_exec_multiprocess_test.cpp:325 + + (begin + (set m (msg-recv "oob.chan")) + (get-attr m "payload")) + +@@@ script src/cvc/tests/state_exec_multiprocess_test.cpp:394 + + (begin + (msg-send "oob.chan" "hello-from-A") + "sent") + +@@@ script src/cvc/tests/state_exec_multiprocess_test.cpp:489 + + (begin + (set reader (generator (lambda () + (for i (range 5) + (yield (state-get (str-concat "items.v" (str i)))))))) + (set count 0) + (set concat "") + (for val reader + (begin + (set count (+ count 1)) + (set concat (str-concat concat val ":")))) + (state-set "result.consumer" (str-concat (str count) "|" concat)) + count) + +@@@ script src/cvc/tests/state_exec_multiprocess_test.cpp:553 + + (begin + (set val-gen (generator (lambda () + (for i (range 5) + (yield (str-concat "v" (str i))))))) + (for val val-gen + (begin + (set idx (str-concat "items." val)) + (state-set idx val))) + (state-set "items.ready" "true") + "produced") + +@@@ script src/cvc/tests/state_exec_multiprocess_test.cpp:655 + + (begin + (set count 0) + (for p (range 3) + (for i (range 3) + (begin + (set key (str-concat "producer." (str p) ".item." (str i))) + (if (state-exists key) + (set count (+ count 1)) + 0)))) + count) + +@@@ script src/cvc/tests/state_exec_multiprocess_test.cpp:807 + + (begin + (set result "") + (set i 0) + (while (< i 4) + (begin + (set m (msg-recv "pipe.data")) + (set result (str-concat result (get-attr m "payload") ",")) + (set i (+ i 1)))) + (state-set "pipeline.result" result) + result) + +@@@ script src/cvc/tests/state_exec_multiprocess_test.cpp:892 + + (begin + (set payload-gen (generator (lambda () + (for tag (list "aa" "bb" "cc" "dd") + (yield tag))))) + (for payload payload-gen + (msg-send "pipe.data" payload)) + "all-sent") + +@@@ script src/cvc/tests/state_exec_multiprocess_test.cpp:988 + + (begin + (state-set "grpc.x" "alpha") + (state-set "grpc.y" "beta") + "written") + +@@@ script src/cvc/tests/state_exec_multiprocess_test.cpp:1012 + + (begin + (str-concat (state-get "grpc.x") ":" (state-get "grpc.y"))) + +@@@ script src/cvc/tests/state_exec_multiprocess_test.cpp:1064 + + (begin + (set m (msg-recv "grpc.chan")) + (get-attr m "payload")) + +@@@ script src/cvc/tests/state_exec_multiprocess_test.cpp:1086 + + (begin + (msg-send "grpc.chan" "hello-grpc") + "sent") + +@@@ script src/cvc/tests/state_exec_multiprocess_test.cpp:1147 + + (begin + (set squares (generator (lambda () + (for i (range 5) + (yield (* i i)))))) + (let ((i 0)) + (for sq squares + (begin + (state-set (str-concat "data.v" (str i)) (str sq)) + (set i (+ i 1))))) + (state-set "data.ready" "true") + "produced") + +@@@ script src/cvc/tests/state_exec_multiprocess_test.cpp:1179 + + (begin + (set reader (generator (lambda () + (for i (range 5) + (yield (int (state-get (str-concat "data.v" (str i))))))))) + (set sum 0) + (for val reader + (set sum (+ sum val))) + sum) + +@@@ script src/cvc/tests/state_exec_parser_test.cpp:83 +(if (> x 0) "pos" "neg") +@@@ script src/cvc/tests/state_exec_parser_test.cpp:97 +(a (b (c (d)))) +@@@ script src/cvc/tests/state_exec_parser_test.cpp:143 +(+ 1 ; add one +2) +@@@ script src/cvc/tests/state_exec_parser_test.cpp:152 +(defun f (x) x) (f 42) +@@@ script src/cvc/tests/state_exec_scheduler_test.cpp:177 +(- 100 50) +@@@ script src/cvc/tests/state_exec_scheduler_test.cpp:280 +(begin 4 5 6) +@@@ script src/cvc/tests/state_exec_scheduler_test.cpp:806 + + (begin + (defun fib (n) + (if (<= n 1) n + (+ (fib (- n 1)) (fib (- n 2))))) + (fib 8)) + +@@@ script src/cvc/tests/state_exec_scheduler_test.cpp:813 + + (begin + (defun fact (n) + (if (<= n 1) 1 + (* n (fact (- n 1))))) + (fact 6)) + +@@@ script src/cvc/tests/state_exec_scheduler_test.cpp:826 + + (begin + (let ((sum 0)) + (for i (list 1 2 3 4 5) + (set sum (+ sum i))) + sum)) + +@@@ script src/cvc/tests/state_exec_scheduler_test.cpp:838 + + (begin + (let ((i 0) (sum 0)) + (while (< i 10) + (begin + (set sum (+ sum i)) + (set i (+ i 1)))) + sum)) + +@@@ script src/cvc/tests/state_exec_scheduler_test.cpp:852 + + (begin + (let ((double (lambda (x) (* x 2)))) + (double 21))) + +@@@ script src/cvc/tests/state_exec_scheduler_test.cpp:1112 +(if #t 42 99) +@@@ script src/cvc/tests/state_exec_scheduler_test.cpp:1113 +(let ((x 10)) (* x x)) +@@@ script src/cvc/tests/state_exec_scheduler_test.cpp:1113 +(length "hello") +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:157 +(begin (set i 0) (while (< i 1000) (set i (+ i 1))) i) +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:220 + + (let ((g (generator (lambda () + (yield 10) + (yield 20) + (yield 30))))) + (list (next g) (next g) (next g))) + +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:236 + + (let ((g (generator (lambda () (yield 1))))) + (list (next g) (next g))) + +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:248 + + (let ((g (generator (lambda () + (yield (* 2 3)) + (yield (+ 10 5)))))) + (+ (next g) (next g))) + +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:259 + + (let ((r (range 0 5))) + (+ (next r) (next r) (next r))) + +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:267 +(collect (range 5)) +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:274 + + (let ((total 0)) + (for x (range 1 6) + (set total (+ total x))) + total) + +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:284 + + (let ((accum (list))) + (for x (generator (lambda () + (yield "a") + (yield "b") + (yield "c"))) + (append accum x)) + (length accum)) + +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:298 + + (collect (generator (lambda () + (let ((i 0)) + (while (< i 4) + (begin (yield i) + (set i (+ i 1)))))))) + +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:316 +(while t (break)) +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:322 +(while t (break 42)) +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:326 +(begin (set i 0) (while t (if (= i 5) (break i) (set i (+ i 1))))) +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:335 +(for x (list 1 2 3 4 5) (if (= x 3) (break x) nil)) +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:342 +(for x (range 1000) (if (= x 5) (break x) nil)) +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:349 +(begin (set sum 0) (for i (list 1 2 3) (for j (list 10 20 30) (if (= j 20) (break nil) (set sum (+ sum j))))) sum) +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:360 +(begin (set i 0) (while t (begin (set i (+ i 1)) (if (= i 3) (break i) nil)))) +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:370 +(for x (list 1 2 3 4 5) (let ((doubled (* x 2))) (if (> doubled 6) (break doubled) nil))) +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:382 +(if #t 1 2) +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:383 +(begin 1 2) +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:384 +(quote 1) +@@@ script src/cvc/tests/state_exec_stackless_evaluator_test.cpp:385 +(defclass C (init (self) 1)) +@@@ script src/cvc/tests/state_exec_types_test.cpp:72 +(1 2 3) +@@@ script src/cvc/tests/state_exec_types_test.cpp:83 +(1 ("a" "b")) +@@@ script src/cvcGL/test/cvcgl_media_audio.cpp:50 +(if (= (state-get "cap.done") "1") #f (begin (stream-capture-audio (get-attr (stream-open (dict "id" "mic" "codec" "f32" "rate" 48000 "ch" 1 "frames_per_chunk" 256)) "token") (dict "rate" 48000 "ch" 1 "frames" 256 "mix" (list (dict "tone" (dict "hz" 440)) (dict "tone" (dict "hz" 660))))) (state-set "cap.done" "1"))) +@@@ script src/cvcGL/test/cvcgl_media_capture.cpp:52 +(if (= (state-get "cap.done") "1") #f (begin (stream-capture-camera (get-attr (stream-open (dict "id" "cam" "w" 64 "h" 48)) "token") (dict "synthetic" #t "w" 64 "h" 48 "hz" 240)) (state-set "cap.done" "1"))) +@@@ script src/cvcGL/test/cvcgl_media_play.cpp:52 +(if (= (state-get "play.done") "1") #f (begin (stream-open (dict "id" "mic" "codec" "f32" "rate" 48000 "ch" 1 "frames_per_chunk" 256 "subscriber_depth" 8)) (stream-capture-audio "streams.mic" (dict "rate" 48000 "ch" 1 "frames" 256 "tone" (dict "hz" 440))) (stream-play-audio "streams.mic") (state-set "play.done" "1"))) +@@@ script src/cvcGL/test/cvcgl_stream_bind.cpp:63 +(begin (set h (gl-bind-stream "screen" "streams.cam0")) (state-set "ok" (get-attr h "ok")) (state-set "node" (get-attr h "node")) (state-set "token" (get-attr h "token")) (state-set "done" "1")) +@@@ script src/cvcGL/test/cvcgl_stream_bind.cpp:81 +(begin (set h (gl-bind-stream "screen" (dict "token" "demo.streams.cam0"))) (state-set "ok" (get-attr h "ok")) (state-set "done" "1")) +@@@ script src/cvcGL/test/cvcgl_stream_bind.cpp:96 +(begin (set h (gl-bind-stream "ghost" "streams.cam0")) (state-set "ok" (get-attr h "ok")) (state-set "err" (get-attr h "error")) (state-set "done" "1")) +@@@ script src/cvcGL/test/cvcgl_stream_bind.cpp:109 +(begin (set h (gl-bind-stream "screen" "streams.cam0")) (state-set "ok" (get-attr h "ok")) (state-set "err" (get-attr h "error")) (state-set "done" "1")) +@@@ script src/cvcGL/test/cvcgl_stream_bind.cpp:123 +(begin (set h (gl-bind-stream "screen")) (state-set "ok" (get-attr h "ok")) (state-set "err" (get-attr h "error")) (state-set "done" "1")) +@@@ script bindings/pycvc/test_pycvc_exec.py:49 +(state-set "dsl.out" "wrote-from-dsl") +@@@ script bindings/pycvc/test_pycvc_exec.py:123 +(capture #t) +@@@ script bindings/pycvc/test_pycvc_exec.py:147 +(capture (make-bytes)) +@@@ script bindings/pycvc/test_pycvc_exec.py:21 +(* 6 7) +@@@ script bindings/pycvc/test_pycvc_exec.py:22 +(begin (set x 5) x) +@@@ script bindings/pycvc/test_pycvc_exec.py:23 +(str-concat "ab" "cd") +@@@ script bindings/pycvc/test_pycvc_exec.py:25 +(> 1 2) +@@@ script bindings/pycvc/test_pycvc_exec.py:47 +(state-get "shared.x") +@@@ script bindings/pycvc/test_pycvc_exec.py:64 +(py-double 21) +@@@ script bindings/pycvc/test_pycvc_exec.py:65 +(py-add 40 2) +@@@ script bindings/pycvc/test_pycvc_exec.py:66 +(py-greet "cvc") +@@@ script bindings/pycvc/test_pycvc_exec.py:68 +(+ 1 (py-double 20)) +@@@ script bindings/pycvc/test_pycvc_exec.py:81 +(stash "py.k" "py.v") +@@@ script bindings/pycvc/test_pycvc_exec.py:86 +(fetch "py.k") +@@@ script bindings/pycvc/test_pycvc_exec.py:98 +(boom) +@@@ script bindings/pycvc/test_pycvc_exec.py:119 +(capture 5) +@@@ script bindings/pycvc/test_pycvc_exec.py:121 +(capture "text") +@@@ script bindings/pycvc/test_pycvc_exec_async.py:29 +(begin (adouble 21)) +@@@ script bindings/pycvc/test_pycvc_exec_async.py:62 +(begin (slow 7)) +@@@ script bindings/pycvc/test_pycvc_exec_async.py:78 +(begin (get-attr (boom 1) "__async_error__")) +@@@ script bindings/pycvc/test_pycvc_exec_async.py:91 +(begin (inc (triple 4))) +@@@ script bindings/pycvc/test_pycvc_exec_async.py:110 +(begin (spawn "(while t 1)") (state-set "r" (str (slow 7))) (state-get "r")) +@@@ script bindings/pycvc/test_pycvc_exec_async.py:82 +(begin (idn 5)) +@@@ script bindings/pycvc/test_pycvc_exec_coro.py:58 +(begin (spawn "(while t 1)") (state-set "r" (str (acount 6))) (state-get "r")) +@@@ script bindings/pycvc/test_pycvc_exec_coro.py:25 +(begin (adbl 21)) +@@@ script bindings/pycvc/test_pycvc_exec_coro.py:40 +(begin (acount 10)) +@@@ script bindings/pycvc/test_pycvc_exec_coro.py:96 +(begin (plus10 (inc (triple 4)))) +@@@ script bindings/pycvc/test_pycvc_exec_coro.py:128 +(begin (reenter 1)) +@@@ script bindings/pycvc/test_pycvc_integration.py:60 +(state-set "dsl.done" "yes") +@@@ script bindings/pycvc/test_pycvc_integration.py:58 +(+ 0 (verts)) diff --git a/src/cvc/tests/state_exec_corpus_test.cpp b/src/cvc/tests/state_exec_corpus_test.cpp new file mode 100644 index 000000000..2ed2fffe1 --- /dev/null +++ b/src/cvc/tests/state_exec_corpus_test.cpp @@ -0,0 +1,102 @@ +// Parses every DSL script the repo carries (state_exec_corpus/scripts.txt, a frozen snapshot made +// by gen_state_exec_corpus.py) and compares each parse with state_exec_corpus/expected.txt. The +// grammar work (docs/STATE_EXEC_GRAMMAR.md) replaces the parser; this pins how every existing +// script reads, so a change to that is visible in the same PR as an expected.txt diff. +// +// Regenerate expected.txt after an intended change: CVC_UPDATE_GOLDENS=1 state_exec_corpus_test + +#include "state_exec_grammar_dump.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace fs = std::filesystem; +namespace dump = cvc::state_exec::grammar_dump; + +namespace { + +const std::string kMark = "@@@ script "; + +fs::path corpus_dir() { return fs::path(__FILE__).parent_path() / "state_exec_corpus"; } + +std::string read_file(const fs::path &p) { + std::ifstream in(p, std::ios::binary); + std::stringstream ss; + ss << in.rdbuf(); + std::string s = ss.str(); + // The files are LF-only (.gitattributes); undo a CRLF checkout anyway. + for (auto i = s.find("\r\n"); i != std::string::npos; i = s.find("\r\n", i)) + s.erase(i, 1); + return s; +} + +// Split "@@@ script \n\n" entries into (source, body) pairs. A body is everything up +// to the next entry's mark, minus the one '\n' that ends it. +std::vector> read_entries(const fs::path &p) { + const std::string text = read_file(p); + std::vector> out; + // The mark only counts at the start of a line (the header comment mentions it). + std::size_t pos = text.compare(0, kMark.size(), kMark) == 0 ? 0 : text.find("\n" + kMark); + if (pos != std::string::npos && text[pos] == '\n') + ++pos; + while (pos != std::string::npos) { + const std::size_t eol = text.find('\n', pos); + std::string source = text.substr(pos + kMark.size(), eol - pos - kMark.size()); + std::size_t next = text.find("\n" + kMark, eol); + std::string body = + text.substr(eol + 1, (next == std::string::npos ? text.size() : next + 1) - (eol + 1)); + if (!body.empty() && body.back() == '\n') + body.pop_back(); + out.emplace_back(std::move(source), std::move(body)); + pos = next == std::string::npos ? next : next + 1; + } + return out; +} + +} // namespace + +TEST(StateExecCorpus, EveryScriptParsesAsRecorded) { + const auto scripts = read_entries(corpus_dir() / "scripts.txt"); + ASSERT_GT(scripts.size(), 100u) << "corpus missing: " << corpus_dir() / "scripts.txt"; + + std::vector actual; + std::size_t errors = 0; + for (const auto &[source, script] : scripts) { + actual.push_back(dump::dump_all(script)); + errors += actual.back().rfind("!error", 0) == 0; + } + std::cout << scripts.size() << " scripts, " << errors << " parse errors\n"; + + const fs::path expected_path = corpus_dir() / "expected.txt"; + if (const char *update = std::getenv("CVC_UPDATE_GOLDENS"); update && *update == '1') { + std::ofstream out(expected_path, std::ios::binary); + out << "# Expected parses of scripts.txt, in the grammar_dump format of\n" + "# state_exec_grammar_dump.h. Written by state_exec_corpus_test with\n" + "# CVC_UPDATE_GOLDENS=1; do not edit by hand.\n"; + for (std::size_t i = 0; i < scripts.size(); ++i) + out << kMark << scripts[i].first << '\n' << actual[i] << '\n'; + GTEST_SKIP() << "wrote " << expected_path; + } + + const auto expected = read_entries(expected_path); + ASSERT_EQ(expected.size(), scripts.size()) + << expected_path << " is stale; regenerate it with CVC_UPDATE_GOLDENS=1"; + int shown = 0; + for (std::size_t i = 0; i < scripts.size(); ++i) { + ASSERT_EQ(expected[i].first, scripts[i].first) + << expected_path << " is stale; regenerate it with CVC_UPDATE_GOLDENS=1"; + if (actual[i] != expected[i].second && ++shown <= 20) + ADD_FAILURE() << scripts[i].first << "\n--- script\n" + << scripts[i].second << "\n--- expected\n" + << expected[i].second << "\n--- actual\n" + << actual[i]; + } + EXPECT_EQ(shown, 0) << shown << " scripts parse differently from " << expected_path; +} diff --git a/src/cvc/tests/state_exec_grammar_dump.h b/src/cvc/tests/state_exec_grammar_dump.h new file mode 100644 index 000000000..908fb6dc0 --- /dev/null +++ b/src/cvc/tests/state_exec_grammar_dump.h @@ -0,0 +1,240 @@ +/** + * @file state_exec_grammar_dump.h + * @brief Exact, type-tagged rendering of parser output for the grammar tests. + * + * state_exec_grammar_test and state_exec_corpus_test compare parses through this dump, not + * through to_string(): to_string is lossy (unescaped strings, 6-digit floats) and changes in + * phase P5 of docs/STATE_EXEC_GRAMMAR.md, while a golden must mean the same thing before and + * after every phase. The format: + * + * nil / #t / #f nil and the bools + * 42, -7 int64 + * 1.5, 31.0, 1e+20 double: whole numbers below 1e16 as N.0, others in the shortest + * form that reads back exactly; always with '.', 'e' or a special + * +inf.0 -inf.0 +nan.0 + * "..." string: \\ \" \n \t \r, other ASCII controls as \xNN, invalid UTF-8 + * bytes as \xNN (always >= \x80), invisible or control code points as + * \u{X}, everything else verbatim + * b"..." bytes: printable ASCII verbatim, the rest \xNN + * foo, |1+| symbol; |...| (same escapes, plus \|) when the bare text would read + * as something else + * (a b) list + * + * dump_all() joins top-level values with '\n' and renders a parse_error as "!error line:col". + */ +#ifndef CVC_STATE_EXEC_GRAMMAR_DUMP_H +#define CVC_STATE_EXEC_GRAMMAR_DUMP_H + +#include +#include +#include +#include +#include +#include +#include +#include + +namespace cvc::state_exec::grammar_dump { + +/// Code points a dump writes as \u{X}: C1 controls, bidi controls, U+FEFF, and the +/// whitespace that is not ASCII (so an NBSP in a golden is visible). +inline bool invisible(char32_t cp) { + return (cp >= 0x80 && cp <= 0x9F) || cp == 0x61C || cp == 0xA0 || cp == 0x1680 || + (cp >= 0x2000 && cp <= 0x200F) || (cp >= 0x2028 && cp <= 0x202F) || cp == 0x205F || + (cp >= 0x2066 && cp <= 0x2069) || cp == 0x3000 || cp == 0xFEFF; +} + +inline void hex_byte(unsigned char b, std::string &out) { + static const char kHex[] = "0123456789abcdef"; + out += "\\x"; + out += kHex[b >> 4]; + out += kHex[b & 0x0F]; +} + +/// Escape UTF-8 text for a string or |symbol| dump; `quote` is the closing delimiter. +inline void text(std::string_view s, char quote, std::string &out) { + std::size_t pos = 0; + while (pos < s.size()) { + char32_t cp; + std::size_t len; + if (!utf8::decode(s, pos, cp, len)) { + hex_byte(static_cast(s[pos]), out); + pos += 1; + continue; + } + if (cp == '\\' || cp == static_cast(quote)) { + out += '\\'; + out += static_cast(cp); + } else if (cp == '\n') { + out += "\\n"; + } else if (cp == '\t') { + out += "\\t"; + } else if (cp == '\r') { + out += "\\r"; + } else if (cp < 0x20 || cp == 0x7F) { + hex_byte(static_cast(cp), out); + } else if (invisible(cp)) { + char buf[16]; + std::snprintf(buf, sizeof buf, "\\u{%X}", static_cast(cp)); + out += buf; + } else { + out.append(s.substr(pos, len)); + } + pos += len; + } +} + +/// Shortest decimal form that reads back to exactly `d`, marked as a double. +inline std::string number(double d) { + if (std::isnan(d)) + return "+nan.0"; + if (std::isinf(d)) + return d > 0 ? "+inf.0" : "-inf.0"; + char buf[40]; + if (d == std::trunc(d) && std::fabs(d) < 1e16) { // whole numbers read as 1000.0, not 1e+03 + std::snprintf(buf, sizeof buf, "%.1f", d); + return buf; + } + for (int prec = 1; prec <= 17; ++prec) { + std::snprintf(buf, sizeof buf, "%.*g", prec, d); + if (std::strtod(buf, nullptr) == d) + break; + } + std::string s(buf); + if (s.find_first_of(".e") == std::string::npos) + s += ".0"; + return s; +} + +/// True when a symbol's bare name would read back as something other than that symbol. +inline bool needs_bars(std::string_view name) { + if (name.empty() || name == "nil" || name == "true" || name == "false" || name[0] == '#') + return true; + const char c0 = name[0]; + if (c0 >= '0' && c0 <= '9') + return true; + if ((c0 == '+' || c0 == '-' || c0 == '.') && name.size() > 1) { + const char c1 = name[1]; + if ((c1 >= '0' && c1 <= '9') || c1 == '.' || c1 == 'i' || c1 == 'n') + return true; + } + std::size_t pos = 0; + while (pos < name.size()) { + char32_t cp; + std::size_t len; + if (!utf8::decode(name, pos, cp, len) || cp <= 0x20 || cp == 0x7F || invisible(cp)) + return true; + switch (cp) { + case '(': + case ')': + case '"': + case '\'': + case ';': + case ',': + case '[': + case ']': + case '\\': + case '`': + case '{': + case '|': + case '}': + return true; + default: + break; + } + pos += len; + } + return false; +} + +inline void value(const value_t &v, std::string &out) { + std::visit( + [&](auto &&arg) { + using T = std::decay_t; + if constexpr (std::is_same_v) { + out += "nil"; + } else if constexpr (std::is_same_v) { + out += arg ? "#t" : "#f"; + } else if constexpr (std::is_same_v) { + out += std::to_string(arg); + } else if constexpr (std::is_same_v) { + out += number(arg); + } else if constexpr (std::is_same_v) { + out += '"'; + text(arg, '"', out); + out += '"'; + } else if constexpr (std::is_same_v) { + out += "b\""; + for (unsigned char b : arg.data) { + if (b == '"' || b == '\\') { + out += '\\'; + out += static_cast(b); + } else if (b >= 0x20 && b < 0x7F) { + out += static_cast(b); + } else { + hex_byte(b, out); + } + } + out += '"'; + } else if constexpr (std::is_same_v) { + if (needs_bars(arg.name)) { + out += '|'; + text(arg.name, '|', out); + out += '|'; + } else { + out += arg.name; + } + } else if constexpr (std::is_same_v) { + out += '('; + if (arg) { + for (std::size_t i = 0; i < arg->size(); ++i) { + if (i > 0) + out += ' '; + value((*arg)[i], out); + } + } + out += ')'; + } else { + out += "#<" + v.type_name() + ">"; // the parser never produces these + } + }, + v.v); +} + +inline std::string value(const value_t &v) { + std::string out; + value(v, out); + return out; +} + +inline std::string error(const parse_error &e) { + return "!error " + std::to_string(e.line()) + ":" + std::to_string(e.col()); +} + +/// parse_all(src), one dumped value per line. +inline std::string dump_all(const std::string &src) { + try { + std::string out; + for (const value_t &v : parse_all(src)) { + if (!out.empty()) + out += '\n'; + value(v, out); + } + return out; + } catch (const parse_error &e) { + return error(e); + } +} + +/// parse(src), dumped. +inline std::string dump_one(const std::string &src) { + try { + return value(parse(src)); + } catch (const parse_error &e) { + return error(e); + } +} + +} // namespace cvc::state_exec::grammar_dump + +#endif // CVC_STATE_EXEC_GRAMMAR_DUMP_H diff --git a/src/cvc/tests/state_exec_grammar_test.cpp b/src/cvc/tests/state_exec_grammar_test.cpp new file mode 100644 index 000000000..f8b0ca1a2 --- /dev/null +++ b/src/cvc/tests/state_exec_grammar_test.cpp @@ -0,0 +1,284 @@ +// Conformance tests for the state_exec grammar (docs/STATE_EXEC_GRAMMAR.md). +// +// Every case names the EBNF productions it exercises and the phase from which it holds: +// 0 holds for the pre-grammar (hand-written) parser too; +// N holds once phase PN of the grammar work lands (P2 = parser built from the grammar, +// P3 = Unicode source rules + strict escapes, P4 = octal escapes, C escapes, raw strings). +// Cases for landed phases (<= kLandedPhase) must hold. Cases for pending phases must still +// DIFFER, so the table stays an exact list of what each phase changes; when a phase lands, bump +// kLandedPhase in the same PR. The doc's EBNF is checked for closure, and every production must +// have a case. +// +// Inputs and expectations are byte-exact: non-ASCII is written as \xNN escapes (MSVC builds +// without /utf-8), split with "" where a hex letter follows an escape. + +#include "state_exec_grammar_dump.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace fs = std::filesystem; +namespace dump = cvc::state_exec::grammar_dump; + +namespace { + +constexpr int kLandedPhase = 0; + +enum class mode { all, one }; // parse_all() or parse() + +struct grammar_case { + int phase; + std::string rules; // space-separated EBNF production names + mode how; + std::string input; + std::string expect; // grammar_dump output; "!error" matches any error position +}; + +std::string nest(int depth) { return std::string(depth, '(') + std::string(depth, ')'); } + +const std::vector &cases() { + static const std::vector kCases = { + // --- holds today ---------------------------------------------------------------------- + {0, "program", mode::all, "", ""}, + {0, "program atmosphere", mode::all, "1 2\n3", "1\n2\n3"}, + {0, "single_datum", mode::one, "", "nil"}, + {0, "single_datum line_comment", mode::one, " ; note\n 42 ", "42"}, + {0, "datum list", mode::all, "(a (b c) ())", "(a (b c) ())"}, + {0, "list", mode::all, nest(256), nest(256)}, + {0, "list", mode::all, "(a", "!error"}, + {0, "quoted", mode::all, "'x ' (1 2) ''a", "(quote x)\n(quote (1 2))\n(quote (quote a))"}, + {0, "line_comment line_end", mode::all, "; c\n(a ; c2\n b)", "(a b)"}, + {0, "whitespace", mode::all, "\t\n\v\f\r a", "a"}, + {0, "text_literal text_string string_char", mode::all, "\"hi there\"", "\"hi there\""}, + {0, "string_char", mode::all, "\"a\nb\"", R"("a\nb")"}, + {0, "text_string", mode::all, "\"abc", "!error"}, + {0, "text_escape simple_escape", mode::all, R"("\n\t\r\\\"")", R"("\n\t\r\\\"")"}, + {0, "hex_escape hex", mode::all, R"("\x41\xe9")", "\"A\xC3\xA9\""}, + {0, "hex_escape", mode::all, R"("\xZ1")", "!error"}, + {0, "unicode_escape", mode::all, R"("é\u{1F600}")", "\"\xC3\xA9\xF0\x9F\x98\x80\""}, + {0, "unicode_escape", mode::all, R"("\u12")", "!error"}, + {0, "unicode_escape", mode::all, R"("\u{}")", "!error"}, + {0, "unicode_escape", mode::all, R"("\u{1234567}")", "!error"}, + {0, "octal_escape", mode::all, R"("\0")", R"("\x00")"}, + {0, "bytes_literal byte_string byte_escape", mode::all, R"(b"ab\x00\xff\n")", + R"(b"ab\x00\xff\x0a")"}, + {0, "byte_string", mode::all, "b\"\xC3\xA9\"", R"(b"\xc3\xa9")"}, + {0, "byte_string symbol", mode::all, "b (b) b\"\"", "b\n(b)\nb\"\""}, + {0, "boolean", mode::all, "#t true", "#t\n#t"}, + {0, "nil", mode::all, "#f false nil", "nil\nnil\nnil"}, + {0, "number integer dec_digits sign", mode::all, "42 -7 007 -9223372036854775808", + "42\n-7\n7\n-9223372036854775808"}, + {0, "decimal_float exponent", mode::all, "1.5 .5 1. 1e3 -2.5E-3", + "1.5\n0.5\n1.0\n1000.0\n-0.0025"}, + {0, "atom token constituent symbol", mode::all, + "foo-bar? -> + - caf\xC3\xA9 \xCE\xBB \xE6\x97\xA5\xE6\x9C\xAC a.b:c", + "foo-bar?\n->\n+\n-\ncaf\xC3\xA9\n\xCE\xBB\n\xE6\x97\xA5\xE6\x9C\xAC\na.b:c"}, + {0, "symbol", mode::all, "1+ ... #foo", "|1+|\n|...|\n|#foo|"}, + {0, "delimiter constituent", mode::all, "a\"b\"(c)'d;e", "a\n\"b\"\n(c)\n(quote d)"}, + {0, "raw_text symbol", mode::all, "r#foo", "r#foo"}, + + // --- P2: parser built from the grammar, numbers included ----------------------------- + {2, "program", mode::all, "a)", "!error"}, + {2, "reserved", mode::all, "[a]", "!error"}, + {2, "reserved", mode::all, "a,b", "!error"}, + {2, "reserved", mode::all, "`a", "!error"}, + {2, "reserved", mode::all, "{}", "!error"}, + {2, "reserved", mode::all, "a|b", "!error"}, + {2, "reserved", mode::all, "a\\b", "!error"}, + {2, "single_datum", mode::one, "1 2", "!error"}, + {2, "quoted", mode::all, "'", "!error"}, + {2, "list", mode::all, nest(257), "!error"}, + {2, "line_end line_comment", mode::all, "; c\r1", "1"}, + {2, "radix_integer hex_digits", mode::all, "0x1F 0X1f #x1F #X1f 0xFF_FF", + "31\n31\n31\n31\n65535"}, + {2, "radix_integer oct_digits bin_digits sign", mode::all, "0o17 #o17 0b101 #b101 -0x10 +0b1", + "15\n15\n5\n5\n-16\n1"}, + {2, "radix_prefixed", mode::all, "0xZZ", "!error"}, + {2, "radix_prefixed", mode::all, "#bar", "!error"}, + {2, "radix_prefixed", mode::all, "0x", "!error"}, + {2, "integer sign", mode::all, "+5", "5"}, + {2, "dec_digits", mode::all, "1_000", "1000"}, + {2, "integer", mode::all, "9223372036854775808", "!error"}, + {2, "float_special", mode::all, "+inf.0 -inf.0 +nan.0 -nan.0", + "+inf.0\n-inf.0\n+nan.0\n+nan.0"}, + {2, "symbol", mode::all, "inf nan infinity -INF", "inf\nnan\ninfinity\n-INF"}, + {2, "decimal_float", mode::all, "1e999", "!error"}, + {2, "decimal_float dec_digits", mode::all, "6.022_140e23", "6.02214e+23"}, + + // --- P3: Unicode source rules, strict escapes ---------------------------------------- + {3, "scalar_value", mode::all, + "a\xff" + "b", + "!error"}, + {3, "bom", mode::all, "\xEF\xBB\xBF(a)", "(a)"}, + {3, "other_space constituent", mode::all, + "a\xC2\xA0" + "b", + "!error"}, + {3, "whitespace", mode::all, + "a\xC2\x85" + "b a\xE2\x80\xA8" + "b", + "a\nb\na\nb"}, + {3, "source_char bidi_control", mode::all, + "\"a\xE2\x80\xAE" + "b\"", + "!error"}, + {3, "bidi_control line_comment", mode::all, "; \xE2\x81\xA6\n1", "!error"}, + {3, "control", mode::all, + "\"a\x01" + "b\"", + "!error"}, + {3, "text_escape", mode::all, R"("\q")", "!error"}, + {3, "unicode_escape", mode::all, R"("\uD800")", "!error"}, + {3, "unicode_escape", mode::all, R"("\u{110000}")", "!error"}, + // The error is reported at the offending '"', counted in code points: 1:6, not 1:8. + {3, "hex_escape", mode::all, "\"\xC3\xA9\\x4\"", "!error 1:6"}, + + // --- P4: octal escapes, C escapes, raw strings --------------------------------------- + {4, "octal_escape", mode::all, R"("\101\012")", R"("A\n")"}, + {4, "byte_escape octal_escape", mode::all, R"(b"\377")", R"(b"\xff")"}, + {4, "byte_escape", mode::all, R"(b"\400")", "!error"}, + {4, "simple_escape", mode::all, R"("\a\b\f\v")", R"("\x07\x08\x0c\x0b")"}, + {4, "raw_text raw_body raw_char", mode::all, R"(r"a\nb")", R"("a\\nb")"}, + {4, "hash_fence", mode::all, R"(r#"say "hi""#)", R"("say \"hi\"")"}, + {4, "hash_fence", mode::all, R"(r##"a"#b"##)", R"("a\"#b")"}, + {4, "raw_bytes", mode::all, R"(br"\x00")", R"(b"\\x00")"}, + {4, "raw_body", mode::all, R"(r#"abc")", "!error"}, + }; + return kCases; +} + +std::string run(const grammar_case &c) { + return c.how == mode::all ? dump::dump_all(c.input) : dump::dump_one(c.input); +} + +bool matches(const std::string &actual, const std::string &expect) { + if (expect == "!error") + return actual.rfind("!error", 0) == 0; + return actual == expect; +} + +std::string describe(const grammar_case &c) { + return "[P" + std::to_string(c.phase) + " " + c.rules + "] input: " + c.input; +} + +fs::path grammar_doc() { + // src/cvc/tests/ -> repo root -> docs/ + return fs::path(__FILE__).parent_path().parent_path().parent_path().parent_path() / "docs" / + "STATE_EXEC_GRAMMAR.md"; +} + +struct ebnf { + std::map defined; // production -> times defined + std::map> // production -> names its body references + uses; +}; + +// Read the ```ebnf block of the grammar doc: `name ::= body` lines, continuation lines, and +// /* comments */. Names inside quotes, character classes and #xN code points are not references. +ebnf read_ebnf(const fs::path &doc) { + std::ifstream in(doc); + std::stringstream ss; + ss << in.rdbuf(); + const std::string text = ss.str(); + const auto open = text.find("```ebnf\n"); + ebnf g; + if (open == std::string::npos) + return g; + const auto close = text.find("```", open + 8); + std::string body = text.substr(open + 8, close - open - 8); + + for (auto c = body.find("/*"); c != std::string::npos; c = body.find("/*")) // strip comments + body.erase(c, body.find("*/", c) + 2 - c); + + auto is_name_char = [](char ch) { return (ch >= 'a' && ch <= 'z') || ch == '_'; }; + std::istringstream lines(body); + std::string line, current; + while (std::getline(lines, line)) { + std::size_t i = 0; + const auto def = line.find("::="); + if (def != std::string::npos) { + std::size_t end = line.find_first_of(" \t"); + current = line.substr(0, end); + ++g.defined[current]; + i = def + 3; + } + while (i < line.size()) { + const char ch = line[i]; + if (ch == '\'' || ch == '"') { + i = line.find(ch, i + 1) + 1; + } else if (ch == '[') { + i = line.find(']', i + 1) + 1; + } else if (ch == '#' && i + 1 < line.size() && line[i + 1] == 'x') { + i += 2; + while (i < line.size() && std::isxdigit(static_cast(line[i]))) + ++i; + } else if (is_name_char(ch)) { + std::size_t j = i; + while (j < line.size() && is_name_char(line[j])) + ++j; + g.uses[current].insert(line.substr(i, j - i)); + i = j; + } else { + ++i; + } + } + } + return g; +} + +} // namespace + +TEST(StateExecGrammar, CasesForLandedPhasesHold) { + for (const grammar_case &c : cases()) { + if (c.phase > kLandedPhase) + continue; + EXPECT_PRED2(matches, run(c), c.expect) << describe(c); + } +} + +TEST(StateExecGrammar, CasesForPendingPhasesStillDiffer) { + for (const grammar_case &c : cases()) { + if (c.phase <= kLandedPhase) + continue; + EXPECT_FALSE(matches(run(c), c.expect)) + << describe(c) << "\nalready holds; set its phase to the one that made it hold"; + } +} + +TEST(StateExecGrammar, DocGrammarIsClosed) { + const ebnf g = read_ebnf(grammar_doc()); + ASSERT_GT(g.defined.size(), 40u) << "no ```ebnf block found in " << grammar_doc(); + std::set referenced; + for (const auto &[name, names] : g.uses) + referenced.insert(names.begin(), names.end()); + for (const auto &[name, count] : g.defined) { + EXPECT_EQ(count, 1) << name << " is defined more than once"; + if (name != "program" && name != "single_datum") + EXPECT_TRUE(referenced.count(name)) << name << " is defined but never used"; + } + for (const std::string &name : referenced) + EXPECT_TRUE(g.defined.count(name)) << name << " is used but never defined"; +} + +TEST(StateExecGrammar, EveryProductionHasACase) { + const ebnf g = read_ebnf(grammar_doc()); + std::set covered; + for (const grammar_case &c : cases()) { + std::istringstream names(c.rules); + for (std::string n; names >> n;) { + EXPECT_TRUE(g.defined.count(n)) << "case names unknown production '" << n << "'"; + covered.insert(n); + } + } + for (const auto &[name, count] : g.defined) + EXPECT_TRUE(covered.count(name)) << name << " has no case"; +} From 1381d6de5121acb49d155668c645203fa714c726 Mon Sep 17 00:00:00 2001 From: Joe Rivera Date: Sat, 17 Oct 2026 00:00:28 -0500 Subject: [PATCH 02/17] state_exec grammar: make the nesting limit configurable (default 256) Spec only: parse_options::max_depth on parse()/parse_all()/parser and execute_options::max_parse_depth for submitted scripts, both from P2. Records why the limit exists (the reader recurses on the native stack; the evaluators are stackless) and the corpus's real depths (max 13, median 2). --- docs/STATE_EXEC_GRAMMAR.md | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/docs/STATE_EXEC_GRAMMAR.md b/docs/STATE_EXEC_GRAMMAR.md index c979d0abb..cef1095a9 100644 --- a/docs/STATE_EXEC_GRAMMAR.md +++ b/docs/STATE_EXEC_GRAMMAR.md @@ -123,9 +123,20 @@ raw_bytes ::= 'br' raw_body - **Positions.** `parse_error::line()` and `col()` are 1-based, and point at the character where the error was found, such as the non-hex character in `"\x4"`. A new line starts after LF, after CR, and once after CRLF. Columns count code points, not bytes. -- **Nesting.** At most 256 lists and quotes may be nested. Deeper input is an error, not a stack - overflow. The limit is sized for the 64 KiB default stack of a wasm build; real scripts nest far - less. +- **Nesting.** Lists and quotes may nest at most `max_depth` levels, 256 by default. + - `(a (b (c)))` is three levels, and `'x` counts as one, since it reads as `(quote x)`. + - The parser recurses on the native stack once per level. So deeper input is a `parse_error`, + not a stack overflow, even when the script comes from an untrusted source (a message, an HTTP + body, the multiprocess wire). The evaluators are stackless, so this limit is only about + reading. + - The default fits the 64 KiB default stack of a wasm build. Real scripts nest far less: the + deepest in the corpus is 13 levels, and the median is 2. + - The limit is configurable, from P2 on: + - `parse_options::max_depth`, an optional argument to `parse()`, `parse_all()` and `parser`; + - `execute_options::max_parse_depth`, for scripts submitted to a scheduler or coordinator, + next to `max_steps` and `max_memory`. 0 means the default. + + A host that raises it must give the parsing thread a big enough stack. ## Data @@ -228,7 +239,7 @@ strings. Every row is a case in `state_exec_grammar_test`. | `[a]`, `a,b`, `` `a ``, `{}`, `a\|b`, `a\b` | odd one-character or glued symbols | error (reserved) | P2 | | `parse("1 2")` | `1`; the rest is ignored | error | P2 | | `'` at end of input | `(quote nil)` | error | P2 | -| 257 nested lists | parsed | error | P2 | +| 257 nested lists | parsed (no limit) | error at the default `max_depth` | P2 | | `; c` CR `1` | comment runs to LF, so no datum | `1` | P2 | | `0x1F` | `31.0`, a double (via `strtod`) | `31` | P2 | | `#x1F`, `0o17`, `0b101`, `1_000` | symbols | integers | P2 | From c6486e242477963759968ce06f49b83dd3904cac Mon Sep 17 00:00:00 2001 From: Joe Rivera Date: Sat, 17 Oct 2026 11:07:21 -0500 Subject: [PATCH 03/17] state_exec grammar: parser settings live on the state tree Spec only. max_depth resolves from state_exec.schedulers..parser.max_depth, then state_exec.defaults.parser.max_depth, then 256, matching scheduler::load_settings(); parse_options remains for callers without a tree. Drops the execute_options::max_parse_depth idea so the tree is the single source. --- docs/STATE_EXEC_GRAMMAR.md | 39 +++++++++++++++++++++++++++++++++----- 1 file changed, 34 insertions(+), 5 deletions(-) diff --git a/docs/STATE_EXEC_GRAMMAR.md b/docs/STATE_EXEC_GRAMMAR.md index cef1095a9..77cf67060 100644 --- a/docs/STATE_EXEC_GRAMMAR.md +++ b/docs/STATE_EXEC_GRAMMAR.md @@ -131,12 +131,41 @@ raw_bytes ::= 'br' raw_body reading. - The default fits the 64 KiB default stack of a wasm build. Real scripts nest far less: the deepest in the corpus is 13 levels, and the median is 2. - - The limit is configurable, from P2 on: - - `parse_options::max_depth`, an optional argument to `parse()`, `parse_all()` and `parser`; - - `execute_options::max_parse_depth`, for scripts submitted to a scheduler or coordinator, - next to `max_steps` and `max_memory`. 0 means the default. + - The limit is a parser setting. Like state_exec's other settings, it lives on the state tree; + see [Parser settings](#parser-settings). A host that raises it must give the parsing thread a + big enough stack. - A host that raises it must give the parsing thread a big enough stack. +## Parser settings + +**Status:** not implemented yet. Today's parser has no nesting limit and reads no settings. + +The parser's settings live on the state tree, next to the scheduler's (`state_exec.defaults.*` +and `state_exec.schedulers..*`, see `scheduler::load_settings()`). Each setting resolves in +this order: + +1. `state_exec.schedulers..parser.`, for a scheduler that has an id; +2. `state_exec.defaults.parser.`, for everything on that tree; +3. the built-in default. + +A value that is missing, empty, not a positive integer or out of range is skipped, and the next +one in the order applies. + +| Key | Meaning | Default | +|---|---|---| +| `max_depth` | how many lists and quotes may nest | 256 | + +Who reads them: + +- **`scheduler`** resolves them in `load_settings()`, like `max_pending_messages`. It publishes + the effective values under `state_exec.schedulers..parser.*` and uses them for every + script submitted to it afterwards. +- **`async_scheduler`** has no settings loader. It reads `state_exec.defaults.parser.*` from its + watch root each time a script is submitted. +- **Ariadne** reads `state_exec.defaults.parser.*` from the app's tree when it compiles a + predicate. +- **Code without a state tree** (a standalone evaluator, the `.ari` channel lint) uses the + built-in defaults, or passes `parse_options` to `parse()` and `parse_all()` explicitly. + `parse_options_from_state()` does the resolution above for any host that has a tree. ## Data From e221ac7c62086966362351dd925b55ab2b204c41 Mon Sep 17 00:00:00 2001 From: Joe Rivera Date: Sat, 17 Oct 2026 11:24:34 -0500 Subject: [PATCH 04/17] state_exec: parser settings on the state tree, with a nesting limit (grammar P1b) The parser now stops at parser.max_depth nested lists and quotes (default 256) with a parse_error, instead of recursing until the native stack overflows. Like the scheduler's other settings, the limit lives on the state tree: state_exec.schedulers..parser.max_depth, then state_exec.defaults.parser.max_depth, then 256; a missing, malformed or zero value is skipped. - parser: parse_options (max_depth) on parse()/parse_all()/parser, a depth guard on lists and quotes, and parse_options_from_state(). - scheduler: load_settings() resolves parser.* and publishes the effective value under state_exec.schedulers..parser.max_depth; execute() parses with it. - async_scheduler (the app's exec scheduler) reads the defaults key on every submit; Ariadne predicates read it from the app tree. - stackless evaluators: create_state(script, env, parse_options). The grammar test moves from a numeric kLandedPhase to named phases; the 257-deep case is P1b and holds now. Tests cover the default, a custom depth, quotes, 100000 open parens, the error position, tree resolution, bad values, and both schedulers. --- docs/STATE_EXEC_GRAMMAR.md | 12 +- .../state_exec/async_stackless_evaluator.h | 3 +- inc/cvc/state/state_exec/parser.h | 30 ++- inc/cvc/state/state_exec/scheduler.h | 9 + .../state/state_exec/stackless_evaluator.h | 7 +- src/cvc/ariadne/ariadne.cpp | 6 +- src/cvc/state/state_exec/async_scheduler.cpp | 5 +- .../state_exec/async_stackless_evaluator.cpp | 5 +- src/cvc/state/state_exec/parser.cpp | 62 +++++- src/cvc/state/state_exec/scheduler.cpp | 6 +- .../state/state_exec/stackless_evaluator.cpp | 5 +- src/cvc/tests/state_exec_grammar_test.cpp | 178 +++++++++--------- src/cvc/tests/state_exec_integration_test.cpp | 71 +++++++ src/cvc/tests/state_exec_parser_test.cpp | 48 +++++ 14 files changed, 333 insertions(+), 114 deletions(-) diff --git a/docs/STATE_EXEC_GRAMMAR.md b/docs/STATE_EXEC_GRAMMAR.md index 77cf67060..38b0426aa 100644 --- a/docs/STATE_EXEC_GRAMMAR.md +++ b/docs/STATE_EXEC_GRAMMAR.md @@ -5,7 +5,8 @@ This is the formal grammar of the state_exec s-expression language: the text tha `.ari` documents embed it in `init`, `on`, `bind` and `*_when` fields. **Status: target grammar.** The parser in `src/cvc/state/state_exec/parser.cpp` is still the -hand-written one. It moves onto this grammar in phases (P2 to P4, see +hand-written one, now with the nesting limit and its [settings](#parser-settings) (P1b). It +moves onto this grammar in phases (P2 to P4, see [Changes from the pre-grammar parser](#changes-from-the-pre-grammar-parser)). Until then it differs from this document exactly where that table says. `state_exec_grammar_test` checks both sides: cases for landed phases must hold, and cases for pending phases must still differ. @@ -137,8 +138,6 @@ raw_bytes ::= 'br' raw_body ## Parser settings -**Status:** not implemented yet. Today's parser has no nesting limit and reads no settings. - The parser's settings live on the state tree, next to the scheduler's (`state_exec.defaults.*` and `state_exec.schedulers..*`, see `scheduler::load_settings()`). Each setting resolves in this order: @@ -258,7 +257,8 @@ hashes and `"`, it is a string literal, and leaving it unterminated is an error. ## Changes from the pre-grammar parser -Phase P2 replaces the hand-written lexer with the grammar, including the number syntax. P3 adds +Phase P1b adds the nesting limit and the parser settings to the hand-written parser. Phase P2 +replaces the hand-written lexer with the grammar, including the number syntax. P3 adds the Unicode source rules and strict escapes. P4 adds octal escapes, the extra C escapes and raw strings. Every row is a case in `state_exec_grammar_test`. @@ -268,7 +268,7 @@ strings. Every row is a case in `state_exec_grammar_test`. | `[a]`, `a,b`, `` `a ``, `{}`, `a\|b`, `a\b` | odd one-character or glued symbols | error (reserved) | P2 | | `parse("1 2")` | `1`; the rest is ignored | error | P2 | | `'` at end of input | `(quote nil)` | error | P2 | -| 257 nested lists | parsed (no limit) | error at the default `max_depth` | P2 | +| 257 nested lists | parsed (no limit) | error at the default `max_depth` | P1b | | `; c` CR `1` | comment runs to LF, so no datum | `1` | P2 | | `0x1F` | `31.0`, a double (via `strtod`) | `31` | P2 | | `#x1F`, `0o17`, `0b101`, `1_000` | symbols | integers | P2 | @@ -316,7 +316,7 @@ These were considered and left out. Each would need its own decision: `single_datum` is used; - every production has at least one accept or reject case; - each case holds once its phase has landed, and still differs while its phase is pending. - `kLandedPhase` in the test says which phases have landed. + `kLanded` in the test lists the phases that have landed. - `src/cvc/tests/state_exec_corpus_test.cpp` parses the corpus: every DSL script the repo carries, drawn from the `.ari` files, the docs' `lisp` blocks, the C++ test sources and the pycvc tests. It compares each parse with `state_exec_corpus/expected.txt`, so a parser change cannot alter diff --git a/inc/cvc/state/state_exec/async_stackless_evaluator.h b/inc/cvc/state/state_exec/async_stackless_evaluator.h index 5be65c1d9..4aee7c92d 100644 --- a/inc/cvc/state/state_exec/async_stackless_evaluator.h +++ b/inc/cvc/state/state_exec/async_stackless_evaluator.h @@ -36,7 +36,8 @@ class async_stackless_evaluator { /// Create a fresh evaluator_state (delegates to inner evaluator). evaluator_state create_state(const value_t &expr, environment_ptr env = nullptr); - evaluator_state create_state(const std::string &script, environment_ptr env = nullptr); + evaluator_state create_state(const std::string &script, environment_ptr env = nullptr, + const parse_options &popts = {}); /// Single step as a coroutine — yields after the step. task step(evaluator_state &state); diff --git a/inc/cvc/state/state_exec/parser.h b/inc/cvc/state/state_exec/parser.h index 8a96aec50..b231f1403 100644 --- a/inc/cvc/state/state_exec/parser.h +++ b/inc/cvc/state/state_exec/parser.h @@ -11,12 +11,17 @@ #ifndef CVC_STATE_EXEC_PARSER_H #define CVC_STATE_EXEC_PARSER_H +#include #include #include #include #include #include +namespace cvc { +class state; +} + namespace cvc::state_exec { /// Parse error with location information. @@ -33,20 +38,35 @@ class parse_error : public std::runtime_error { std::size_t _col; }; +/// Parser settings ("Parser settings" in docs/STATE_EXEC_GRAMMAR.md). Hosts keep them on the +/// state tree under `parser.*`; parse_options_from_state() reads them from there. +struct parse_options { + /// How many lists and quotes may nest (`'x` reads as `(quote x)`, one level). The parser + /// recurses on the native stack once per level, so deeper input is a parse_error rather than a + /// stack overflow. State-tree key: `parser.max_depth`. + std::size_t max_depth = 256; +}; + +/// Resolve parser settings from a state tree, per key: `state_exec.schedulers.. +/// parser.` (when scheduler_id is not empty), then `state_exec.defaults.parser.`, then +/// the parse_options default. A value that is missing, empty, not a positive integer or out of +/// range is skipped. +parse_options parse_options_from_state(cvc::state &root, const std::string &scheduler_id = {}); + /// Parse a single S-expression from the input string. /// Returns the parsed value_t. /// @throws parse_error on syntax errors. -value_t parse(const std::string &input); +value_t parse(const std::string &input, const parse_options &opts = {}); /// Parse all S-expressions from the input string. /// Returns a vector of parsed value_t values. /// @throws parse_error on syntax errors. -std::vector parse_all(const std::string &input); +std::vector parse_all(const std::string &input, const parse_options &opts = {}); /// Internal tokenizer/parser — exposed for testing. class parser { public: - explicit parser(std::string_view input); + explicit parser(std::string_view input, const parse_options &opts = {}); /// Parse one expression. Returns nil if at end-of-input. value_t parse_expr(); @@ -58,10 +78,14 @@ class parser { std::size_t col() const { return _col; } private: + class depth_guard; + std::string_view _input; + parse_options _opts; std::size_t _pos = 0; std::size_t _line = 1; std::size_t _col = 1; + std::size_t _depth = 0; // lists and quotes open around the current position char peek() const; char advance(); diff --git a/inc/cvc/state/state_exec/scheduler.h b/inc/cvc/state/state_exec/scheduler.h index 4bb9e5989..e69ecd20c 100644 --- a/inc/cvc/state/state_exec/scheduler.h +++ b/inc/cvc/state/state_exec/scheduler.h @@ -180,8 +180,16 @@ class scheduler : public scheduler_base { /// After resolution, effective values are published back to the /// per-scheduler subtree so they are visible in the state tree. /// Requires set_watch_root() and (for per-scheduler) set_id() first. + /// + /// Settings: `max_pending_messages`, and the parser's `parser.*` + /// (parse_options_from_state), which apply to every script submitted + /// afterwards. void load_settings(); + /// The parser settings execute() parses scripts with (defaults until + /// load_settings() runs). + const parse_options &parser_settings() const { return parse_options_; } + /// Well-known state tree paths for scheduler configuration. static constexpr const char *settings_root = "state_exec"; static constexpr const char *defaults_prefix = "state_exec.defaults"; @@ -190,6 +198,7 @@ class scheduler : public scheduler_base { private: std::string id_; scheduling_policy policy_; + parse_options parse_options_; int next_pid_ = 1; int rr_index_ = 0; bool running_ = false; diff --git a/inc/cvc/state/state_exec/stackless_evaluator.h b/inc/cvc/state/state_exec/stackless_evaluator.h index 3ce531dd2..17a6b1b08 100644 --- a/inc/cvc/state/state_exec/stackless_evaluator.h +++ b/inc/cvc/state/state_exec/stackless_evaluator.h @@ -15,6 +15,7 @@ #include #include #include +#include // parse_options #include #include #include @@ -92,8 +93,10 @@ class stackless_evaluator { /// Create a fresh evaluator_state from a parsed expression. evaluator_state create_state(const value_t &expr, environment_ptr env = nullptr); - /// Create a fresh evaluator_state from a script string. - evaluator_state create_state(const std::string &script, environment_ptr env = nullptr); + /// Create a fresh evaluator_state from a script string, parsed with `popts` (a scheduler passes + /// the parser settings it resolved from the state tree). + evaluator_state create_state(const std::string &script, environment_ptr env = nullptr, + const parse_options &popts = {}); /// Execute one evaluation step. Returns true when done. bool step(evaluator_state &state); diff --git a/src/cvc/ariadne/ariadne.cpp b/src/cvc/ariadne/ariadne.cpp index 0660e8b38..85657179c 100644 --- a/src/cvc/ariadne/ariadne.cpp +++ b/src/cvc/ariadne/ariadne.cpp @@ -374,6 +374,7 @@ class ReactiveEngine { proc_->pid = 1; proc_->status = se::process_status::ready; cvc::state &root = cvc::state::instance(app); + tree_ = &root; ctx_.sched = &sched_; ctx_.tracker = &tracker_; ctx_.proc = proc_; @@ -577,7 +578,7 @@ class ReactiveEngine { if (it != compiled_.end()) return it->second ? &*it->second : nullptr; try { - se::value_t v = se::parse(src); + se::value_t v = se::parse(src, se::parse_options_from_state(*tree_)); // parser.* settings auto ins = compiled_.emplace(src, std::move(v)); return &*ins.first->second; } catch (const se::parse_error &) { @@ -592,7 +593,8 @@ class ReactiveEngine { static constexpr std::size_t kMaxOptions = 1024; // cap on computed combo options se::time_budget frame_{kFrameBudgetSeconds}; // the current frame's predicate evals' time - std::string root_path_; // §12: the scope the ctx_ is currently chrooted to (set_root_path) + std::string root_path_; // §12: the scope the ctx_ is currently chrooted to (set_root_path) + cvc::state *tree_ = nullptr; // the app's state tree, for state_exec.defaults.parser.* se::scheduler sched_; se::memory_tracker tracker_; std::shared_ptr proc_; diff --git a/src/cvc/state/state_exec/async_scheduler.cpp b/src/cvc/state/state_exec/async_scheduler.cpp index 8895ed041..d76728284 100644 --- a/src/cvc/state/state_exec/async_scheduler.cpp +++ b/src/cvc/state/state_exec/async_scheduler.cpp @@ -37,7 +37,10 @@ int async_scheduler::execute(const std::string &script, const execute_options &o proc.signal_handlers = opts.signal_handlers; proc.on_complete = opts.on_complete; proc.create_time = std::chrono::steady_clock::now(); - proc.state = evaluator_.create_state(script, opts.env); + // No load_settings() here: read the parser settings from the tree on every submission. + const parse_options popts = + watch_root_ ? parse_options_from_state(*watch_root_) : parse_options{}; + proc.state = evaluator_.create_state(script, opts.env, popts); proc.state.stats.start(); processes_.emplace(pid, std::make_shared(std::move(proc))); return pid; diff --git a/src/cvc/state/state_exec/async_stackless_evaluator.cpp b/src/cvc/state/state_exec/async_stackless_evaluator.cpp index 813283d2f..2c1007cf6 100644 --- a/src/cvc/state/state_exec/async_stackless_evaluator.cpp +++ b/src/cvc/state/state_exec/async_stackless_evaluator.cpp @@ -28,8 +28,9 @@ evaluator_state async_stackless_evaluator::create_state(const value_t &expr, env } evaluator_state async_stackless_evaluator::create_state(const std::string &script, - environment_ptr env) { - return inner_.create_state(script, env); + environment_ptr env, + const parse_options &popts) { + return inner_.create_state(script, env, popts); } // --------------------------------------------------------------------------- diff --git a/src/cvc/state/state_exec/parser.cpp b/src/cvc/state/state_exec/parser.cpp index d69c2c8c0..3ad94118a 100644 --- a/src/cvc/state/state_exec/parser.cpp +++ b/src/cvc/state/state_exec/parser.cpp @@ -1,22 +1,25 @@ #include #include #include +#include #include +#include // defaults_prefix / sched_prefix #include // SE-2: \u/\x escapes -> UTF-8, and UTF-8 identifier bytes +#include #include namespace cvc::state_exec { // -- Free functions ---------------------------------------------------------- -value_t parse(const std::string &input) { - parser p(input); +value_t parse(const std::string &input, const parse_options &opts) { + parser p(input, opts); auto result = p.parse_expr(); return result; } -std::vector parse_all(const std::string &input) { - parser p(input); +std::vector parse_all(const std::string &input, const parse_options &opts) { + parser p(input, opts); std::vector results; while (!p.at_end()) { results.push_back(p.parse_expr()); @@ -24,9 +27,53 @@ std::vector parse_all(const std::string &input) { return results; } +// A positive integer setting at `path`, or nullopt when it is missing, empty, malformed, zero or +// out of range (so the caller falls through to the next place in the resolution order). +static std::optional read_positive_setting(cvc::state &root, const std::string &path) { + cvc::state *node = root.findDescendant(path); + if (!node) + return std::nullopt; + const std::string v = node->value(); + std::size_t n = 0; + const auto [end, ec] = std::from_chars(v.data(), v.data() + v.size(), n); + if (ec != std::errc{} || end != v.data() + v.size() || n == 0) + return std::nullopt; + return n; +} + +parse_options parse_options_from_state(cvc::state &root, const std::string &scheduler_id) { + parse_options opts; + const std::string key = ".parser.max_depth"; + if (auto v = read_positive_setting(root, std::string(scheduler::defaults_prefix) + key)) + opts.max_depth = *v; + if (!scheduler_id.empty()) + if (auto v = read_positive_setting(root, std::string(scheduler::sched_prefix) + "." + + scheduler_id + key)) + opts.max_depth = *v; + return opts; +} + // -- parser ------------------------------------------------------------------ -parser::parser(std::string_view input) : _input(input) {} +parser::parser(std::string_view input, const parse_options &opts) : _input(input), _opts(opts) {} + +// One level of list/quote nesting for the lifetime of the guard. The check comes before the +// increment, so a guard that throws leaves the depth unchanged. +class parser::depth_guard { +public: + explicit depth_guard(parser &p) : _p(p) { + if (_p._depth >= _p._opts.max_depth) + _p.error("nesting deeper than " + std::to_string(_p._opts.max_depth) + + " levels (state_exec setting parser.max_depth)"); + ++_p._depth; + } + ~depth_guard() { --_p._depth; } + depth_guard(const depth_guard &) = delete; + depth_guard &operator=(const depth_guard &) = delete; + +private: + parser &_p; +}; bool parser::at_end() const { // Skip whitespace conceptually @@ -88,10 +135,13 @@ value_t parser::parse_expr() { char c = peek(); - if (c == '(') + if (c == '(') { + depth_guard level(*this); return parse_list(); + } if (c == '\'') { // Quote: 'expr → (quote expr) + depth_guard level(*this); advance(); auto quoted = parse_expr(); return make_list({value_t(symbol{"quote"}), quoted}); diff --git a/src/cvc/state/state_exec/scheduler.cpp b/src/cvc/state/state_exec/scheduler.cpp index a62aacca0..28f2584c0 100644 --- a/src/cvc/state/state_exec/scheduler.cpp +++ b/src/cvc/state/state_exec/scheduler.cpp @@ -55,10 +55,14 @@ void scheduler::load_settings() { max_pending_messages = effective; + // --- parser.* (same resolution order; see parse_options_from_state) --- + parse_options_ = parse_options_from_state(*watch_root_, id_); + // Publish effective values to per-scheduler subtree if (!id_.empty()) { std::string base = std::string(sched_prefix) + "." + id_; (*watch_root_)(base + ".max_pending_messages").value(std::to_string(max_pending_messages)); + (*watch_root_)(base + ".parser.max_depth").value(std::to_string(parse_options_.max_depth)); (*watch_root_)(base + ".policy") .value(policy_ == scheduling_policy::round_robin ? "round_robin" : policy_ == scheduling_policy::priority ? "priority" @@ -89,7 +93,7 @@ int scheduler::execute(const std::string &script, const execute_options &opts) { proc.signal_handlers = opts.signal_handlers; proc.on_complete = opts.on_complete; proc.create_time = std::chrono::steady_clock::now(); - proc.state = evaluator_.create_state(script, opts.env); + proc.state = evaluator_.create_state(script, opts.env, parse_options_); proc.state.stats.start(); processes_.emplace(pid, std::make_shared(std::move(proc))); return pid; diff --git a/src/cvc/state/state_exec/stackless_evaluator.cpp b/src/cvc/state/state_exec/stackless_evaluator.cpp index ec55b1e5e..f498e79bd 100644 --- a/src/cvc/state/state_exec/stackless_evaluator.cpp +++ b/src/cvc/state/state_exec/stackless_evaluator.cpp @@ -39,8 +39,9 @@ evaluator_state stackless_evaluator::create_state(const value_t &expr, environme return st; } -evaluator_state stackless_evaluator::create_state(const std::string &script, environment_ptr env) { - auto exprs = parse_all(script); +evaluator_state stackless_evaluator::create_state(const std::string &script, environment_ptr env, + const parse_options &popts) { + auto exprs = parse_all(script, popts); if (exprs.empty()) { evaluator_state st; st.done = true; diff --git a/src/cvc/tests/state_exec_grammar_test.cpp b/src/cvc/tests/state_exec_grammar_test.cpp index f8b0ca1a2..7653b6ab4 100644 --- a/src/cvc/tests/state_exec_grammar_test.cpp +++ b/src/cvc/tests/state_exec_grammar_test.cpp @@ -1,13 +1,14 @@ // Conformance tests for the state_exec grammar (docs/STATE_EXEC_GRAMMAR.md). // // Every case names the EBNF productions it exercises and the phase from which it holds: -// 0 holds for the pre-grammar (hand-written) parser too; -// N holds once phase PN of the grammar work lands (P2 = parser built from the grammar, -// P3 = Unicode source rules + strict escapes, P4 = octal escapes, C escapes, raw strings). -// Cases for landed phases (<= kLandedPhase) must hold. Cases for pending phases must still -// DIFFER, so the table stays an exact list of what each phase changes; when a phase lands, bump -// kLandedPhase in the same PR. The doc's EBNF is checked for closure, and every production must -// have a case. +// P0 holds for the pre-grammar (hand-written) parser too; +// P1b parser settings on the state tree (the nesting limit); +// P2 parser built from the grammar, numbers included; +// P3 Unicode source rules + strict escapes; +// P4 octal escapes, C escapes, raw strings. +// Cases for landed phases (kLanded) must hold. Cases for pending phases must still DIFFER, so +// the table stays an exact list of what each phase changes; add a phase to kLanded in the PR that +// lands it. The doc's EBNF is checked for closure, and every production must have a case. // // Inputs and expectations are byte-exact: non-ASCII is written as \xNN escapes (MSVC builds // without /utf-8), split with "" where a hex letter follows an escape. @@ -29,12 +30,13 @@ namespace dump = cvc::state_exec::grammar_dump; namespace { -constexpr int kLandedPhase = 0; +// Phases whose cases must hold. Add a phase here in the PR that lands it. +const std::set kLanded = {"P0", "P1b"}; enum class mode { all, one }; // parse_all() or parse() struct grammar_case { - int phase; + std::string phase; // "P0", "P1b", "P2", ... std::string rules; // space-separated EBNF production names mode how; std::string input; @@ -46,112 +48,112 @@ std::string nest(int depth) { return std::string(depth, '(') + std::string(depth const std::vector &cases() { static const std::vector kCases = { // --- holds today ---------------------------------------------------------------------- - {0, "program", mode::all, "", ""}, - {0, "program atmosphere", mode::all, "1 2\n3", "1\n2\n3"}, - {0, "single_datum", mode::one, "", "nil"}, - {0, "single_datum line_comment", mode::one, " ; note\n 42 ", "42"}, - {0, "datum list", mode::all, "(a (b c) ())", "(a (b c) ())"}, - {0, "list", mode::all, nest(256), nest(256)}, - {0, "list", mode::all, "(a", "!error"}, - {0, "quoted", mode::all, "'x ' (1 2) ''a", "(quote x)\n(quote (1 2))\n(quote (quote a))"}, - {0, "line_comment line_end", mode::all, "; c\n(a ; c2\n b)", "(a b)"}, - {0, "whitespace", mode::all, "\t\n\v\f\r a", "a"}, - {0, "text_literal text_string string_char", mode::all, "\"hi there\"", "\"hi there\""}, - {0, "string_char", mode::all, "\"a\nb\"", R"("a\nb")"}, - {0, "text_string", mode::all, "\"abc", "!error"}, - {0, "text_escape simple_escape", mode::all, R"("\n\t\r\\\"")", R"("\n\t\r\\\"")"}, - {0, "hex_escape hex", mode::all, R"("\x41\xe9")", "\"A\xC3\xA9\""}, - {0, "hex_escape", mode::all, R"("\xZ1")", "!error"}, - {0, "unicode_escape", mode::all, R"("é\u{1F600}")", "\"\xC3\xA9\xF0\x9F\x98\x80\""}, - {0, "unicode_escape", mode::all, R"("\u12")", "!error"}, - {0, "unicode_escape", mode::all, R"("\u{}")", "!error"}, - {0, "unicode_escape", mode::all, R"("\u{1234567}")", "!error"}, - {0, "octal_escape", mode::all, R"("\0")", R"("\x00")"}, - {0, "bytes_literal byte_string byte_escape", mode::all, R"(b"ab\x00\xff\n")", + {"P0", "program", mode::all, "", ""}, + {"P0", "program atmosphere", mode::all, "1 2\n3", "1\n2\n3"}, + {"P0", "single_datum", mode::one, "", "nil"}, + {"P0", "single_datum line_comment", mode::one, " ; note\n 42 ", "42"}, + {"P0", "datum list", mode::all, "(a (b c) ())", "(a (b c) ())"}, + {"P0", "list", mode::all, nest(256), nest(256)}, + {"P0", "list", mode::all, "(a", "!error"}, + {"P0", "quoted", mode::all, "'x ' (1 2) ''a", "(quote x)\n(quote (1 2))\n(quote (quote a))"}, + {"P0", "line_comment line_end", mode::all, "; c\n(a ; c2\n b)", "(a b)"}, + {"P0", "whitespace", mode::all, "\t\n\v\f\r a", "a"}, + {"P0", "text_literal text_string string_char", mode::all, "\"hi there\"", "\"hi there\""}, + {"P0", "string_char", mode::all, "\"a\nb\"", R"("a\nb")"}, + {"P0", "text_string", mode::all, "\"abc", "!error"}, + {"P0", "text_escape simple_escape", mode::all, R"("\n\t\r\\\"")", R"("\n\t\r\\\"")"}, + {"P0", "hex_escape hex", mode::all, R"("\x41\xe9")", "\"A\xC3\xA9\""}, + {"P0", "hex_escape", mode::all, R"("\xZ1")", "!error"}, + {"P0", "unicode_escape", mode::all, R"("é\u{1F600}")", "\"\xC3\xA9\xF0\x9F\x98\x80\""}, + {"P0", "unicode_escape", mode::all, R"("\u12")", "!error"}, + {"P0", "unicode_escape", mode::all, R"("\u{}")", "!error"}, + {"P0", "unicode_escape", mode::all, R"("\u{1234567}")", "!error"}, + {"P0", "octal_escape", mode::all, R"("\0")", R"("\x00")"}, + {"P0", "bytes_literal byte_string byte_escape", mode::all, R"(b"ab\x00\xff\n")", R"(b"ab\x00\xff\x0a")"}, - {0, "byte_string", mode::all, "b\"\xC3\xA9\"", R"(b"\xc3\xa9")"}, - {0, "byte_string symbol", mode::all, "b (b) b\"\"", "b\n(b)\nb\"\""}, - {0, "boolean", mode::all, "#t true", "#t\n#t"}, - {0, "nil", mode::all, "#f false nil", "nil\nnil\nnil"}, - {0, "number integer dec_digits sign", mode::all, "42 -7 007 -9223372036854775808", + {"P0", "byte_string", mode::all, "b\"\xC3\xA9\"", R"(b"\xc3\xa9")"}, + {"P0", "byte_string symbol", mode::all, "b (b) b\"\"", "b\n(b)\nb\"\""}, + {"P0", "boolean", mode::all, "#t true", "#t\n#t"}, + {"P0", "nil", mode::all, "#f false nil", "nil\nnil\nnil"}, + {"P0", "number integer dec_digits sign", mode::all, "42 -7 007 -9223372036854775808", "42\n-7\n7\n-9223372036854775808"}, - {0, "decimal_float exponent", mode::all, "1.5 .5 1. 1e3 -2.5E-3", + {"P0", "decimal_float exponent", mode::all, "1.5 .5 1. 1e3 -2.5E-3", "1.5\n0.5\n1.0\n1000.0\n-0.0025"}, - {0, "atom token constituent symbol", mode::all, + {"P0", "atom token constituent symbol", mode::all, "foo-bar? -> + - caf\xC3\xA9 \xCE\xBB \xE6\x97\xA5\xE6\x9C\xAC a.b:c", "foo-bar?\n->\n+\n-\ncaf\xC3\xA9\n\xCE\xBB\n\xE6\x97\xA5\xE6\x9C\xAC\na.b:c"}, - {0, "symbol", mode::all, "1+ ... #foo", "|1+|\n|...|\n|#foo|"}, - {0, "delimiter constituent", mode::all, "a\"b\"(c)'d;e", "a\n\"b\"\n(c)\n(quote d)"}, - {0, "raw_text symbol", mode::all, "r#foo", "r#foo"}, + {"P0", "symbol", mode::all, "1+ ... #foo", "|1+|\n|...|\n|#foo|"}, + {"P0", "delimiter constituent", mode::all, "a\"b\"(c)'d;e", "a\n\"b\"\n(c)\n(quote d)"}, + {"P0", "raw_text symbol", mode::all, "r#foo", "r#foo"}, // --- P2: parser built from the grammar, numbers included ----------------------------- - {2, "program", mode::all, "a)", "!error"}, - {2, "reserved", mode::all, "[a]", "!error"}, - {2, "reserved", mode::all, "a,b", "!error"}, - {2, "reserved", mode::all, "`a", "!error"}, - {2, "reserved", mode::all, "{}", "!error"}, - {2, "reserved", mode::all, "a|b", "!error"}, - {2, "reserved", mode::all, "a\\b", "!error"}, - {2, "single_datum", mode::one, "1 2", "!error"}, - {2, "quoted", mode::all, "'", "!error"}, - {2, "list", mode::all, nest(257), "!error"}, - {2, "line_end line_comment", mode::all, "; c\r1", "1"}, - {2, "radix_integer hex_digits", mode::all, "0x1F 0X1f #x1F #X1f 0xFF_FF", + {"P2", "program", mode::all, "a)", "!error"}, + {"P2", "reserved", mode::all, "[a]", "!error"}, + {"P2", "reserved", mode::all, "a,b", "!error"}, + {"P2", "reserved", mode::all, "`a", "!error"}, + {"P2", "reserved", mode::all, "{}", "!error"}, + {"P2", "reserved", mode::all, "a|b", "!error"}, + {"P2", "reserved", mode::all, "a\\b", "!error"}, + {"P2", "single_datum", mode::one, "1 2", "!error"}, + {"P2", "quoted", mode::all, "'", "!error"}, + {"P1b", "list", mode::all, nest(257), "!error"}, + {"P2", "line_end line_comment", mode::all, "; c\r1", "1"}, + {"P2", "radix_integer hex_digits", mode::all, "0x1F 0X1f #x1F #X1f 0xFF_FF", "31\n31\n31\n31\n65535"}, - {2, "radix_integer oct_digits bin_digits sign", mode::all, "0o17 #o17 0b101 #b101 -0x10 +0b1", - "15\n15\n5\n5\n-16\n1"}, - {2, "radix_prefixed", mode::all, "0xZZ", "!error"}, - {2, "radix_prefixed", mode::all, "#bar", "!error"}, - {2, "radix_prefixed", mode::all, "0x", "!error"}, - {2, "integer sign", mode::all, "+5", "5"}, - {2, "dec_digits", mode::all, "1_000", "1000"}, - {2, "integer", mode::all, "9223372036854775808", "!error"}, - {2, "float_special", mode::all, "+inf.0 -inf.0 +nan.0 -nan.0", + {"P2", "radix_integer oct_digits bin_digits sign", mode::all, + "0o17 #o17 0b101 #b101 -0x10 +0b1", "15\n15\n5\n5\n-16\n1"}, + {"P2", "radix_prefixed", mode::all, "0xZZ", "!error"}, + {"P2", "radix_prefixed", mode::all, "#bar", "!error"}, + {"P2", "radix_prefixed", mode::all, "0x", "!error"}, + {"P2", "integer sign", mode::all, "+5", "5"}, + {"P2", "dec_digits", mode::all, "1_000", "1000"}, + {"P2", "integer", mode::all, "9223372036854775808", "!error"}, + {"P2", "float_special", mode::all, "+inf.0 -inf.0 +nan.0 -nan.0", "+inf.0\n-inf.0\n+nan.0\n+nan.0"}, - {2, "symbol", mode::all, "inf nan infinity -INF", "inf\nnan\ninfinity\n-INF"}, - {2, "decimal_float", mode::all, "1e999", "!error"}, - {2, "decimal_float dec_digits", mode::all, "6.022_140e23", "6.02214e+23"}, + {"P2", "symbol", mode::all, "inf nan infinity -INF", "inf\nnan\ninfinity\n-INF"}, + {"P2", "decimal_float", mode::all, "1e999", "!error"}, + {"P2", "decimal_float dec_digits", mode::all, "6.022_140e23", "6.02214e+23"}, // --- P3: Unicode source rules, strict escapes ---------------------------------------- - {3, "scalar_value", mode::all, + {"P3", "scalar_value", mode::all, "a\xff" "b", "!error"}, - {3, "bom", mode::all, "\xEF\xBB\xBF(a)", "(a)"}, - {3, "other_space constituent", mode::all, + {"P3", "bom", mode::all, "\xEF\xBB\xBF(a)", "(a)"}, + {"P3", "other_space constituent", mode::all, "a\xC2\xA0" "b", "!error"}, - {3, "whitespace", mode::all, + {"P3", "whitespace", mode::all, "a\xC2\x85" "b a\xE2\x80\xA8" "b", "a\nb\na\nb"}, - {3, "source_char bidi_control", mode::all, + {"P3", "source_char bidi_control", mode::all, "\"a\xE2\x80\xAE" "b\"", "!error"}, - {3, "bidi_control line_comment", mode::all, "; \xE2\x81\xA6\n1", "!error"}, - {3, "control", mode::all, + {"P3", "bidi_control line_comment", mode::all, "; \xE2\x81\xA6\n1", "!error"}, + {"P3", "control", mode::all, "\"a\x01" "b\"", "!error"}, - {3, "text_escape", mode::all, R"("\q")", "!error"}, - {3, "unicode_escape", mode::all, R"("\uD800")", "!error"}, - {3, "unicode_escape", mode::all, R"("\u{110000}")", "!error"}, + {"P3", "text_escape", mode::all, R"("\q")", "!error"}, + {"P3", "unicode_escape", mode::all, R"("\uD800")", "!error"}, + {"P3", "unicode_escape", mode::all, R"("\u{110000}")", "!error"}, // The error is reported at the offending '"', counted in code points: 1:6, not 1:8. - {3, "hex_escape", mode::all, "\"\xC3\xA9\\x4\"", "!error 1:6"}, + {"P3", "hex_escape", mode::all, "\"\xC3\xA9\\x4\"", "!error 1:6"}, // --- P4: octal escapes, C escapes, raw strings --------------------------------------- - {4, "octal_escape", mode::all, R"("\101\012")", R"("A\n")"}, - {4, "byte_escape octal_escape", mode::all, R"(b"\377")", R"(b"\xff")"}, - {4, "byte_escape", mode::all, R"(b"\400")", "!error"}, - {4, "simple_escape", mode::all, R"("\a\b\f\v")", R"("\x07\x08\x0c\x0b")"}, - {4, "raw_text raw_body raw_char", mode::all, R"(r"a\nb")", R"("a\\nb")"}, - {4, "hash_fence", mode::all, R"(r#"say "hi""#)", R"("say \"hi\"")"}, - {4, "hash_fence", mode::all, R"(r##"a"#b"##)", R"("a\"#b")"}, - {4, "raw_bytes", mode::all, R"(br"\x00")", R"(b"\\x00")"}, - {4, "raw_body", mode::all, R"(r#"abc")", "!error"}, + {"P4", "octal_escape", mode::all, R"("\101\012")", R"("A\n")"}, + {"P4", "byte_escape octal_escape", mode::all, R"(b"\377")", R"(b"\xff")"}, + {"P4", "byte_escape", mode::all, R"(b"\400")", "!error"}, + {"P4", "simple_escape", mode::all, R"("\a\b\f\v")", R"("\x07\x08\x0c\x0b")"}, + {"P4", "raw_text raw_body raw_char", mode::all, R"(r"a\nb")", R"("a\\nb")"}, + {"P4", "hash_fence", mode::all, R"(r#"say "hi""#)", R"("say \"hi\"")"}, + {"P4", "hash_fence", mode::all, R"(r##"a"#b"##)", R"("a\"#b")"}, + {"P4", "raw_bytes", mode::all, R"(br"\x00")", R"(b"\\x00")"}, + {"P4", "raw_body", mode::all, R"(r#"abc")", "!error"}, }; return kCases; } @@ -167,7 +169,7 @@ bool matches(const std::string &actual, const std::string &expect) { } std::string describe(const grammar_case &c) { - return "[P" + std::to_string(c.phase) + " " + c.rules + "] input: " + c.input; + return "[" + c.phase + " " + c.rules + "] input: " + c.input; } fs::path grammar_doc() { @@ -239,7 +241,7 @@ ebnf read_ebnf(const fs::path &doc) { TEST(StateExecGrammar, CasesForLandedPhasesHold) { for (const grammar_case &c : cases()) { - if (c.phase > kLandedPhase) + if (!kLanded.count(c.phase)) continue; EXPECT_PRED2(matches, run(c), c.expect) << describe(c); } @@ -247,7 +249,7 @@ TEST(StateExecGrammar, CasesForLandedPhasesHold) { TEST(StateExecGrammar, CasesForPendingPhasesStillDiffer) { for (const grammar_case &c : cases()) { - if (c.phase <= kLandedPhase) + if (kLanded.count(c.phase)) continue; EXPECT_FALSE(matches(run(c), c.expect)) << describe(c) << "\nalready holds; set its phase to the one that made it hold"; diff --git a/src/cvc/tests/state_exec_integration_test.cpp b/src/cvc/tests/state_exec_integration_test.cpp index 4e8f792c4..0fc33685e 100644 --- a/src/cvc/tests/state_exec_integration_test.cpp +++ b/src/cvc/tests/state_exec_integration_test.cpp @@ -5,6 +5,7 @@ #include #include #include +#include #include #include #include @@ -2332,6 +2333,76 @@ TEST_F(SchedulerSettingsTest, TwoSchedulersWithDifferentOverrides) { EXPECT_EQ(sched_fast.max_pending_messages, 100u); EXPECT_EQ(sched_heavy.max_pending_messages, 5000u); } + +// --- parser.* settings (docs/STATE_EXEC_GRAMMAR.md, "Parser settings") --- + +TEST_F(SchedulerSettingsTest, ParserSettingsResolveFromTree) { + auto &root = cvc::state::instance(app_ctx); + EXPECT_EQ(parse_options_from_state(root).max_depth, 256u); // nothing set: built-in default + + root("state_exec.defaults.parser.max_depth").value("64"); + EXPECT_EQ(parse_options_from_state(root).max_depth, 64u); + EXPECT_EQ(parse_options_from_state(root, "other").max_depth, 64u); + + root("state_exec.schedulers.deep.parser.max_depth").value("1024"); + EXPECT_EQ(parse_options_from_state(root, "deep").max_depth, 1024u); + EXPECT_EQ(parse_options_from_state(root).max_depth, + 64u); // no id: the per-scheduler key is unused +} + +TEST_F(SchedulerSettingsTest, BadParserSettingsAreSkipped) { + auto &root = cvc::state::instance(app_ctx); + for (const char *bad : {"", "abc", "0", "-5", "12x", "99999999999999999999999"}) { + root("state_exec.defaults.parser.max_depth").value(bad); + EXPECT_EQ(parse_options_from_state(root).max_depth, 256u) << "value: " << bad; + } + // A bad per-scheduler value falls through to the defaults key, not to the built-in default. + root("state_exec.defaults.parser.max_depth").value("32"); + root("state_exec.schedulers.s.parser.max_depth").value("nope"); + EXPECT_EQ(parse_options_from_state(root, "s").max_depth, 32u); +} + +TEST_F(SchedulerSettingsTest, SchedulerParsesWithTreeDepthAndPublishesIt) { + auto &root = cvc::state::instance(app_ctx); + root("state_exec.defaults.parser.max_depth").value("8"); + root("state_exec.schedulers.strict.parser.max_depth").value("3"); + + scheduler sched; + sched.set_watch_root(&root); + sched.set_id("strict"); + EXPECT_EQ(sched.parser_settings().max_depth, 256u); // until load_settings() + sched.load_settings(); + EXPECT_EQ(sched.parser_settings().max_depth, 3u); + EXPECT_EQ(root("state_exec.schedulers.strict.parser.max_depth").value(), "3"); + + EXPECT_NO_THROW(sched.execute(std::string("(+ 1 (+ 1 (+ 1 1)))"))); // 3 levels + EXPECT_THROW(sched.execute(std::string("(+ 1 (+ 1 (+ 1 (+ 1 1))))")), parse_error); // 4 +} + +TEST_F(SchedulerSettingsTest, SchedulerWithoutIdUsesDefaultsKey) { + auto &root = cvc::state::instance(app_ctx); + root("state_exec.defaults.parser.max_depth").value("2"); + + scheduler sched; + sched.set_watch_root(&root); + sched.load_settings(); + EXPECT_EQ(sched.parser_settings().max_depth, 2u); + EXPECT_THROW(sched.execute(std::string("(+ 1 (+ 1 (+ 1 1)))")), parse_error); +} + +TEST_F(SchedulerSettingsTest, AsyncSchedulerReadsParserSettingsOnEachSubmit) { + auto &root = cvc::state::instance(app_ctx); + async_scheduler sched; + sched.set_watch_root(&root); + const std::string three_deep = "(+ 1 (+ 1 (+ 1 1)))"; + + EXPECT_NO_THROW(sched.execute(three_deep)); + root("state_exec.defaults.parser.max_depth").value("2"); + EXPECT_THROW(sched.execute(three_deep), parse_error); + root("state_exec.defaults.parser.max_depth").value("16"); + EXPECT_NO_THROW(sched.execute(three_deep)); +} + // =========================================================================== // §12 message-channel scoping — end-to-end through the scheduler + chroot // =========================================================================== diff --git a/src/cvc/tests/state_exec_parser_test.cpp b/src/cvc/tests/state_exec_parser_test.cpp index 5db1c3308..6b490eb88 100644 --- a/src/cvc/tests/state_exec_parser_test.cpp +++ b/src/cvc/tests/state_exec_parser_test.cpp @@ -238,3 +238,51 @@ TEST(StateExecParserTest, FullProgram) { EXPECT_EQ(std::get((**call)[0].v).name, "fib"); EXPECT_EQ(std::get((**call)[1].v), 10); } + +// -- Nesting limit (parse_options::max_depth; state-tree key parser.max_depth) -- + +static std::string nested(std::size_t depth) { + return std::string(depth, '(') + std::string(depth, ')'); +} + +TEST(StateExecParserTest, NestingLimitDefaultsTo256) { + EXPECT_EQ(parse_options{}.max_depth, 256u); + EXPECT_NO_THROW(parse(nested(256))); + EXPECT_THROW(parse(nested(257)), parse_error); +} + +TEST(StateExecParserTest, NestingLimitIsConfigurable) { + parse_options opts; + opts.max_depth = 3; + EXPECT_NO_THROW(parse("(((1)))", opts)); + EXPECT_THROW(parse("((((1))))", opts), parse_error); + EXPECT_NO_THROW(parse("'''x", opts)); // each quote is one level: (quote (quote (quote x))) + EXPECT_THROW(parse("''''x", opts), parse_error); + EXPECT_THROW(parse_all("(a) ((((b))))", opts), parse_error); + + opts.max_depth = 1000; + EXPECT_NO_THROW(parse(nested(1000), opts)); +} + +TEST(StateExecParserTest, NestingLimitStopsRunawayInputBeforeTheStackDoes) { + // A hundred thousand open parens would recurse that deep; the limit stops it at 257. + EXPECT_THROW(parse(std::string(100000, '(')), parse_error); +} + +TEST(StateExecParserTest, NestingErrorPointsAtTheParenThatGoesTooDeep) { + parse_options opts; + opts.max_depth = 2; + try { + parse("(a\n ((b)))", opts); + FAIL() << "expected a parse_error"; + } catch (const parse_error &e) { + EXPECT_EQ(e.line(), 2u); + EXPECT_EQ(e.col(), 4u); + } +} + +TEST(StateExecParserTest, NestingDepthResetsBetweenTopLevelForms) { + parse_options opts; + opts.max_depth = 2; + EXPECT_EQ(parse_all("((a)) ((b)) ((c))", opts).size(), 3u); +} From 7a4b4b1868690294bfbd8582b1f153ea946345bd Mon Sep 17 00:00:00 2001 From: Joe Rivera Date: Sat, 17 Oct 2026 13:02:07 -0500 Subject: [PATCH 05/17] state_exec: make the parser stackless (explicit frame stack, resumable read) Lists and quotes were parsed by native recursion, so input depth was bounded by the thread's stack. The parser now keeps open lists and quotes on a heap stack of frames and reads in a loop: native stack depth is constant however deeply the input nests (a test reads a million open parens and reports the unclosed list instead of crashing). The loop is a resumable read(out, budget): it reads up to `budget` tokens and returns datum / end / more, carrying open frames between calls, so a scheduler can parse a script a slice at a time. parse() and parse_all() call it with no budget. Output is unchanged, quirks included (a stray top-level `)` still reads as a symbol until P2): the conformance cases, the 597-script corpus and the evaluator suites all pass. parser.max_depth stays on the state tree; it now bounds parsed data for the code that still walks values recursively (list destruction, the snapshot codec, to_string) rather than the parser's own stack. parse_fib20_expr: about 3.4 us vs 3.1 us for the recursive parser (median of 6, MSVC Release, loaded machine). --- docs/STATE_EXEC_GRAMMAR.md | 16 +-- inc/cvc/state/state_exec/parser.h | 35 +++++-- src/cvc/state/state_exec/parser.cpp | 124 ++++++++++++----------- src/cvc/tests/state_exec_parser_test.cpp | 56 ++++++++++ 4 files changed, 161 insertions(+), 70 deletions(-) diff --git a/docs/STATE_EXEC_GRAMMAR.md b/docs/STATE_EXEC_GRAMMAR.md index 38b0426aa..c9e2ce74b 100644 --- a/docs/STATE_EXEC_GRAMMAR.md +++ b/docs/STATE_EXEC_GRAMMAR.md @@ -126,15 +126,17 @@ raw_bytes ::= 'br' raw_body after CR, and once after CRLF. Columns count code points, not bytes. - **Nesting.** Lists and quotes may nest at most `max_depth` levels, 256 by default. - `(a (b (c)))` is three levels, and `'x` counts as one, since it reads as `(quote x)`. - - The parser recurses on the native stack once per level. So deeper input is a `parse_error`, - not a stack overflow, even when the script comes from an untrusted source (a message, an HTTP - body, the multiprocess wire). The evaluators are stackless, so this limit is only about - reading. + - The parser is stackless. Open lists and quotes live on a heap stack, not on the native + stack, so no input depth can overflow it. + - The limit is for what happens to the data afterwards. Some code still walks values on the + native stack: destroying a nested list, the snapshot codec, `to_string`. Deeper input is a + `parse_error`, so a script from an untrusted source (a message, an HTTP body, the + multiprocess wire) can't produce data those would crash on. - The default fits the 64 KiB default stack of a wasm build. Real scripts nest far less: the - deepest in the corpus is 13 levels, and the median is 2. + deepest in the corpus is 13 levels, and the median is 2. Once that downstream code is + iterative too, the default can rise. - The limit is a parser setting. Like state_exec's other settings, it lives on the state tree; - see [Parser settings](#parser-settings). A host that raises it must give the parsing thread a - big enough stack. + see [Parser settings](#parser-settings). ## Parser settings diff --git a/inc/cvc/state/state_exec/parser.h b/inc/cvc/state/state_exec/parser.h index b231f1403..102d447c1 100644 --- a/inc/cvc/state/state_exec/parser.h +++ b/inc/cvc/state/state_exec/parser.h @@ -41,9 +41,10 @@ class parse_error : public std::runtime_error { /// Parser settings ("Parser settings" in docs/STATE_EXEC_GRAMMAR.md). Hosts keep them on the /// state tree under `parser.*`; parse_options_from_state() reads them from there. struct parse_options { - /// How many lists and quotes may nest (`'x` reads as `(quote x)`, one level). The parser - /// recurses on the native stack once per level, so deeper input is a parse_error rather than a - /// stack overflow. State-tree key: `parser.max_depth`. + /// How many lists and quotes may nest (`'x` reads as `(quote x)`, one level); deeper input is + /// a parse_error. The parser itself is stackless, but code downstream still walks values + /// recursively (destroying a nested list, the snapshot codec, to_string), so this keeps parsed + /// data within what they can handle. State-tree key: `parser.max_depth`. std::size_t max_depth = 256; }; @@ -64,6 +65,11 @@ value_t parse(const std::string &input, const parse_options &opts = {}); std::vector parse_all(const std::string &input, const parse_options &opts = {}); /// Internal tokenizer/parser — exposed for testing. +/// +/// Stackless: lists and quotes are tracked on a heap stack of open frames, not by native +/// recursion, so the native stack depth is constant however deeply the input nests. Reading is +/// also resumable (read() with a token budget), which lets a scheduler parse a script a slice at +/// a time inside a process's steps. class parser { public: explicit parser(std::string_view input, const parse_options &opts = {}); @@ -71,6 +77,19 @@ class parser { /// Parse one expression. Returns nil if at end-of-input. value_t parse_expr(); + /// What read() stopped at. + enum class read_status { + datum, ///< a top-level datum is complete; it is in `out` + end, ///< only whitespace and comments remain; nothing is open + more, ///< the token budget ran out; call read() again to continue + }; + + /// Read until a top-level datum completes, the input ends, or `budget` tokens (parens, quotes, + /// atoms, strings) have been read; `budget` is decremented by the tokens read. Open lists and + /// quotes carry over between calls. + /// @throws parse_error on syntax errors, including nesting past parse_options::max_depth. + read_status read(value_t &out, std::size_t &budget); + /// True if all input has been consumed. bool at_end() const; @@ -78,22 +97,26 @@ class parser { std::size_t col() const { return _col; } private: - class depth_guard; + /// An open list (`(` read, `)` not yet) or quote (`'` read, its datum not yet complete). + struct frame { + bool quote = false; + std::vector elements; // a list's elements so far + }; std::string_view _input; parse_options _opts; std::size_t _pos = 0; std::size_t _line = 1; std::size_t _col = 1; - std::size_t _depth = 0; // lists and quotes open around the current position + std::vector _frames; // innermost last; its size is the current nesting depth char peek() const; char advance(); void skip_whitespace_and_comments(); [[noreturn]] void error(const std::string &msg); + void open_frame(bool quote); value_t parse_atom(); - value_t parse_list(); value_t parse_string(); value_t parse_bytes_string(); // SE-3: b"..." byte-string literal (\xNN = a raw byte, no \u) value_t parse_number_or_symbol(char first); diff --git a/src/cvc/state/state_exec/parser.cpp b/src/cvc/state/state_exec/parser.cpp index 3ad94118a..340003b07 100644 --- a/src/cvc/state/state_exec/parser.cpp +++ b/src/cvc/state/state_exec/parser.cpp @@ -55,25 +55,9 @@ parse_options parse_options_from_state(cvc::state &root, const std::string &sche // -- parser ------------------------------------------------------------------ -parser::parser(std::string_view input, const parse_options &opts) : _input(input), _opts(opts) {} - -// One level of list/quote nesting for the lifetime of the guard. The check comes before the -// increment, so a guard that throws leaves the depth unchanged. -class parser::depth_guard { -public: - explicit depth_guard(parser &p) : _p(p) { - if (_p._depth >= _p._opts.max_depth) - _p.error("nesting deeper than " + std::to_string(_p._opts.max_depth) + - " levels (state_exec setting parser.max_depth)"); - ++_p._depth; - } - ~depth_guard() { --_p._depth; } - depth_guard(const depth_guard &) = delete; - depth_guard &operator=(const depth_guard &) = delete; - -private: - parser &_p; -}; +parser::parser(std::string_view input, const parse_options &opts) : _input(input), _opts(opts) { + _frames.reserve(32); // one allocation covers real scripts (the corpus nests at most 13 deep) +} bool parser::at_end() const { // Skip whitespace conceptually @@ -129,49 +113,75 @@ void parser::skip_whitespace_and_comments() { void parser::error(const std::string &msg) { throw parse_error(msg, _line, _col); } value_t parser::parse_expr() { - skip_whitespace_and_comments(); - if (_pos >= _input.size()) - return nil_value; - - char c = peek(); - - if (c == '(') { - depth_guard level(*this); - return parse_list(); - } - if (c == '\'') { - // Quote: 'expr → (quote expr) - depth_guard level(*this); - advance(); - auto quoted = parse_expr(); - return make_list({value_t(symbol{"quote"}), quoted}); - } - if (c == '"') - return parse_string(); - // SE-3: a `b` immediately followed by `"` (no space) is a bytes literal b"...". A lone `b`, or - // `b` followed by anything else, stays an ordinary symbol. - if (c == 'b' && _pos + 1 < _input.size() && _input[_pos + 1] == '"') { - advance(); // consume the 'b' prefix - return parse_bytes_string(); - } - - return parse_atom(); + _frames.clear(); // a datum left half-read by an earlier parse_error is abandoned + std::size_t unlimited = static_cast(-1); + value_t out; + return read(out, unlimited) == read_status::datum ? out : nil_value; } -value_t parser::parse_list() { - advance(); // consume '(' - std::vector elements; +// Push an open list or quote. The depth check comes first, so the error points at the `(` or `'` +// that would go one level too deep. +void parser::open_frame(bool quote) { + if (_frames.size() >= _opts.max_depth) + error("nesting deeper than " + std::to_string(_opts.max_depth) + + " levels (state_exec setting parser.max_depth)"); + _frames.push_back(frame{quote, {}}); +} - skip_whitespace_and_comments(); - while (peek() != ')') { - if (_pos >= _input.size()) - error("unclosed parenthesis"); - elements.push_back(parse_expr()); +parser::read_status parser::read(value_t &out, std::size_t &budget) { + while (budget > 0) { skip_whitespace_and_comments(); - } - advance(); // consume ')' + value_t datum; + if (_pos >= _input.size()) { + if (_frames.empty()) + return read_status::end; + if (!_frames.back().quote) + error("unclosed parenthesis"); + datum = nil_value; // a quote at end of input quotes nil (the pre-grammar behavior) + } else { + --budget; + const char c = peek(); + if (c == '(') { + open_frame(false); + advance(); + continue; + } + if (c == '\'') { // 'expr -> (quote expr) + open_frame(true); + advance(); + continue; + } + if (c == ')' && !_frames.empty() && !_frames.back().quote) { + advance(); + datum = make_list(std::move(_frames.back().elements)); + _frames.pop_back(); + } else if (c == '"') { + datum = parse_string(); + } else if (c == 'b' && _pos + 1 < _input.size() && _input[_pos + 1] == '"') { + // SE-3: a `b` immediately followed by `"` (no space) is a bytes literal b"...". A lone + // `b`, or `b` followed by anything else, stays an ordinary symbol. + advance(); // consume the 'b' prefix + datum = parse_bytes_string(); + } else { + // Everything else is an atom, including a `)` with no open list around it (it reads as + // a symbol until the grammar's P2 makes it an error). + datum = parse_atom(); + } + } - return make_list(std::move(elements)); + // A complete datum closes every quote waiting on it, then joins the innermost open list, or + // is the result when nothing is open. + while (!_frames.empty() && _frames.back().quote) { + _frames.pop_back(); + datum = make_list({value_t(symbol{"quote"}), std::move(datum)}); + } + if (_frames.empty()) { + out = std::move(datum); + return read_status::datum; + } + _frames.back().elements.push_back(std::move(datum)); + } + return read_status::more; } value_t parser::parse_string() { diff --git a/src/cvc/tests/state_exec_parser_test.cpp b/src/cvc/tests/state_exec_parser_test.cpp index 6b490eb88..ce00c4fee 100644 --- a/src/cvc/tests/state_exec_parser_test.cpp +++ b/src/cvc/tests/state_exec_parser_test.cpp @@ -286,3 +286,59 @@ TEST(StateExecParserTest, NestingDepthResetsBetweenTopLevelForms) { opts.max_depth = 2; EXPECT_EQ(parse_all("((a)) ((b)) ((c))", opts).size(), 3u); } + +// -- Stackless, resumable reading -- + +TEST(StateExecParserTest, ParserIsStacklessAtAnyDepth) { + // A million open parens with the limit out of the way: a recursive parser would overflow the + // native stack long before reaching the end. This one reads them all and reports the unclosed + // list. (The input never closes, so no deeply nested value is built and later destroyed.) + parse_options opts; + opts.max_depth = 2'000'000; + try { + parse(std::string(1'000'000, '('), opts); + FAIL() << "expected a parse_error"; + } catch (const parse_error &e) { + EXPECT_STREQ(e.what(), "unclosed parenthesis"); + } +} + +TEST(StateExecParserTest, ReadResumesAcrossTokenBudgets) { + const std::string src = "(a (b c)) 'd \"e\" (f '(g))"; + std::vector whole; + for (const value_t &v : parse_all(src)) + whole.push_back(to_string(v)); + + parser p(src); + std::vector sliced; + int pauses = 0; + for (;;) { + std::size_t budget = 1; // one token per call + value_t v; + const auto status = p.read(v, budget); + if (status == parser::read_status::end) + break; + if (status == parser::read_status::more) { + ++pauses; + continue; + } + sliced.push_back(to_string(v)); + } + EXPECT_EQ(sliced, whole); + EXPECT_GT(pauses, 5); // the lists really were read across several calls +} + +TEST(StateExecParserTest, ReadCountsTokensAgainstTheBudget) { + parser p("(a b) c"); + std::size_t budget = 3; // '(' 'a' 'b' + value_t v; + EXPECT_EQ(p.read(v, budget), parser::read_status::more); + EXPECT_EQ(budget, 0u); + budget = 10; + EXPECT_EQ(p.read(v, budget), parser::read_status::datum); // ')' closes the list + EXPECT_EQ(to_string(v), "(a b)"); + EXPECT_EQ(budget, 9u); + EXPECT_EQ(p.read(v, budget), parser::read_status::datum); + EXPECT_EQ(to_string(v), "c"); + EXPECT_EQ(p.read(v, budget), parser::read_status::end); +} From c040bddf065de397cad62cd301836a851aeb0803 Mon Sep 17 00:00:00 2001 From: Joe Rivera Date: Sat, 17 Oct 2026 13:22:57 -0500 Subject: [PATCH 06/17] state_exec: read submitted scripts inside the process, a slice per step The stackless parser's state is an input position plus a heap stack of open frames, so it can pause anywhere. New setting parser.step_tokens (on the state tree, resolved like parser.max_depth; default 0) lets a scheduler parse a submitted script inside the new process instead of at submit: each scheduler step reads N tokens, and the step that reaches the end sets up evaluation exactly as parsing up front would have. - Reading steps are process steps: they count against max_steps and max_time and interleave with other processes. - A syntax error surfaces like a runtime error: the async scheduler kills the process with it as the exit error; the sync scheduler throws it from step()/run(). With step_tokens = 0, execute() still throws. - Signals and watch events wait until the script is read (their handlers run on the evaluation stack, which doesn't exist yet). - The snapshot codec stores a mid-read process's source and settings and re-reads it after a restore. Adds parse_options::step_tokens, script_reader (owns its source and drives parser::read() a budget at a time) and evaluator_state::reading; stackless_evaluator::step() reads until the script is read. Tests cover the reader, the evaluator, the codec and both schedulers (results, syntax errors, max_steps, deferred signals). --- docs/STATE_EXEC_GRAMMAR.md | 27 ++++- inc/cvc/state/state_exec/parser.h | 32 ++++++ .../state/state_exec/stackless_evaluator.h | 13 ++- src/cvc/state/state_exec/async_scheduler.cpp | 9 +- src/cvc/state/state_exec/parser.cpp | 49 +++++++--- src/cvc/state/state_exec/scheduler.cpp | 10 +- .../state/state_exec/stackless_evaluator.cpp | 42 ++++++-- .../state/state_exec/state_value_codec.cpp | 27 +++++ src/cvc/tests/state_exec_codec_test.cpp | 31 ++++++ src/cvc/tests/state_exec_integration_test.cpp | 98 +++++++++++++++++++ src/cvc/tests/state_exec_parser_test.cpp | 29 ++++++ .../state_exec_stackless_evaluator_test.cpp | 40 ++++++++ 12 files changed, 383 insertions(+), 24 deletions(-) diff --git a/docs/STATE_EXEC_GRAMMAR.md b/docs/STATE_EXEC_GRAMMAR.md index c9e2ce74b..8a5835397 100644 --- a/docs/STATE_EXEC_GRAMMAR.md +++ b/docs/STATE_EXEC_GRAMMAR.md @@ -148,12 +148,13 @@ this order: 2. `state_exec.defaults.parser.`, for everything on that tree; 3. the built-in default. -A value that is missing, empty, not a positive integer or out of range is skipped, and the next -one in the order applies. +A value that is missing, empty, not a non-negative integer or out of range is skipped, and the +next one in the order applies. `max_depth` must also be positive. | Key | Meaning | Default | |---|---|---| | `max_depth` | how many lists and quotes may nest | 256 | +| `step_tokens` | 0: a scheduler parses a submitted script all at once, when it is submitted. N: it parses the script inside the new process, N tokens per step (see [Reading inside the scheduler](#reading-inside-the-scheduler)) | 0 | Who reads them: @@ -168,6 +169,28 @@ Who reads them: built-in defaults, or passes `parse_options` to `parse()` and `parse_all()` explicitly. `parse_options_from_state()` does the resolution above for any host that has a tree. +### Reading inside the scheduler + +The parser is stackless and resumable. Its whole state is an input position plus a heap stack +of open lists and quotes, and `parser::read(out, budget)` stops after `budget` tokens. So a +scheduler can parse a script the same way it evaluates one: a slice per step. With +`parser.step_tokens` = N: + +- **Submitting.** `execute(script)` returns at once. The new process starts in a reading phase. + Each of its scheduler steps reads N more tokens (parens, quotes, atoms, strings). The step + after the last token sets up evaluation, exactly as parsing at submit would have, and + evaluation steps follow. +- **Budgets.** Reading steps are process steps. They count against `max_steps` and `max_time`, + and they interleave with other processes. A huge script costs its own process's budget instead + of stalling the scheduler before the process exists. +- **Errors.** A syntax error surfaces where a runtime error would. The async scheduler kills the + process, with the parse error as its exit error. The sync scheduler throws it from `step()` or + `run()`. With step_tokens = 0, `execute()` throws it, as before. +- **Signals and watch events** wait until the script is read, because their handlers run on the + evaluation stack. +- **Snapshots.** A process snapshotted while reading stores its source and parser settings. After + a restore it reads the script again from the start. + ## Data | Production | Value (`value_t` alternative) | diff --git a/inc/cvc/state/state_exec/parser.h b/inc/cvc/state/state_exec/parser.h index 102d447c1..a3d4ab5a3 100644 --- a/inc/cvc/state/state_exec/parser.h +++ b/inc/cvc/state/state_exec/parser.h @@ -46,6 +46,13 @@ struct parse_options { /// recursively (destroying a nested list, the snapshot codec, to_string), so this keeps parsed /// data within what they can handle. State-tree key: `parser.max_depth`. std::size_t max_depth = 256; + + /// How a scheduler parses a submitted script. 0 parses all of it when it is submitted, so + /// execute() throws a parse_error. N > 0 parses it inside the new process, N tokens per + /// scheduler step, before evaluation starts. Those steps count against the process's max_steps + /// and max_time, and a syntax error then surfaces like a runtime error. State-tree key: + /// `parser.step_tokens`. + std::size_t step_tokens = 0; }; /// Resolve parser settings from a state tree, per key: `state_exec.schedulers.. @@ -122,6 +129,31 @@ class parser { value_t parse_number_or_symbol(char first); }; +/// Reads a whole script a slice at a time. A scheduler drives it from a process's steps when +/// parse_options::step_tokens is set. It owns its copy of the source, which its parser views, so +/// it can be neither copied nor moved; hold it by pointer. +class script_reader { +public: + script_reader(std::string source, const parse_options &opts); + script_reader(const script_reader &) = delete; + script_reader &operator=(const script_reader &) = delete; + + /// Read up to `budget` more tokens. Returns true once the whole script has been read; forms() + /// then holds every top-level datum, as parse_all() would return them. + /// @throws parse_error on syntax errors. + bool step(std::size_t budget); + + std::vector &forms() { return _forms; } + const std::string &source() const { return _source; } + const parse_options &options() const { return _opts; } + +private: + std::string _source; // declared before _parser, which views it + parse_options _opts; + parser _parser; + std::vector _forms; +}; + } // namespace cvc::state_exec #endif // CVC_STATE_EXEC_PARSER_H diff --git a/inc/cvc/state/state_exec/stackless_evaluator.h b/inc/cvc/state/state_exec/stackless_evaluator.h index 17a6b1b08..10902797c 100644 --- a/inc/cvc/state/state_exec/stackless_evaluator.h +++ b/inc/cvc/state/state_exec/stackless_evaluator.h @@ -74,6 +74,9 @@ struct evaluator_state { evaluator::macro_map user_macros; evaluation_stats stats; bool yielded = false; // Set by (yield val) — signals generator_next to stop + /// Set while the script is still being read, a slice per step (parse_options::step_tokens); + /// step() reads instead of evaluating until the whole script is read. + std::shared_ptr reading; }; /// Synchronous stackless evaluator (Tier 2 — local contexts only). @@ -94,7 +97,9 @@ class stackless_evaluator { evaluator_state create_state(const value_t &expr, environment_ptr env = nullptr); /// Create a fresh evaluator_state from a script string, parsed with `popts` (a scheduler passes - /// the parser settings it resolved from the state tree). + /// the parser settings it resolved from the state tree). With popts.step_tokens > 0 nothing is + /// parsed yet: the state reads the script popts.step_tokens tokens per step() first, and a + /// syntax error is thrown from step() rather than from here. evaluator_state create_state(const std::string &script, environment_ptr env = nullptr, const parse_options &popts = {}); @@ -137,6 +142,12 @@ class stackless_evaluator { private: void check_interrupted(); + // A state that evaluates `forms` as a script: done when empty, the form itself when one, a + // (begin ...) of them otherwise. + evaluator_state create_program_state(std::vector forms, environment_ptr env); + // One step of a state that is still reading its script (evaluator_state::reading). + bool step_read(evaluator_state &state); + // Frame management void pop_frame(evaluator_state &state, value_t result); void push_frame(evaluator_state &state, value_t expr, environment_ptr env); diff --git a/src/cvc/state/state_exec/async_scheduler.cpp b/src/cvc/state/state_exec/async_scheduler.cpp index d76728284..1f7e6b31d 100644 --- a/src/cvc/state/state_exec/async_scheduler.cpp +++ b/src/cvc/state/state_exec/async_scheduler.cpp @@ -139,12 +139,17 @@ void async_scheduler::execute_process_step(process &proc) { } } reset_guard{current_pid_, current_proc_}; - if (!proc.pending_signals.empty() && !proc.in_signal_handler && !proc.in_watch_handler) { + // A process still reading its script (parser.step_tokens) takes signals and watch events once + // the script is read: their handlers run on the evaluation stack, which doesn't exist yet. + const bool can_interrupt = + !proc.in_signal_handler && !proc.in_watch_handler && !proc.state.reading; + + if (!proc.pending_signals.empty() && can_interrupt) { handle_signal(proc); return; } - if (!proc.pending_watch_events.empty() && !proc.in_signal_handler && !proc.in_watch_handler) { + if (!proc.pending_watch_events.empty() && can_interrupt) { handle_watch_event(proc); return; } diff --git a/src/cvc/state/state_exec/parser.cpp b/src/cvc/state/state_exec/parser.cpp index 340003b07..8f3964a2b 100644 --- a/src/cvc/state/state_exec/parser.cpp +++ b/src/cvc/state/state_exec/parser.cpp @@ -27,29 +27,36 @@ std::vector parse_all(const std::string &input, const parse_options &op return results; } -// A positive integer setting at `path`, or nullopt when it is missing, empty, malformed, zero or -// out of range (so the caller falls through to the next place in the resolution order). -static std::optional read_positive_setting(cvc::state &root, const std::string &path) { +// A non-negative integer setting at `path` (positive unless `allow_zero`), or nullopt when it is +// missing, empty, malformed or out of range, so the caller falls through to the next place in +// the resolution order. +static std::optional read_size_setting(cvc::state &root, const std::string &path, + bool allow_zero) { cvc::state *node = root.findDescendant(path); if (!node) return std::nullopt; const std::string v = node->value(); std::size_t n = 0; const auto [end, ec] = std::from_chars(v.data(), v.data() + v.size(), n); - if (ec != std::errc{} || end != v.data() + v.size() || n == 0) + if (ec != std::errc{} || end != v.data() + v.size() || v.empty() || (n == 0 && !allow_zero)) return std::nullopt; return n; } parse_options parse_options_from_state(cvc::state &root, const std::string &scheduler_id) { parse_options opts; - const std::string key = ".parser.max_depth"; - if (auto v = read_positive_setting(root, std::string(scheduler::defaults_prefix) + key)) - opts.max_depth = *v; - if (!scheduler_id.empty()) - if (auto v = read_positive_setting(root, std::string(scheduler::sched_prefix) + "." + - scheduler_id + key)) - opts.max_depth = *v; + const auto resolve = [&](const char *key, bool allow_zero, std::size_t &out) { + const std::string suffix = std::string(".parser.") + key; + if (auto v = + read_size_setting(root, std::string(scheduler::defaults_prefix) + suffix, allow_zero)) + out = *v; + if (!scheduler_id.empty()) + if (auto v = read_size_setting( + root, std::string(scheduler::sched_prefix) + "." + scheduler_id + suffix, allow_zero)) + out = *v; + }; + resolve("max_depth", /*allow_zero=*/false, opts.max_depth); + resolve("step_tokens", /*allow_zero=*/true, opts.step_tokens); // 0 = parse at submit return opts; } @@ -184,6 +191,26 @@ parser::read_status parser::read(value_t &out, std::size_t &budget) { return read_status::more; } +// -- script_reader ------------------------------------------------------------- + +script_reader::script_reader(std::string source, const parse_options &opts) + : _source(std::move(source)), _opts(opts), _parser(_source, opts) {} + +bool script_reader::step(std::size_t budget) { + for (;;) { + value_t form; + switch (_parser.read(form, budget)) { + case parser::read_status::datum: + _forms.push_back(std::move(form)); + break; + case parser::read_status::end: + return true; + case parser::read_status::more: + return false; + } + } +} + value_t parser::parse_string() { advance(); // consume opening '"' std::string result; diff --git a/src/cvc/state/state_exec/scheduler.cpp b/src/cvc/state/state_exec/scheduler.cpp index 28f2584c0..6d1d09872 100644 --- a/src/cvc/state/state_exec/scheduler.cpp +++ b/src/cvc/state/state_exec/scheduler.cpp @@ -63,6 +63,7 @@ void scheduler::load_settings() { std::string base = std::string(sched_prefix) + "." + id_; (*watch_root_)(base + ".max_pending_messages").value(std::to_string(max_pending_messages)); (*watch_root_)(base + ".parser.max_depth").value(std::to_string(parse_options_.max_depth)); + (*watch_root_)(base + ".parser.step_tokens").value(std::to_string(parse_options_.step_tokens)); (*watch_root_)(base + ".policy") .value(policy_ == scheduling_policy::round_robin ? "round_robin" : policy_ == scheduling_policy::priority ? "priority" @@ -195,14 +196,19 @@ void scheduler::execute_process_step(process &proc) { } } reset_guard{current_pid_, current_proc_}; + // A process still reading its script (parser.step_tokens) takes signals and watch events once + // the script is read: their handlers run on the evaluation stack, which doesn't exist yet. + const bool can_interrupt = + !proc.in_signal_handler && !proc.in_watch_handler && !proc.state.reading; + // 1. Handle pending signals first - if (!proc.pending_signals.empty() && !proc.in_signal_handler && !proc.in_watch_handler) { + if (!proc.pending_signals.empty() && can_interrupt) { handle_signal(proc); return; // Signal setup counts as a step } // 1b. Handle pending watch events - if (!proc.pending_watch_events.empty() && !proc.in_signal_handler && !proc.in_watch_handler) { + if (!proc.pending_watch_events.empty() && can_interrupt) { handle_watch_event(proc); return; } diff --git a/src/cvc/state/state_exec/stackless_evaluator.cpp b/src/cvc/state/state_exec/stackless_evaluator.cpp index f498e79bd..2aa10014b 100644 --- a/src/cvc/state/state_exec/stackless_evaluator.cpp +++ b/src/cvc/state/state_exec/stackless_evaluator.cpp @@ -3,6 +3,7 @@ #include #include #include +#include #include namespace cvc::state_exec { @@ -41,23 +42,52 @@ evaluator_state stackless_evaluator::create_state(const value_t &expr, environme evaluator_state stackless_evaluator::create_state(const std::string &script, environment_ptr env, const parse_options &popts) { - auto exprs = parse_all(script, popts); - if (exprs.empty()) { + if (popts.step_tokens > 0) { + // Read inside the process: step_read() parses a slice per step, then sets up evaluation. + evaluator_state st; + st.global_env = env ? env : environment::extend(global_env_); + st.reading = std::make_shared(script, popts); + return st; + } + return create_program_state(parse_all(script, popts), env); +} + +evaluator_state stackless_evaluator::create_program_state(std::vector forms, + environment_ptr env) { + if (forms.empty()) { evaluator_state st; st.done = true; return st; } - if (exprs.size() == 1) - return create_state(exprs[0], env); + if (forms.size() == 1) + return create_state(forms[0], env); std::vector begin_exprs; - begin_exprs.reserve(1 + exprs.size()); + begin_exprs.reserve(1 + forms.size()); begin_exprs.push_back(value_t{symbol{"begin"}}); - begin_exprs.insert(begin_exprs.end(), exprs.begin(), exprs.end()); + begin_exprs.insert(begin_exprs.end(), std::make_move_iterator(forms.begin()), + std::make_move_iterator(forms.end())); return create_state(make_list(std::move(begin_exprs)), env); } +// Read the next slice of the script. Once all of it is read, the state becomes exactly what +// create_state(script) would have built, keeping the step count and timing of the reading. A +// parse_error propagates like any error from step(): the sync scheduler rethrows it to its +// caller, and the async scheduler kills the process with it. +bool stackless_evaluator::step_read(evaluator_state &state) { + state.stats.increment_step(); + script_reader &reader = *state.reading; + if (!reader.step(reader.options().step_tokens)) + return false; + evaluator_state program = create_program_state(std::move(reader.forms()), state.global_env); + program.stats = std::move(state.stats); + state = std::move(program); // also releases the reader + return state.done; +} + bool stackless_evaluator::step(evaluator_state &state) { check_interrupted(); + if (state.reading) + return step_read(state); if (state.done || state.stack.empty()) { state.done = true; return true; diff --git a/src/cvc/state/state_exec/state_value_codec.cpp b/src/cvc/state/state_exec/state_value_codec.cpp index 6d5c9c5bf..0555ff7dc 100644 --- a/src/cvc/state/state_exec/state_value_codec.cpp +++ b/src/cvc/state/state_exec/state_value_codec.cpp @@ -5,6 +5,7 @@ cvc::state subtrees. */ +#include #include #include #include @@ -552,6 +553,15 @@ void encode_evaluator_state(cvc::state &node, const evaluator_state &es) { } } + // A script still being read (parser.step_tokens): keep its source and parser settings; the + // restored state reads it again from the start. + if (es.reading) { + auto &r = node("reading"); + r("source").value(es.reading->source()); + r("max_depth").value(std::to_string(es.reading->options().max_depth)); + r("step_tokens").value(std::to_string(es.reading->options().step_tokens)); + } + // Stats snapshot node("step_count").value(std::to_string(es.stats.get_step_count())); } @@ -600,6 +610,23 @@ evaluator_state decode_evaluator_state(cvc::state &node) { } } + // A script still being read restarts its reading from the beginning. + if (has_child(node, "reading")) { + auto &r = node("reading"); + parse_options opts; + const auto size_of = [&](const char *key, std::size_t fallback) { + const std::string v = r(key).value(); + std::size_t n = fallback; + const auto [end, ec] = std::from_chars(v.data(), v.data() + v.size(), n); + return ec == std::errc{} && end == v.data() + v.size() ? n : fallback; + }; + opts.max_depth = size_of("max_depth", opts.max_depth); + opts.step_tokens = size_of("step_tokens", 1); // a reading state always has a positive slice + if (opts.step_tokens == 0) + opts.step_tokens = 1; + es.reading = std::make_shared(r("source").value(), opts); + } + return es; } diff --git a/src/cvc/tests/state_exec_codec_test.cpp b/src/cvc/tests/state_exec_codec_test.cpp index 277c27d05..20fea8d41 100644 --- a/src/cvc/tests/state_exec_codec_test.cpp +++ b/src/cvc/tests/state_exec_codec_test.cpp @@ -291,6 +291,37 @@ TEST_F(EvaluatorStateCodecTest, CompletedState) { EXPECT_TRUE(restored.stack.empty()); } +TEST_F(EvaluatorStateCodecTest, StateStillReadingItsScript) { + // parser.step_tokens: a process snapshotted mid-read keeps its source and settings, and the + // restored state reads the script again and evaluates it. + parse_options popts; + popts.step_tokens = 2; + popts.max_depth = 64; + auto state = ev->create_state(std::string("(set x 20) (+ x 22)"), nullptr, popts); + ev->step(state); // part way through reading + ASSERT_TRUE(state.reading); + + auto &node = ctx.root()("__es_reading__"); + encode_evaluator_state(node, state); + auto restored = decode_evaluator_state(node); + + ASSERT_TRUE(restored.reading); + EXPECT_EQ(restored.reading->source(), "(set x 20) (+ x 22)"); + EXPECT_EQ(restored.reading->options().step_tokens, 2u); + EXPECT_EQ(restored.reading->options().max_depth, 64u); + + // Native builtins aren't serialized; put them back on the restored environment's root, as + // PauseResumeProducesCorrectResult does. Nothing is on the frame stack yet. + ASSERT_TRUE(restored.global_env); + environment_ptr root_env = restored.global_env; + while (root_env->outer) + root_env = root_env->outer; + const auto builtins_env = builtins::make_default_environment(); // must outlive the loop + for (auto &[name, val] : builtins_env->bindings) + root_env->set(name, val); + EXPECT_EQ(std::get(ev->run(restored).v), 42); +} + TEST_F(EvaluatorStateCodecTest, PausedState) { // Run a program partially (10 steps of a loop) auto state = diff --git a/src/cvc/tests/state_exec_integration_test.cpp b/src/cvc/tests/state_exec_integration_test.cpp index 0fc33685e..b3235c3a6 100644 --- a/src/cvc/tests/state_exec_integration_test.cpp +++ b/src/cvc/tests/state_exec_integration_test.cpp @@ -2403,6 +2403,104 @@ TEST_F(SchedulerSettingsTest, AsyncSchedulerReadsParserSettingsOnEachSubmit) { EXPECT_NO_THROW(sched.execute(three_deep)); } +// --- parser.step_tokens: reading a script inside the process, a slice per step --- + +TEST_F(SchedulerSettingsTest, StepTokensResolveFromTreeAndAllowZero) { + auto &root = cvc::state::instance(app_ctx); + EXPECT_EQ(parse_options_from_state(root).step_tokens, 0u); // built-in: parse at submit + root("state_exec.defaults.parser.step_tokens").value("64"); + EXPECT_EQ(parse_options_from_state(root).step_tokens, 64u); + root("state_exec.schedulers.eager.parser.step_tokens").value("0"); // 0 is a real setting here + EXPECT_EQ(parse_options_from_state(root, "eager").step_tokens, 0u); + root("state_exec.schedulers.eager.parser.step_tokens").value("-1"); // bad: falls through + EXPECT_EQ(parse_options_from_state(root, "eager").step_tokens, 64u); +} + +TEST_F(SchedulerSettingsTest, SyncSchedulerEvaluatesAScriptReadInSlices) { + auto &root = cvc::state::instance(app_ctx); + root("state_exec.defaults.parser.step_tokens").value("2"); + scheduler sched; + sched.set_watch_root(&root); + sched.set_id("reader"); + sched.load_settings(); + EXPECT_EQ(sched.parser_settings().step_tokens, 2u); + EXPECT_EQ(root("state_exec.schedulers.reader.parser.step_tokens").value(), "2"); + + const int pid = sched.execute(std::string("(set a 40) (+ a 2)")); + auto results = sched.run(); + EXPECT_EQ(std::get(results[pid].v), 42); +} + +TEST_F(SchedulerSettingsTest, SyncSchedulerThrowsSyntaxErrorsFromRunWhenReadingInSlices) { + auto &root = cvc::state::instance(app_ctx); + root("state_exec.defaults.parser.step_tokens").value("2"); + scheduler sched; + sched.set_watch_root(&root); + sched.load_settings(); + EXPECT_NO_THROW(sched.execute(std::string("(+ 1 2) (+ 3"))); // nothing is parsed yet + EXPECT_THROW(sched.run(), parse_error); // the process reads it +} + +TEST_F(SchedulerSettingsTest, ReadingStepsCountAgainstMaxSteps) { + auto &root = cvc::state::instance(app_ctx); + root("state_exec.defaults.parser.step_tokens").value("1"); + scheduler sched; + sched.set_watch_root(&root); + sched.load_settings(); + std::string big = "(list"; + for (int i = 0; i < 100; ++i) + big += " " + std::to_string(i); + big += ")"; + execute_options opts; + opts.max_steps = 5; // far fewer than the 103 tokens to read + const int pid = sched.execute(big, opts); + sched.run(); + auto info = sched.get_process_info(pid); + ASSERT_TRUE(info); + EXPECT_EQ(info->status, process_status::killed); + EXPECT_EQ(info->step_count, 5u); +} + +TEST_F(SchedulerSettingsTest, SignalsWaitUntilTheScriptIsRead) { + auto &root = cvc::state::instance(app_ctx); + root("state_exec.defaults.parser.step_tokens").value("1"); + scheduler sched; + sched.set_watch_root(&root); + sched.load_settings(); + execute_options opts; + opts.signal_handlers["SIGUSR1"] = make_list({value_t{symbol{"begin"}}, value_t{int64_t(7)}}); + const int pid = sched.execute(std::string("(begin 1 2 (+ 40 2))"), opts); + sched.step(); // reading has begun + EXPECT_TRUE(sched.send_signal(pid, "SIGUSR1")); + auto results = sched.run(); + EXPECT_EQ(sched.get_process_info(pid)->status, process_status::terminated); + EXPECT_EQ(std::get(results[pid].v), 42); +} + +TEST_F(SchedulerSettingsTest, AsyncSchedulerKillsAProcessWhoseScriptDoesNotParse) { + auto &root = cvc::state::instance(app_ctx); + root("state_exec.defaults.parser.step_tokens").value("4"); + async_scheduler sched; + sched.set_watch_root(&root); + int pid = -1; + ASSERT_NO_THROW(pid = sched.execute(std::string("(+ 1 2) (+ 3"))); + sched.sync_run(1000, 1.0); + EXPECT_EQ(sched.get_process_info(pid)->status, process_status::killed); + const auto err = sched.get_exit_error(pid); + ASSERT_TRUE(err); + EXPECT_NE(err->find("unclosed parenthesis"), std::string::npos) << *err; +} + +TEST_F(SchedulerSettingsTest, AsyncSchedulerEvaluatesAScriptReadInSlices) { + auto &root = cvc::state::instance(app_ctx); + root("state_exec.defaults.parser.step_tokens").value("3"); + async_scheduler sched; + sched.set_watch_root(&root); + const int pid = sched.execute(std::string("(set a 40) (+ a 2)")); + auto results = sched.sync_run(1000, 1.0); + EXPECT_EQ(std::get(results[pid].v), 42); +} + // =========================================================================== // §12 message-channel scoping — end-to-end through the scheduler + chroot // =========================================================================== diff --git a/src/cvc/tests/state_exec_parser_test.cpp b/src/cvc/tests/state_exec_parser_test.cpp index ce00c4fee..888bc8b86 100644 --- a/src/cvc/tests/state_exec_parser_test.cpp +++ b/src/cvc/tests/state_exec_parser_test.cpp @@ -328,6 +328,35 @@ TEST(StateExecParserTest, ReadResumesAcrossTokenBudgets) { EXPECT_GT(pauses, 5); // the lists really were read across several calls } +TEST(StateExecParserTest, ScriptReaderReadsAWholeScriptInSlices) { + const std::string src = "(define x 1) ; a comment\n(+ x '(2 3)) \"s\""; + std::vector whole; + for (const value_t &v : parse_all(src)) + whole.push_back(to_string(v)); + + script_reader reader(src, parse_options{}); + int slices = 1; + while (!reader.step(2)) // two tokens per slice + ++slices; + std::vector sliced; + for (const value_t &v : reader.forms()) + sliced.push_back(to_string(v)); + EXPECT_EQ(sliced, whole); + EXPECT_GT(slices, 5); + EXPECT_EQ(reader.source(), src); // owns its copy +} + +TEST(StateExecParserTest, ScriptReaderReportsSyntaxErrorsWhenItReachesThem) { + script_reader reader("(a) (b", parse_options{}); + EXPECT_FALSE(reader.step(2)); // '(' 'a' + EXPECT_THROW( + { + while (!reader.step(2)) { + } + }, + parse_error); +} + TEST(StateExecParserTest, ReadCountsTokensAgainstTheBudget) { parser p("(a b) c"); std::size_t budget = 3; // '(' 'a' 'b' diff --git a/src/cvc/tests/state_exec_stackless_evaluator_test.cpp b/src/cvc/tests/state_exec_stackless_evaluator_test.cpp index 906f59c63..3728ce8ed 100644 --- a/src/cvc/tests/state_exec_stackless_evaluator_test.cpp +++ b/src/cvc/tests/state_exec_stackless_evaluator_test.cpp @@ -416,3 +416,43 @@ TEST_F(StacklessEvaluatorTest, DeadlineIsNotSpentWhileTheThreadDoesNotRun) { auto st = napper.create_state(parse("(nap)")); EXPECT_EQ(std::get(napper.run(st, std::nullopt, /*timeout_sec=*/0.02).v), 7); } + +// ─── Reading inside the steps (parse_options::step_tokens) ───────────────── + +TEST_F(StacklessEvaluatorTest, SlicedReadingEvaluatesLikeParsingUpFront) { + const std::string script = "(defun sq (x) (* x x)) (set a (sq 7)) (+ a 1)"; + auto eager = ev->create_state(script); + ASSERT_FALSE(eager.reading); + const value_t expected = ev->run(eager); + + parse_options popts; + popts.step_tokens = 3; + auto sliced = ev->create_state(script, nullptr, popts); + ASSERT_TRUE(sliced.reading); // nothing parsed yet + EXPECT_TRUE(sliced.stack.empty()); + const value_t got = ev->run(sliced); + EXPECT_TRUE(sliced.done); + EXPECT_FALSE(sliced.reading); + EXPECT_EQ(to_string(got), to_string(expected)); + EXPECT_EQ(std::get(got.v), 50); + // The reading steps are steps: 25 tokens at 3 per step take 9 (the ninth reads the last token, + // finds the end of the script and sets up evaluation). + EXPECT_EQ(sliced.stats.get_step_count(), eager.stats.get_step_count() + 9); +} + +TEST_F(StacklessEvaluatorTest, SlicedReadingThrowsSyntaxErrorsFromStep) { + parse_options popts; + popts.step_tokens = 2; + evaluator_state st; + ASSERT_NO_THROW(st = ev->create_state(std::string("(+ 1 2) (+ 3"), nullptr, popts)); + EXPECT_THROW(ev->run(st), parse_error); +} + +TEST_F(StacklessEvaluatorTest, SlicedReadingOfAnEmptyScriptFinishesWithNil) { + parse_options popts; + popts.step_tokens = 4; + auto st = ev->create_state(std::string(" ; nothing here\n"), nullptr, popts); + EXPECT_TRUE(ev->step(st)); + EXPECT_TRUE(st.done); + EXPECT_TRUE(st.result.is_nil()); +} From 97c2a921054a60013ce831a4fd99cc25c11b84dc Mon Sep 17 00:00:00 2001 From: Joe Rivera Date: Sat, 17 Oct 2026 13:32:09 -0500 Subject: [PATCH 07/17] state_exec: the app's exec scheduler reads scripts in steps (1024 tokens) app::exec_scheduler(), which runs Ariadne's actions and residents, now defaults parser.step_tokens to 1024, so it parses submitted scripts inside their processes. async_scheduler gains set_parser_defaults(): the built-in values under its watch root's state_exec.defaults.parser.* keys. A plain async_scheduler (pycvc's Exec) and the sync scheduler stay at 0, so they still report syntax errors at submit. Why 1024: every script the repo carries reads in one slice (the largest is 159 tokens, .ari scripts at most 24), so a normal action pays one extra step of its 5000; a slice costs about 80 us at the parser's measured ~80 ns/token, so a pathologically large script is spread over steps and frames instead of stalling a drain. Ariadne now says why a process died: a killed action's warning carries its exit error, and the per-frame sweep reports a killed resident (a syntax error in its body, a runtime error, its per-step cap) once, instead of letting it vanish silently. --- docs/STATE_EXEC_GRAMMAR.md | 18 +++++++++-- inc/cvc/state/state_exec/async_scheduler.h | 8 +++++ inc/cvc/state/state_exec/parser.h | 7 +++-- src/cvc/ariadne/ariadne.cpp | 18 ++++++++++- src/cvc/core/app.cpp | 9 ++++++ src/cvc/state/state_exec/async_scheduler.cpp | 5 +-- src/cvc/state/state_exec/parser.cpp | 5 +-- src/cvc/tests/ariadne_runtime_test.cpp | 23 +++++++++++++- src/cvc/tests/state_exec_integration_test.cpp | 31 +++++++++++++++++++ 9 files changed, 112 insertions(+), 12 deletions(-) diff --git a/docs/STATE_EXEC_GRAMMAR.md b/docs/STATE_EXEC_GRAMMAR.md index 8a5835397..30605c60d 100644 --- a/docs/STATE_EXEC_GRAMMAR.md +++ b/docs/STATE_EXEC_GRAMMAR.md @@ -154,7 +154,7 @@ next one in the order applies. `max_depth` must also be positive. | Key | Meaning | Default | |---|---|---| | `max_depth` | how many lists and quotes may nest | 256 | -| `step_tokens` | 0: a scheduler parses a submitted script all at once, when it is submitted. N: it parses the script inside the new process, N tokens per step (see [Reading inside the scheduler](#reading-inside-the-scheduler)) | 0 | +| `step_tokens` | 0: a scheduler parses a submitted script all at once, when it is submitted. N: it parses the script inside the new process, N tokens per step (see [Reading inside the scheduler](#reading-inside-the-scheduler)) | 0; **1024** for the app's exec scheduler | Who reads them: @@ -162,7 +162,17 @@ Who reads them: the effective values under `state_exec.schedulers..parser.*` and uses them for every script submitted to it afterwards. - **`async_scheduler`** has no settings loader. It reads `state_exec.defaults.parser.*` from its - watch root each time a script is submitted. + watch root each time a script is submitted. Keys that aren't set fall back to the scheduler's + own defaults (`set_parser_defaults()`), which are the built-in ones unless the host changes + them. + - **The app's exec scheduler** (`app::exec_scheduler()`, which runs Ariadne's actions and + residents) sets `step_tokens` = 1024, so it reads scripts inside their processes. + - **Why 1024:** every script in the corpus fits in one slice (the largest is 159 tokens, and + `.ari` scripts are at most 24). So a normal action pays one extra step out of its 5000. A + slice costs about 80 µs at the parser's measured ~80 ns per token, so a pathologically large + script is spread over steps and frames instead of stalling a drain. + - pycvc's `Exec` keeps a private `async_scheduler` and stays at 0, so its `run()` still raises + on a syntax error straight away. - **Ariadne** reads `state_exec.defaults.parser.*` from the app's tree when it compiles a predicate. - **Code without a state tree** (a standalone evaluator, the `.ari` channel lint) uses the @@ -185,7 +195,9 @@ scheduler can parse a script the same way it evaluates one: a slice per step. Wi of stalling the scheduler before the process exists. - **Errors.** A syntax error surfaces where a runtime error would. The async scheduler kills the process, with the parse error as its exit error. The sync scheduler throws it from `step()` or - `run()`. With step_tokens = 0, `execute()` throws it, as before. + `run()`. With step_tokens = 0, `execute()` throws it, as before. Ariadne reports a killed + action or resident with its exit error ("ari: on: action program did not complete: …", + "ari: resident [...] stopped: …"). - **Signals and watch events** wait until the script is read, because their handlers run on the evaluation stack. - **Snapshots.** A process snapshotted while reading stores its source and parser settings. After diff --git a/inc/cvc/state/state_exec/async_scheduler.h b/inc/cvc/state/state_exec/async_scheduler.h index 750c61d68..bb440d3de 100644 --- a/inc/cvc/state/state_exec/async_scheduler.h +++ b/inc/cvc/state/state_exec/async_scheduler.h @@ -163,6 +163,13 @@ class async_scheduler : public scheduler_base { void set_watch_root(cvc::state *root) override { watch_root_ = root; } + /// This scheduler's built-in parser settings: what a submitted script is parsed with when the + /// watch root's `state_exec.defaults.parser.*` keys don't say otherwise. parse_options{} (parse + /// at submit) unless the host sets it; the app's exec scheduler reads in steps + /// (app::exec_scheduler()). + void set_parser_defaults(const parse_options &defaults) { parser_defaults_ = defaults; } + const parse_options &parser_defaults() const { return parser_defaults_; } + private: scheduling_policy policy_; int next_pid_ = 1; @@ -180,6 +187,7 @@ class async_scheduler : public scheduler_base { async_stackless_evaluator evaluator_; memory_tracker mem_tracker_; cvc::state *watch_root_ = nullptr; + parse_options parser_defaults_; std::unordered_map processes_; diff --git a/inc/cvc/state/state_exec/parser.h b/inc/cvc/state/state_exec/parser.h index a3d4ab5a3..5267c0c3d 100644 --- a/inc/cvc/state/state_exec/parser.h +++ b/inc/cvc/state/state_exec/parser.h @@ -57,9 +57,10 @@ struct parse_options { /// Resolve parser settings from a state tree, per key: `state_exec.schedulers.. /// parser.` (when scheduler_id is not empty), then `state_exec.defaults.parser.`, then -/// the parse_options default. A value that is missing, empty, not a positive integer or out of -/// range is skipped. -parse_options parse_options_from_state(cvc::state &root, const std::string &scheduler_id = {}); +/// `base` (a host's built-in defaults; parse_options{} unless it has its own). A value that is +/// missing, empty, malformed or out of range is skipped; max_depth must also be positive. +parse_options parse_options_from_state(cvc::state &root, const std::string &scheduler_id = {}, + const parse_options &base = {}); /// Parse a single S-expression from the input string. /// Returns the parsed value_t. diff --git a/src/cvc/ariadne/ariadne.cpp b/src/cvc/ariadne/ariadne.cpp index 85657179c..dba3f7fc9 100644 --- a/src/cvc/ariadne/ariadne.cpp +++ b/src/cvc/ariadne/ariadne.cpp @@ -1467,12 +1467,28 @@ void Runtime::Impl::pump_and_sweep_actions() { if (!info || info->status == se::process_status::terminated) { it = live_actions_.erase(it); // normal completion (or already reaped from the process map) } else if (info->status == se::process_status::killed) { - warn_once("ari: on: action program did not complete (a runtime error or resource limit)"); + // The exit error says why: a syntax error (scripts are read inside the process), a runtime + // error or a resource limit. + warn_once("ari: on: action program did not complete: " + + sched.get_exit_error(it->first).value_or("killed")); it = live_actions_.erase(it); } else { ++it; // ready/running/waiting — still in flight (e.g. suspended on await/msg-recv/sleep) } } + // A resident never finishes on its own, so one that was killed (a syntax error in its body, + // read inside the process; a runtime error; its per-step cap) is reported, once per reason. + const auto report_killed_resident = [&](int pid, const std::string &channel) { + if (pid < 0) + return; + auto info = sched.get_process_info(pid); + if (info && info->status == se::process_status::killed) + warn_once("ari: resident [" + channel + + "] stopped: " + sched.get_exit_error(pid).value_or("killed")); + }; + report_killed_resident(tick_resident_pid_, tick_channel()); + report_killed_resident(key_resident_pid_, key_channel()); + report_killed_resident(pointer_resident_pid_, pointer_channel()); } void Runtime::Impl::ensure_resident(const std::string &channel, const std::string &body, diff --git a/src/cvc/core/app.cpp b/src/cvc/core/app.cpp index d776c16bf..7df1bb94f 100644 --- a/src/cvc/core/app.cpp +++ b/src/cvc/core/app.cpp @@ -1022,6 +1022,15 @@ cvc::state_exec::async_scheduler &app::exec_scheduler() { // Wire it to this app's state root so state-watches (state-watch/on_change:) // resolve against the app tree. Processes are chrooted per-owner on submit. _execScheduler->set_watch_root(&root()); + // Read submitted scripts inside their processes, 1024 tokens per step (docs/ + // STATE_EXEC_GRAMMAR.md, "Reading inside the scheduler"). Every script the repo carries fits + // in one slice (the largest is 159 tokens; .ari scripts are at most 24), so a normal + // action costs one extra step out of its 5000. A slice costs about 80 us at the parser's + // measured ~80 ns per token, so a pathologically large script is spread over steps and frames + // instead of stalling a drain. state_exec.defaults.parser.step_tokens overrides it. + cvc::state_exec::parse_options parser_defaults; + parser_defaults.step_tokens = 1024; + _execScheduler->set_parser_defaults(parser_defaults); } return *_execScheduler; } diff --git a/src/cvc/state/state_exec/async_scheduler.cpp b/src/cvc/state/state_exec/async_scheduler.cpp index 1f7e6b31d..4de30dd99 100644 --- a/src/cvc/state/state_exec/async_scheduler.cpp +++ b/src/cvc/state/state_exec/async_scheduler.cpp @@ -37,9 +37,10 @@ int async_scheduler::execute(const std::string &script, const execute_options &o proc.signal_handlers = opts.signal_handlers; proc.on_complete = opts.on_complete; proc.create_time = std::chrono::steady_clock::now(); - // No load_settings() here: read the parser settings from the tree on every submission. + // No load_settings() here: read the parser settings from the tree on every submission, over + // this scheduler's own defaults (set_parser_defaults). const parse_options popts = - watch_root_ ? parse_options_from_state(*watch_root_) : parse_options{}; + watch_root_ ? parse_options_from_state(*watch_root_, {}, parser_defaults_) : parser_defaults_; proc.state = evaluator_.create_state(script, opts.env, popts); proc.state.stats.start(); processes_.emplace(pid, std::make_shared(std::move(proc))); diff --git a/src/cvc/state/state_exec/parser.cpp b/src/cvc/state/state_exec/parser.cpp index 8f3964a2b..dd024c405 100644 --- a/src/cvc/state/state_exec/parser.cpp +++ b/src/cvc/state/state_exec/parser.cpp @@ -43,8 +43,9 @@ static std::optional read_size_setting(cvc::state &root, const std: return n; } -parse_options parse_options_from_state(cvc::state &root, const std::string &scheduler_id) { - parse_options opts; +parse_options parse_options_from_state(cvc::state &root, const std::string &scheduler_id, + const parse_options &base) { + parse_options opts = base; const auto resolve = [&](const char *key, bool allow_zero, std::size_t &out) { const std::string suffix = std::string(".parser.") + key; if (auto v = diff --git a/src/cvc/tests/ariadne_runtime_test.cpp b/src/cvc/tests/ariadne_runtime_test.cpp index 970158b3a..1ae4c670d 100644 --- a/src/cvc/tests/ariadne_runtime_test.cpp +++ b/src/cvc/tests/ariadne_runtime_test.cpp @@ -16,7 +16,7 @@ #include // §13.8 (fetch uri): register a custom scheme handler #include #include -#include // launch_pool_task generalization test +#include // launch_pool_task generalization test #include // item 2: full type for app.computePool().parallel_for #include // §13.8: swap the transport for a fake in the http-get test #include @@ -520,6 +520,27 @@ TEST(AriadneResident, TickProgramRunsOncePerDrain) { EXPECT_TRUE(rt.take_reactive_warnings().empty()); } +// The app's exec scheduler reads a submitted script inside its process (parser.step_tokens), so a +// resident whose body doesn't parse is killed by the syntax error once it is read, and the sweep +// reports it with the parse error rather than letting it vanish. +TEST(AriadneResident, ResidentWithASyntaxErrorIsReported) { + cvc::app app; + Runtime rt(app, ""); + MockBackend mb; + rt.set_backend(&mb); + rt.set_tick_program("(state-set \"tick.flag\" \"1\""); // unclosed + rt.render(); + rt.drain(); + rt.drain(); + const std::vector warns = rt.take_reactive_warnings(); + bool reported = false; + for (const std::string &w : warns) + reported = reported || (w.find("stopped") != std::string::npos && + w.find("unclosed parenthesis") != std::string::npos); + EXPECT_TRUE(reported) << "warnings: " << ::testing::PrintToString(warns); + EXPECT_NE(cvc::state::instance(app)("tick.flag").value(), "1"); // the body never ran +} + // set_tick_program("") clears the resident: after clearing, a drain no longer re-fires the body. TEST(AriadneResident, ClearingTickProgramStopsIt) { cvc::app app; diff --git a/src/cvc/tests/state_exec_integration_test.cpp b/src/cvc/tests/state_exec_integration_test.cpp index b3235c3a6..4ae0e0808 100644 --- a/src/cvc/tests/state_exec_integration_test.cpp +++ b/src/cvc/tests/state_exec_integration_test.cpp @@ -2491,6 +2491,37 @@ TEST_F(SchedulerSettingsTest, AsyncSchedulerKillsAProcessWhoseScriptDoesNotParse EXPECT_NE(err->find("unclosed parenthesis"), std::string::npos) << *err; } +TEST_F(SchedulerSettingsTest, AsyncSchedulerParserDefaultsSitUnderTheTree) { + auto &root = cvc::state::instance(app_ctx); + async_scheduler sched; + sched.set_watch_root(&root); + EXPECT_EQ(sched.parser_defaults().step_tokens, 0u); // a plain async_scheduler parses at submit + EXPECT_THROW(sched.execute(std::string("(+ 1")), parse_error); + + parse_options defaults; + defaults.step_tokens = 1024; + sched.set_parser_defaults(defaults); + int pid = -1; + ASSERT_NO_THROW(pid = sched.execute(std::string("(+ 1"))); // now read inside the process + sched.sync_run(1000, 1.0); + EXPECT_EQ(sched.get_process_info(pid)->status, process_status::killed); + + root("state_exec.defaults.parser.step_tokens").value("0"); // the tree overrides the defaults + EXPECT_THROW(sched.execute(std::string("(+ 1")), parse_error); +} + +TEST_F(SchedulerSettingsTest, AppExecSchedulerReadsScriptsInSteps) { + auto &sched = app_ctx.exec_scheduler(); + EXPECT_EQ(sched.parser_defaults().step_tokens, 1024u); + int pid = -1; + ASSERT_NO_THROW(pid = sched.execute(std::string("(+ 1 (+ 2"))); + sched.sync_run(1000, 1.0); + EXPECT_EQ(sched.get_process_info(pid)->status, process_status::killed); + const auto err = sched.get_exit_error(pid); + ASSERT_TRUE(err); + EXPECT_NE(err->find("unclosed parenthesis"), std::string::npos) << *err; +} + TEST_F(SchedulerSettingsTest, AsyncSchedulerEvaluatesAScriptReadInSlices) { auto &root = cvc::state::instance(app_ctx); root("state_exec.defaults.parser.step_tokens").value("3"); From 9ed8702fa4e497c5d464fe4b8f620f2b3c79f752 Mon Sep 17 00:00:00 2001 From: Joe Rivera Date: Sat, 17 Oct 2026 13:41:38 -0500 Subject: [PATCH 08/17] pycvc: give Exec the same state_exec defaults as the app's exec scheduler pycvc's Exec runs programs on a private async_scheduler that already watches the app's state tree, but its built-in parser defaults were plain parse_options{} while app::exec_scheduler() read scripts in 1024-token steps. Both now take async_scheduler::host_parser_defaults(), one definition (with the reasoning for 1024) for every libcvc host, and the app tree's state_exec.defaults.parser.* keys override both. Exec.run() still raises a RuntimeError with the parse error's message: from the killed process's exit error when the script is read in steps, or from execute() when the tree sets step_tokens to 0. Tests: test_pycvc_exec.py checks the parse error's message and that max_depth / step_tokens on the app tree configure Exec; the integration test pins the app scheduler to host_parser_defaults(). --- bindings/pycvc/pycvc_exec.cpp | 9 +++- bindings/pycvc/test_pycvc_exec.py | 42 +++++++++++++++++++ docs/STATE_EXEC_GRAMMAR.md | 15 ++++--- inc/cvc/state/state_exec/async_scheduler.h | 16 ++++++- src/cvc/core/app.cpp | 12 ++---- src/cvc/tests/state_exec_integration_test.cpp | 5 ++- 6 files changed, 81 insertions(+), 18 deletions(-) diff --git a/bindings/pycvc/pycvc_exec.cpp b/bindings/pycvc/pycvc_exec.cpp index 27ea4d652..441dfb940 100644 --- a/bindings/pycvc/pycvc_exec.cpp +++ b/bindings/pycvc/pycvc_exec.cpp @@ -378,6 +378,10 @@ Exec::Exec(const std::shared_ptr &app) : impl_(std::make_sharedcomputePool(); impl_->sched.set_watch_root(impl_->root); + // The same state_exec defaults as the app's exec scheduler, so a program behaves the same from + // Python as from Ariadne: scripts are read inside the process (host_parser_defaults), and the + // app tree's state_exec.defaults.parser.* keys override them for both. + impl_->sched.set_parser_defaults(async_scheduler::host_parser_defaults()); impl_->host_proc = make_process(); impl_->host_proc->pid = 0; @@ -554,7 +558,10 @@ std::string Exec::run(const std::string &src) { execute_options opts; opts.name = "pycvc"; opts.env = impl_->env; - const int pid = sched.execute(src, opts); // may throw on parse error (nothing to clean up yet) + // A syntax error is read inside the process (parser.step_tokens) and surfaces below as the + // killed process's exit error; with step_tokens = 0 on the tree, execute() throws it here + // instead (nothing to clean up yet). Either way run() raises with the parse error's message. + const int pid = sched.execute(src, opts); // Drive the private scheduler to completion. A purely synchronous program finishes in the first // slice; a program that parks (await / an async fn) yields, and the pump waits for a compute-pool diff --git a/bindings/pycvc/test_pycvc_exec.py b/bindings/pycvc/test_pycvc_exec.py index dff90e373..eb627bcbf 100644 --- a/bindings/pycvc/test_pycvc_exec.py +++ b/bindings/pycvc/test_pycvc_exec.py @@ -36,6 +36,48 @@ def test_parse_error_raises(): raise AssertionError("a parse error must raise") +def _run_error(ex, src): + """run(src) must raise; return the message.""" + try: + ex.run(src) + except Exception as e: # noqa: BLE001 - pycvc surfaces C++ errors as RuntimeError + return str(e) + raise AssertionError(f"{src!r} must raise") + + +def test_parse_error_names_the_problem(): + # Exec uses the same state_exec defaults as the app's exec scheduler: the script + # is read inside its process, so a syntax error comes back as the process's exit + # error. run() raises with the parse error's own message, as it did when the + # script was parsed at submit. + app = pycvc.make_app() + ex = pycvc.Exec(app) + assert "unclosed parenthesis" in _run_error(ex, "(begin (set x") + + +def test_parser_settings_come_from_the_app_tree(): + # Exec's scheduler watches the app's state tree, so state_exec.defaults.parser.* + # configures it exactly as it configures the app's exec scheduler + # (docs/STATE_EXEC_GRAMMAR.md). + app = pycvc.make_app() + ex = pycvc.Exec(app) + deep = "(+ 1 (+ 2 (+ 3 (+ 4 5))))" # four levels of nesting + assert ex.run(deep) == "15" + + pycvc.state_set(app, "state_exec.defaults.parser.max_depth", "3") + assert "nesting deeper than 3" in _run_error(ex, deep) + assert ex.run("(+ 1 (+ 2 3))") == "6" # three levels still parse + + # Parsing at submit (step_tokens 0) gives the same results and the same errors. + pycvc.state_set(app, "state_exec.defaults.parser.step_tokens", "0") + assert "nesting deeper than 3" in _run_error(ex, deep) + assert "unclosed parenthesis" in _run_error(ex, "(begin (set x") + assert ex.run("(+ 1 (+ 2 3))") == "6" + + pycvc.state_set(app, "state_exec.defaults.parser.max_depth", "256") + assert ex.run(deep) == "15" + + # ── DSL <-> state <-> Python round trips (one shared app) ─────────────── diff --git a/docs/STATE_EXEC_GRAMMAR.md b/docs/STATE_EXEC_GRAMMAR.md index 30605c60d..c6045b560 100644 --- a/docs/STATE_EXEC_GRAMMAR.md +++ b/docs/STATE_EXEC_GRAMMAR.md @@ -154,7 +154,7 @@ next one in the order applies. `max_depth` must also be positive. | Key | Meaning | Default | |---|---|---| | `max_depth` | how many lists and quotes may nest | 256 | -| `step_tokens` | 0: a scheduler parses a submitted script all at once, when it is submitted. N: it parses the script inside the new process, N tokens per step (see [Reading inside the scheduler](#reading-inside-the-scheduler)) | 0; **1024** for the app's exec scheduler | +| `step_tokens` | 0: a scheduler parses a submitted script all at once, when it is submitted. N: it parses the script inside the new process, N tokens per step (see [Reading inside the scheduler](#reading-inside-the-scheduler)) | 0; **1024** for the app's exec scheduler and pycvc's `Exec` | Who reads them: @@ -165,14 +165,19 @@ Who reads them: watch root each time a script is submitted. Keys that aren't set fall back to the scheduler's own defaults (`set_parser_defaults()`), which are the built-in ones unless the host changes them. - - **The app's exec scheduler** (`app::exec_scheduler()`, which runs Ariadne's actions and - residents) sets `step_tokens` = 1024, so it reads scripts inside their processes. + - **libcvc's own hosts all use `async_scheduler::host_parser_defaults()`** (`step_tokens` = + 1024), so a script behaves the same in each: + - the app's exec scheduler (`app::exec_scheduler()`, which runs Ariadne's actions and + residents); + - pycvc's `Exec`, on its private scheduler. That scheduler watches the same app tree, so the + same keys override both. - **Why 1024:** every script in the corpus fits in one slice (the largest is 159 tokens, and `.ari` scripts are at most 24). So a normal action pays one extra step out of its 5000. A slice costs about 80 µs at the parser's measured ~80 ns per token, so a pathologically large script is spread over steps and frames instead of stalling a drain. - - pycvc's `Exec` keeps a private `async_scheduler` and stays at 0, so its `run()` still raises - on a syntax error straight away. + - pycvc's `Exec.run()` raises a `RuntimeError` carrying the parse error either way: from the + killed process's exit error, or straight from `execute()` when the tree sets + `step_tokens` = 0. - **Ariadne** reads `state_exec.defaults.parser.*` from the app's tree when it compiles a predicate. - **Code without a state tree** (a standalone evaluator, the `.ari` channel lint) uses the diff --git a/inc/cvc/state/state_exec/async_scheduler.h b/inc/cvc/state/state_exec/async_scheduler.h index bb440d3de..8af85f86f 100644 --- a/inc/cvc/state/state_exec/async_scheduler.h +++ b/inc/cvc/state/state_exec/async_scheduler.h @@ -165,11 +165,23 @@ class async_scheduler : public scheduler_base { /// This scheduler's built-in parser settings: what a submitted script is parsed with when the /// watch root's `state_exec.defaults.parser.*` keys don't say otherwise. parse_options{} (parse - /// at submit) unless the host sets it; the app's exec scheduler reads in steps - /// (app::exec_scheduler()). + /// at submit) unless the host sets it. void set_parser_defaults(const parse_options &defaults) { parser_defaults_ = defaults; } const parse_options &parser_defaults() const { return parser_defaults_; } + /// The parser defaults libcvc's own hosts give their async schedulers, so a script behaves the + /// same in each: app::exec_scheduler() (Ariadne) and pycvc's Exec. Scripts are read inside + /// their process, 1024 tokens per step (docs/STATE_EXEC_GRAMMAR.md, "Reading inside the + /// scheduler"). Every script the repo carries fits in one slice (the largest is 159 tokens; + /// .ari scripts are at most 24), so a normal action pays one extra step of its 5000. A slice + /// costs about 80 us at the parser's measured ~80 ns per token, so a pathologically large + /// script is spread over steps instead of stalling a drain. + static parse_options host_parser_defaults() { + parse_options defaults; + defaults.step_tokens = 1024; + return defaults; + } + private: scheduling_policy policy_; int next_pid_ = 1; diff --git a/src/cvc/core/app.cpp b/src/cvc/core/app.cpp index 7df1bb94f..5af747911 100644 --- a/src/cvc/core/app.cpp +++ b/src/cvc/core/app.cpp @@ -1022,15 +1022,9 @@ cvc::state_exec::async_scheduler &app::exec_scheduler() { // Wire it to this app's state root so state-watches (state-watch/on_change:) // resolve against the app tree. Processes are chrooted per-owner on submit. _execScheduler->set_watch_root(&root()); - // Read submitted scripts inside their processes, 1024 tokens per step (docs/ - // STATE_EXEC_GRAMMAR.md, "Reading inside the scheduler"). Every script the repo carries fits - // in one slice (the largest is 159 tokens; .ari scripts are at most 24), so a normal - // action costs one extra step out of its 5000. A slice costs about 80 us at the parser's - // measured ~80 ns per token, so a pathologically large script is spread over steps and frames - // instead of stalling a drain. state_exec.defaults.parser.step_tokens overrides it. - cvc::state_exec::parse_options parser_defaults; - parser_defaults.step_tokens = 1024; - _execScheduler->set_parser_defaults(parser_defaults); + // Read submitted scripts inside their processes, like pycvc's Exec (see host_parser_defaults + // for why 1024 tokens a step). state_exec.defaults.parser.* on this tree overrides it. + _execScheduler->set_parser_defaults(cvc::state_exec::async_scheduler::host_parser_defaults()); } return *_execScheduler; } diff --git a/src/cvc/tests/state_exec_integration_test.cpp b/src/cvc/tests/state_exec_integration_test.cpp index 4ae0e0808..561f38ca7 100644 --- a/src/cvc/tests/state_exec_integration_test.cpp +++ b/src/cvc/tests/state_exec_integration_test.cpp @@ -2512,7 +2512,10 @@ TEST_F(SchedulerSettingsTest, AsyncSchedulerParserDefaultsSitUnderTheTree) { TEST_F(SchedulerSettingsTest, AppExecSchedulerReadsScriptsInSteps) { auto &sched = app_ctx.exec_scheduler(); - EXPECT_EQ(sched.parser_defaults().step_tokens, 1024u); + EXPECT_EQ(async_scheduler::host_parser_defaults().step_tokens, 1024u); + EXPECT_EQ(sched.parser_defaults().step_tokens, + async_scheduler::host_parser_defaults().step_tokens); // shared with pycvc's Exec + EXPECT_EQ(sched.parser_defaults().max_depth, async_scheduler::host_parser_defaults().max_depth); int pid = -1; ASSERT_NO_THROW(pid = sched.execute(std::string("(+ 1 (+ 2"))); sched.sync_run(1000, 1.0); From 753abb17fe1098c6a19707a960c768f4124bf43c Mon Sep 17 00:00:00 2001 From: Joe Rivera Date: Sat, 17 Oct 2026 14:07:25 -0500 Subject: [PATCH 09/17] pycvc: expose the state_exec parser-settings surface under its C++ names pycvc's state_exec surface was only Exec; the parser settings these PRs add had no Python counterpart, so the defaults weren't even observable. In pycvc's direct-wrap style (the real C++ types, no facade): - pycvc.parse_options: the real cvc::state_exec::parse_options struct (max_depth, step_tokens), C++ defaults included. - pycvc.async_scheduler.host_parser_defaults(): the real static, declared to SWIG on its own (async_scheduler isn't wrapped yet, so no constructor). - pycvc.parse_options_from_state(app, scheduler_id="", base=...): the C++ resolver, taking the app where C++ takes the root (pycvc's convention). - Exec.parser_defaults() / Exec.set_parser_defaults(): the same calls as the async_scheduler Exec wraps. test_pycvc_exec.py covers each against the C++ behaviour; the README gains a state_exec section. Built locally (SWIG 4.4.1, Python 3.11): exec, exec_async, exec_coro and ari tests, 26 passed. --- bindings/pycvc/README.md | 24 +++++++++++++++ bindings/pycvc/pycvc.i | 26 ++++++++++++++++ bindings/pycvc/pycvc_exec.cpp | 16 ++++++++++ bindings/pycvc/pycvc_exec.h | 15 ++++++++++ bindings/pycvc/test_pycvc_exec.py | 49 +++++++++++++++++++++++++++++++ 5 files changed, 130 insertions(+) diff --git a/bindings/pycvc/README.md b/bindings/pycvc/README.md index c77d513de..2d3952d22 100644 --- a/bindings/pycvc/README.md +++ b/bindings/pycvc/README.md @@ -184,3 +184,27 @@ Every widget reads/writes `cvc::state`, so the same value is drivable from a script, a config file or a replicated peer. Raw custom-widget ImGui (free-form windows) is a separate, deferred design (a curated subset compiled inside cvcGL). Contract test: `test_pycvc_gl_imgui.py`. + +## state_exec DSL (`Exec`) and parser settings + +`pycvc.Exec(app)` runs state_exec programs over the app's state tree. The parser +settings are the C++ surface, under the C++ names (see +`docs/STATE_EXEC_GRAMMAR.md`, "Parser settings"): + +```python +import pycvc +app = pycvc.make_app() +ex = pycvc.Exec(app) +ex.run("(+ 1 2)") # "3" + +pycvc.parse_options() # max_depth 256, step_tokens 0 +pycvc.async_scheduler.host_parser_defaults() # step_tokens 1024 +ex.parser_defaults() # == host_parser_defaults() +pycvc.state_set(app, "state_exec.defaults.parser.max_depth", "64") # tree overrides +pycvc.parse_options_from_state(app, "", ex.parser_defaults()) # what Exec uses +``` + +`Exec` starts from the same defaults as the app's exec scheduler (Ariadne), and +the same `state_exec.defaults.parser.*` keys override both. A syntax error +raises a `RuntimeError` with the parse error's message. Contract test: +`test_pycvc_exec.py`. diff --git a/bindings/pycvc/pycvc.i b/bindings/pycvc/pycvc.i index 535391b20..404a59e87 100644 --- a/bindings/pycvc/pycvc.i +++ b/bindings/pycvc/pycvc.i @@ -1437,6 +1437,32 @@ class AsyncStateObserver(state_observer): self._workers = [] %} +// ── state_exec parser settings (docs/STATE_EXEC_GRAMMAR.md, "Parser settings") ── +// DIRECT WRAP, under the C++ names: the real cvc::state_exec::parse_options +// struct and async_scheduler::host_parser_defaults(). Only these members are +// declared to SWIG; the rest of parser.h / async_scheduler.h (value_t, the +// parser, the scheduler itself) is not wrapped yet, so async_scheduler has no +// Python constructor. parse_options_from_state and Exec's parser_defaults / +// set_parser_defaults are in pycvc_exec.h. +%{ +#include +#include +%} +namespace cvc { +namespace state_exec { +struct parse_options { + size_t max_depth; // lists and quotes that may nest (default 256) + size_t step_tokens; // 0: parse at submit; N: read N tokens per process step +}; +%nodefaultctor async_scheduler; +%nodefaultdtor async_scheduler; +class async_scheduler { +public: + static parse_options host_parser_defaults(); +}; +} +} + // ── Phase 4: DSL execution + Python-authored DSL functions ────────────── // pycvc.Exec(app) runs state_exec DSL programs over app's state root and lets a // Python callable be registered as a DSL function (register_fn). PyObject* is diff --git a/bindings/pycvc/pycvc_exec.cpp b/bindings/pycvc/pycvc_exec.cpp index 441dfb940..94f5c8d28 100644 --- a/bindings/pycvc/pycvc_exec.cpp +++ b/bindings/pycvc/pycvc_exec.cpp @@ -648,4 +648,20 @@ std::string Exec::run(const std::string &src) { return render_result(*result); } +cvc::state_exec::parse_options Exec::parser_defaults() const { + return impl_->sched.parser_defaults(); +} + +void Exec::set_parser_defaults(const cvc::state_exec::parse_options &defaults) { + impl_->sched.set_parser_defaults(defaults); +} + +cvc::state_exec::parse_options +parse_options_from_state(const std::shared_ptr &app, const std::string &scheduler_id, + const cvc::state_exec::parse_options &base) { + if (!app) + throw std::invalid_argument("pycvc.parse_options_from_state: null app handle"); + return cvc::state_exec::parse_options_from_state(cvc::state::instance(*app), scheduler_id, base); +} + } // namespace pycvc diff --git a/bindings/pycvc/pycvc_exec.h b/bindings/pycvc/pycvc_exec.h index 192de81d5..55c3c9482 100644 --- a/bindings/pycvc/pycvc_exec.h +++ b/bindings/pycvc/pycvc_exec.h @@ -7,6 +7,7 @@ // registered here becomes callable from DSL source like any builtin. #pragma once +#include // parse_options #include #include @@ -20,6 +21,13 @@ typedef _object PyObject; namespace pycvc { +// cvc::state_exec::parse_options_from_state on `app`'s state root (pycvc passes the app where +// C++ passes the root): per key, state_exec.schedulers..parser. (when +// scheduler_id is not empty), then state_exec.defaults.parser., then `base`. +cvc::state_exec::parse_options parse_options_from_state( + const std::shared_ptr &app, const std::string &scheduler_id = "", + const cvc::state_exec::parse_options &base = cvc::state_exec::parse_options()); + class Exec { public: // Build a DSL environment + scheduler over `app`'s state root. @@ -86,6 +94,13 @@ class Exec { // (each owns its own scheduler, so they do not interfere). std::string run(const std::string &src); + // The parser settings this Exec's scheduler parses programs with when the app tree's + // state_exec.defaults.parser.* keys don't say otherwise; the same calls as + // async_scheduler::parser_defaults() / set_parser_defaults(). They start as + // async_scheduler::host_parser_defaults(), as the app's exec scheduler's do. + cvc::state_exec::parse_options parser_defaults() const; + void set_parser_defaults(const cvc::state_exec::parse_options &defaults); + private: struct ExecImpl; std::shared_ptr impl_; diff --git a/bindings/pycvc/test_pycvc_exec.py b/bindings/pycvc/test_pycvc_exec.py index eb627bcbf..6d932392b 100644 --- a/bindings/pycvc/test_pycvc_exec.py +++ b/bindings/pycvc/test_pycvc_exec.py @@ -78,6 +78,55 @@ def test_parser_settings_come_from_the_app_tree(): assert ex.run(deep) == "15" +# ── state_exec parser settings: the C++ surface, under the C++ names ──── + + +def test_parse_options_is_the_cpp_struct(): + o = pycvc.parse_options() + assert (o.max_depth, o.step_tokens) == (256, 0) # C++ defaults + o.max_depth = 32 + o.step_tokens = 8 + assert (o.max_depth, o.step_tokens) == (32, 8) + + +def test_exec_starts_from_the_host_parser_defaults(): + # The same defaults as the app's exec scheduler: + # async_scheduler::host_parser_defaults(). + host = pycvc.async_scheduler.host_parser_defaults() + assert (host.max_depth, host.step_tokens) == (256, 1024) + ex = pycvc.Exec(pycvc.make_app()) + d = ex.parser_defaults() + assert (d.max_depth, d.step_tokens) == (host.max_depth, host.step_tokens) + + +def test_parse_options_from_state_resolves_like_cpp(): + app = pycvc.make_app() + # Nothing set: the C++ default. + assert pycvc.parse_options_from_state(app).step_tokens == 0 + # What Exec reads with: its own defaults under the tree. + ex = pycvc.Exec(app) + effective = pycvc.parse_options_from_state(app, "", ex.parser_defaults()) + assert effective.step_tokens == 1024 + + pycvc.state_set(app, "state_exec.defaults.parser.max_depth", "40") + pycvc.state_set(app, "state_exec.schedulers.s1.parser.max_depth", "50") + assert pycvc.parse_options_from_state(app).max_depth == 40 + assert pycvc.parse_options_from_state(app, "s1").max_depth == 50 + pycvc.state_set(app, "state_exec.defaults.parser.max_depth", "nope") # bad: skipped + assert pycvc.parse_options_from_state(app).max_depth == 256 + + +def test_set_parser_defaults_changes_how_exec_parses(): + ex = pycvc.Exec(pycvc.make_app()) + o = pycvc.parse_options() + o.max_depth = 2 + ex.set_parser_defaults(o) + assert ex.parser_defaults().max_depth == 2 + assert ex.parser_defaults().step_tokens == 0 + assert "nesting deeper than 2" in _run_error(ex, "(+ 1 (+ 2 (+ 3 4)))") + assert ex.run("(+ 1 (+ 2 3))") == "6" + + # ── DSL <-> state <-> Python round trips (one shared app) ─────────────── From ffa9a26f08c3208f2a84d5b50bf6b05aa3e35eb1 Mon Sep 17 00:00:00 2001 From: Joe Rivera Date: Sat, 17 Oct 2026 15:41:03 -0500 Subject: [PATCH 10/17] pycvc: parse_options_from_state takes the State root, as in C++ Wrap the real cvc::state_exec::parse_options_from_state(state&, id, base) instead of a pycvc adapter that took the app: Python passes the tree root, pycvc.State.instance(app), exactly as C++ passes cvc::state&. This is the first decision of the pycvc/C++ parity approach (C++ cvc::state& parameters take a pycvc.State in Python). --- bindings/pycvc/README.md | 3 ++- bindings/pycvc/pycvc.i | 12 ++++++++---- bindings/pycvc/pycvc_exec.cpp | 8 -------- bindings/pycvc/pycvc_exec.h | 7 ------- bindings/pycvc/test_pycvc_exec.py | 18 ++++++++++-------- 5 files changed, 20 insertions(+), 28 deletions(-) diff --git a/bindings/pycvc/README.md b/bindings/pycvc/README.md index 2d3952d22..3624fa02b 100644 --- a/bindings/pycvc/README.md +++ b/bindings/pycvc/README.md @@ -201,7 +201,8 @@ pycvc.parse_options() # max_depth 256, step_toke pycvc.async_scheduler.host_parser_defaults() # step_tokens 1024 ex.parser_defaults() # == host_parser_defaults() pycvc.state_set(app, "state_exec.defaults.parser.max_depth", "64") # tree overrides -pycvc.parse_options_from_state(app, "", ex.parser_defaults()) # what Exec uses +root = pycvc.State.instance(app) # the tree root, as in C++ +pycvc.parse_options_from_state(root, "", ex.parser_defaults()) # what Exec uses ``` `Exec` starts from the same defaults as the app's exec scheduler (Ariadne), and diff --git a/bindings/pycvc/pycvc.i b/bindings/pycvc/pycvc.i index 404a59e87..50fc32b62 100644 --- a/bindings/pycvc/pycvc.i +++ b/bindings/pycvc/pycvc.i @@ -1439,10 +1439,11 @@ class AsyncStateObserver(state_observer): // ── state_exec parser settings (docs/STATE_EXEC_GRAMMAR.md, "Parser settings") ── // DIRECT WRAP, under the C++ names: the real cvc::state_exec::parse_options -// struct and async_scheduler::host_parser_defaults(). Only these members are -// declared to SWIG; the rest of parser.h / async_scheduler.h (value_t, the -// parser, the scheduler itself) is not wrapped yet, so async_scheduler has no -// Python constructor. parse_options_from_state and Exec's parser_defaults / +// struct, parse_options_from_state (which takes a State, the tree root, as in +// C++: pycvc.State.instance(app)) and async_scheduler::host_parser_defaults(). +// Only these members are declared to SWIG; the rest of parser.h / +// async_scheduler.h (value_t, the parser, the scheduler itself) is not wrapped +// yet, so async_scheduler has no Python constructor. Exec's parser_defaults / // set_parser_defaults are in pycvc_exec.h. %{ #include @@ -1454,6 +1455,9 @@ struct parse_options { size_t max_depth; // lists and quotes that may nest (default 256) size_t step_tokens; // 0: parse at submit; N: read N tokens per process step }; +parse_options parse_options_from_state(cvc::state &root, + const std::string &scheduler_id = std::string(), + const parse_options &base = parse_options()); %nodefaultctor async_scheduler; %nodefaultdtor async_scheduler; class async_scheduler { diff --git a/bindings/pycvc/pycvc_exec.cpp b/bindings/pycvc/pycvc_exec.cpp index 94f5c8d28..dd3893939 100644 --- a/bindings/pycvc/pycvc_exec.cpp +++ b/bindings/pycvc/pycvc_exec.cpp @@ -656,12 +656,4 @@ void Exec::set_parser_defaults(const cvc::state_exec::parse_options &defaults) { impl_->sched.set_parser_defaults(defaults); } -cvc::state_exec::parse_options -parse_options_from_state(const std::shared_ptr &app, const std::string &scheduler_id, - const cvc::state_exec::parse_options &base) { - if (!app) - throw std::invalid_argument("pycvc.parse_options_from_state: null app handle"); - return cvc::state_exec::parse_options_from_state(cvc::state::instance(*app), scheduler_id, base); -} - } // namespace pycvc diff --git a/bindings/pycvc/pycvc_exec.h b/bindings/pycvc/pycvc_exec.h index 55c3c9482..b774ac2c4 100644 --- a/bindings/pycvc/pycvc_exec.h +++ b/bindings/pycvc/pycvc_exec.h @@ -21,13 +21,6 @@ typedef _object PyObject; namespace pycvc { -// cvc::state_exec::parse_options_from_state on `app`'s state root (pycvc passes the app where -// C++ passes the root): per key, state_exec.schedulers..parser. (when -// scheduler_id is not empty), then state_exec.defaults.parser., then `base`. -cvc::state_exec::parse_options parse_options_from_state( - const std::shared_ptr &app, const std::string &scheduler_id = "", - const cvc::state_exec::parse_options &base = cvc::state_exec::parse_options()); - class Exec { public: // Build a DSL environment + scheduler over `app`'s state root. diff --git a/bindings/pycvc/test_pycvc_exec.py b/bindings/pycvc/test_pycvc_exec.py index 6d932392b..10e9f4392 100644 --- a/bindings/pycvc/test_pycvc_exec.py +++ b/bindings/pycvc/test_pycvc_exec.py @@ -100,20 +100,22 @@ def test_exec_starts_from_the_host_parser_defaults(): def test_parse_options_from_state_resolves_like_cpp(): + # The real C++ function: it takes the tree root, a State, as in C++. app = pycvc.make_app() + root = pycvc.State.instance(app) # Nothing set: the C++ default. - assert pycvc.parse_options_from_state(app).step_tokens == 0 + assert pycvc.parse_options_from_state(root).step_tokens == 0 # What Exec reads with: its own defaults under the tree. ex = pycvc.Exec(app) - effective = pycvc.parse_options_from_state(app, "", ex.parser_defaults()) + effective = pycvc.parse_options_from_state(root, "", ex.parser_defaults()) assert effective.step_tokens == 1024 - pycvc.state_set(app, "state_exec.defaults.parser.max_depth", "40") - pycvc.state_set(app, "state_exec.schedulers.s1.parser.max_depth", "50") - assert pycvc.parse_options_from_state(app).max_depth == 40 - assert pycvc.parse_options_from_state(app, "s1").max_depth == 50 - pycvc.state_set(app, "state_exec.defaults.parser.max_depth", "nope") # bad: skipped - assert pycvc.parse_options_from_state(app).max_depth == 256 + root.child("state_exec.defaults.parser.max_depth").value("40") + root.child("state_exec.schedulers.s1.parser.max_depth").value("50") + assert pycvc.parse_options_from_state(root).max_depth == 40 + assert pycvc.parse_options_from_state(root, "s1").max_depth == 50 + root.child("state_exec.defaults.parser.max_depth").value("nope") # bad: skipped + assert pycvc.parse_options_from_state(root).max_depth == 256 def test_set_parser_defaults_changes_how_exec_parses(): From 2a697c0597bbe0d1a1fdca7f31f5544e5fd493e3 Mon Sep 17 00:00:00 2001 From: Joe Rivera Date: Sat, 17 Oct 2026 15:56:11 -0500 Subject: [PATCH 11/17] pycvc: state_exec parity part 1 (values, parsing, parse_error, value codec) Start bringing pycvc's state_exec API to the C++ surface under the C++ names (docs/PYCVC_STATE_EXEC_PARITY.md: the audit of all 20 state_exec headers, the agreed approach, and the PR plan). - One strict value_t <-> Python conversion (pycvc_value.h), used by every API and by Exec: native types for data, pycvc.symbol (the real struct) for symbols, and pycvc.value (the real value_t, opaque) for closures, native fns, data objects and generators, which convert back unchanged. Non-str dict keys and unsupported types raise TypeError, ints beyond 64 bits OverflowError, and nesting past 1000 levels or a cycle ValueError. This replaces the lossy converter (symbols became str, closures "", anything else str(obj)), which could also put a null item in a list on invalid UTF-8 and leave an OverflowError pending. - pycvc.parse / parse_all, pycvc.parse_error (a RuntimeError with .line and .col), pycvc.script_reader, to_string, values_equal, and the value codec encode_value / decode_value on a State node. - Exec.run raises pycvc.parse_error for a syntax error whether the script was parsed at submit or read inside the process (it re-parses a killed process's script to recover the position). - The pycvc README's Status/Design sections described a v0 facade; they now describe the direct-wrap design. test_pycvc_state_exec.py (13 tests) covers each against the C++ behaviour. Locally: those plus pycvc's exec, async, coro, ari, state and core suites pass (87), except test_volume_rawiv_roundtrip, which fails the same way on the unmodified build (a Windows temp-file lock). --- bindings/pycvc/CMakeLists.txt | 1 + bindings/pycvc/README.md | 47 ++- bindings/pycvc/pycvc.i | 386 +++++++++++++++++++++++- bindings/pycvc/pycvc_exec.cpp | 135 ++------- bindings/pycvc/pycvc_value.h | 38 +++ bindings/pycvc/test_pycvc_state_exec.py | 183 +++++++++++ docs/PYCVC_STATE_EXEC_PARITY.md | 107 +++++++ 7 files changed, 769 insertions(+), 128 deletions(-) create mode 100644 bindings/pycvc/pycvc_value.h create mode 100644 bindings/pycvc/test_pycvc_state_exec.py create mode 100644 docs/PYCVC_STATE_EXEC_PARITY.md diff --git a/bindings/pycvc/CMakeLists.txt b/bindings/pycvc/CMakeLists.txt index 4d42f00f8..410d11d79 100644 --- a/bindings/pycvc/CMakeLists.txt +++ b/bindings/pycvc/CMakeLists.txt @@ -271,6 +271,7 @@ if(CVC_BUILD_PYCVC_CORE) pycvc_exec:test_pycvc_exec.py pycvc_exec_async:test_pycvc_exec_async.py pycvc_exec_coro:test_pycvc_exec_coro.py + pycvc_state_exec:test_pycvc_state_exec.py pycvc_world_clock:test_pycvc_world_clock.py pycvc_world_units:test_pycvc_world_units.py pycvc_dataprep:test_pycvc_dataprep.py diff --git a/bindings/pycvc/README.md b/bindings/pycvc/README.md index 3624fa02b..b1aadec08 100644 --- a/bindings/pycvc/README.md +++ b/bindings/pycvc/README.md @@ -12,20 +12,24 @@ Python package that imports `pycvc`. ## Status -- **v0 (this): `Geometry`** — build meshes/polylines incrementally or in - bulk, set per-vertex colors, compute normals, and read/write files - (`.off`, CVC-raw, …) via libcvc's `geometry_file_io`. -- Planned: `Volume` (cvc::volume + SDF), `State` (cvc::state tree), and - numpy fast-paths; then a `vr3` module (exposed by VolumeRover3's embedded - interpreter) that turns a `pycvc.Geometry` into a live scene node. +pycvc covers geometry and volumes (with numpy views), the state tree (`State`), +images, models, world units and clocks, LOD, state_exec (below), and, in the +`pycvc_gl` module, cvcGL scenes, Ariadne and Dear ImGui HUDs. Each area has a +contract test (`test_pycvc_*.py`). ## Design -`pycvc_geometry.h` is a SWIG-safe facade: it *forward-declares* -`cvc::geometry` behind a `shared_ptr`, so SWIG only ever parses -`std::string`/`std::vector`/primitive signatures — never libcvc's heavy -headers. Only the `.cpp` includes libcvc. C++ host apps reach the wrapped -`cvc::geometry` via `native()` (which is `%ignore`d on the Python side). +pycvc wraps the **real** libcvc types under their C++ names; there is no +parallel facade layer (`pycvc.i`). `pycvc.volume` is a `cvc::volume`, `pycvc.State` +is a `cvc::state`, `pycvc.parse` is `cvc::state_exec::parse`. The libcvc headers +are `%include`d and curated with `%ignore` for members that can't cross. A thin +`%extend` layer adds Python conveniences such as numpy views. + +Every constructor or operation that needs an app takes it explicitly +(`pycvc.volume(app)`, `pycvc.State.instance(app)`). There is no module-global +"current app". Where C++ takes a `cvc::state&`, Python passes a `pycvc.State`. +The goal is for the Python API to match the C++ API. For state_exec the audit +and plan are in `docs/PYCVC_STATE_EXEC_PARITY.md`. ## Build @@ -207,5 +211,22 @@ pycvc.parse_options_from_state(root, "", ex.parser_defaults()) # what Exec `Exec` starts from the same defaults as the app's exec scheduler (Ariadne), and the same `state_exec.defaults.parser.*` keys override both. A syntax error -raises a `RuntimeError` with the parse error's message. Contract test: -`test_pycvc_exec.py`. +raises `pycvc.parse_error`, a `RuntimeError` with `.line` and `.col`. + +The rest of state_exec is the C++ API under the C++ names +(`docs/PYCVC_STATE_EXEC_PARITY.md` lists what is wrapped and what comes next): + +```python +from pycvc import symbol +pycvc.parse("(f 'x \"s\")") # [symbol('f'), [symbol('quote'), symbol('x')], 's'] +pycvc.parse_all("1 2") # [1, 2] +pycvc.to_string([symbol("a"), 1]) # '(a 1)' +pycvc.encode_value(root.child("snap"), {"k": [1, 2]}) +pycvc.decode_value(root.child("snap")) # {'k': [1, 2]} +``` + +Values convert to native Python types, with `pycvc.symbol` for symbols and +`pycvc.value` as an opaque handle for closures and other non-data values. Such +a handle converts back to the same value when passed into the DSL. Conversion +is strict: non-`str` dict keys, ints outside 64 bits and unsupported types +raise. Contract tests: `test_pycvc_exec.py`, `test_pycvc_state_exec.py`. diff --git a/bindings/pycvc/pycvc.i b/bindings/pycvc/pycvc.i index 50fc32b62..e9e055812 100644 --- a/bindings/pycvc/pycvc.i +++ b/bindings/pycvc/pycvc.i @@ -1437,20 +1437,368 @@ class AsyncStateObserver(state_observer): self._workers = [] %} -// ── state_exec parser settings (docs/STATE_EXEC_GRAMMAR.md, "Parser settings") ── -// DIRECT WRAP, under the C++ names: the real cvc::state_exec::parse_options -// struct, parse_options_from_state (which takes a State, the tree root, as in -// C++: pycvc.State.instance(app)) and async_scheduler::host_parser_defaults(). -// Only these members are declared to SWIG; the rest of parser.h / -// async_scheduler.h (value_t, the parser, the scheduler itself) is not wrapped -// yet, so async_scheduler has no Python constructor. Exec's parser_defaults / -// set_parser_defaults are in pycvc_exec.h. +// ── state_exec (docs/PYCVC_STATE_EXEC_PARITY.md) ───────────────────────── +// DIRECT WRAP of cvc::state_exec under the C++ names. Where C++ takes a +// cvc::state&, Python passes a State (State.instance(app) is the root). Only +// the declarations below are visible to SWIG; the parity doc lists what is +// wrapped, what comes next, and what is deliberately not wrapped (the +// coroutine API, the string_view-based parser class, ...). %{ #include #include +#include +#include +#include "pycvc_value.h" +#include +#include +#include +#include +#include %} + +// ── values: the one value_t <-> Python conversion (pycvc_value.h) ────── +// Defined here because it creates and recognises the pycvc.symbol and +// pycvc.value proxies, which need the SWIG runtime; pycvc_exec.cpp calls it +// through pycvc_value.h. +%{ +namespace pycvc { +namespace { +using cvc::state_exec::value_t; +constexpr int kMaxValueDepth = 1000; + +swig_type_info *symbol_swig_type() { + static swig_type_info *t = SWIG_TypeQuery("cvc::state_exec::symbol *"); + return t; +} +swig_type_info *value_swig_type() { + static swig_type_info *t = SWIG_TypeQuery("cvc::state_exec::value_tag *"); + return t; +} + +bool too_deep(int depth) { + if (depth <= kMaxValueDepth) + return false; + PyErr_SetString(PyExc_ValueError, + "pycvc: value nests deeper than 1000 levels (or contains itself)"); + return true; +} + +PyObject *text_to_py(const std::string &s) { + return PyUnicode_DecodeUTF8(s.data(), static_cast(s.size()), "strict"); +} + +PyObject *to_py(const value_t &v, int depth) { + using namespace cvc::state_exec; + if (too_deep(depth)) + return nullptr; + if (v.is_nil()) + Py_RETURN_NONE; + if (const auto *b = std::get_if(&v.v)) + return PyBool_FromLong(*b); + if (const auto *i = std::get_if(&v.v)) + return PyLong_FromLongLong(static_cast(*i)); + if (const auto *d = std::get_if(&v.v)) + return PyFloat_FromDouble(*d); + if (const auto *s = std::get_if(&v.v)) + return text_to_py(*s); + if (const auto *by = std::get_if(&v.v)) + return PyBytes_FromStringAndSize(by->data.data(), static_cast(by->data.size())); + if (const auto *sym = std::get_if(&v.v)) + return SWIG_NewPointerObj(new symbol(*sym), symbol_swig_type(), SWIG_POINTER_OWN); + if (const auto *l = std::get_if(&v.v)) { + const Py_ssize_t n = *l ? static_cast((*l)->size()) : 0; + PyObject *out = PyList_New(n); + if (out == nullptr) + return nullptr; + for (Py_ssize_t k = 0; k < n; ++k) { + PyObject *item = to_py((**l)[static_cast(k)], depth + 1); + if (item == nullptr) { + Py_DECREF(out); + return nullptr; + } + PyList_SET_ITEM(out, k, item); // steals the new reference + } + return out; + } + if (const auto *dd = std::get_if(&v.v)) { + PyObject *out = PyDict_New(); + if (out == nullptr) + return nullptr; + if (*dd) { + for (const auto &kv : **dd) { + PyObject *key = text_to_py(kv.first); + PyObject *val = key ? to_py(kv.second, depth + 1) : nullptr; + const int rc = val ? PyDict_SetItem(out, key, val) : -1; + Py_XDECREF(key); + Py_XDECREF(val); + if (rc < 0) { + Py_DECREF(out); + return nullptr; + } + } + } + return out; + } + // closure / native_fn / data_object / generator: an opaque handle that converts back to the + // same value, so Python can hold it and pass it into the DSL again. + return SWIG_NewPointerObj(new value_t(v), value_swig_type(), SWIG_POINTER_OWN); +} + +bool from_py(PyObject *o, value_t &out, int depth) { + using namespace cvc::state_exec; + if (too_deep(depth)) + return false; + if (o == nullptr || o == Py_None) { + out = value_t(); + return true; + } + if (PyBool_Check(o)) { // before int: bool is an int subclass + out = value_t(o == Py_True); + return true; + } + if (PyLong_Check(o)) { + int overflow = 0; + const long long x = PyLong_AsLongLongAndOverflow(o, &overflow); + if (overflow != 0) { + PyErr_SetString(PyExc_OverflowError, "pycvc: int does not fit in 64 bits"); + return false; + } + if (x == -1 && PyErr_Occurred()) + return false; + out = value_t(static_cast(x)); + return true; + } + if (PyFloat_Check(o)) { + const double d = PyFloat_AsDouble(o); + if (d == -1.0 && PyErr_Occurred()) + return false; + out = value_t(d); + return true; + } + if (PyUnicode_Check(o)) { + Py_ssize_t n = 0; + const char *s = PyUnicode_AsUTF8AndSize(o, &n); // fails (and raises) on lone surrogates + if (s == nullptr) + return false; + out = value_t(std::string(s, static_cast(n))); + return true; + } + if (PyBytes_Check(o)) { + out = make_bytes(std::string(PyBytes_AS_STRING(o), static_cast(PyBytes_GET_SIZE(o)))); + return true; + } + if (PyByteArray_Check(o)) { + out = make_bytes( + std::string(PyByteArray_AS_STRING(o), static_cast(PyByteArray_GET_SIZE(o)))); + return true; + } + if (PyList_Check(o) || PyTuple_Check(o)) { + const bool tup = PyTuple_Check(o); + const Py_ssize_t n = tup ? PyTuple_GET_SIZE(o) : PyList_GET_SIZE(o); + std::vector elems; + elems.reserve(static_cast(n)); + for (Py_ssize_t k = 0; k < n; ++k) { + value_t e; + if (!from_py(tup ? PyTuple_GET_ITEM(o, k) : PyList_GET_ITEM(o, k), e, depth + 1)) + return false; + elems.push_back(std::move(e)); + } + out = make_list(std::move(elems)); + return true; + } + if (PyDict_Check(o)) { + std::vector> entries; + PyObject *k = nullptr, *v = nullptr; + Py_ssize_t pos = 0; + while (PyDict_Next(o, &pos, &k, &v)) { + if (!PyUnicode_Check(k)) { + PyErr_Format(PyExc_TypeError, "pycvc: dict keys must be str, not %.200s", + Py_TYPE(k)->tp_name); + return false; + } + Py_ssize_t n = 0; + const char *ks = PyUnicode_AsUTF8AndSize(k, &n); + if (ks == nullptr) + return false; + value_t e; + if (!from_py(v, e, depth + 1)) + return false; + entries.emplace_back(std::string(ks, static_cast(n)), std::move(e)); + } + out = make_dict(std::move(entries)); + return true; + } + void *p = nullptr; + if (SWIG_IsOK(SWIG_ConvertPtr(o, &p, symbol_swig_type(), 0)) && p != nullptr) { + out = value_t(*static_cast(p)); + return true; + } + if (SWIG_IsOK(SWIG_ConvertPtr(o, &p, value_swig_type(), 0)) && p != nullptr) { + out = *static_cast(p); + return true; + } + PyErr_Clear(); // a failed proxy lookup above may leave an attribute error behind + PyErr_Format(PyExc_TypeError, "pycvc: cannot convert %.200s to a state_exec value", + Py_TYPE(o)->tp_name); + return false; +} +} // namespace + +PyObject *value_to_py(const cvc::state_exec::value_t &v) { return to_py(v, 0); } + +PyObject *values_to_py_tuple(std::span values) { + PyObject *t = PyTuple_New(static_cast(values.size())); + if (t == nullptr) + return nullptr; + for (size_t i = 0; i < values.size(); ++i) { + PyObject *item = to_py(values[i], 0); + if (item == nullptr) { + Py_DECREF(t); + return nullptr; + } + PyTuple_SET_ITEM(t, static_cast(i), item); + } + return t; +} + +bool py_to_value(PyObject *o, cvc::state_exec::value_t &out) { return from_py(o, out, 0); } +} // namespace pycvc +%} + +%typemap(in) cvc::state_exec::value_tag (cvc::state_exec::value_tag temp) %{ + if (!pycvc::py_to_value($input, temp)) + SWIG_fail; + $1 = temp; +%} +%typemap(in) const cvc::state_exec::value_tag & (cvc::state_exec::value_tag temp) %{ + if (!pycvc::py_to_value($input, temp)) + SWIG_fail; + $1 = &temp; +%} +%typemap(typecheck, precedence=SWIG_TYPECHECK_POINTER) cvc::state_exec::value_tag, + const cvc::state_exec::value_tag & "$1 = 1;" +%typemap(out) cvc::state_exec::value_tag %{ + $result = pycvc::value_to_py(static_cast($1)); + if (!$result) + SWIG_fail; +%} +%typemap(out) std::vector %{ + { // a block, so the wrapper's later `goto fail` doesn't jump over this initialization + const std::vector &pycvc_vals = $1; + $result = PyList_New(static_cast(pycvc_vals.size())); + if (!$result) + SWIG_fail; + for (size_t i = 0; i < pycvc_vals.size(); ++i) { + PyObject *item = pycvc::value_to_py(pycvc_vals[i]); + if (!item) { + Py_DECREF($result); + $result = nullptr; + SWIG_fail; + } + PyList_SET_ITEM($result, static_cast(i), item); + } + } +%} + +// ── parse_error: pycvc.parse_error, a RuntimeError with .line and .col ── +%{ +static PyObject *pycvc_parse_error_type = nullptr; + +static void pycvc_raise_parse_error(const cvc::state_exec::parse_error &e) { + PyObject *inst = PyObject_CallFunction(pycvc_parse_error_type, "s", e.what()); + if (inst == nullptr) + return; + PyObject *line = PyLong_FromSize_t(e.line()); + PyObject *col = PyLong_FromSize_t(e.col()); + if (line != nullptr) + PyObject_SetAttrString(inst, "line", line); + if (col != nullptr) + PyObject_SetAttrString(inst, "col", col); + Py_XDECREF(line); + Py_XDECREF(col); + PyErr_SetObject(pycvc_parse_error_type, inst); + Py_DECREF(inst); +} +%} +%init %{ + pycvc_parse_error_type = PyErr_NewException("pycvc.parse_error", PyExc_RuntimeError, NULL); + if (pycvc_parse_error_type != NULL) + PyDict_SetItemString(d, "parse_error", pycvc_parse_error_type); +%} +%pythoncode %{ +parse_error = _pycvc.parse_error +%} + +// For the state_exec declarations (and Exec, below): the default mapping, with +// parse_error first so it raises pycvc.parse_error. Scoped to this section, so +// the modules that %import pycvc.i keep PYCVC_DEFAULT_EXCEPTION. +%exception { + try { + $action + } catch (const cvc::state_exec::parse_error &e) { + pycvc_raise_parse_error(e); + SWIG_fail; + } catch (const cvc::exception &e) { + SWIG_exception(SWIG_RuntimeError, e.what()); + } catch (const std::exception &e) { + SWIG_exception(SWIG_RuntimeError, e.what()); + } catch (...) { + SWIG_exception(SWIG_RuntimeError, "pycvc: C++ exception (see libcvc)"); + } +} + namespace cvc { namespace state_exec { + +// symbol: the real struct (name), so a symbol stays a symbol in Python. +struct symbol { + std::string name; +}; +%extend symbol { + symbol(const std::string &name) { return new cvc::state_exec::symbol{name}; } + %pythoncode %{ + def __eq__(self, other): + return isinstance(other, symbol) and self.name == other.name + def __ne__(self, other): + return not self.__eq__(other) + def __hash__(self): + return hash(("pycvc.symbol", self.name)) + def __repr__(self): + return "symbol(%r)" % (self.name,) + %} +} + +// value: the real value_t, as an opaque handle. Plain data converts to native +// Python types; a closure, native fn, data object or generator arrives as one of +// these and converts back to the same value. Not constructible from Python. +%rename(value) value_tag; +%nodefaultctor value_tag; +struct value_tag { + bool is_nil() const; + bool is_truthy() const; + std::string type_name() const; +}; +%extend value_tag { + %pythoncode %{ + def __repr__(self): + return "" % (self.type_name(), to_string(self)) + def __eq__(self, other): + try: + return values_equal(self, other) + except (TypeError, ValueError, OverflowError): + return NotImplemented + def __ne__(self, other): + r = self.__eq__(other) + return r if r is NotImplemented else not r + __hash__ = None + %} +} +typedef value_tag value_t; + +std::string to_string(const value_t &val); +bool values_equal(const value_t &a, const value_t &b); + +// Parser (docs/STATE_EXEC_GRAMMAR.md). struct parse_options { size_t max_depth; // lists and quotes that may nest (default 256) size_t step_tokens; // 0: parse at submit; N: read N tokens per process step @@ -1458,6 +1806,26 @@ struct parse_options { parse_options parse_options_from_state(cvc::state &root, const std::string &scheduler_id = std::string(), const parse_options &base = parse_options()); +value_t parse(const std::string &input, const parse_options &opts = parse_options()); +std::vector parse_all(const std::string &input, + const parse_options &opts = parse_options()); + +// script_reader owns its source, so it is the Python way to read a script a +// slice at a time (parser views a string_view and isn't wrapped). forms(), +// source() and options() return copies here; C++ returns references. +class script_reader { +public: + script_reader(std::string source, const parse_options &opts); + bool step(size_t budget); + std::vector forms(); + std::string source() const; + parse_options options() const; +}; + +// Snapshot codec for one value: write it under a State node, read it back. +void encode_value(cvc::state &node, const value_t &val); +value_t decode_value(cvc::state &node); + %nodefaultctor async_scheduler; %nodefaultdtor async_scheduler; class async_scheduler { @@ -1473,3 +1841,5 @@ public: // SWIG's raw-passthrough type (the callable crosses uninspected; the .cpp // INCREFs and manages its lifetime). %include "pycvc_exec.h" + +PYCVC_DEFAULT_EXCEPTION diff --git a/bindings/pycvc/pycvc_exec.cpp b/bindings/pycvc/pycvc_exec.cpp index dd3893939..e80639c01 100644 --- a/bindings/pycvc/pycvc_exec.cpp +++ b/bindings/pycvc/pycvc_exec.cpp @@ -6,6 +6,8 @@ #include "pycvc_exec.h" +#include "pycvc_value.h" // the shared value_t <-> Python conversion + #include #include #include @@ -30,101 +32,6 @@ namespace pycvc { namespace { using namespace cvc::state_exec; -// ── value_t <-> Python marshaling ─────────────────────────────────────── -// Convert a DSL value to a NEW Python reference. Scalars + list/dict cross; -// opaque DSL types (symbol/closure/native_fn/data/generator) degrade to their -// printed form so a Python fn still receives something sensible. -PyObject *value_to_py(const value_t &v) { - if (v.is_nil()) - Py_RETURN_NONE; - if (const auto *b = std::get_if(&v.v)) - return PyBool_FromLong(*b); - if (const auto *i = std::get_if(&v.v)) - return PyLong_FromLongLong(static_cast(*i)); - if (const auto *d = std::get_if(&v.v)) - return PyFloat_FromDouble(*d); - if (const auto *s = std::get_if(&v.v)) - return PyUnicode_FromStringAndSize(s->data(), static_cast(s->size())); - if (const auto *by = std::get_if(&v.v)) - return PyBytes_FromStringAndSize(by->data.data(), static_cast(by->data.size())); - if (const auto *l = std::get_if(&v.v)) { - const auto &vec = **l; - PyObject *out = PyList_New(static_cast(vec.size())); - for (Py_ssize_t k = 0; k < static_cast(vec.size()); ++k) - PyList_SET_ITEM(out, k, value_to_py(vec[k])); // steals the new ref - return out; - } - if (const auto *dd = std::get_if(&v.v)) { - const auto &pairs = **dd; - PyObject *out = PyDict_New(); - for (const auto &kv : pairs) { - PyObject *pv = value_to_py(kv.second); - PyDict_SetItemString(out, kv.first.c_str(), pv); - Py_DECREF(pv); - } - return out; - } - // Fallback: the DSL's printed form. - std::string s = to_string(v); - return PyUnicode_FromStringAndSize(s.data(), static_cast(s.size())); -} - -// Convert a Python object (borrowed) to a DSL value. bool BEFORE int (bool is a -// subclass of int in Python). dict order is preserved. -value_t py_to_value(PyObject *o) { - if (o == nullptr || o == Py_None) - return value_t(std::monostate{}); - if (PyBool_Check(o)) - return value_t(o == Py_True); - if (PyLong_Check(o)) - return value_t(static_cast(PyLong_AsLongLong(o))); - if (PyFloat_Check(o)) - return value_t(PyFloat_AsDouble(o)); - if (PyUnicode_Check(o)) { - Py_ssize_t n = 0; - const char *s = PyUnicode_AsUTF8AndSize(o, &n); - return value_t(std::string(s ? s : "", s ? static_cast(n) : 0)); - } - // Python `bytes` -> a first-class binary value (bytes_value), the counterpart - // of the text `str` path above. Lets a Python DSL fn receive/return a raw - // binary msg-send/msg-recv payload without lossy UTF-8 coercion. - if (PyBytes_Check(o)) { - char *buf = nullptr; - Py_ssize_t n = 0; - PyBytes_AsStringAndSize(o, &buf, &n); - return make_bytes(std::string(buf ? buf : "", buf ? static_cast(n) : 0)); - } - if (PyList_Check(o) || PyTuple_Check(o)) { - bool tup = PyTuple_Check(o); - Py_ssize_t n = tup ? PyTuple_Size(o) : PyList_Size(o); - std::vector elems; - elems.reserve(static_cast(n)); - for (Py_ssize_t k = 0; k < n; ++k) - elems.push_back(py_to_value(tup ? PyTuple_GetItem(o, k) : PyList_GetItem(o, k))); - return value_t(std::make_shared>(std::move(elems))); - } - if (PyDict_Check(o)) { - auto pairs = std::make_shared>>(); - PyObject *k = nullptr, *v = nullptr; - Py_ssize_t pos = 0; - while (PyDict_Next(o, &pos, &k, &v)) { - PyObject *ks = PyObject_Str(k); - Py_ssize_t n = 0; - const char *s = ks ? PyUnicode_AsUTF8AndSize(ks, &n) : ""; - pairs->emplace_back(std::string(s ? s : "", s ? static_cast(n) : 0), py_to_value(v)); - Py_XDECREF(ks); - } - return value_t(dict_ptr(std::move(pairs))); - } - // Anything else: its str(). - PyObject *s = PyObject_Str(o); - Py_ssize_t n = 0; - const char *cs = s ? PyUnicode_AsUTF8AndSize(s, &n) : ""; - value_t out(std::string(cs ? cs : "", cs ? static_cast(n) : 0)); - Py_XDECREF(s); - return out; -} - std::string fetch_py_error() { if (!PyErr_Occurred()) return "unknown error"; @@ -189,18 +96,23 @@ std::shared_ptr own_callable(PyObject *callable) { value_t call_python(const std::shared_ptr &holder, std::span args, const std::string &name) { PyGILState_STATE gil = PyGILState_Ensure(); - PyObject *pyargs = PyTuple_New(static_cast(args.size())); - for (Py_ssize_t i = 0; i < static_cast(args.size()); ++i) - PyTuple_SET_ITEM(pyargs, i, value_to_py(args[i])); // steals the new ref - PyObject *res = PyObject_CallObject(holder.get(), pyargs); - Py_DECREF(pyargs); + PyObject *pyargs = values_to_py_tuple(args); // nullptr (error set) if an arg can't convert + PyObject *res = pyargs ? PyObject_CallObject(holder.get(), pyargs) : nullptr; + Py_XDECREF(pyargs); if (res == nullptr) { std::string msg = fetch_py_error(); PyGILState_Release(gil); throw std::runtime_error("python DSL fn '" + name + "': " + msg); } - value_t out = py_to_value(res); + value_t out; + const bool converted = py_to_value(res, out); // strict: e.g. a set or non-str dict key fails Py_DECREF(res); + if (!converted) { + std::string msg = fetch_py_error(); + PyGILState_Release(gil); + throw std::runtime_error("python DSL fn '" + name + + "' returned an unconvertible value: " + msg); + } PyGILState_Release(gil); return out; } @@ -287,7 +199,10 @@ void drive_coros(PyObject *loop, std::vector> } else if (exc == Py_None) { Py_DECREF(exc); PyObject *res = PyObject_CallMethod(task, "result", nullptr); - payload = res ? py_to_value(res) : async_error_dict("python coro: " + fetch_py_error()); + value_t converted; + payload = res && py_to_value(res, converted) + ? converted + : async_error_dict("python coro: " + fetch_py_error()); Py_XDECREF(res); } else { PyObject *s = PyObject_Str(exc); @@ -501,11 +416,9 @@ void Exec::register_async_coro(const std::string &name, PyObject *coro_fn) { const std::string done = resolve_channel_key(root, chan); // The verb runs during sync_run (GIL released by the pump) — re-acquire it to touch Python. PyGILState_STATE gil = PyGILState_Ensure(); - PyObject *pyargs = PyTuple_New(static_cast(args.size())); - for (Py_ssize_t i = 0; i < static_cast(args.size()); ++i) - PyTuple_SET_ITEM(pyargs, i, value_to_py(args[i])); - PyObject *coro = PyObject_CallObject(holder.get(), pyargs); - Py_DECREF(pyargs); + PyObject *pyargs = values_to_py_tuple(args); // nullptr (error set) on an unconvertible arg + PyObject *coro = pyargs ? PyObject_CallObject(holder.get(), pyargs) : nullptr; + Py_XDECREF(pyargs); PyObject *task = nullptr; if (coro != nullptr) { task = PyObject_CallMethod(impl->loop, "create_task", "O", coro); @@ -575,6 +488,7 @@ std::string Exec::run(const std::string &src) { auto idle_deadline = clock::now() + std::chrono::duration(kIdleTimeoutSecs); std::optional result; std::optional error; // set -> throw AFTER the pool is quiesced + bool killed = false; // the scheduler killed the process (error has its exit error) try { while (true) { sched.drain_ingress(); // deliver any worker post_message FIRST, so a parked receiver readies @@ -606,6 +520,7 @@ std::string Exec::run(const std::string &src) { error = (ex && !ex->empty()) ? *ex : ("pycvc.Exec.run: program killed (pid " + std::to_string(pid) + ")"); + killed = true; break; } // Fast-path only on OUR OWN pid's runnability — has_runnable() scans siblings too, so gating @@ -640,6 +555,12 @@ std::string Exec::run(const std::string &src) { // to Python, so a worker can never post to / be destroyed against a torn-down scheduler. inflight.wait_idle(); sched.reap(pid); // drop the finished/killed process from the private scheduler (no leak) + if (killed) { + // A script read inside its process (parser.step_tokens) reports a syntax error only as the + // exit error's text. Re-parse it, so a syntax error raises pycvc.parse_error with its position + // either way, as it does when the script is parsed at submit. + (void)parse_all(src, parse_options_from_state(*impl_->root, {}, sched.parser_defaults())); + } if (error) throw std::runtime_error(*error); if (!result.has_value()) diff --git a/bindings/pycvc/pycvc_value.h b/bindings/pycvc/pycvc_value.h new file mode 100644 index 000000000..c9f4bf28a --- /dev/null +++ b/bindings/pycvc/pycvc_value.h @@ -0,0 +1,38 @@ +// pycvc_value.h — the one conversion between state_exec values (cvc::state_exec::value_t) and +// Python, used by every pycvc API that takes or returns a value (pycvc.parse, encode_value, +// Exec's registered functions, ...). docs/PYCVC_STATE_EXEC_PARITY.md has the full table. +// +// value_t -> Python (and back): +// - nil, bool, int64, double -> None, bool, int, float. +// - string -> str. The text must be valid UTF-8; invalid text raises UnicodeDecodeError. +// - bytes_value -> bytes (bytes or bytearray coming in). +// - list -> list (list or tuple coming in). +// - dict -> dict (keys must be str coming in; others raise TypeError). +// - symbol -> pycvc.symbol. +// - closure, native_fn, data_object, generator -> pycvc.value, an opaque handle that converts +// back to the same value, so it can be passed into the DSL again. +// +// Strict: an int outside 64 bits raises OverflowError, any other Python type raises TypeError, and +// a value nested deeper than 1000 levels (or a cyclic one) raises ValueError. Defined in the SWIG +// wrapper (pycvc.i), which owns the pycvc.symbol / pycvc.value types. Call with the GIL held. +#pragma once + +#include +#include + +struct _object; +typedef _object PyObject; + +namespace pycvc { + +/// A new reference, or nullptr with a Python exception set. +PyObject *value_to_py(const cvc::state_exec::value_t &v); + +/// A new tuple of the converted values (a DSL call's arguments), or nullptr with a Python +/// exception set. +PyObject *values_to_py_tuple(std::span values); + +/// Convert `o` into `out`. Returns false, with a Python exception set, when it can't. +bool py_to_value(PyObject *o, cvc::state_exec::value_t &out); + +} // namespace pycvc diff --git a/bindings/pycvc/test_pycvc_state_exec.py b/bindings/pycvc/test_pycvc_state_exec.py new file mode 100644 index 000000000..4c2da3dc6 --- /dev/null +++ b/bindings/pycvc/test_pycvc_state_exec.py @@ -0,0 +1,183 @@ +"""state_exec parity, part 1: parsing, values, parse_error and the snapshot codec. + +pycvc wraps cvc::state_exec under the C++ names; docs/PYCVC_STATE_EXEC_PARITY.md has +the approach and the full C++ vs Python inventory. Values convert to native Python +types, with pycvc.symbol for symbols and the opaque pycvc.value for closures and +other non-data values. +""" + +import pycvc +from pycvc import symbol + + +def _raises(exc, fn, *args): + try: + fn(*args) + except exc as e: + return e + raise AssertionError(f"{fn.__name__}{args!r} did not raise {exc.__name__}") + + +# ── parse / parse_all ─────────────────────────────────────────────────── + + +def test_parse_returns_python_values(): + assert pycvc.parse("42") == 42 + assert pycvc.parse("-2.5") == -2.5 + assert pycvc.parse('"café"') == "café" + assert pycvc.parse("#t") is True + assert pycvc.parse("nil") is None + assert pycvc.parse("") is None # nil, as in C++ + assert pycvc.parse('b"\\x00\\xff"') == b"\x00\xff" + assert pycvc.parse('(define x "s")') == [symbol("define"), symbol("x"), "s"] + assert pycvc.parse("'x") == [symbol("quote"), symbol("x")] + + +def test_parse_all_returns_every_form(): + assert pycvc.parse_all("1 (2 3) ; c\n x") == [1, [2, 3], symbol("x")] + assert pycvc.parse_all(" ; nothing\n") == [] + + +def test_parse_options_apply(): + o = pycvc.parse_options() + o.max_depth = 2 + assert pycvc.parse("((1))", o) == [[1]] + _raises(pycvc.parse_error, pycvc.parse, "(((1)))", o) + + +# ── parse_error ───────────────────────────────────────────────────────── + + +def test_parse_error_carries_its_position(): + o = pycvc.parse_options() + o.max_depth = 2 + e = _raises(pycvc.parse_error, pycvc.parse, "(a\n ((b)))", o) + assert isinstance(e, RuntimeError) # existing `except RuntimeError` still works + assert (e.line, e.col) == (2, 4) + assert "nesting deeper than 2" in str(e) + e = _raises(pycvc.parse_error, pycvc.parse_all, '(a "unterminated') + assert "unterminated string" in str(e) + + +def test_exec_run_raises_parse_error_either_way(): + ex = pycvc.Exec(pycvc.make_app()) + # Exec reads a script inside its process by default (step_tokens 1024)... + e = _raises(pycvc.parse_error, ex.run, "(begin\n (set x") + assert "unclosed parenthesis" in str(e) and e.line == 2 + # ...or parses it at submit (step_tokens 0): the same error, at the same place. + o = pycvc.parse_options() + o.step_tokens = 0 + ex.set_parser_defaults(o) + e2 = _raises(pycvc.parse_error, ex.run, "(begin\n (set x") + assert (str(e2), e2.line, e2.col) == (str(e), e.line, e.col) + + +# ── symbol ────────────────────────────────────────────────────────────── + + +def test_symbol_is_the_cpp_struct(): + s = symbol("x") + assert s.name == "x" + assert s == symbol("x") and s != symbol("y") + assert s != "x" # a symbol is not a string + assert hash(s) == hash(symbol("x")) + assert repr(s) == "symbol('x')" + assert {symbol("a"): 1}[symbol("a")] == 1 + + +# ── to_string / values_equal ──────────────────────────────────────────── + + +def test_to_string_and_values_equal(): + assert pycvc.to_string([symbol("a"), 1, "s", None]) == '(a 1 "s" nil)' + assert pycvc.to_string(b"\x00A") == 'b"\\x00A"' + assert pycvc.to_string((1, 2)) == "(1 2)" # a tuple converts as a list + assert pycvc.to_string(bytearray(b"ab")) == 'b"ab"' + assert pycvc.values_equal([1, {"k": "v"}], [1, {"k": "v"}]) + assert not pycvc.values_equal([1], [2]) + + +# ── strict conversion ─────────────────────────────────────────────────── + + +def test_conversion_is_strict(): + _raises(TypeError, pycvc.to_string, {1: 2}) # dict keys must be str + _raises(OverflowError, pycvc.to_string, 2**70) + _raises(TypeError, pycvc.to_string, {1, 2}) # a set has no state_exec equivalent + deep = [] + for _ in range(1100): + deep = [deep] + _raises(ValueError, pycvc.to_string, deep) + loop = [] + loop.append(loop) + _raises(ValueError, pycvc.to_string, loop) + + +def test_registered_fns_get_symbols_and_strict_returns(): + ex = pycvc.Exec(pycvc.make_app()) + seen = [] + ex.register_fn("see", lambda x: seen.append(x) or x) + assert ex.run("(see 'abc)") == "abc" + assert seen[0] == symbol("abc") # was the plain string "abc" + ex.register_fn("bad", lambda: {1, 2}) + e = _raises(RuntimeError, ex.run, "(bad)") + assert "cannot convert set" in str(e), str(e) + + +# ── opaque values ─────────────────────────────────────────────────────── + + +def test_closures_cross_as_opaque_values_and_come_back(): + ex = pycvc.Exec(pycvc.make_app()) + kept = [] + ex.register_fn("keep", lambda f: kept.append(f) or 1) + ex.register_fn("give", lambda: kept[0]) + ex.run("(keep (lambda (x) (+ x 1)))") + f = kept[0] + assert isinstance(f, pycvc.value) # was the text "" + assert f.type_name() == "closure" + assert not f.is_nil() and f.is_truthy() + assert "closure" in repr(f) + assert ex.run("(begin (set g (give)) (g 41))") == "42" + + +# ── snapshot codec ────────────────────────────────────────────────────── + + +def test_encode_and_decode_value_on_a_state_node(): + app = pycvc.make_app() + node = pycvc.State.instance(app).child("snap.v") + original = { + "name": "café", + "items": [1, 2.5, None, True, symbol("s")], + "blob": b"\x00\x01\xff", + } + pycvc.encode_value(node, original) + assert pycvc.decode_value(node) == original + + +# ── script_reader ─────────────────────────────────────────────────────── + + +def test_script_reader_reads_in_slices(): + r = pycvc.script_reader('(a 1) (b "two")', pycvc.parse_options()) + steps = 1 + while not r.step(1): + steps += 1 + assert r.forms() == [[symbol("a"), 1], [symbol("b"), "two"]] + assert steps > 3 + assert r.source() == '(a 1) (b "two")' + assert r.options().max_depth == 256 + + +def test_script_reader_raises_parse_error(): + r = pycvc.script_reader("(a", pycvc.parse_options()) + _raises(pycvc.parse_error, r.step, 100) + + +if __name__ == "__main__": + tests = [v for k, v in sorted(globals().items()) if k.startswith("test_") and callable(v)] + for t in tests: + t() + print(" ok:", t.__name__) + print("pycvc state_exec parity tests: OK") diff --git a/docs/PYCVC_STATE_EXEC_PARITY.md b/docs/PYCVC_STATE_EXEC_PARITY.md new file mode 100644 index 000000000..6e9ce0813 --- /dev/null +++ b/docs/PYCVC_STATE_EXEC_PARITY.md @@ -0,0 +1,107 @@ +# pycvc ↔ C++ API parity for state_exec + +Goal: pycvc's state_exec API is the C++ API, under the same names, so Python code reads like the +C++ it drives. pycvc already wraps libcvc this way: `pycvc.i` wraps the real types and adds no +facade layer. This doc records the audit of `inc/cvc/state/state_exec/*.h` against pycvc, the +approach agreed for closing the gap, and the plan. + +## Approach + +1. **Real types and functions, C++ names.** SWIG wraps the actual declarations: + `cvc::state_exec::parse` is `pycvc.parse`, `symbol` is `pycvc.symbol`, and so on. Members + that can't cross are listed below as exceptions, with the reason. +2. **Tree arguments are `State`s.** Where C++ takes a `cvc::state&`, Python passes a + `pycvc.State` (`pycvc.State.instance(app)` is the root, `root.child("a.b")` a node). The + older `state_set(app, …)` functions stay as a convenience layer. +3. **Values convert to native Python types**, with two small types for what Python lacks + (`bindings/pycvc/pycvc_value.h`): + + | `value_t` | Python | + |---|---| + | nil, bool, int64, double | `None`, `bool`, `int`, `float` | + | string | `str`. The text must be valid UTF-8, or `UnicodeDecodeError` is raised | + | `bytes_value` | `bytes`; `bytes` or `bytearray` coming in | + | list | `list`; a `list` or `tuple` coming in | + | dict | `dict`; keys must be `str` coming in | + | symbol | `pycvc.symbol`: the real struct, with `.name`, equality and hashing | + | closure, native_fn, data_object, generator | `pycvc.value`: the real `value_t` as an opaque handle that converts back to the same value | + + Conversion is strict: + - an int outside 64 bits raises `OverflowError`; + - a non-`str` dict key or any other Python type raises `TypeError`; + - a value nested deeper than 1000 levels, or one that contains itself, raises `ValueError`. + + Before this, symbols became plain strings, closures became the text `""`, and any + object became its `str()`. The old converter could also pass a null object into a list when a + string wasn't valid UTF-8, and it left an `OverflowError` pending on a big int. +4. **Errors keep their type.** `parse_error` is `pycvc.parse_error`, a `RuntimeError` subclass + with `.line` and `.col`. Other C++ exceptions stay `RuntimeError`. +5. **`Exec` stays** as the Python-integration layer on top of `async_scheduler`. It handles the + GIL around Python callbacks, an asyncio loop for coroutines, and compute-pool offload. It uses + the same value conversion as everything else. `Exec.run` raises `pycvc.parse_error` for a + syntax error whether the script was parsed at submit or read inside the process. + +## Inventory + +The audit (2026-10-17) found 97 public items in the 20 headers. Counted individually, with the 38 +`builtin_*` functions and the 7 `MIME_*` constants, it is 140. Status after PR 1: + +| Header | Item | Python | Status | +|---|---|---|---| +| parser.h | `parse_options` | `pycvc.parse_options` | wrapped (#653) | +| | `parse_options_from_state(state&, id, base)` | `pycvc.parse_options_from_state(State, …)` | wrapped (#653) | +| | `parse`, `parse_all` | `pycvc.parse`, `pycvc.parse_all` | wrapped (PR 1) | +| | `parse_error` (line, col) | `pycvc.parse_error` | wrapped (PR 1) | +| | `script_reader` | `pycvc.script_reader` | wrapped (PR 1). `forms()`, `source()` and `options()` return copies | +| | `parser` (`read`, `parse_expr`) | none | exception: it holds a `string_view` of its input, which Python can't keep alive. `script_reader` owns its source and gives the same slice-by-slice reading | +| types.h | `symbol` | `pycvc.symbol` | wrapped (PR 1) | +| | `value_t` | native types, or `pycvc.value` | wrapped (PR 1), see the table above | +| | `to_string`, `values_equal` | `pycvc.to_string`, `pycvc.values_equal` | wrapped (PR 1) | +| | `make_list`, `make_dict`, `make_bytes`, `nil_value`, `true_value`, `false_value` | `list`, `dict`, `bytes`, `None`, `True`, `False` | exception: the native Python values are the equivalents | +| | `environment` | | PR 2 | +| | `closure`, `data_object`, `generator` | `pycvc.value` (opaque) | wrapped as opaque handles; no member access | +| | `time_budget`, `eval_deadline_guard`, `eval_deadline_expired`, `thread_cpu_seconds` | none | exception: thread-local evaluator internals | +| state_value_codec.h | `encode_value`, `decode_value` | `pycvc.encode_value(State, v)`, `pycvc.decode_value(State)` | wrapped (PR 1) | +| | `encode_environment`/`decode_environment`, `encode_evaluator_state`/`decode_evaluator_state` | | PR 3, with the environment and evaluator types | +| scheduler_base.h | `execute_options`, `process_info`, `scheduler_stats`, `scheduling_policy` | | PR 2 | +| scheduler.h | `scheduler` (about 45 members) | | PR 2 | +| async_scheduler.h | `async_scheduler` (about 55 members) | `pycvc.async_scheduler.host_parser_defaults()` only | PR 2. Coroutine `step()`/`run()` are an exception: Python gets the existing `sync_step()`/`sync_run()` | +| process.h | `process_status`, `to_string(process_status)`, `parse_process_status` | | PR 2 | +| | `process` (about 35 fields), `make_process` | | PR 2, read-only where safe | +| builtins.h | `builtins::make_default_environment`, `builtins::register_fn` | `Exec.register_fn` | PR 2. `register_fn` takes a Python callable | +| | 38 `builtin_*` functions | none | exception: DSL builtins, called from DSL source | +| | `percent_encode`, `percent_decode` | | PR 2 | +| stackless_evaluator.h, evaluator.h | `stackless_evaluator`, `evaluator`, `evaluator_state`, `evaluation_stats`, the evaluation exceptions | | PR 3 | +| async_evaluator.h, async_stackless_evaluator.h | the async evaluators | none | exception: coroutine API | +| resource_policy.h | `resource_policy`, `process_limits`, `validate_limits` | | PR 3 | +| intrinsics.h | `register_intrinsics`, `intrinsics_context`, `apply_chroot`, `channel_policy` | `Exec` uses them internally | PR 3, with the intrinsics context's raw-pointer fields as `State`/scheduler references | +| | `document_scope`, `park_on_channel`, `make_future`, `future_channel_of` | none | exception: Ariadne/runtime internals (templates, raw process pointers) | +| stdlib.h | `stdlib_registry` | | after the stdlib is wired into the DSL (separate work: today only tests import it) | +| exec_coordinator.h | `exec_coordinator` and its result types | none | deferred: it needs the cluster shard, membership and message-bus objects, which pycvc doesn't wrap | +| memory_tracker.h | `memory_tracker` | none | exception: per-node byte accounting, internal to the schedulers | +| generator.h, task.h | `generator`, `generator_next`, `task<>` | none | exception: coroutine and evaluator internals; a generator crosses as a `pycvc.value` | +| utf8.h | `decode`, `count`, `byte_offset`, `encode`, `is_valid` | none | exception: Python's `str` already counts and indexes code points | + +## Plan + +Each PR is stacked on the previous one and includes pytest checks against the C++ behaviour. + +1. **PR 1 (this):** values, parsing and the value codec. + - `pycvc_value.h`: the strict shared converter, used by `Exec` too. + - `pycvc.symbol` and `pycvc.value`. + - `parse`, `parse_all`, `parse_error` with position, `script_reader`, `to_string`, + `values_equal`, `encode_value`, `decode_value`. + - `bindings/pycvc/test_pycvc_state_exec.py`. + - The pycvc README's outdated Status and Design sections are rewritten. +2. **PR 2:** `environment` and `builtins`; `execute_options`, `process_info`, `scheduler_stats` and + the enums; `scheduler` and `async_scheduler` as real classes, with their non-coroutine + methods. `Exec` exposes its scheduler. +3. **PR 3:** the evaluators and `evaluator_state`, the environment and evaluator-state codec, + `resource_policy`, and `register_intrinsics` / `intrinsics_context`. + +Known gaps after PR 1: +- Invalid UTF-8 in a DSL string raises when it crosses to Python, but no Python test produces such + a string. Python can't create one, and the stdlib's `bytes.decode` isn't reachable from DSL code + yet. +- `Exec.run` finds a syntax error in a script read inside its process by parsing it again after + the process is killed. That costs one extra parse, and only on failure. From 73b3f7a9e5f3746437db6cd678a4c2ac2966ff6f Mon Sep 17 00:00:00 2001 From: Joe Rivera Date: Sat, 17 Oct 2026 19:16:32 -0500 Subject: [PATCH 12/17] pycvc: state_exec parity part 2 (environments, builtins, schedulers) Wrap the scheduler surface of cvc::state_exec under the C++ names: - environment (shared, as environment_ptr) and builtins: make_default_environment, and register_fn taking a Python callable or a pycvc.value holding a native fn. - execute_options (on_complete is a Python callable, signal_handlers a dict), process_info, scheduler_stats, percent_encode/percent_decode. - process_status and scheduling_policy as enum.IntEnum classes; an enum argument must be a member, so to_string(process_status) and to_string(value_t) keep their shared name. - scheduler and async_scheduler as real classes with their non-coroutine members. run/step and sync_run/sync_step release the GIL; Python callbacks retake it. - Exec.scheduler() returns Exec's async_scheduler, which keeps the Exec alive. Empty optionals are None, except for values: get_result, pop_pending_message and environment.lookup raise KeyError, since None is nil. The value_t typecheck now runs last, so execute(str) picks the script overload. test_pycvc_state_exec_sched.py repeats the C++ scheduler tests on both classes. The parity doc and README are updated. --- bindings/pycvc/CMakeLists.txt | 1 + bindings/pycvc/README.md | 26 +- bindings/pycvc/pycvc.i | 590 +++++++++++++++++- bindings/pycvc/pycvc_exec.cpp | 60 +- bindings/pycvc/pycvc_exec.h | 8 + bindings/pycvc/pycvc_value.h | 22 + bindings/pycvc/test_pycvc_state_exec_sched.py | 481 ++++++++++++++ docs/PYCVC_STATE_EXEC_PARITY.md | 54 +- 8 files changed, 1198 insertions(+), 44 deletions(-) create mode 100644 bindings/pycvc/test_pycvc_state_exec_sched.py diff --git a/bindings/pycvc/CMakeLists.txt b/bindings/pycvc/CMakeLists.txt index 410d11d79..7b7db7368 100644 --- a/bindings/pycvc/CMakeLists.txt +++ b/bindings/pycvc/CMakeLists.txt @@ -272,6 +272,7 @@ if(CVC_BUILD_PYCVC_CORE) pycvc_exec_async:test_pycvc_exec_async.py pycvc_exec_coro:test_pycvc_exec_coro.py pycvc_state_exec:test_pycvc_state_exec.py + pycvc_state_exec_sched:test_pycvc_state_exec_sched.py pycvc_world_clock:test_pycvc_world_clock.py pycvc_world_units:test_pycvc_world_units.py pycvc_dataprep:test_pycvc_dataprep.py diff --git a/bindings/pycvc/README.md b/bindings/pycvc/README.md index b1aadec08..b9bd100d6 100644 --- a/bindings/pycvc/README.md +++ b/bindings/pycvc/README.md @@ -229,4 +229,28 @@ Values convert to native Python types, with `pycvc.symbol` for symbols and `pycvc.value` as an opaque handle for closures and other non-data values. Such a handle converts back to the same value when passed into the DSL. Conversion is strict: non-`str` dict keys, ints outside 64 bits and unsupported types -raise. Contract tests: `test_pycvc_exec.py`, `test_pycvc_state_exec.py`. +raise. + +The schedulers are the real classes too, with environments, builtins and the +process types: + +```python +from pycvc import process_status, scheduling_policy +env = pycvc.builtins.make_default_environment() +pycvc.builtins.register_fn(env, "greet", lambda n: "hi " + n) # a Python callable +opts = pycvc.execute_options() +opts.env = env +opts.on_complete = print # called with the result +sched = pycvc.scheduler(scheduling_policy.round_robin) # or pycvc.async_scheduler() +pid = sched.execute('(greet "ann")', opts) +sched.run() # {pid: 'hi ann'}; releases the GIL +sched.get_process_info(pid).status # process_status.terminated +ex.scheduler().list_processes() # Exec's own async_scheduler +``` + +The enums are `enum.IntEnum` classes, and an enum argument must be one of their +members. Where C++ returns an empty `std::optional` or a null pointer, Python gets +`None`, except for a value: `get_result`, `pop_pending_message` and +`environment.lookup` raise `KeyError` instead, because `None` is the nil value. +Contract tests: `test_pycvc_exec.py`, `test_pycvc_state_exec.py`, +`test_pycvc_state_exec_sched.py`. diff --git a/bindings/pycvc/pycvc.i b/bindings/pycvc/pycvc.i index e9e055812..2e8269c48 100644 --- a/bindings/pycvc/pycvc.i +++ b/bindings/pycvc/pycvc.i @@ -1445,14 +1445,24 @@ class AsyncStateObserver(state_observer): // coroutine API, the string_view-based parser class, ...). %{ #include +#include #include +#include +#include +#include #include #include #include "pycvc_value.h" +#include +#include #include +#include #include +#include #include +#include #include +#include #include %} @@ -1675,7 +1685,10 @@ bool py_to_value(PyObject *o, cvc::state_exec::value_t &out) { return from_py(o, SWIG_fail; $1 = &temp; %} -%typemap(typecheck, precedence=SWIG_TYPECHECK_POINTER) cvc::state_exec::value_tag, +// Any Python object may be a value, so the value overload is tried last (SWIG tries the lowest +// precedence first): a str goes to execute(script) and a process_status to +// to_string(process_status), not to their value_t overloads. +%typemap(typecheck, precedence=10000) cvc::state_exec::value_tag, const cvc::state_exec::value_tag & "$1 = 1;" %typemap(out) cvc::state_exec::value_tag %{ $result = pycvc::value_to_py(static_cast($1)); @@ -1700,6 +1713,276 @@ bool py_to_value(PyObject *o, cvc::state_exec::value_t &out) { return from_py(o, } %} +// ── maps and optionals (the scheduler surface) ────────────────────────── +// A name -> value map (environment::bindings, execute_options::signal_handlers) is a dict with str +// keys, in key order. A pid -> value map (run()'s results) is a dict with int keys. +%{ +namespace pycvc { +namespace { +using value_map = std::unordered_map; + +PyObject *value_map_to_py(const value_map &m) { + std::vector entries; + entries.reserve(m.size()); + for (const auto &kv : m) + entries.push_back(&kv); + std::sort(entries.begin(), entries.end(), + [](const auto *a, const auto *b) { return a->first < b->first; }); + PyObject *dict = PyDict_New(); + if (dict == nullptr) + return nullptr; + for (const auto *kv : entries) { + PyObject *key = text_to_py(kv->first); + PyObject *val = key != nullptr ? value_to_py(kv->second) : nullptr; + const int rc = val != nullptr ? PyDict_SetItem(dict, key, val) : -1; + Py_XDECREF(key); + Py_XDECREF(val); + if (rc != 0) { + Py_DECREF(dict); + return nullptr; + } + } + return dict; +} + +bool py_to_value_map(PyObject *o, value_map &out) { + if (!PyDict_Check(o)) { + PyErr_Format(PyExc_TypeError, "pycvc: expected a dict of str to values, got %.200s", + Py_TYPE(o)->tp_name); + return false; + } + value_map m; + PyObject *key = nullptr; + PyObject *val = nullptr; + Py_ssize_t pos = 0; + while (PyDict_Next(o, &pos, &key, &val)) { + if (!PyUnicode_Check(key)) { + PyErr_Format(PyExc_TypeError, "pycvc: dict keys must be str, got %.200s", + Py_TYPE(key)->tp_name); + return false; + } + Py_ssize_t n = 0; + const char *s = PyUnicode_AsUTF8AndSize(key, &n); + cvc::state_exec::value_t v; + if (s == nullptr || !py_to_value(val, v)) + return false; + m[std::string(s, static_cast(n))] = std::move(v); + } + out = std::move(m); + return true; +} +} // namespace +} // namespace pycvc +%} + +%typemap(in) const std::unordered_map & + (std::unordered_map temp) %{ + if (!pycvc::py_to_value_map($input, temp)) + SWIG_fail; + $1 = &temp; +%} +%typemap(out) const std::unordered_map & %{ + $result = pycvc::value_map_to_py(*$1); + if (!$result) + SWIG_fail; +%} +%typemap(out) std::unordered_map %{ + { // a block, so the wrapper's later `goto fail` doesn't jump over these initializations + const std::unordered_map &pycvc_map = $1; + $result = PyDict_New(); + if (!$result) + SWIG_fail; + for (const auto &kv : pycvc_map) { + PyObject *k = PyLong_FromLong(kv.first); + PyObject *v = k ? pycvc::value_to_py(kv.second) : nullptr; + const int rc = v ? PyDict_SetItem($result, k, v) : -1; + Py_XDECREF(k); + Py_XDECREF(v); + if (rc != 0) { + Py_DECREF($result); + $result = nullptr; + SWIG_fail; + } + } + } +%} + +// Limits passed as std::optional: None is std::nullopt. +%typemap(in) std::optional %{ + if ($input != Py_None) { + const unsigned long long pycvc_v = PyLong_AsUnsignedLongLong($input); + if (pycvc_v == static_cast(-1) && PyErr_Occurred()) + SWIG_fail; + $1 = pycvc_v; + } +%} +%typemap(typecheck, precedence=SWIG_TYPECHECK_UINT64) std::optional + "$1 = ($input == Py_None || PyLong_Check($input)) ? 1 : 0;" +%typemap(in) std::optional %{ + if ($input != Py_None) { + const double pycvc_v = PyFloat_AsDouble($input); + if (pycvc_v == -1.0 && PyErr_Occurred()) + SWIG_fail; + $1 = pycvc_v; + } +%} +%typemap(typecheck, precedence=SWIG_TYPECHECK_DOUBLE) std::optional + "$1 = ($input == Py_None || PyFloat_Check($input) || PyLong_Check($input)) ? 1 : 0;" + +// Optional results other than values are None when empty (a value can be nil, so get_result and +// pop_pending_message raise KeyError instead; see their %extend below). +%typemap(out) std::optional %{ + { // a block, so the wrapper's later `goto fail` doesn't jump over this initialization + const std::optional &pycvc_opt = $1; + if (pycvc_opt) { + $result = PyUnicode_DecodeUTF8(pycvc_opt->data(), static_cast(pycvc_opt->size()), + "surrogateescape"); + if (!$result) + SWIG_fail; + } else { + $result = Py_None; + Py_INCREF(Py_None); + } + } +%} +%typemap(out) std::optional %{ + { // a block, so the wrapper's later `goto fail` doesn't jump over this initialization + const std::optional &pycvc_opt = $1; + if (pycvc_opt) { + $result = SWIG_NewPointerObj(new cvc::state_exec::process_info(*pycvc_opt), + $descriptor(cvc::state_exec::process_info *), SWIG_POINTER_OWN); + if (!$result) + SWIG_fail; + } else { + $result = Py_None; + Py_INCREF(Py_None); + } + } +%} +%typemap(out) std::vector %{ + { // a block, so the wrapper's later `goto fail` doesn't jump over this initialization + const std::vector &pycvc_infos = $1; + $result = PyList_New(static_cast(pycvc_infos.size())); + if (!$result) + SWIG_fail; + for (size_t i = 0; i < pycvc_infos.size(); ++i) { + PyObject *item = SWIG_NewPointerObj(new cvc::state_exec::process_info(pycvc_infos[i]), + $descriptor(cvc::state_exec::process_info *), + SWIG_POINTER_OWN); + if (!item) { + Py_DECREF($result); + $result = nullptr; + SWIG_fail; + } + PyList_SET_ITEM($result, static_cast(i), item); + } + } +%} + +// ── enums: pycvc.process_status and pycvc.scheduling_policy are enum.IntEnum classes ── +// pycvc.process_status.killed is process_status::killed. An enum argument must be a member of its +// IntEnum (strict, like values), which is also what lets to_string(process_status) and +// to_string(value) share their C++ name. +%{ +static PyObject *pycvc_enum_process_status = nullptr; +static PyObject *pycvc_enum_scheduling_policy = nullptr; + +// enum.IntEnum(name, members) in module pycvc, or nullptr with a Python error set. +static PyObject *pycvc_make_int_enum(const char *name, + std::initializer_list> members) { + PyObject *enum_mod = PyImport_ImportModule("enum"); + PyObject *int_enum = enum_mod ? PyObject_GetAttrString(enum_mod, "IntEnum") : nullptr; + Py_XDECREF(enum_mod); + PyObject *pairs = int_enum ? PyList_New(0) : nullptr; + if (pairs == nullptr) { + Py_XDECREF(int_enum); + return nullptr; + } + for (const auto &m : members) { + PyObject *pair = Py_BuildValue("(si)", m.first, m.second); + if (pair == nullptr || PyList_Append(pairs, pair) != 0) { + Py_XDECREF(pair); + Py_DECREF(pairs); + Py_DECREF(int_enum); + return nullptr; + } + Py_DECREF(pair); + } + PyObject *args = Py_BuildValue("(sO)", name, pairs); + PyObject *kwargs = Py_BuildValue("{s:s}", "module", "pycvc"); + PyObject *cls = args && kwargs ? PyObject_Call(int_enum, args, kwargs) : nullptr; + Py_XDECREF(args); + Py_XDECREF(kwargs); + Py_DECREF(pairs); + Py_DECREF(int_enum); + return cls; +} + +static bool pycvc_is_enum(PyObject *o, PyObject *type) { + const int is = type != nullptr ? PyObject_IsInstance(o, type) : 0; + if (is < 0) + PyErr_Clear(); + return is == 1; +} + +// An enum argument's value: `o` must be a member of `type`. False, with a TypeError set, otherwise. +static bool pycvc_enum_from_py(PyObject *o, PyObject *type, const char *name, int &out) { + if (!pycvc_is_enum(o, type)) { + PyErr_Format(PyExc_TypeError, "pycvc: expected a pycvc.%s member, got %.200s", name, + Py_TYPE(o)->tp_name); + return false; + } + const long v = PyLong_AsLong(o); + if (v == -1 && PyErr_Occurred()) + return false; + out = static_cast(v); + return true; +} +%} + +%define PYCVC_STATE_EXEC_ENUM(TYPE, PYTYPE, NAME) +%typemap(in) TYPE (int pycvc_ev) { + if (!pycvc_enum_from_py($input, PYTYPE, NAME, pycvc_ev)) + SWIG_fail; + $1 = static_cast(pycvc_ev); +} +%typemap(out) TYPE { + $result = PyObject_CallFunction(PYTYPE, "i", static_cast($1)); + if (!$result) + SWIG_fail; +} +%typemap(typecheck, precedence=SWIG_TYPECHECK_INT32) TYPE { + $1 = pycvc_is_enum($input, PYTYPE) ? 1 : 0; +} +%enddef +PYCVC_STATE_EXEC_ENUM(cvc::state_exec::process_status, pycvc_enum_process_status, "process_status") +PYCVC_STATE_EXEC_ENUM(cvc::state_exec::scheduling_policy, pycvc_enum_scheduling_policy, + "scheduling_policy") + +%init %{ + pycvc_enum_process_status = pycvc_make_int_enum( + "process_status", + {{"ready", static_cast(cvc::state_exec::process_status::ready)}, + {"running", static_cast(cvc::state_exec::process_status::running)}, + {"paused", static_cast(cvc::state_exec::process_status::paused)}, + {"waiting", static_cast(cvc::state_exec::process_status::waiting)}, + {"terminated", static_cast(cvc::state_exec::process_status::terminated)}, + {"killed", static_cast(cvc::state_exec::process_status::killed)}}); + if (pycvc_enum_process_status != NULL) + PyDict_SetItemString(d, "process_status", pycvc_enum_process_status); + pycvc_enum_scheduling_policy = pycvc_make_int_enum( + "scheduling_policy", + {{"round_robin", static_cast(cvc::state_exec::scheduling_policy::round_robin)}, + {"priority", static_cast(cvc::state_exec::scheduling_policy::priority)}, + {"priority_rr", static_cast(cvc::state_exec::scheduling_policy::priority_rr)}}); + if (pycvc_enum_scheduling_policy != NULL) + PyDict_SetItemString(d, "scheduling_policy", pycvc_enum_scheduling_policy); +%} +%pythoncode %{ +process_status = _pycvc.process_status +scheduling_policy = _pycvc.scheduling_policy +%} + // ── parse_error: pycvc.parse_error, a RuntimeError with .line and .col ── %{ static PyObject *pycvc_parse_error_type = nullptr; @@ -1729,12 +2012,39 @@ static void pycvc_raise_parse_error(const cvc::state_exec::parse_error &e) { parse_error = _pycvc.parse_error %} +// ── no value: KeyError ── +// Where C++ returns an empty std::optional or a null value_t* (an unbound name, a pid with +// no result, an empty message queue), the wrapper throws pycvc::no_value and Python gets +// KeyError(key). It can't return None: None is the nil value. +%{ +namespace pycvc { +struct no_value { + std::variant key; +}; +} // namespace pycvc + +static void pycvc_raise_no_value(const pycvc::no_value &e) { + PyObject *key = nullptr; + if (const int *pid = std::get_if(&e.key)) { + key = PyLong_FromLong(*pid); + } else { + const std::string &s = std::get(e.key); + key = PyUnicode_DecodeUTF8(s.data(), static_cast(s.size()), "surrogateescape"); + } + if (key != nullptr) { + PyErr_SetObject(PyExc_KeyError, key); + Py_DECREF(key); + } +} +%} + // For the state_exec declarations (and Exec, below): the default mapping, with // parse_error first so it raises pycvc.parse_error. Scoped to this section, so // the modules that %import pycvc.i keep PYCVC_DEFAULT_EXCEPTION. -%exception { - try { - $action +%define PYCVC_STATE_EXEC_CATCH + catch (const pycvc::no_value &e) { + pycvc_raise_no_value(e); + SWIG_fail; } catch (const cvc::state_exec::parse_error &e) { pycvc_raise_parse_error(e); SWIG_fail; @@ -1745,8 +2055,36 @@ parse_error = _pycvc.parse_error } catch (...) { SWIG_exception(SWIG_RuntimeError, "pycvc: C++ exception (see libcvc)"); } +%enddef +%exception { + try { + $action + } + PYCVC_STATE_EXEC_CATCH } +// Stepping and running a scheduler release the GIL around the C++ call (pycvc::gil_release, from +// pycvc_lod.i; a throw restores it before the catch arms run), so other Python threads keep going +// while DSL programs run. A Python function a program calls takes the GIL back for its call. +%define PYCVC_STATE_EXEC_NOGIL(NAME) +%exception NAME { + try { + pycvc::gil_release pycvc_nogil; + $action + } + PYCVC_STATE_EXEC_CATCH +} +%enddef +PYCVC_STATE_EXEC_NOGIL(cvc::state_exec::scheduler::step) +PYCVC_STATE_EXEC_NOGIL(cvc::state_exec::scheduler::run) +PYCVC_STATE_EXEC_NOGIL(cvc::state_exec::async_scheduler::sync_step) +PYCVC_STATE_EXEC_NOGIL(cvc::state_exec::async_scheduler::sync_run) + +// environment is shared (environment_ptr), as in C++. The map members read and write as dicts. +%shared_ptr(cvc::state_exec::environment) +%naturalvar cvc::state_exec::environment::bindings; +%naturalvar cvc::state_exec::execute_options::signal_handlers; + namespace cvc { namespace state_exec { @@ -1826,15 +2164,255 @@ public: void encode_value(cvc::state &node, const value_t &val); value_t decode_value(cvc::state &node); -%nodefaultctor async_scheduler; -%nodefaultdtor async_scheduler; +// ── environment and builtins ── +// lookup() returns the bound value, or raises KeyError (C++ returns nullptr). bindings is the +// scope's own map, read and written as a dict (a copy: change bindings with set()). +struct environment { + std::unordered_map bindings; + std::shared_ptr outer; + void set(const std::string &name, value_t val); + void set_existing(const std::string &name, value_t val); + static std::shared_ptr extend(std::shared_ptr parent); +}; +%extend environment { + value_t lookup(const std::string &name) { + if (const cvc::state_exec::value_t *v = $self->lookup(name)) + return *v; + throw pycvc::no_value{name}; + } +} +typedef std::shared_ptr environment_ptr; + +// register_fn takes a Python callable (or a pycvc.value holding a native fn) where C++ takes a +// native_fn. The callable runs when a program calls it, on the thread stepping the scheduler; it +// gets the arguments as Python values and its result converts back, as for Exec.register_fn. +%nodefaultctor builtins; +%nodefaultdtor builtins; +class builtins { +public: + static environment_ptr make_default_environment(); +}; +%extend builtins { + static void register_fn(environment_ptr env, const std::string &name, PyObject *fn) { + if (!env) + throw std::invalid_argument("pycvc.builtins.register_fn: env is None"); + cvc::state_exec::value_t held; + void *p = nullptr; + if (SWIG_IsOK(SWIG_ConvertPtr(fn, &p, pycvc::value_swig_type(), 0)) && p != nullptr) + held = *static_cast(p); + if (const auto *native = std::get_if(&held.v)) { + cvc::state_exec::builtins::register_fn(env, name, *native); + return; + } + cvc::state_exec::builtins::register_fn( + env, name, pycvc::python_native_fn("pycvc.builtins.register_fn", name, fn)); + } +} + +std::string percent_encode(const std::string &s); +std::string percent_decode(const std::string &s); + +// ── processes and schedulers ── +// process_status and scheduling_policy are the IntEnums above; declared opaque here so SWIG adds +// no flat constants of its own. +enum class process_status; +enum class scheduling_policy; +const char *to_string(process_status s); +process_status parse_process_status(const std::string &s); + +// on_complete is a Python callable (or None) called with the process's result when it terminates, +// on the thread stepping the scheduler. Its return value is ignored; an exception it raises is +// raised from that step()/run() call. The uint64_t limits are Python ints. +struct execute_options { + std::string name; + int priority; + std::string uid; + std::string gid; + std::string root_path; + std::string owner; + unsigned long long max_steps; + double max_time; + double max_step_time; + unsigned long long max_memory; + unsigned long long max_messages; + unsigned long long max_message_bytes; + std::unordered_map signal_handlers; + environment_ptr env; +}; +%extend execute_options { + PyObject *_on_complete_get() const { + if (!$self->on_complete) { + Py_INCREF(Py_None); + return Py_None; + } + if (const auto *f = $self->on_complete.target()) { + Py_INCREF(f->callable.get()); + return f->callable.get(); + } + throw std::runtime_error("pycvc.execute_options.on_complete: holds a C++ callback"); + } + void _on_complete_set(PyObject *fn) { + if (fn == Py_None) + $self->on_complete = nullptr; + else + $self->on_complete = pycvc::make_python_on_complete(fn); + } + %pythoncode %{ + on_complete = property(_on_complete_get, _on_complete_set) + %} +} + +struct process_info { + int pid; + std::string name; + process_status status; + int priority; + std::string uid; + std::string gid; + unsigned long long step_count; + double elapsed_time; + unsigned long long current_memory; + unsigned long long peak_memory; + unsigned long long max_memory; + double max_time; + unsigned long long message_count; + unsigned long long max_messages; + unsigned long long message_bytes; + unsigned long long max_message_bytes; + int parent_pid; +}; +%extend process_info { + %pythoncode %{ + def __repr__(self): + return "" % ( + self.pid, self.name, self.status.name) + %} +} + +struct scheduler_stats { + int total_processes; + int running; + int ready; + int paused; + int terminated; + int killed; + unsigned long long total_steps; +}; +%extend scheduler_stats { + %pythoncode %{ + def __repr__(self): + return ("" % ( + self.total_processes, self.running, self.ready, self.paused, + self.terminated, self.killed, self.total_steps)) + %} +} + +// The members scheduler and async_scheduler share (scheduler_base plus their common extras). +// Returns of a C++ reference come back as copies. Not wrapped: mem_tracker(), and +// current_process() / queue_watch_event(), which take the process struct (PR 3). +%define PYCVC_STATE_EXEC_SCHEDULER_MEMBERS + int execute(const std::string &script, const execute_options &opts = execute_options()); + int execute(const value_t &expr, const execute_options &opts = execute_options()); + void stop(); + bool is_running() const; + bool pause(int pid); + bool resume(int pid); + bool kill(int pid); + int fork(int pid); + bool sleep(int pid, double seconds); + bool yield_frame(int pid); + bool receive_message(int pid, const std::string &path); + int deliver_to_receivers(const std::string &path, const value_t &msg); + size_t pending_message_count(const std::string &path) const; + size_t total_pending_messages() const; + size_t max_pending_messages; + bool set_priority(int pid, int priority); + bool set_max_steps(int pid, unsigned long long max_steps); + bool set_max_time(int pid, double seconds); + bool set_max_memory(int pid, unsigned long long bytes); + bool set_max_messages(int pid, unsigned long long count); + bool set_max_message_bytes(int pid, unsigned long long bytes); + bool send_signal(int pid, const std::string &signal); + std::vector list_processes() const; + std::optional get_process_info(int pid) const; + std::unordered_map get_results() const; + scheduler_stats get_stats() const; + scheduling_policy policy() const; + int process_count() const; + bool has_runnable() const; + int current_pid() const; + void register_watch_handler(int pid, int watch_id, value_t handler, const std::string &path); + void unregister_watch_handler(int pid, int watch_id); + void set_watch_root(cvc::state *root); +%enddef + +// get_result and pop_pending_message raise KeyError(pid / path) where C++ returns std::nullopt. +%define PYCVC_STATE_EXEC_SCHEDULER_EXTEND(CLASS) +%extend CLASS { + value_t get_result(int pid) const { + if (std::optional r = $self->get_result(pid)) + return *r; + throw pycvc::no_value{pid}; + } + value_t pop_pending_message(const std::string &path) { + if (std::optional m = $self->pop_pending_message(path)) + return *m; + throw pycvc::no_value{path}; + } +} +%enddef + +// The synchronous scheduler. run() and step() release the GIL while programs run. +class scheduler { +public: + scheduler(); + scheduler(scheduling_policy policy); + PYCVC_STATE_EXEC_SCHEDULER_MEMBERS + int step(); + std::unordered_map run(std::optional max_steps = std::nullopt, + std::optional max_time = std::nullopt); + void set_id(const std::string &id); + std::string id() const; + void load_settings(); + parse_options parser_settings() const; + static constexpr const char *settings_root = "state_exec"; + static constexpr const char *defaults_prefix = "state_exec.defaults"; + static constexpr const char *sched_prefix = "state_exec.schedulers"; +}; +PYCVC_STATE_EXEC_SCHEDULER_EXTEND(scheduler) + +// The scheduler libcvc's hosts use (app::exec_scheduler(), Exec). The coroutine step()/run() aren't +// wrapped; Python drives it with sync_step()/sync_run(), which release the GIL while programs run. +// post_message() is the one thread-safe call. class async_scheduler { public: + async_scheduler(); + async_scheduler(scheduling_policy policy); + PYCVC_STATE_EXEC_SCHEDULER_MEMBERS + int sync_step(); + std::unordered_map sync_run( + std::optional max_steps = std::nullopt, + std::optional max_time = std::nullopt); + int kill_owner(const std::string &owner); + int wake_awaiting(); + void post_message(const std::string &path, const value_t &msg); + void drain_ingress(); + std::optional get_exit_error(int pid) const; + bool reap(int pid); + void set_parser_defaults(const parse_options &defaults); + parse_options parser_defaults() const; static parse_options host_parser_defaults(); }; +PYCVC_STATE_EXEC_SCHEDULER_EXTEND(async_scheduler) } } +// Exec.scheduler() returns a reference into the Exec, so the scheduler keeps the Exec alive. +%pythonappend pycvc::Exec::scheduler %{ + val._pycvc_exec = self +%} + // ── Phase 4: DSL execution + Python-authored DSL functions ────────────── // pycvc.Exec(app) runs state_exec DSL programs over app's state root and lets a // Python callable be registered as a DSL function (register_fn). PyObject* is diff --git a/bindings/pycvc/pycvc_exec.cpp b/bindings/pycvc/pycvc_exec.cpp index e80639c01..e342d2302 100644 --- a/bindings/pycvc/pycvc_exec.cpp +++ b/bindings/pycvc/pycvc_exec.cpp @@ -92,9 +92,10 @@ std::shared_ptr own_callable(PyObject *callable) { // async fn): PyGILState_Ensure creates/binds a thread state and blocks until the // GIL is free, so the only rule is that whoever is idle-waiting for this worker // must have released the GIL (run()'s pump does). Contains a Python exception as -// a C++ error carrying the message. +// a C++ error carrying the message, prefixed with `what`. With `want_result` +// false the return value is ignored (an on_complete callback) and nil returned. value_t call_python(const std::shared_ptr &holder, std::span args, - const std::string &name) { + const std::string &what, bool want_result = true) { PyGILState_STATE gil = PyGILState_Ensure(); PyObject *pyargs = values_to_py_tuple(args); // nullptr (error set) if an arg can't convert PyObject *res = pyargs ? PyObject_CallObject(holder.get(), pyargs) : nullptr; @@ -102,30 +103,21 @@ value_t call_python(const std::shared_ptr &holder, std::span holder = own_callable(callable); - return [name, holder](std::span args) -> value_t { - return call_python(holder, args, name); - }; -} - std::string render_result(const value_t &v) { if (const auto *s = std::get_if(&v.v)) return *s; @@ -260,6 +252,32 @@ struct EndGuard { }; } // namespace +// A SYNCHRONOUS DSL native_fn: it runs inline on the thread stepping the scheduler (Exec::run's, +// or the caller of a wrapped scheduler's run/step, which release the GIL; call_python retakes it). +native_fn python_native_fn(const std::string &who, const std::string &name, PyObject *callable) { + if (callable == nullptr || !PyCallable_Check(callable)) + throw std::invalid_argument(who + ": '" + name + "' is not callable"); + if (is_coroutine_function(callable)) + throw std::invalid_argument(who + ": '" + name + + "' is an async def — use Exec.register_async_coro"); + std::shared_ptr holder = own_callable(callable); + const std::string what = "python DSL fn '" + name + "'"; + return [what, holder](std::span args) -> value_t { + return call_python(holder, args, what); + }; +} + +void python_on_complete::operator()(value_t result) const { + call_python(callable, std::span(&result, 1), "python on_complete", + /*want_result=*/false); +} + +python_on_complete make_python_on_complete(PyObject *callable) { + if (callable == nullptr || !PyCallable_Check(callable)) + throw std::invalid_argument("pycvc.execute_options.on_complete: not callable"); + return python_on_complete{own_callable(callable)}; +} + // Owns everything an evaluation needs. The intrinsics capture &ictx (a raw // pointer), so ictx must outlive any run — holding it here (Exec-lifetime) // satisfies that. Mirrors the C++ tests' make_exec_env(). @@ -339,12 +357,8 @@ Exec::~Exec() { } void Exec::register_fn(const std::string &name, PyObject *callable) { - if (callable == nullptr || !PyCallable_Check(callable)) - throw std::invalid_argument("pycvc.Exec.register_fn: '" + name + "' is not callable"); - if (is_coroutine_function(callable)) - throw std::invalid_argument("pycvc.Exec.register_fn: '" + name + - "' is an async def — use register_async_coro"); - builtins::register_fn(impl_->env, name, make_python_native_fn(name, callable)); + builtins::register_fn(impl_->env, name, + python_native_fn("pycvc.Exec.register_fn", name, callable)); } void Exec::register_async_fn(const std::string &name, PyObject *callable) { @@ -377,7 +391,7 @@ void Exec::register_async_fn(const std::string &name, PyObject *callable) { [holder, argv, name, inflight]() -> value_t { EndGuard guard{ inflight}; // --in-flight when this body returns/throws, before the post - return call_python(holder, *argv, name); + return call_python(holder, *argv, "python DSL fn '" + name + "'"); }, [name](const std::string &e) -> value_t { return async_error_dict("python async fn '" + name + "': " + e); @@ -577,4 +591,6 @@ void Exec::set_parser_defaults(const cvc::state_exec::parse_options &defaults) { impl_->sched.set_parser_defaults(defaults); } +cvc::state_exec::async_scheduler &Exec::scheduler() { return impl_->sched; } + } // namespace pycvc diff --git a/bindings/pycvc/pycvc_exec.h b/bindings/pycvc/pycvc_exec.h index b774ac2c4..3adb1d888 100644 --- a/bindings/pycvc/pycvc_exec.h +++ b/bindings/pycvc/pycvc_exec.h @@ -13,7 +13,10 @@ namespace cvc { class app; +namespace state_exec { +class async_scheduler; } +} // namespace cvc // Forward-declare PyObject so this SWIG-visible header needs no . struct _object; @@ -94,6 +97,11 @@ class Exec { cvc::state_exec::parse_options parser_defaults() const; void set_parser_defaults(const cvc::state_exec::parse_options &defaults); + // This Exec's private scheduler, the one run() drives. A program submitted to it directly + // (scheduler().execute(...)) runs alongside run()'s programs, and run() steps it too. In Python + // the returned scheduler keeps this Exec alive. + cvc::state_exec::async_scheduler &scheduler(); + private: struct ExecImpl; std::shared_ptr impl_; diff --git a/bindings/pycvc/pycvc_value.h b/bindings/pycvc/pycvc_value.h index c9f4bf28a..8cc17b305 100644 --- a/bindings/pycvc/pycvc_value.h +++ b/bindings/pycvc/pycvc_value.h @@ -18,7 +18,9 @@ #pragma once #include +#include #include +#include struct _object; typedef _object PyObject; @@ -35,4 +37,24 @@ PyObject *values_to_py_tuple(std::span values); /// Convert `o` into `out`. Returns false, with a Python exception set, when it can't. bool py_to_value(PyObject *o, cvc::state_exec::value_t &out); +// Python callables as state_exec callbacks. These are defined in pycvc_exec.cpp. Each one owns a +// reference to its callable and takes the GIL for the call, so it can run on any thread. + +/// A DSL native function that calls `callable`: its arguments convert to Python and its return +/// value converts back, and a Python exception becomes a C++ error naming `name`. Throws +/// std::invalid_argument, prefixed with `who`, when `callable` isn't callable or is an async def. +cvc::state_exec::native_fn python_native_fn(const std::string &who, const std::string &name, + PyObject *callable); + +/// An execute_options::on_complete that calls `callable` with the process's result and ignores +/// what it returns. A Python exception becomes a C++ error, thrown from the scheduler call that +/// finished the process. It keeps the callable, so pycvc can hand it back. +struct python_on_complete { + std::shared_ptr callable; + void operator()(cvc::state_exec::value_t result) const; +}; + +/// The python_on_complete for `callable`. Throws std::invalid_argument when it isn't callable. +python_on_complete make_python_on_complete(PyObject *callable); + } // namespace pycvc diff --git a/bindings/pycvc/test_pycvc_state_exec_sched.py b/bindings/pycvc/test_pycvc_state_exec_sched.py new file mode 100644 index 000000000..8de82aa61 --- /dev/null +++ b/bindings/pycvc/test_pycvc_state_exec_sched.py @@ -0,0 +1,481 @@ +"""state_exec parity, part 2: environments, builtins and the schedulers. + +pycvc.scheduler and pycvc.async_scheduler are the real C++ classes, with +execute_options, process_info, scheduler_stats, and the process_status and +scheduling_policy enums (enum.IntEnum classes). Most tests here repeat a C++ test +from src/cvc/tests/state_exec_scheduler_test.cpp or state_exec_async_test.cpp, +on both schedulers where both have the call. docs/PYCVC_STATE_EXEC_PARITY.md has +the inventory. +""" + +import threading +import time + +import pycvc +from pycvc import process_status, scheduling_policy, symbol + + +def _raises(exc, fn, *args): + try: + fn(*args) + except exc as e: + return e + raise AssertionError(f"{fn.__name__}{args!r} did not raise {exc.__name__}") + + +def _schedulers(): + """A fresh sync and async scheduler, each with its run and step calls.""" + s = pycvc.scheduler() + a = pycvc.async_scheduler() + return [(s, s.run, s.step), (a, a.sync_run, a.sync_step)] + + +def _opts(**fields): + o = pycvc.execute_options() + for k, v in fields.items(): + setattr(o, k, v) + return o + + +# ── enums ─────────────────────────────────────────────────────────────── + + +def test_enums_are_int_enums_under_the_cpp_names(): + assert [m.name for m in process_status] == [ + "ready", "running", "paused", "waiting", "terminated", "killed"] + assert [m.name for m in scheduling_policy] == ["round_robin", "priority", "priority_rr"] + assert process_status.killed == 5 # an IntEnum, with the C++ values + assert pycvc.to_string(process_status.waiting) == "waiting" + assert pycvc.parse_process_status("paused") is process_status.paused + assert pycvc.parse_process_status("bogus") is process_status.ready # as in C++ + # A plain int is a value, not an enum, so to_string keeps both C++ overloads. + assert pycvc.to_string(2) == "2" + # An enum argument must be the enum. + _raises(TypeError, pycvc.scheduler, 1) + assert pycvc.scheduler(scheduling_policy.priority).policy() is scheduling_policy.priority + assert pycvc.async_scheduler().policy() is scheduling_policy.round_robin + + +# ── environment and builtins ──────────────────────────────────────────── + + +def test_environment_is_the_scope_chain(): + env = pycvc.environment() + env.set("x", 1) + assert env.lookup("x") == 1 + child = pycvc.environment.extend(env) + assert child.outer is not None and child.outer.lookup("x") == 1 + assert child.lookup("x") == 1 # lookup walks the chain + child.set("y", [1, "two"]) + assert child.bindings == {"y": [1, "two"]} # this scope's own map + child.set_existing("x", 5) # the nearest scope that has it + assert env.lookup("x") == 5 and "x" not in child.bindings + # Bound nowhere: the C++ sets it in the outermost scope (its header comment says this + # scope, but the recursion ends at the root). + child.set_existing("z", None) + assert "z" not in child.bindings and env.bindings == {"x": 5, "z": None} + # C++ returns nullptr for an unbound name; Python raises KeyError (None is nil). + assert _raises(KeyError, child.lookup, "nope").args == ("nope",) + env.bindings = {"a": symbol("b")} # replaces the map, as assigning it does in C++ + assert env.bindings == {"a": symbol("b")} + _raises(KeyError, child.lookup, "x") + _raises(TypeError, setattr, env, "bindings", {1: 2}) + assert pycvc.environment.extend(None).outer is None + + +def test_builtins_and_register_fn(): + env = pycvc.builtins.make_default_environment() + plus = env.lookup("+") + assert isinstance(plus, pycvc.value) and plus.type_name() == "native_fn" + calls = [] + + def greet(name): + calls.append(name) + return "hi " + name + + pycvc.builtins.register_fn(env, "greet", greet) + pycvc.builtins.register_fn(env, "plus", plus) # a native fn moves as a pycvc.value + for sched, run, _ in _schedulers(): + pid = sched.execute('(list (greet "ann") (plus 2 3))', _opts(env=env)) + assert run()[pid] == ["hi ann", 5] + assert calls == ["ann", "ann"] + # Not callable, no environment, or an async def: refused at registration. + _raises(RuntimeError, pycvc.builtins.register_fn, env, "bad", 42) + _raises(RuntimeError, pycvc.builtins.register_fn, None, "f", greet) + + async def coro(): + return 1 + + _raises(RuntimeError, pycvc.builtins.register_fn, env, "c", coro) + + +def test_a_python_fn_error_surfaces_as_each_scheduler_reports_it(): + env = pycvc.builtins.make_default_environment() + pycvc.builtins.register_fn(env, "boom", lambda: 1 / 0) + # The sync scheduler throws from run(), as a throwing native fn does in C++. + s = pycvc.scheduler() + s.execute("(boom)", _opts(env=env)) + assert "division by zero" in str(_raises(RuntimeError, s.run)) + # The async scheduler kills the process and keeps the reason as its exit error. + a = pycvc.async_scheduler() + pid = a.execute("(boom)", _opts(env=env)) + assert a.sync_run() == {} + assert a.get_process_info(pid).status is process_status.killed + assert "division by zero" in a.get_exit_error(pid) + assert a.reap(pid) and a.get_process_info(pid) is None + assert a.get_exit_error(999) is None + + +def test_percent_encode_and_decode(): + assert pycvc.percent_encode("a b/é~") == "a%20b%2F%C3%A9~" + assert pycvc.percent_decode("a%20b%2F%C3%A9~") == "a b/é~" + assert pycvc.percent_decode("x+y%zz") == "x y%zz" # '+' is a space; a bad escape stays + + +# ── execute_options, process_info, scheduler_stats ────────────────────── + + +def test_execute_options_has_the_cpp_fields_and_defaults(): + o = pycvc.execute_options() + assert (o.name, o.priority, o.uid, o.gid, o.root_path, o.owner) == ("", 0, "", "", "", "") + assert (o.max_steps, o.max_time, o.max_step_time, o.max_memory) == (0, 0.0, 0.0, 0) + assert (o.max_messages, o.max_message_bytes) == (0, 0) + assert o.signal_handlers == {} and o.env is None and o.on_complete is None + o.max_memory = 2**63 # the uint64_t limits take the full range + assert o.max_memory == 2**63 + _raises(OverflowError, setattr, o, "max_steps", -1) + + +def test_process_info_and_stats_report_each_process(): + for sched, run, _ in _schedulers(): + p1 = sched.execute("42", _opts(name="my-process", uid="user1", gid="group1")) + p2 = sched.execute("(+ 1 2)") + info = sched.get_process_info(p1) + assert (info.pid, info.name, info.uid, info.gid) == (p1, "my-process", "user1", "group1") + assert info.status is process_status.ready and info.parent_pid == -1 + assert sched.get_process_info(p2).name == "proc-%d" % p2 + assert sched.get_process_info(999) is None + stats = sched.get_stats() + assert (stats.total_processes, stats.ready, stats.terminated) == (2, 2, 0) + run() + stats = sched.get_stats() + assert stats.terminated == 2 and stats.total_steps > 0 + procs = sched.list_processes() + assert [p.pid for p in procs] == [p1, p2] # sorted by pid + assert all(p.status is process_status.terminated for p in procs) + assert procs[1].step_count > 0 + + +# ── running programs ──────────────────────────────────────────────────── + + +def test_both_schedulers_run_programs(): + for sched, run, step in _schedulers(): + assert not sched.has_runnable() and sched.process_count() == 0 + p1 = sched.execute("(+ 1 2)") + p2 = sched.execute("(* 3 4)") + fib = sched.execute( + "(begin (defun fib (n) (if (<= n 1) n (+ (fib (- n 1)) (fib (- n 2))))) (fib 10))") + nil = sched.execute("nil") + assert sched.has_runnable() and sched.process_count() == 4 + # No result yet: C++ returns std::nullopt, Python raises KeyError(pid). + assert _raises(KeyError, sched.get_result, p1).args == (p1,) + assert run() == {p1: 3, p2: 12, fib: 55, nil: None} + assert sched.get_result(p1) == 3 + assert sched.get_result(nil) is None # nil is a result, not a missing one + assert sched.get_results() == {p1: 3, p2: 12, fib: 55, nil: None} + assert not sched.has_runnable() and not sched.is_running() + + +def test_execute_takes_a_script_or_a_value(): + for sched, run, _ in _schedulers(): + script = sched.execute("(* 6 7)") + expr = sched.execute(pycvc.parse("(* 6 7)")) # an expression, as execute(value_t) + quoted = sched.execute([symbol("quote"), [1, symbol("a")]]) + assert run() == {script: 42, expr: 42, quoted: [1, symbol("a")]} + + +def test_step_by_step_and_run_limits(): + for sched, run, step in _schedulers(): + pid = sched.execute("(+ 1 2)") + stepped = 0 + while sched.has_runnable(): + step() + stepped += 1 + assert stepped > 0 and sched.get_result(pid) == 3 + sched.execute("(begin 1 2 3 4 5 6 7 8 9 10)") + run(3) # at most three steps in all + assert sched.has_runnable() + _raises(OverflowError, run, -1) + run(None, 30.0) + assert not sched.has_runnable() + + +def test_priority_policies(): + for policy in (scheduling_policy.priority, scheduling_policy.priority_rr): + for sched in (pycvc.scheduler(policy), pycvc.async_scheduler(policy)): + run = sched.run if isinstance(sched, pycvc.scheduler) else sched.sync_run + low = sched.execute("(begin 1 2 3)", _opts(priority=10)) + high = sched.execute("(begin 4 5 6)", _opts(priority=-10)) + assert run() == {low: 3, high: 6} + + +def test_process_control(): + for sched, run, step in _schedulers(): + pid = sched.execute("(begin 1 2 3 4 5)") + step() + assert sched.pause(pid) + assert sched.get_process_info(pid).status is process_status.paused + assert step() == 0 # nothing runnable while paused + assert sched.resume(pid) and not sched.resume(pid) + assert sched.get_process_info(pid).status is process_status.ready + assert sched.set_priority(pid, -5) and sched.get_process_info(pid).priority == -5 + assert sched.sleep(pid, 0.01) + assert sched.get_process_info(pid).status is process_status.waiting + # A sleeper counts as runnable for the sync scheduler only; the async one's driver + # steps it after the deadline (as the C++ tests do). + assert sched.has_runnable() is isinstance(sched, pycvc.scheduler) + time.sleep(0.03) + for _ in range(64): + if sched.get_process_info(pid).status is process_status.terminated: + break + step() + assert sched.get_result(pid) == 5 + assert not sched.kill(pid) # already terminated + for call in (sched.pause, sched.kill, sched.yield_frame): + assert not call(999) + assert sched.fork(999) == -1 + + doomed = sched.execute("(begin 1 2 3 4 5)") + step() + assert sched.kill(doomed) + assert sched.get_process_info(doomed).status is process_status.killed + + +def test_fork_copies_the_process(): + for sched, run, step in _schedulers(): + parent = sched.execute("(begin 1 2 3)", _opts(priority=-5, uid="user1", gid="group1")) + step() + child = sched.fork(parent) + assert child > 0 and child != parent + info = sched.get_process_info(child) + assert (info.parent_pid, info.priority) == (parent, -5) + assert (info.uid, info.gid) == ("user1", "group1") + assert info.status is process_status.ready + run() + assert sched.get_result(parent) == 3 and sched.get_result(child) == 3 + + +def test_limits_kill_the_process(): + for sched, run, step in _schedulers(): + capped = sched.execute("(begin 1 2 3 4 5 6 7 8 9 10)", _opts(max_steps=5)) + later = sched.execute("(begin 1 2 3 4 5 6 7 8 9 10)") + assert sched.set_max_steps(later, 3) + assert sched.set_max_time(later, 10.0) and sched.set_max_memory(later, 1 << 20) + assert sched.set_max_messages(later, 100) and sched.set_max_message_bytes(later, 4096) + assert not sched.set_max_steps(999, 1) + info = sched.get_process_info(later) + assert (info.max_time, info.max_memory, info.max_messages, info.max_message_bytes) == ( + 10.0, 1 << 20, 100, 4096) + run() + assert sched.get_process_info(capped).status is process_status.killed + assert sched.get_process_info(later).status is process_status.killed + + +def test_signals(): + handler = [symbol("begin"), 42] + for sched, run, step in _schedulers(): + o = _opts(signal_handlers={"SIGUSR1": handler}) + assert o.signal_handlers == {"SIGUSR1": handler} + pid = sched.execute("(begin 1 2 3 4 5)", o) + step() + assert sched.send_signal(pid, "SIGUSR1") + run() + assert sched.get_process_info(pid).status is process_status.terminated + assert not sched.send_signal(pid, "SIGUSR1") # terminated + doomed = sched.execute("(begin 1 2 3 4 5 6 7 8 9 10)") + step() + assert sched.send_signal(doomed, "SIGKILL") + assert sched.get_process_info(doomed).status is process_status.killed + assert not sched.send_signal(999, "SIGUSR1") + + +def test_on_complete_is_a_python_callable(): + results = [] + + def done(value): + results.append(value) + return {1, 2} # ignored, so not converted + + o = _opts(on_complete=done) + assert o.on_complete is done + for sched, run, _ in _schedulers(): + sched.execute("(list 1 2)", o) + run() + assert results == [[1, 2], [1, 2]] + o.on_complete = None + assert o.on_complete is None + _raises(RuntimeError, setattr, o, "on_complete", 42) + # A raising on_complete is raised from the call that finished the process. + s = pycvc.scheduler() + s.execute("7", _opts(on_complete=lambda v: 1 / 0)) + assert "division by zero" in str(_raises(RuntimeError, s.run)) + + +def test_messages_queue_until_received(): + for sched, run, step in _schedulers(): + # No receiver: the message queues under its path (FIFO). + assert sched.deliver_to_receivers("test.q", {"n": 1}) == 0 + sched.deliver_to_receivers("test.q", {"n": 2}) + assert sched.pending_message_count("test.q") == 2 and sched.total_pending_messages() == 2 + assert sched.pop_pending_message("test.q") == {"n": 1} + assert sched.pop_pending_message("test.q") == {"n": 2} + # Empty: C++ returns std::nullopt, Python raises KeyError(path). + assert _raises(KeyError, sched.pop_pending_message, "test.q").args == ("test.q",) + # A waiting receiver gets the message instead, and wakes. + pid = sched.execute("(begin 1 2 3 4 5)") + step() + assert sched.receive_message(pid, "test.inbox") + assert sched.get_process_info(pid).status is process_status.waiting + assert sched.deliver_to_receivers("test.inbox", "hello") == 1 + assert sched.get_process_info(pid).status is process_status.ready + assert not sched.receive_message(999, "test.inbox") + # The queue is bounded by max_pending_messages. + assert sched.max_pending_messages == 1024 + sched.max_pending_messages = 2 + for n in range(4): + sched.deliver_to_receivers("test.b", n) + assert sched.pending_message_count("test.b") == 2 + + +def test_watch_handlers_register_and_unregister(): + app = pycvc.make_app() + for sched, run, step in _schedulers(): + sched.set_watch_root(pycvc.State.instance(app)) + pid = sched.execute("(begin 1 2 3)") + sched.register_watch_handler(pid, 1, [symbol("begin"), 1], "watched.path") + sched.unregister_watch_handler(pid, 1) + run() + assert sched.get_result(pid) == 3 + sched.set_watch_root(None) + + +def test_scheduler_settings_come_from_the_tree(): + app = pycvc.make_app() + root = pycvc.State.instance(app) + root.child("state_exec.defaults.max_pending_messages").value("512") + root.child("state_exec.schedulers.py-sched.max_pending_messages").value("2048") + root.child("state_exec.schedulers.py-sched.parser.max_depth").value("12") + s = pycvc.scheduler() + assert s.parser_settings().max_depth == 256 + s.set_watch_root(root) + s.set_id("py-sched") + assert s.id() == "py-sched" + s.load_settings() + assert s.max_pending_messages == 2048 + assert s.parser_settings().max_depth == 12 + assert pycvc.scheduler.settings_root == "state_exec" + assert pycvc.scheduler.defaults_prefix == "state_exec.defaults" + assert pycvc.scheduler.sched_prefix == "state_exec.schedulers" + + +# ── async_scheduler's own calls ───────────────────────────────────────── + + +def test_async_scheduler_extras(): + a = pycvc.async_scheduler() + assert a.parser_defaults().step_tokens == 0 # a bare async_scheduler parses at submit + assert pycvc.async_scheduler.host_parser_defaults().step_tokens == 1024 + a.set_parser_defaults(pycvc.async_scheduler.host_parser_defaults()) + assert a.parser_defaults().step_tokens == 1024 + + a1 = a.execute("(begin 1 2 3 4 5)", _opts(owner="docA")) + a2 = a.execute("(begin 1 2 3 4 5)", _opts(owner="docA")) + b1 = a.execute("(begin 1 2 3 4 5)", _opts(owner="docB")) + a.sync_step() + assert a.kill_owner("docA") == 2 and a.kill_owner("docA") == 0 + assert a.get_process_info(a1).status is process_status.killed + assert a.get_process_info(a2).status is process_status.killed + assert a.get_process_info(b1).status is not process_status.killed + + # (await ...) parking: only wake_awaiting, once per frame, readies it. + assert a.yield_frame(b1) + a.sync_step() + assert a.get_process_info(b1).status is process_status.waiting + assert a.wake_awaiting() == 1 + assert a.get_process_info(b1).status is not process_status.waiting + + +def test_post_message_from_another_thread(): + a = pycvc.async_scheduler() + pid = a.execute("(begin 1 2 3 4 5)") + a.sync_step() + assert a.receive_message(pid, "chan.x") + worker = threading.Thread(target=a.post_message, args=("chan.x", "done")) + worker.start() + worker.join() + a.drain_ingress() # applied on this (the scheduler's) thread + assert a.get_process_info(pid).status is not process_status.waiting + assert a.sync_run()[pid] == 5 + + +def test_run_releases_the_gil(): + # A Python thread keeps running while a scheduler runs a long program: run() and + # sync_run() release the GIL around the C++ call. + for sched, run, _ in _schedulers(): + sched.execute("(while 1 1)") + ticks = [] + stop = threading.Event() + + def spin(): + while not stop.is_set(): + ticks.append(time.perf_counter()) + + t = threading.Thread(target=spin) + t.start() + while not ticks: + time.sleep(0.001) + t0 = time.perf_counter() + run(None, 0.3) + t1 = time.perf_counter() + stop.set() + t.join() + assert t1 - t0 >= 0.1 + # A call that held the GIL would stop the thread for its whole duration; it could + # only run at the call's edges. So look in the middle third. + third = (t1 - t0) / 3 + assert any(t0 + third < x < t1 - third for x in ticks) + + +# ── Exec ──────────────────────────────────────────────────────────────── + + +def test_exec_exposes_its_scheduler(): + ex = pycvc.Exec(pycvc.make_app()) + sched = ex.scheduler() + assert isinstance(sched, pycvc.async_scheduler) + assert sched.parser_defaults().step_tokens == 1024 # host_parser_defaults, as set by Exec + o = pycvc.parse_options() + o.max_depth = 40 + ex.set_parser_defaults(o) + assert sched.parser_defaults().max_depth == 40 # the same scheduler + o.max_depth = 256 + sched.set_parser_defaults(o) + assert ex.parser_defaults().max_depth == 256 + # A program submitted to it directly runs while run() drives the scheduler. + pid = sched.execute("(+ 20 22)") + assert ex.run("1") == "1" + assert sched.get_result(pid) == 42 + # run() reaps its own processes, so only ours is left. + assert [p.pid for p in sched.list_processes()] == [pid] + # The scheduler keeps its Exec alive. + del ex + assert sched.process_count() == 1 and sched.reap(pid) and sched.process_count() == 0 + + +if __name__ == "__main__": + tests = [v for k, v in sorted(globals().items()) if k.startswith("test_") and callable(v)] + for t in tests: + t() + print(" ok:", t.__name__) + print("pycvc state_exec scheduler parity tests: OK") diff --git a/docs/PYCVC_STATE_EXEC_PARITY.md b/docs/PYCVC_STATE_EXEC_PARITY.md index 6e9ce0813..4bb041bdc 100644 --- a/docs/PYCVC_STATE_EXEC_PARITY.md +++ b/docs/PYCVC_STATE_EXEC_PARITY.md @@ -40,11 +40,28 @@ approach agreed for closing the gap, and the plan. GIL around Python callbacks, an asyncio loop for coroutines, and compute-pool offload. It uses the same value conversion as everything else. `Exec.run` raises `pycvc.parse_error` for a syntax error whether the script was parsed at submit or read inside the process. + `Exec.scheduler()` returns its `async_scheduler`, which keeps the `Exec` alive. +6. **Enums are `enum.IntEnum` classes** with the C++ names and values: + `pycvc.process_status.killed` is `process_status::killed`. An enum argument must be a member + of its enum, so a plain int stays a value: `to_string(process_status)` and `to_string(value_t)` + keep their shared C++ name. +7. **Empty results.** Where C++ returns an empty `std::optional` or a null pointer, Python gets + `None`, except for a value: `None` is nil, so `get_result`, `pop_pending_message` and + `environment::lookup` raise `KeyError(pid / path / name)` instead. +8. **Callables cross as Python callables.** Where C++ takes a `native_fn` or a + `std::function` (`builtins::register_fn`, `execute_options::on_complete`), Python passes a + callable. It runs on the thread stepping the scheduler and takes the GIL for its call; a + Python exception becomes the C++ error a throwing callback would be. `on_complete` reads back + as the callable that was set. +9. **The GIL is released while programs run:** `scheduler.run`/`step` and + `async_scheduler.sync_run`/`sync_step`, so other Python threads keep going. The schedulers' + thread safety is the C++ one: while a scheduler runs, another thread may call only + `async_scheduler.post_message`. ## Inventory The audit (2026-10-17) found 97 public items in the 20 headers. Counted individually, with the 38 -`builtin_*` functions and the 7 `MIME_*` constants, it is 140. Status after PR 1: +`builtin_*` functions and the 7 `MIME_*` constants, it is 140. Status after PR 2: | Header | Item | Python | Status | |---|---|---|---| @@ -58,19 +75,21 @@ The audit (2026-10-17) found 97 public items in the 20 headers. Counted individu | | `value_t` | native types, or `pycvc.value` | wrapped (PR 1), see the table above | | | `to_string`, `values_equal` | `pycvc.to_string`, `pycvc.values_equal` | wrapped (PR 1) | | | `make_list`, `make_dict`, `make_bytes`, `nil_value`, `true_value`, `false_value` | `list`, `dict`, `bytes`, `None`, `True`, `False` | exception: the native Python values are the equivalents | -| | `environment` | | PR 2 | +| | `environment` (`bindings`, `outer`, `lookup`, `set`, `set_existing`, `extend`) | `pycvc.environment` | wrapped (PR 2), shared like `environment_ptr`. `bindings` reads and writes as a dict (a copy); `lookup` raises `KeyError` where C++ returns nullptr | | | `closure`, `data_object`, `generator` | `pycvc.value` (opaque) | wrapped as opaque handles; no member access | | | `time_budget`, `eval_deadline_guard`, `eval_deadline_expired`, `thread_cpu_seconds` | none | exception: thread-local evaluator internals | | state_value_codec.h | `encode_value`, `decode_value` | `pycvc.encode_value(State, v)`, `pycvc.decode_value(State)` | wrapped (PR 1) | | | `encode_environment`/`decode_environment`, `encode_evaluator_state`/`decode_evaluator_state` | | PR 3, with the environment and evaluator types | -| scheduler_base.h | `execute_options`, `process_info`, `scheduler_stats`, `scheduling_policy` | | PR 2 | -| scheduler.h | `scheduler` (about 45 members) | | PR 2 | -| async_scheduler.h | `async_scheduler` (about 55 members) | `pycvc.async_scheduler.host_parser_defaults()` only | PR 2. Coroutine `step()`/`run()` are an exception: Python gets the existing `sync_step()`/`sync_run()` | -| process.h | `process_status`, `to_string(process_status)`, `parse_process_status` | | PR 2 | -| | `process` (about 35 fields), `make_process` | | PR 2, read-only where safe | -| builtins.h | `builtins::make_default_environment`, `builtins::register_fn` | `Exec.register_fn` | PR 2. `register_fn` takes a Python callable | +| scheduler_base.h | `execute_options`, `process_info`, `scheduler_stats` | `pycvc.execute_options`, `pycvc.process_info`, `pycvc.scheduler_stats` | wrapped (PR 2). `on_complete` is a Python callable, `signal_handlers` a dict, the `uint64_t` fields Python ints | +| | `scheduling_policy` | `pycvc.scheduling_policy` | wrapped (PR 2), an `IntEnum` | +| | `scheduler_base` | none | exception: the abstract interface the intrinsics use; Python has the two concrete classes | +| scheduler.h | `scheduler` (about 45 members) | `pycvc.scheduler` | wrapped (PR 2). `run`/`step` release the GIL. `execute(str)` is the script overload, so a string can't be passed as a `value_t` expression (write it as the script `"\"text\""`). `id()`, `parser_settings()` return copies. Not wrapped: `mem_tracker()` (see memory_tracker.h), and `current_process()` / `queue_watch_event()`, which take the `process` struct (PR 3) | +| async_scheduler.h | `async_scheduler` (about 55 members) | `pycvc.async_scheduler` | wrapped (PR 2), with the same notes as `scheduler`; `sync_run`/`sync_step` release the GIL, and `parser_defaults()` returns a copy. Coroutine `step()`/`run()` are an exception: Python gets `sync_step()`/`sync_run()` | +| process.h | `process_status`, `to_string(process_status)`, `parse_process_status` | `pycvc.process_status` (an `IntEnum`), `pycvc.to_string`, `pycvc.parse_process_status` | wrapped (PR 2) | +| | `process` (about 35 fields), `make_process` | | PR 3, with `evaluator_state`, which a process holds | +| builtins.h | `builtins::make_default_environment`, `builtins::register_fn` | `pycvc.builtins.make_default_environment()`, `pycvc.builtins.register_fn(env, name, fn)` | wrapped (PR 2). `fn` is a Python callable, or a `pycvc.value` holding a native fn | | | 38 `builtin_*` functions | none | exception: DSL builtins, called from DSL source | -| | `percent_encode`, `percent_decode` | | PR 2 | +| | `percent_encode`, `percent_decode` | `pycvc.percent_encode`, `pycvc.percent_decode` | wrapped (PR 2) | | stackless_evaluator.h, evaluator.h | `stackless_evaluator`, `evaluator`, `evaluator_state`, `evaluation_stats`, the evaluation exceptions | | PR 3 | | async_evaluator.h, async_stackless_evaluator.h | the async evaluators | none | exception: coroutine API | | resource_policy.h | `resource_policy`, `process_limits`, `validate_limits` | | PR 3 | @@ -86,18 +105,23 @@ The audit (2026-10-17) found 97 public items in the 20 headers. Counted individu Each PR is stacked on the previous one and includes pytest checks against the C++ behaviour. -1. **PR 1 (this):** values, parsing and the value codec. +1. **PR 1 (#661):** values, parsing and the value codec. - `pycvc_value.h`: the strict shared converter, used by `Exec` too. - `pycvc.symbol` and `pycvc.value`. - `parse`, `parse_all`, `parse_error` with position, `script_reader`, `to_string`, `values_equal`, `encode_value`, `decode_value`. - `bindings/pycvc/test_pycvc_state_exec.py`. - The pycvc README's outdated Status and Design sections are rewritten. -2. **PR 2:** `environment` and `builtins`; `execute_options`, `process_info`, `scheduler_stats` and - the enums; `scheduler` and `async_scheduler` as real classes, with their non-coroutine - methods. `Exec` exposes its scheduler. -3. **PR 3:** the evaluators and `evaluator_state`, the environment and evaluator-state codec, - `resource_policy`, and `register_intrinsics` / `intrinsics_context`. +2. **PR 2 (this):** `environment` and `builtins`; `execute_options`, `process_info`, + `scheduler_stats` and the enums; `scheduler` and `async_scheduler` as real classes, with their + non-coroutine methods. `Exec.scheduler()` returns its scheduler. + `bindings/pycvc/test_pycvc_state_exec_sched.py` repeats the C++ scheduler tests on both + classes. +3. **PR 3:** the evaluators and `evaluator_state`, the `process` struct and the scheduler calls + that take it, the environment and evaluator-state codec, `resource_policy`, and + `register_intrinsics` / `intrinsics_context`. With `register_intrinsics`, a program submitted + to `Exec.scheduler()` can get the intrinsics `Exec.run` programs have; until then an + environment built in Python has only the builtins and what `register_fn` adds. Known gaps after PR 1: - Invalid UTF-8 in a DSL string raises when it crosses to Python, but no Python test produces such From 0c24a141ca57838683b07fd2f4b30355547abb89 Mon Sep 17 00:00:00 2001 From: Joe Rivera Date: Sat, 17 Oct 2026 20:47:48 -0500 Subject: [PATCH 13/17] Rerun CI on master (the merge push ran while the PR still targeted #653's branch) From b2b772a83b441d5986f0792205834c7131e17147 Mon Sep 17 00:00:00 2001 From: Joe Rivera Date: Sat, 17 Oct 2026 21:30:56 -0500 Subject: [PATCH 14/17] pycvc: environment.frozen (#660), and the set_existing note now that its comment is fixed --- bindings/pycvc/pycvc.i | 4 +++- bindings/pycvc/test_pycvc_state_exec_sched.py | 16 +++++++++++++--- docs/PYCVC_STATE_EXEC_PARITY.md | 2 +- 3 files changed, 17 insertions(+), 5 deletions(-) diff --git a/bindings/pycvc/pycvc.i b/bindings/pycvc/pycvc.i index 2e8269c48..67c0d6855 100644 --- a/bindings/pycvc/pycvc.i +++ b/bindings/pycvc/pycvc.i @@ -2166,10 +2166,12 @@ value_t decode_value(cvc::state &node); // ── environment and builtins ── // lookup() returns the bound value, or raises KeyError (C++ returns nullptr). bindings is the -// scope's own map, read and written as a dict (a copy: change bindings with set()). +// scope's own map, read and written as a dict (a copy: change bindings with set()). set() on a +// frozen scope raises RuntimeError, as C++ throws std::logic_error. struct environment { std::unordered_map bindings; std::shared_ptr outer; + bool frozen; void set(const std::string &name, value_t val); void set_existing(const std::string &name, value_t val); static std::shared_ptr extend(std::shared_ptr parent); diff --git a/bindings/pycvc/test_pycvc_state_exec_sched.py b/bindings/pycvc/test_pycvc_state_exec_sched.py index 8de82aa61..d4008ef59 100644 --- a/bindings/pycvc/test_pycvc_state_exec_sched.py +++ b/bindings/pycvc/test_pycvc_state_exec_sched.py @@ -70,9 +70,7 @@ def test_environment_is_the_scope_chain(): assert child.bindings == {"y": [1, "two"]} # this scope's own map child.set_existing("x", 5) # the nearest scope that has it assert env.lookup("x") == 5 and "x" not in child.bindings - # Bound nowhere: the C++ sets it in the outermost scope (its header comment says this - # scope, but the recursion ends at the root). - child.set_existing("z", None) + child.set_existing("z", None) # bound nowhere: the outermost unfrozen scope assert "z" not in child.bindings and env.bindings == {"x": 5, "z": None} # C++ returns nullptr for an unbound name; Python raises KeyError (None is nil). assert _raises(KeyError, child.lookup, "nope").args == ("nope",) @@ -83,6 +81,18 @@ def test_environment_is_the_scope_chain(): assert pycvc.environment.extend(None).outer is None +def test_a_frozen_scope_is_read_only(): + shared = pycvc.environment() + shared.set("k", 1) + assert not shared.frozen + shared.frozen = True + _raises(RuntimeError, shared.set, "k", 2) # C++ throws std::logic_error + child = pycvc.environment.extend(shared) + assert child.lookup("k") == 1 + child.set_existing("k", 3) # never written into the frozen scope: shadowed here instead + assert child.bindings == {"k": 3} and shared.bindings == {"k": 1} + + def test_builtins_and_register_fn(): env = pycvc.builtins.make_default_environment() plus = env.lookup("+") diff --git a/docs/PYCVC_STATE_EXEC_PARITY.md b/docs/PYCVC_STATE_EXEC_PARITY.md index 4bb041bdc..39b477342 100644 --- a/docs/PYCVC_STATE_EXEC_PARITY.md +++ b/docs/PYCVC_STATE_EXEC_PARITY.md @@ -75,7 +75,7 @@ The audit (2026-10-17) found 97 public items in the 20 headers. Counted individu | | `value_t` | native types, or `pycvc.value` | wrapped (PR 1), see the table above | | | `to_string`, `values_equal` | `pycvc.to_string`, `pycvc.values_equal` | wrapped (PR 1) | | | `make_list`, `make_dict`, `make_bytes`, `nil_value`, `true_value`, `false_value` | `list`, `dict`, `bytes`, `None`, `True`, `False` | exception: the native Python values are the equivalents | -| | `environment` (`bindings`, `outer`, `lookup`, `set`, `set_existing`, `extend`) | `pycvc.environment` | wrapped (PR 2), shared like `environment_ptr`. `bindings` reads and writes as a dict (a copy); `lookup` raises `KeyError` where C++ returns nullptr | +| | `environment` (`bindings`, `outer`, `frozen`, `lookup`, `set`, `set_existing`, `extend`) | `pycvc.environment` | wrapped (PR 2), shared like `environment_ptr`. `bindings` reads and writes as a dict (a copy); `lookup` raises `KeyError` where C++ returns nullptr; `set` on a frozen scope (#660) raises `RuntimeError` | | | `closure`, `data_object`, `generator` | `pycvc.value` (opaque) | wrapped as opaque handles; no member access | | | `time_budget`, `eval_deadline_guard`, `eval_deadline_expired`, `thread_cpu_seconds` | none | exception: thread-local evaluator internals | | state_value_codec.h | `encode_value`, `decode_value` | `pycvc.encode_value(State, v)`, `pycvc.decode_value(State)` | wrapped (PR 1) | From 12f9295810646340115a9282953273074d8c70d6 Mon Sep 17 00:00:00 2001 From: Joe Rivera Date: Sat, 17 Oct 2026 21:34:05 -0500 Subject: [PATCH 15/17] pycvc: state_exec parity part 3 (evaluators, processes, policy, intrinsics) Wrap the rest of cvc::state_exec's public surface under the C++ names: - stackless_evaluator and evaluator, with evaluator_state, eval_frame, eval_phase (an IntEnum) and evaluation_stats; evaluation_interrupted and evaluation_timeout as RuntimeError subclasses. The evaluators release the GIL while they evaluate, so another thread can interrupt, pause or resume them. on_complete is a Python callable. - The process struct (shared, as process_ptr), watch_event and watch_handler_entry, make_process, and the scheduler calls that take them: current_process() and queue_watch_event(). scheduler_base is the common base of the two schedulers. - encode_/decode_environment and encode_/decode_evaluator_state. - resource_policy (mode as an IntEnum), process_limits, validate_limits. - intrinsics_context, channel_policy, register_intrinsics, apply_chroot, resolve_channel_key, make_future, future_channel_of. Every function register_intrinsics adds keeps its context alive, and the context keeps the scheduler and channel policy assigned to it. - From #660: kDefaultMaxAllocBytes, eval_alloc_limit, eval_alloc_check, alloc_bytes, scheduler.max_alloc_bytes(), max_alloc_bytes_from_state. A reference into an object (evaluator_state.stats, evaluator.stats(), process.state) keeps its owner alive; other members read and write as copies. evaluator_state is move-only, so it is moved into Python. test_pycvc_state_exec_eval.py repeats the C++ tests. The parity doc, whose inventory now includes what landed on master since the audit, and the README are updated. --- bindings/pycvc/CMakeLists.txt | 1 + bindings/pycvc/README.md | 19 +- bindings/pycvc/pycvc.i | 849 ++++++++++++++++++- bindings/pycvc/pycvc_exec.cpp | 4 +- bindings/pycvc/pycvc_value.h | 4 + bindings/pycvc/test_pycvc_state_exec_eval.py | 398 +++++++++ docs/PYCVC_STATE_EXEC_PARITY.md | 67 +- 7 files changed, 1309 insertions(+), 33 deletions(-) create mode 100644 bindings/pycvc/test_pycvc_state_exec_eval.py diff --git a/bindings/pycvc/CMakeLists.txt b/bindings/pycvc/CMakeLists.txt index 7b7db7368..266a103b2 100644 --- a/bindings/pycvc/CMakeLists.txt +++ b/bindings/pycvc/CMakeLists.txt @@ -273,6 +273,7 @@ if(CVC_BUILD_PYCVC_CORE) pycvc_exec_coro:test_pycvc_exec_coro.py pycvc_state_exec:test_pycvc_state_exec.py pycvc_state_exec_sched:test_pycvc_state_exec_sched.py + pycvc_state_exec_eval:test_pycvc_state_exec_eval.py pycvc_world_clock:test_pycvc_world_clock.py pycvc_world_units:test_pycvc_world_units.py pycvc_dataprep:test_pycvc_dataprep.py diff --git a/bindings/pycvc/README.md b/bindings/pycvc/README.md index b9bd100d6..56b9ecf50 100644 --- a/bindings/pycvc/README.md +++ b/bindings/pycvc/README.md @@ -252,5 +252,22 @@ The enums are `enum.IntEnum` classes, and an enum argument must be one of their members. Where C++ returns an empty `std::optional` or a null pointer, Python gets `None`, except for a value: `get_result`, `pop_pending_message` and `environment.lookup` raise `KeyError` instead, because `None` is the nil value. + +The evaluators, their state, and the intrinsics are wrapped the same way: + +```python +ev = pycvc.stackless_evaluator(pycvc.builtins.make_default_environment()) +st = ev.create_state("(begin (set x 1) (+ x 41))") +while not ev.step(st): # releases the GIL; ev.interrupt() from + pass # another thread raises evaluation_interrupted +st.result, st.stats.get_step_count() # 42, the step count +pycvc.encode_evaluator_state(root.child("snap"), st) # snapshot, decode_... to restore + +ctx = pycvc.intrinsics_context() # state-get, msg-send, spawn, ... over a tree +ctx.sched, ctx.root = sched, root +env = pycvc.builtins.make_default_environment() +pycvc.register_intrinsics(env, ctx) # each function keeps ctx alive +``` + Contract tests: `test_pycvc_exec.py`, `test_pycvc_state_exec.py`, -`test_pycvc_state_exec_sched.py`. +`test_pycvc_state_exec_sched.py`, `test_pycvc_state_exec_eval.py`. diff --git a/bindings/pycvc/pycvc.i b/bindings/pycvc/pycvc.i index 67c0d6855..12ab1f07b 100644 --- a/bindings/pycvc/pycvc.i +++ b/bindings/pycvc/pycvc.i @@ -1446,17 +1446,24 @@ class AsyncStateObserver(state_observer): %{ #include #include +#include +#include #include #include +#include #include #include +#include #include #include #include "pycvc_value.h" #include +#include #include #include #include +#include +#include #include #include #include @@ -1464,6 +1471,11 @@ class AsyncStateObserver(state_observer): #include #include #include + +namespace pycvc { +// The evaluators' on_complete parameter type, under a name SWIG can match a typemap on. +using value_callback = std::function; +} // namespace pycvc %} // ── values: the one value_t <-> Python conversion (pycvc_value.h) ────── @@ -1879,6 +1891,368 @@ bool py_to_value_map(PyObject *o, value_map &out) { } %} +// ── members and results of the evaluator and process types ── +// A value_t member reads as a Python value. +%typemap(out) const cvc::state_exec::value_tag & %{ + $result = pycvc::value_to_py(*$1); + if (!$result) + SWIG_fail; +%} + +%{ +namespace pycvc { +namespace { +using cvc::state_exec::value_t; + +PyObject *value_list_to_py(const std::vector &vals) { + PyObject *list = PyList_New(static_cast(vals.size())); + if (list == nullptr) + return nullptr; + for (size_t i = 0; i < vals.size(); ++i) { + PyObject *item = value_to_py(vals[i]); + if (item == nullptr) { + Py_DECREF(list); + return nullptr; + } + PyList_SET_ITEM(list, static_cast(i), item); + } + return list; +} + +// Each item of the sequence `o` converted with `convert`, or false with a Python error set. +template bool py_to_list(PyObject *o, std::vector &out, F convert) { + PyObject *seq = PySequence_Fast(o, "pycvc: expected a list"); + if (seq == nullptr) + return false; + const Py_ssize_t n = PySequence_Fast_GET_SIZE(seq); + std::vector items; + items.reserve(static_cast(n)); + for (Py_ssize_t i = 0; i < n; ++i) { + T item; + if (!convert(PySequence_Fast_GET_ITEM(seq, i), item)) { + Py_DECREF(seq); + return false; + } + items.push_back(std::move(item)); + } + Py_DECREF(seq); + out = std::move(items); + return true; +} + +bool py_to_value_list(PyObject *o, std::vector &out) { + return py_to_list(o, out, [](PyObject *item, value_t &v) { return py_to_value(item, v); }); +} + +// A struct proxy of `type` copied out (a list item), or false with a TypeError set. +template bool py_to_struct(PyObject *o, swig_type_info *type, T &out) { + void *p = nullptr; + if (!SWIG_IsOK(SWIG_ConvertPtr(o, &p, type, 0)) || p == nullptr) { + PyErr_Format(PyExc_TypeError, "pycvc: expected a pycvc.%s, got %.200s", + type ? type->str : "?", Py_TYPE(o)->tp_name); + return false; + } + out = *static_cast(p); + return true; +} + +// A new owned proxy of a copy of `v`, or nullptr with a Python error set. +template PyObject *struct_to_py(const T &v, swig_type_info *type) { + return SWIG_NewPointerObj(new T(v), type, SWIG_POINTER_OWN); +} +} // namespace +} // namespace pycvc +%} + +// A list of values, and a list of lists of values (eval_frame members). +%typemap(in) const std::vector & + (std::vector temp) %{ + if (!pycvc::py_to_value_list($input, temp)) + SWIG_fail; + $1 = &temp; +%} +%typemap(out) const std::vector & %{ + $result = pycvc::value_list_to_py(*$1); + if (!$result) + SWIG_fail; +%} +%typemap(in) const std::vector > & + (std::vector > temp) %{ + if (!pycvc::py_to_list($input, temp, [](PyObject *item, std::vector &v) { + return pycvc::py_to_value_list(item, v); + })) + SWIG_fail; + $1 = &temp; +%} +%typemap(out) const std::vector > & %{ + $result = PyList_New(static_cast($1->size())); + if (!$result) + SWIG_fail; + for (size_t i = 0; i < $1->size(); ++i) { + PyObject *item = pycvc::value_list_to_py((*$1)[i]); + if (!item) { + Py_DECREF($result); + $result = nullptr; + SWIG_fail; + } + PyList_SET_ITEM($result, static_cast(i), item); + } +%} + +// A list of structs (eval_frame, process::watch_event) reads as a list of copies and is set from a +// list of them. +%define PYCVC_STATE_EXEC_STRUCT_LIST(TYPE) +%typemap(in) const std::vector & (std::vector temp) { + if (!pycvc::py_to_list($input, temp, [](PyObject *item, TYPE &v) { + return pycvc::py_to_struct(item, $descriptor(TYPE *), v); + })) + SWIG_fail; + $1 = &temp; +} +%typemap(out) const std::vector & { + $result = PyList_New(static_cast($1->size())); + if (!$result) + SWIG_fail; + for (size_t i = 0; i < $1->size(); ++i) { + PyObject *item = pycvc::struct_to_py((*$1)[i], $descriptor(TYPE *)); + if (!item) { + Py_DECREF($result); + $result = nullptr; + SWIG_fail; + } + PyList_SET_ITEM($result, static_cast(i), item); + } +} +%enddef +PYCVC_STATE_EXEC_STRUCT_LIST(cvc::state_exec::eval_frame) +PYCVC_STATE_EXEC_STRUCT_LIST(cvc::state_exec::process::watch_event) + +// process::watch_handlers: a dict of watch id -> watch_handler_entry (copies). +%typemap(in) const std::unordered_map & + (std::unordered_map temp) { + if (!PyDict_Check($input)) { + PyErr_SetString(PyExc_TypeError, "pycvc: expected a dict of int to watch_handler_entry"); + SWIG_fail; + } + PyObject *k = nullptr; + PyObject *v = nullptr; + Py_ssize_t pos = 0; + while (PyDict_Next($input, &pos, &k, &v)) { + const long id = PyLong_AsLong(k); + if (id == -1 && PyErr_Occurred()) + SWIG_fail; + cvc::state_exec::process::watch_handler_entry entry; + if (!pycvc::py_to_struct(v, $descriptor(cvc::state_exec::process::watch_handler_entry *), + entry)) + SWIG_fail; + temp[static_cast(id)] = std::move(entry); + } + $1 = &temp; +} +%typemap(out) const std::unordered_map & { + $result = PyDict_New(); + if (!$result) + SWIG_fail; + for (const auto &kv : *$1) { + PyObject *k = PyLong_FromLong(kv.first); + PyObject *v = k ? pycvc::struct_to_py( + kv.second, $descriptor(cvc::state_exec::process::watch_handler_entry *)) + : nullptr; + const int rc = v ? PyDict_SetItem($result, k, v) : -1; + Py_XDECREF(k); + Py_XDECREF(v); + if (rc != 0) { + Py_DECREF($result); + $result = nullptr; + SWIG_fail; + } + } +} + +// evaluator::macro_map: a dict of macro name -> (params, body), params a list of pycvc.symbol. +%{ +namespace pycvc { +namespace { +using macro_map = cvc::state_exec::evaluator::macro_map; + +PyObject *macro_map_to_py(const macro_map &m) { + PyObject *dict = PyDict_New(); + if (dict == nullptr) + return nullptr; + for (const auto &kv : m) { + std::vector params(kv.second.first.begin(), kv.second.first.end()); + PyObject *key = text_to_py(kv.first); + PyObject *ps = key ? value_list_to_py(params) : nullptr; + PyObject *body = ps ? value_to_py(kv.second.second) : nullptr; + PyObject *pair = body ? PyTuple_Pack(2, ps, body) : nullptr; + const int rc = pair ? PyDict_SetItem(dict, key, pair) : -1; + Py_XDECREF(key); + Py_XDECREF(ps); + Py_XDECREF(body); + Py_XDECREF(pair); + if (rc != 0) { + Py_DECREF(dict); + return nullptr; + } + } + return dict; +} + +bool py_to_macro_map(PyObject *o, macro_map &out) { + if (!PyDict_Check(o)) { + PyErr_SetString(PyExc_TypeError, "pycvc: expected a dict of name to (params, body)"); + return false; + } + macro_map m; + PyObject *key = nullptr; + PyObject *val = nullptr; + Py_ssize_t pos = 0; + while (PyDict_Next(o, &pos, &key, &val)) { + if (!PyUnicode_Check(key) || !PyTuple_Check(val) || PyTuple_GET_SIZE(val) != 2) { + PyErr_SetString(PyExc_TypeError, + "pycvc: a macro is name (str) -> (params, body), params a list of symbols"); + return false; + } + std::vector params; + value_t body; + if (!py_to_value_list(PyTuple_GET_ITEM(val, 0), params) || + !py_to_value(PyTuple_GET_ITEM(val, 1), body)) + return false; + std::vector syms; + for (const value_t &p : params) { + const auto *s = std::get_if(&p.v); + if (s == nullptr) { + PyErr_SetString(PyExc_TypeError, "pycvc: macro params must be pycvc.symbol"); + return false; + } + syms.push_back(*s); + } + const char *name = PyUnicode_AsUTF8(key); + if (name == nullptr) + return false; + m[name] = {std::move(syms), std::move(body)}; + } + out = std::move(m); + return true; +} +} // namespace +} // namespace pycvc +%} +%typemap(in) const cvc::state_exec::evaluator::macro_map & (cvc::state_exec::evaluator::macro_map temp) %{ + if (!pycvc::py_to_macro_map($input, temp)) + SWIG_fail; + $1 = &temp; +%} +%typemap(out) const cvc::state_exec::evaluator::macro_map & %{ + $result = pycvc::macro_map_to_py(*$1); + if (!$result) + SWIG_fail; +%} +%typemap(out) cvc::state_exec::evaluator::macro_map %{ + $result = pycvc::macro_map_to_py(static_cast($1)); + if (!$result) + SWIG_fail; +%} + +// Optional strings as members (process::recv_path, exit_error): None or str. +%typemap(in) const std::optional & (std::optional temp) %{ + if ($input != Py_None) { + Py_ssize_t pycvc_n = 0; + const char *pycvc_s = + PyUnicode_Check($input) ? PyUnicode_AsUTF8AndSize($input, &pycvc_n) : nullptr; + if (pycvc_s == nullptr) { + if (!PyErr_Occurred()) + PyErr_SetString(PyExc_TypeError, "pycvc: expected str or None"); + SWIG_fail; + } + temp = std::string(pycvc_s, static_cast(pycvc_n)); + } + $1 = &temp; +%} +%typemap(out) const std::optional & %{ + if (*$1) { + $result = PyUnicode_DecodeUTF8((*$1)->data(), static_cast((*$1)->size()), + "surrogateescape"); + if (!$result) + SWIG_fail; + } else { + $result = Py_None; + Py_INCREF(Py_None); + } +%} +// process::time_left() and alloc_limit(): None when unlimited. +%typemap(out) std::optional %{ + { // a block, so the wrapper's later `goto fail` doesn't jump over this initialization + const std::optional &pycvc_opt = $1; + $result = pycvc_opt ? PyFloat_FromDouble(*pycvc_opt) : (Py_INCREF(Py_None), Py_None); + } +%} +%typemap(out) std::optional %{ + { // a block, so the wrapper's later `goto fail` doesn't jump over this initialization + const std::optional &pycvc_opt = $1; + $result = pycvc_opt ? PyLong_FromSize_t(*pycvc_opt) : (Py_INCREF(Py_None), Py_None); + } +%} + +// restrict_special_forms takes any iterable of str, or None for no restriction; +// all_special_forms() returns a set. +// $1 is always assigned: SWIG holds this argument in a SwigValueWrapper, which is empty until then. +%typemap(in) std::shared_ptr > %{ + $1 = std::shared_ptr>(); + if ($input != Py_None) { + auto pycvc_forms = std::make_shared>(); + PyObject *pycvc_it = PyObject_GetIter($input); + if (pycvc_it == nullptr) + SWIG_fail; + while (PyObject *pycvc_item = PyIter_Next(pycvc_it)) { + const char *pycvc_s = PyUnicode_Check(pycvc_item) ? PyUnicode_AsUTF8(pycvc_item) : nullptr; + if (pycvc_s != nullptr) + pycvc_forms->insert(pycvc_s); + Py_DECREF(pycvc_item); + if (pycvc_s == nullptr) { + Py_DECREF(pycvc_it); + if (!PyErr_Occurred()) + PyErr_SetString(PyExc_TypeError, "pycvc: special forms must be str"); + SWIG_fail; + } + } + Py_DECREF(pycvc_it); + if (PyErr_Occurred()) + SWIG_fail; + $1 = std::move(pycvc_forms); + } +%} +%typemap(out) const std::set & %{ + $result = PySet_New(nullptr); + if (!$result) + SWIG_fail; + for (const std::string &pycvc_form : *$1) { + PyObject *pycvc_s = PyUnicode_FromStringAndSize(pycvc_form.data(), + static_cast(pycvc_form.size())); + if (!pycvc_s || PySet_Add($result, pycvc_s) != 0) { + Py_XDECREF(pycvc_s); + Py_DECREF($result); + $result = nullptr; + SWIG_fail; + } + Py_DECREF(pycvc_s); + } +%} + +// The evaluators' on_complete: a Python callable or None, called with the result. +%typemap(in) pycvc::value_callback %{ + $1 = pycvc::value_callback(); // always assigned, should SWIG hold it in a SwigValueWrapper + if ($input != Py_None) { + if (!PyCallable_Check($input)) { // checked here: a typemap runs outside the try block + PyErr_SetString(PyExc_TypeError, "pycvc: on_complete must be callable or None"); + SWIG_fail; + } + $1 = pycvc::make_python_on_complete($input); + } +%} +%typemap(typecheck, precedence=SWIG_TYPECHECK_POINTER) pycvc::value_callback + "$1 = ($input == Py_None || PyCallable_Check($input)) ? 1 : 0;" + // ── enums: pycvc.process_status and pycvc.scheduling_policy are enum.IntEnum classes ── // pycvc.process_status.killed is process_status::killed. An enum argument must be a member of its // IntEnum (strict, like values), which is also what lets to_string(process_status) and @@ -1954,6 +2328,19 @@ static bool pycvc_enum_from_py(PyObject *o, PyObject *type, const char *name, in %typemap(typecheck, precedence=SWIG_TYPECHECK_INT32) TYPE { $1 = pycvc_is_enum($input, PYTYPE) ? 1 : 0; } +// By const reference too, for a %naturalvar member of an enum type SWIG wasn't shown +// (resource_policy::enforce). +%typemap(in) const TYPE & (TYPE pycvc_ev_val, int pycvc_ev) { + if (!pycvc_enum_from_py($input, PYTYPE, NAME, pycvc_ev)) + SWIG_fail; + pycvc_ev_val = static_cast(pycvc_ev); + $1 = &pycvc_ev_val; +} +%typemap(out) const TYPE & { + $result = PyObject_CallFunction(PYTYPE, "i", static_cast(*$1)); + if (!$result) + SWIG_fail; +} %enddef PYCVC_STATE_EXEC_ENUM(cvc::state_exec::process_status, pycvc_enum_process_status, "process_status") PYCVC_STATE_EXEC_ENUM(cvc::state_exec::scheduling_policy, pycvc_enum_scheduling_policy, @@ -1983,6 +2370,52 @@ process_status = _pycvc.process_status scheduling_policy = _pycvc.scheduling_policy %} +// eval_phase, and resource_policy::mode (pycvc.resource_policy.mode, set on the class below). +%{ +static PyObject *pycvc_enum_eval_phase = nullptr; +static PyObject *pycvc_enum_resource_policy_mode = nullptr; +%} +PYCVC_STATE_EXEC_ENUM(cvc::state_exec::eval_phase, pycvc_enum_eval_phase, "eval_phase") +PYCVC_STATE_EXEC_ENUM(cvc::state_exec::resource_policy::mode, pycvc_enum_resource_policy_mode, + "resource_policy.mode") +%init %{ + { + using cvc::state_exec::eval_phase; + pycvc_enum_eval_phase = pycvc_make_int_enum( + "eval_phase", + {{"init", static_cast(eval_phase::init)}, + {"eval_args", static_cast(eval_phase::eval_args)}, + {"apply", static_cast(eval_phase::apply)}, + {"if_cond", static_cast(eval_phase::if_cond)}, + {"if_branch", static_cast(eval_phase::if_branch)}, + {"begin_next", static_cast(eval_phase::begin_next)}, + {"while_cond", static_cast(eval_phase::while_cond)}, + {"while_body", static_cast(eval_phase::while_body)}, + {"for_body", static_cast(eval_phase::for_body)}, + {"set_value", static_cast(eval_phase::set_value)}, + {"let_bindings", static_cast(eval_phase::let_bindings)}, + {"let_body", static_cast(eval_phase::let_body)}, + {"return_value", static_cast(eval_phase::return_value)}, + {"eval_inner", static_cast(eval_phase::eval_inner)}, + {"super_args", static_cast(eval_phase::super_args)}, + {"defclass_methods", static_cast(eval_phase::defclass_methods)}, + {"yield_value", static_cast(eval_phase::yield_value)}, + {"break_unwind", static_cast(eval_phase::break_unwind)}}); + if (pycvc_enum_eval_phase != NULL) + PyDict_SetItemString(d, "eval_phase", pycvc_enum_eval_phase); + using rp_mode = cvc::state_exec::resource_policy::mode; + pycvc_enum_resource_policy_mode = + pycvc_make_int_enum("mode", {{"strict", static_cast(rp_mode::strict)}, + {"clamp", static_cast(rp_mode::clamp)}, + {"warn", static_cast(rp_mode::warn)}}); + if (pycvc_enum_resource_policy_mode != NULL) + PyDict_SetItemString(d, "_resource_policy_mode", pycvc_enum_resource_policy_mode); + } +%} +%pythoncode %{ +eval_phase = _pycvc.eval_phase +%} + // ── parse_error: pycvc.parse_error, a RuntimeError with .line and .col ── %{ static PyObject *pycvc_parse_error_type = nullptr; @@ -2012,6 +2445,26 @@ static void pycvc_raise_parse_error(const cvc::state_exec::parse_error &e) { parse_error = _pycvc.parse_error %} +// ── evaluation_interrupted / evaluation_timeout: RuntimeError subclasses, as in C++ ── +%{ +static PyObject *pycvc_evaluation_interrupted_type = nullptr; +static PyObject *pycvc_evaluation_timeout_type = nullptr; +%} +%init %{ + pycvc_evaluation_interrupted_type = + PyErr_NewException("pycvc.evaluation_interrupted", PyExc_RuntimeError, NULL); + if (pycvc_evaluation_interrupted_type != NULL) + PyDict_SetItemString(d, "evaluation_interrupted", pycvc_evaluation_interrupted_type); + pycvc_evaluation_timeout_type = + PyErr_NewException("pycvc.evaluation_timeout", PyExc_RuntimeError, NULL); + if (pycvc_evaluation_timeout_type != NULL) + PyDict_SetItemString(d, "evaluation_timeout", pycvc_evaluation_timeout_type); +%} +%pythoncode %{ +evaluation_interrupted = _pycvc.evaluation_interrupted +evaluation_timeout = _pycvc.evaluation_timeout +%} + // ── no value: KeyError ── // Where C++ returns an empty std::optional or a null value_t* (an unbound name, a pid with // no result, an empty message queue), the wrapper throws pycvc::no_value and Python gets @@ -2048,6 +2501,12 @@ static void pycvc_raise_no_value(const pycvc::no_value &e) { } catch (const cvc::state_exec::parse_error &e) { pycvc_raise_parse_error(e); SWIG_fail; + } catch (const cvc::state_exec::evaluation_interrupted &e) { + PyErr_SetString(pycvc_evaluation_interrupted_type, e.what()); + SWIG_fail; + } catch (const cvc::state_exec::evaluation_timeout &e) { + PyErr_SetString(pycvc_evaluation_timeout_type, e.what()); + SWIG_fail; } catch (const cvc::exception &e) { SWIG_exception(SWIG_RuntimeError, e.what()); } catch (const std::exception &e) { @@ -2079,11 +2538,44 @@ PYCVC_STATE_EXEC_NOGIL(cvc::state_exec::scheduler::step) PYCVC_STATE_EXEC_NOGIL(cvc::state_exec::scheduler::run) PYCVC_STATE_EXEC_NOGIL(cvc::state_exec::async_scheduler::sync_step) PYCVC_STATE_EXEC_NOGIL(cvc::state_exec::async_scheduler::sync_run) - -// environment is shared (environment_ptr), as in C++. The map members read and write as dicts. +// The evaluators, likewise. +PYCVC_STATE_EXEC_NOGIL(cvc::state_exec::stackless_evaluator::step) +PYCVC_STATE_EXEC_NOGIL(cvc::state_exec::stackless_evaluator::run) +PYCVC_STATE_EXEC_NOGIL(cvc::state_exec::stackless_evaluator::evaluate) +PYCVC_STATE_EXEC_NOGIL(cvc::state_exec::stackless_evaluator::evaluate_script) +PYCVC_STATE_EXEC_NOGIL(cvc::state_exec::evaluator::evaluate) +PYCVC_STATE_EXEC_NOGIL(cvc::state_exec::evaluator::evaluate_script) + +// Shared as in C++ (environment_ptr, process_ptr, ...). Struct members that are maps, lists or +// values read and write as Python values (copies). %shared_ptr(cvc::state_exec::environment) +%shared_ptr(cvc::state_exec::script_reader) +%shared_ptr(cvc::state_exec::process) +%shared_ptr(cvc::state_exec::intrinsics_context) +%naturalvar cvc::state_exec::value_tag; %naturalvar cvc::state_exec::environment::bindings; %naturalvar cvc::state_exec::execute_options::signal_handlers; +%naturalvar cvc::state_exec::eval_frame::results; +%naturalvar cvc::state_exec::eval_frame::extra_vals; +%naturalvar cvc::state_exec::eval_frame::extra_strs; +%naturalvar cvc::state_exec::eval_frame::extra_lists; +%naturalvar cvc::state_exec::evaluator_state::stack; +%naturalvar cvc::state_exec::evaluator_state::user_macros; +%naturalvar cvc::state_exec::process::recv_path; +%naturalvar cvc::state_exec::process::exit_error; +%naturalvar cvc::state_exec::process::signal_handlers; +%naturalvar cvc::state_exec::process::pending_signals; +%naturalvar cvc::state_exec::process::saved_stack; +%naturalvar cvc::state_exec::process::pending_watch_events; +%naturalvar cvc::state_exec::process::watch_handlers; +%naturalvar cvc::state_exec::resource_policy::enforce; +%naturalvar cvc::state_exec::channel_policy::declared; +%naturalvar cvc::state_exec::channel_policy::global; +%rename(global_) cvc::state_exec::channel_policy::global; // `global` is a Python keyword +// process::watch_event and process::watch_handler_entry are pycvc.watch_event and +// pycvc.watch_handler_entry (Python has no nested classes here); process.watch_event names them too. +%feature("flatnested") cvc::state_exec::process::watch_event; +%feature("flatnested") cvc::state_exec::process::watch_handler_entry; namespace cvc { namespace state_exec { @@ -2214,6 +2706,122 @@ public: std::string percent_encode(const std::string &s); std::string percent_decode(const std::string &s); +// ── evaluators ── +// evaluation_stats is the C++ class; it can't be copied (it holds a mutex), so the evaluators and +// states hand out references, which keep their owner alive in Python. evaluation_interrupted and +// evaluation_timeout are pycvc exceptions (RuntimeError subclasses). The evaluators release the +// GIL while they evaluate, so another thread can interrupt(), pause() or resume() them. +class evaluation_stats { +public: + evaluation_stats(); + void start(); + void mark_complete(); + void increment_step(); + unsigned long long get_step_count() const; + bool is_complete() const; + double elapsed_seconds() const; +}; + +// eval_phase is the IntEnum above. A frame reads and writes as copies of its parts. +enum class eval_phase; +struct eval_frame { + value_t expr; + environment_ptr env; + eval_phase phase; + long long index; + std::vector results; + std::vector extra_vals; + std::vector extra_strs; + std::vector > extra_lists; + environment_ptr extra_env; +}; + +// The recursive evaluator. user_macros() returns a copy of the macro map: a dict of name -> +// (params, body). +%pythonappend cvc::state_exec::evaluator::stats %{ + val._pycvc_owner = self +%} +class evaluator { +public: + typedef std::unordered_map, value_t> > macro_map; + evaluator(environment_ptr global_env); + value_t evaluate(const value_t &expr, environment_ptr env = environment_ptr(), + std::optional timeout_sec = std::nullopt, + pycvc::value_callback on_complete = pycvc::value_callback()); + value_t evaluate_script(const std::string &script, environment_ptr env = environment_ptr(), + std::optional timeout_sec = std::nullopt, + pycvc::value_callback on_complete = pycvc::value_callback()); + void interrupt(); + void reset_interrupt(); + void pause(); + void resume(); + bool is_paused() const; + evaluation_stats &stats(); + macro_map user_macros(); +}; + +// A stackless evaluator's whole state. The members read and write as copies; stats is a reference +// into the state. +%pythonappend cvc::state_exec::evaluator_state::_stats %{ + val._pycvc_owner = self +%} +struct evaluator_state { + std::vector stack; + value_t result; + bool done; + value_t root_expr; + environment_ptr global_env; + evaluator::macro_map user_macros; + bool yielded; + std::shared_ptr reading; +}; +%extend evaluator_state { + evaluation_stats *_stats() { return &$self->stats; } + %pythoncode %{ + stats = property(_stats) + %} +} + +// evaluator_state can't be copied (its stats hold a mutex), and SWIG returns objects by copy: +// move a returned state into its Python object instead. +%typemap(out) cvc::state_exec::evaluator_state %{ + $result = SWIG_NewPointerObj(new cvc::state_exec::evaluator_state(std::move($1)), + $descriptor(cvc::state_exec::evaluator_state *), SWIG_POINTER_OWN); +%} + +// The stackless evaluator. on_complete is a Python callable or None. A str passed to +// create_state is a script (the std::string overload), as in C++. +class stackless_evaluator { +public: + stackless_evaluator(environment_ptr global_env); + evaluator_state create_state(const value_t &expr, environment_ptr env = environment_ptr()); + evaluator_state create_state(const std::string &script, environment_ptr env = environment_ptr(), + const parse_options &popts = parse_options()); + bool step(evaluator_state &state); + value_t run(evaluator_state &state, std::optional max_steps = std::nullopt, + std::optional timeout_sec = std::nullopt, + pycvc::value_callback on_complete = pycvc::value_callback()); + value_t evaluate(const value_t &expr, environment_ptr env = environment_ptr(), + std::optional timeout_sec = std::nullopt, + pycvc::value_callback on_complete = pycvc::value_callback()); + value_t evaluate_script(const std::string &script, environment_ptr env = environment_ptr(), + std::optional timeout_sec = std::nullopt, + pycvc::value_callback on_complete = pycvc::value_callback()); + void interrupt(); + void reset_interrupt(); + void pause(); + void resume(); + bool is_paused() const; + void restrict_special_forms(std::shared_ptr > allowed); + static const std::set &all_special_forms(); +}; + +// Snapshot codec for an environment chain and for a whole evaluator state. +void encode_environment(cvc::state &node, const environment_ptr &env); +environment_ptr decode_environment(cvc::state &node); +void encode_evaluator_state(cvc::state &node, const evaluator_state &es); +evaluator_state decode_evaluator_state(cvc::state &node); + // ── processes and schedulers ── // process_status and scheduling_policy are the IntEnums above; declared opaque here so SWIG adds // no flat constants of its own. @@ -2241,7 +2849,8 @@ struct execute_options { std::unordered_map signal_handlers; environment_ptr env; }; -%extend execute_options { +%define PYCVC_STATE_EXEC_ON_COMPLETE(CLASS) +%extend CLASS { PyObject *_on_complete_get() const { if (!$self->on_complete) { Py_INCREF(Py_None); @@ -2251,7 +2860,7 @@ struct execute_options { Py_INCREF(f->callable.get()); return f->callable.get(); } - throw std::runtime_error("pycvc.execute_options.on_complete: holds a C++ callback"); + throw std::runtime_error("pycvc: on_complete holds a C++ callback"); } void _on_complete_set(PyObject *fn) { if (fn == Py_None) @@ -2263,6 +2872,8 @@ struct execute_options { on_complete = property(_on_complete_get, _on_complete_set) %} } +%enddef +PYCVC_STATE_EXEC_ON_COMPLETE(execute_options) struct process_info { int pid; @@ -2310,10 +2921,105 @@ struct scheduler_stats { %} } +// ── process: a scheduler's record of one program (shared, as process_ptr) ── +// Its members read and write as copies, except state, a reference into the process. The +// steady_clock time points (create_time, last_run_start, wake_time) aren't wrapped; elapsed_time() +// and accumulated_time give the same information. inbox reads as a list. +const size_t kDefaultMaxAllocBytes = 16777216; +// The per-call allocation limit (#660). A Python function a program calls runs on the thread +// stepping the scheduler, where the step's limit is armed, so it can honour the limit the way a +// builtin does: eval_alloc_check() before building a large result. eval_alloc_guard (RAII, which +// must nest last-in first-out) isn't wrapped. +std::optional eval_alloc_limit(); +void eval_alloc_check(size_t bytes, const char *who); +size_t alloc_bytes(size_t count, size_t each); +%pythonappend cvc::state_exec::process::_state %{ + val._pycvc_owner = self +%} +struct process { + struct watch_event { + int watch_id; + }; + struct watch_handler_entry { + value_t handler; + std::string path; + std::string last_value; + }; + int pid; + std::string name; + process_status status; + int priority; + std::string uid; + std::string gid; + std::string root_path; + std::string owner; + bool awaiting_frame; + unsigned long long max_steps; + double max_time; + unsigned long long max_memory; + unsigned long long max_messages; + unsigned long long max_message_bytes; + double max_step_time; + unsigned long long max_alloc_bytes; + double accumulated_time; + double cpu_time_charged; + std::optional recv_path; + value_t exit_code; + std::optional exit_error; + std::unordered_map signal_handlers; + std::vector pending_signals; + bool in_signal_handler; + bool in_watch_handler; + std::vector saved_stack; + value_t saved_result; + std::vector pending_watch_events; + int next_watch_id; + std::unordered_map watch_handlers; + unsigned long long message_count; + unsigned long long message_bytes; + int parent_pid; + double elapsed_time() const; + bool charges_cpu_time() const; + void charge_cpu_time(double seconds); + bool over_time_limit() const; + std::optional time_left() const; + std::optional alloc_limit(size_t used_bytes) const; + unsigned long long step_count() const; +}; +%extend process { + evaluator_state *_state() { return &$self->state; } + PyObject *_inbox_get() const { + std::queue q = $self->inbox; + std::vector items; + for (; !q.empty(); q.pop()) + items.push_back(q.front()); + return pycvc::value_list_to_py(items); + } + %pythoncode %{ + state = property(_state) + inbox = property(_inbox_get) + %} +} +PYCVC_STATE_EXEC_ON_COMPLETE(process) +%pythoncode %{ +process.watch_event = watch_event +process.watch_handler_entry = watch_handler_entry +%} +typedef std::shared_ptr process_ptr; +process_ptr make_process(); + +// The common base of the two schedulers, which intrinsics_context.sched points to. Its members are +// on the two concrete classes. +%nodefaultctor scheduler_base; +%nodefaultdtor scheduler_base; +class scheduler_base {}; + // The members scheduler and async_scheduler share (scheduler_base plus their common extras). -// Returns of a C++ reference come back as copies. Not wrapped: mem_tracker(), and -// current_process() / queue_watch_event(), which take the process struct (PR 3). +// Returns of a C++ reference come back as copies (current_process() shares the process, as +// process_ptr does). Not wrapped: mem_tracker(). %define PYCVC_STATE_EXEC_SCHEDULER_MEMBERS + process_ptr current_process() const; + void queue_watch_event(int pid, process::watch_event evt); int execute(const std::string &script, const execute_options &opts = execute_options()); int execute(const value_t &expr, const execute_options &opts = execute_options()); void stop(); @@ -2366,7 +3072,7 @@ struct scheduler_stats { %enddef // The synchronous scheduler. run() and step() release the GIL while programs run. -class scheduler { +class scheduler : public scheduler_base { public: scheduler(); scheduler(scheduling_policy policy); @@ -2378,16 +3084,19 @@ public: std::string id() const; void load_settings(); parse_options parser_settings() const; + unsigned long long max_alloc_bytes() const; static constexpr const char *settings_root = "state_exec"; static constexpr const char *defaults_prefix = "state_exec.defaults"; static constexpr const char *sched_prefix = "state_exec.schedulers"; }; PYCVC_STATE_EXEC_SCHEDULER_EXTEND(scheduler) +unsigned long long max_alloc_bytes_from_state(cvc::state &root, const std::string &scheduler_id, + unsigned long long base = kDefaultMaxAllocBytes); // The scheduler libcvc's hosts use (app::exec_scheduler(), Exec). The coroutine step()/run() aren't // wrapped; Python drives it with sync_step()/sync_run(), which release the GIL while programs run. // post_message() is the one thread-safe call. -class async_scheduler { +class async_scheduler : public scheduler_base { public: async_scheduler(); async_scheduler(scheduling_policy policy); @@ -2407,9 +3116,133 @@ public: static parse_options host_parser_defaults(); }; PYCVC_STATE_EXEC_SCHEDULER_EXTEND(async_scheduler) + +// ── resource_policy ── +// mode is pycvc.resource_policy.mode, an IntEnum. validate_limits raises RuntimeError for +// out-of-range limits in strict mode, as C++ throws. +struct resource_policy { + double max_time_min; + double max_time_max; + double max_time_default; + unsigned long long max_memory_min; + unsigned long long max_memory_max; + unsigned long long max_memory_default; + unsigned long long max_messages_min; + unsigned long long max_messages_max; + unsigned long long max_messages_default; + unsigned long long max_message_bytes_min; + unsigned long long max_message_bytes_max; + unsigned long long max_message_bytes_default; + unsigned long long max_steps_min; + unsigned long long max_steps_max; + unsigned long long max_steps_default; + int max_processes; + unsigned long long max_total_memory; + cvc::state_exec::resource_policy::mode enforce; +}; +%pythoncode %{ +resource_policy.mode = _pycvc._resource_policy_mode +%} +struct process_limits { + unsigned long long max_steps; + double max_time; + unsigned long long max_memory; + unsigned long long max_messages; + unsigned long long max_message_bytes; +}; +process_limits validate_limits(const resource_policy &policy, const process_limits &requested); + +// ── intrinsics ── +// channel_policy.global is global_ (a Python keyword). intrinsics_context is shared; its sched and +// channels are the Python objects assigned to them, which the context keeps alive. Its root is a +// tree node, which lives as long as its app (keep the app alive, as in C++). Not wrapped: tracker +// (memory_tracker is internal), document (document_scope's typed slots are C++ templates), and +// watches (the intrinsics' own registry). +struct channel_policy { + bool enforce; + bool quiet; + std::vector declared; + std::vector global; +}; +%rename(_sched) intrinsics_context::sched; +%rename(_channels) intrinsics_context::channels; +struct intrinsics_context { + scheduler_base *sched; + cvc::state *root; + std::shared_ptr proc; + int pid; + std::string uid; + std::string cluster_id; + std::string node_id; + std::string root_path; + const channel_policy *channels; +}; +%extend intrinsics_context { + %pythoncode %{ + def _get_sched(self): + return self.__dict__.get("_pycvc_sched") if self._sched is not None else None + def _set_sched(self, sched): + self._sched = sched + self.__dict__["_pycvc_sched"] = sched + sched = property(_get_sched, _set_sched) + def _get_channels(self): + return self.__dict__.get("_pycvc_channels") if self._channels is not None else None + def _set_channels(self, channels): + self._channels = channels + self.__dict__["_pycvc_channels"] = channels + channels = property(_get_channels, _set_channels) + %} +} +void apply_chroot(intrinsics_context &ctx, cvc::state &tree_root, const std::string &root_path); +std::string resolve_channel_key(const std::string &root_path, const std::string &channel); +value_t make_future(const std::string &channel); +std::optional future_channel_of(const value_t &v); } } +// register_intrinsics(env, ctx): the C++ call, except that every function it adds keeps ctx (and +// so the scheduler and channel policy ctx holds) alive. C++ requires the caller to keep the context +// valid instead; in Python an environment may outlive any object. +%{ +namespace pycvc { +void register_intrinsics_keeping(cvc::state_exec::environment_ptr env, + std::shared_ptr ctx, + PyObject *owner) { + using namespace cvc::state_exec; + if (!env || !ctx) + throw std::invalid_argument("pycvc.register_intrinsics: env and ctx must not be None"); + // register_intrinsics only adds functions, so add them to a scratch scope and copy each one in, + // wrapped to hold the context. + auto scratch = std::make_shared(); + register_intrinsics(scratch, ctx.get()); + std::shared_ptr keep = python_ref(owner); + for (auto &[name, v] : scratch->bindings) { + const auto *fn = std::get_if(&v.v); + if (fn == nullptr) { + env->set(name, v); + continue; + } + env->set(name, value_t(native_fn([inner = *fn, ctx, keep](std::span args) { + return inner(args); + }))); + } +} +} // namespace pycvc +%} +%rename(_register_intrinsics) pycvc::register_intrinsics_keeping; +namespace pycvc { +void register_intrinsics_keeping(cvc::state_exec::environment_ptr env, + std::shared_ptr ctx, + PyObject *owner); +} +%pythoncode %{ +def register_intrinsics(env, ctx): + """cvc::state_exec::register_intrinsics: add the DSL intrinsics over ctx to env. + + Each function added keeps ctx alive, and with it the scheduler and channel policy it holds.""" + _register_intrinsics(env, ctx, ctx) +%} + // Exec.scheduler() returns a reference into the Exec, so the scheduler keeps the Exec alive. %pythonappend pycvc::Exec::scheduler %{ val._pycvc_exec = self diff --git a/bindings/pycvc/pycvc_exec.cpp b/bindings/pycvc/pycvc_exec.cpp index e342d2302..7c971cc0b 100644 --- a/bindings/pycvc/pycvc_exec.cpp +++ b/bindings/pycvc/pycvc_exec.cpp @@ -274,10 +274,12 @@ void python_on_complete::operator()(value_t result) const { python_on_complete make_python_on_complete(PyObject *callable) { if (callable == nullptr || !PyCallable_Check(callable)) - throw std::invalid_argument("pycvc.execute_options.on_complete: not callable"); + throw std::invalid_argument("pycvc: on_complete is not callable"); return python_on_complete{own_callable(callable)}; } +std::shared_ptr python_ref(PyObject *o) { return own_callable(o); } + // Owns everything an evaluation needs. The intrinsics capture &ictx (a raw // pointer), so ictx must outlive any run — holding it here (Exec-lifetime) // satisfies that. Mirrors the C++ tests' make_exec_env(). diff --git a/bindings/pycvc/pycvc_value.h b/bindings/pycvc/pycvc_value.h index 8cc17b305..5fb0fa844 100644 --- a/bindings/pycvc/pycvc_value.h +++ b/bindings/pycvc/pycvc_value.h @@ -57,4 +57,8 @@ struct python_on_complete { /// The python_on_complete for `callable`. Throws std::invalid_argument when it isn't callable. python_on_complete make_python_on_complete(PyObject *callable); +/// A shared owner of a new reference to `o`. The last owner to let go drops the reference under +/// the GIL, so C++ can keep a Python object alive from any thread. +std::shared_ptr python_ref(PyObject *o); + } // namespace pycvc diff --git a/bindings/pycvc/test_pycvc_state_exec_eval.py b/bindings/pycvc/test_pycvc_state_exec_eval.py new file mode 100644 index 000000000..8e897818c --- /dev/null +++ b/bindings/pycvc/test_pycvc_state_exec_eval.py @@ -0,0 +1,398 @@ +"""state_exec parity, part 3: evaluators, evaluator state, processes, policy, intrinsics. + +pycvc.stackless_evaluator and pycvc.evaluator are the real C++ evaluators, with +evaluator_state, eval_frame, evaluation_stats, the process struct, the environment +and evaluator-state codec, resource_policy, and register_intrinsics over an +intrinsics_context. Most tests repeat a C++ test from src/cvc/tests +(state_exec_stackless_evaluator_test.cpp, state_exec_evaluator_test.cpp, +state_exec_codec_test.cpp, state_exec_intrinsics_test.cpp). +docs/PYCVC_STATE_EXEC_PARITY.md has the inventory. +""" + +import gc +import threading +import time + +import pycvc +from pycvc import eval_phase, process_status, symbol + + +def _raises(exc, fn, *args): + try: + fn(*args) + except exc as e: + return e + raise AssertionError(f"{fn.__name__}{args!r} did not raise {exc.__name__}") + + +def _builtins(): + return pycvc.builtins.make_default_environment() + + +LOOP = ( + "(begin (set sum 0) (set i 0)" + " (while (< i 100) (begin (set sum (+ sum i)) (set i (+ i 1)))) sum)" +) + + +# ── evaluators ────────────────────────────────────────────────────────── + + +def test_both_evaluators_evaluate(): + for ev in (pycvc.stackless_evaluator(_builtins()), pycvc.evaluator(_builtins())): + assert ev.evaluate_script("(+ 1 2)") == 3 + assert ev.evaluate([symbol("*"), 6, 7]) == 42 # an expression, as evaluate(value_t) + assert ev.evaluate("text") == "text" # a str value evaluates to itself + done = [] + assert ev.evaluate_script(LOOP, None, None, done.append) == 4950 + assert done == [4950] + env = pycvc.environment.extend(_builtins()) + env.set("x", 40) + assert ev.evaluate_script("(+ x 2)", env) == 42 + + +def test_state_steps_to_a_result(): + ev = pycvc.stackless_evaluator(_builtins()) + st = ev.create_state("(begin (set x 1) (+ x 41))") # a str is a script, as in C++ + assert isinstance(st, pycvc.evaluator_state) and not st.done + steps = 0 + while not ev.step(st): + steps += 1 + assert st.done and st.result == 42 and st.stack == [] + assert st.stats.get_step_count() > steps > 0 + # A value is an expression. + assert ev.run(ev.create_state([symbol("+"), 1, 2])) == 3 + + +def test_run_stops_at_max_steps_and_exposes_the_frames(): + ev = pycvc.stackless_evaluator(_builtins()) + st = ev.create_state(LOOP) + assert ev.run(st, 10) is None and not st.done # nil: not done + frames = st.stack + assert frames and all(isinstance(f, pycvc.eval_frame) for f in frames) + assert all(isinstance(f.phase, eval_phase) for f in frames) + assert frames[0].expr[0] == symbol("begin") + assert ev.run(st) == 4950 and st.done + _raises(OverflowError, ev.run, ev.create_state("1"), -1) + + +def test_state_reading_its_script_in_slices(): + ev = pycvc.stackless_evaluator(_builtins()) + popts = pycvc.parse_options() + popts.step_tokens = 2 + st = ev.create_state("(set x 20) (+ x 22)", None, popts) + assert isinstance(st.reading, pycvc.script_reader) + assert st.reading.source() == "(set x 20) (+ x 22)" + ev.step(st) + assert ev.run(st) == 42 and st.reading is None + + +def test_stats_is_a_reference_that_keeps_its_owner(): + st = pycvc.stackless_evaluator(_builtins()).create_state("(+ 1 2)") + stats = st.stats + del st + gc.collect() + assert stats.get_step_count() == 0 and not stats.is_complete() + rec = pycvc.evaluator(_builtins()) + rec.evaluate_script("(+ 1 2)") + s = rec.stats() + del rec + gc.collect() + assert s.get_step_count() > 0 and s.elapsed_seconds() >= 0.0 + own = pycvc.evaluation_stats() + own.start() + own.increment_step() + own.mark_complete() + assert own.get_step_count() == 1 and own.is_complete() + + +def test_user_macros(): + ev = pycvc.evaluator(_builtins()) + assert ev.evaluate_script("(begin (defmacro double-it (x) (* x 2)) (double-it 21))") == 42 + assert ev.user_macros() == {"double-it": ([symbol("x")], [symbol("*"), symbol("x"), 2])} + st = pycvc.evaluator_state() + st.user_macros = {"m": ([symbol("a")], symbol("a"))} + assert st.user_macros == {"m": ([symbol("a")], symbol("a"))} + _raises(TypeError, setattr, st, "user_macros", {"m": (["a"], 1)}) # params are symbols + + +def test_interrupt_from_another_thread(): + # The evaluators release the GIL while they evaluate, so this thread can interrupt them. + for ev in (pycvc.stackless_evaluator(_builtins()), pycvc.evaluator(_builtins())): + t = threading.Timer(0.1, ev.interrupt) + t.start() + # The 10 s timeout only bounds a failure; the interrupt comes first. + _raises(pycvc.evaluation_interrupted, ev.evaluate_script, "(while 1 1)", None, 10.0) + t.join() + ev.reset_interrupt() + assert ev.evaluate_script("(+ 1 2)") == 3 + + +def test_timeout_and_pause(): + for ev in (pycvc.stackless_evaluator(_builtins()), pycvc.evaluator(_builtins())): + e = _raises(pycvc.evaluation_timeout, ev.evaluate_script, "(while 1 1)", None, 0.05) + assert isinstance(e, RuntimeError) + ev.pause() + assert ev.is_paused() + t = threading.Timer(0.1, ev.resume) + t.start() + t0 = time.perf_counter() + assert ev.evaluate_script("(+ 1 2)") == 3 # waits for resume() + t.join() + assert time.perf_counter() - t0 >= 0.05 and not ev.is_paused() + + +def test_restrict_special_forms(): + assert {"if", "begin", "quote", "defclass"} <= pycvc.stackless_evaluator.all_special_forms() + ev = pycvc.stackless_evaluator(_builtins()) + ev.restrict_special_forms(["if", "begin"]) + assert ev.evaluate_script("(if #t 1 2)") == 1 + _raises(RuntimeError, ev.evaluate_script, "(quote 1)") + assert ev.evaluate_script("(+ 1 2)") == 3 # not a special form + ev.restrict_special_forms(None) + assert ev.evaluate_script("(quote 1)") == 1 + _raises(TypeError, ev.restrict_special_forms, [1]) + + +# ── codec ─────────────────────────────────────────────────────────────── + + +def test_environment_and_evaluator_state_codec(): + app = pycvc.make_app() + root = pycvc.State.instance(app) + env = pycvc.environment.extend(pycvc.environment()) + env.set("a", [1, "two"]) + env.outer.set("b", symbol("c")) + pycvc.encode_environment(root.child("snap.env"), env) + back = pycvc.decode_environment(root.child("snap.env")) + assert back.bindings == {"a": [1, "two"]} and back.lookup("b") == symbol("c") + + ev = pycvc.stackless_evaluator(_builtins()) + paused = ev.create_state(LOOP) + ev.run(paused, 10) + pycvc.encode_evaluator_state(root.child("snap.loop"), paused) + restored = pycvc.decode_evaluator_state(root.child("snap.loop")) + assert not restored.done and len(restored.stack) == len(paused.stack) + + # Resuming, as the C++ PauseResumeProducesCorrectResult does: the codec keeps each frame's + # scope by value (not shared identity), so the expression mutates nothing; and native + # builtins aren't serialized, so they go back on the root of every restored scope chain. + st = ev.create_state("(+ 1 2 3 4 5 6 7 8 9 10)") + ev.run(st, 3) + assert not st.done + pycvc.encode_evaluator_state(root.child("snap.es"), st) + restored = pycvc.decode_evaluator_state(root.child("snap.es")) + builtins = _builtins().bindings + for scope in [restored.global_env] + [f.env for f in restored.stack]: + while scope is not None and scope.outer is not None: + scope = scope.outer + if scope is not None: + for name, fn in builtins.items(): + scope.set(name, fn) + assert ev.run(restored) == 55 + + +# ── process ───────────────────────────────────────────────────────────── + + +def test_process_struct(): + p = pycvc.make_process() + assert isinstance(p, pycvc.process) + assert (p.pid, p.status, p.priority, p.parent_pid) == (-1, process_status.ready, 0, -1) + assert p.max_alloc_bytes == pycvc.kDefaultMaxAllocBytes == 16 << 20 + assert p.recv_path is None and p.exit_error is None and p.inbox == [] + assert p.time_left() is None # no max_time + assert p.alloc_limit(0) == 16 << 20 + p.max_memory = 1000 + assert p.alloc_limit(400) == 600 # the tighter of the two + p.recv_path = "a.b" + p.exit_error = "boom" + p.signal_handlers = {"SIGUSR1": [symbol("begin"), 1]} + p.pending_signals = ["SIGUSR1"] + p.status = process_status.paused + assert (p.recv_path, p.exit_error, p.status) == ("a.b", "boom", process_status.paused) + assert p.signal_handlers == {"SIGUSR1": [symbol("begin"), 1]} + assert list(p.pending_signals) == ["SIGUSR1"] + ev = pycvc.watch_event() + ev.watch_id = 3 + p.pending_watch_events = [ev] + assert [e.watch_id for e in p.pending_watch_events] == [3] + assert pycvc.process.watch_event is pycvc.watch_event + entry = pycvc.watch_handler_entry() + entry.handler = [symbol("begin"), 1] + entry.path = "a.b" + p.watch_handlers = {3: entry} + got = p.watch_handlers[3] + assert (got.handler, got.path) == ([symbol("begin"), 1], "a.b") + seen = [] + p.on_complete = seen.append + assert p.on_complete == seen.append + state = p.state # a reference into the process, which it keeps alive + del p + gc.collect() + assert not state.done and state.stack == [] + + +def test_current_process_during_a_run(): + for sched in (pycvc.scheduler(), pycvc.async_scheduler()): + run = sched.run if isinstance(sched, pycvc.scheduler) else sched.sync_run + assert sched.current_process() is None # nothing is being stepped + seen = [] + env = _builtins() + pycvc.builtins.register_fn(env, "whoami", + lambda: seen.append(sched.current_process()) or 0) + o = pycvc.execute_options() + o.env = env + o.name = "me" + pid = sched.execute("(whoami)", o) + run() + assert len(seen) == 1 and seen[0].pid == pid and seen[0].name == "me" + # Shared, as process_ptr is: it shows the process as it is now, finished. + assert seen[0].status is process_status.terminated and seen[0].exit_code == 0 + sched.queue_watch_event(pid, pycvc.watch_event()) # accepted for a finished pid + assert isinstance(sched, pycvc.scheduler_base) + + +def test_max_alloc_bytes_from_the_tree(): + app = pycvc.make_app() + root = pycvc.State.instance(app) + assert pycvc.scheduler().max_alloc_bytes() == pycvc.kDefaultMaxAllocBytes + assert pycvc.max_alloc_bytes_from_state(root, "") == pycvc.kDefaultMaxAllocBytes + root.child("state_exec.defaults.limits.max_alloc_bytes").value("1024") + root.child("state_exec.schedulers.s1.limits.max_alloc_bytes").value("2048") + assert pycvc.max_alloc_bytes_from_state(root, "") == 1024 + assert pycvc.max_alloc_bytes_from_state(root, "s1") == 2048 + assert pycvc.max_alloc_bytes_from_state(root, "s2", 7) == 1024 + s = pycvc.scheduler() + s.set_watch_root(root) + s.set_id("s1") + s.load_settings() + assert s.max_alloc_bytes() == 2048 + + +def test_allocation_limit_seen_by_a_python_function(): + assert pycvc.eval_alloc_limit() == pycvc.kDefaultMaxAllocBytes # no guard armed + e = _raises(RuntimeError, pycvc.eval_alloc_check, 1 << 40, "py-fn") + assert "py-fn" in str(e) + pycvc.eval_alloc_check(10, "py-fn") # under the limit + assert pycvc.alloc_bytes(3, 4) == 12 + assert pycvc.alloc_bytes(1 << 40, 1 << 40) == 2**64 - 1 # saturates at SIZE_MAX + # During a step, a Python function sees the limit the scheduler armed: the tighter of + # limits.max_alloc_bytes and what is left of the process's max_memory. + seen = [] + env = _builtins() + pycvc.builtins.register_fn(env, "limit", lambda: seen.append(pycvc.eval_alloc_limit()) or 0) + for sched in (pycvc.scheduler(), pycvc.async_scheduler()): + run = sched.run if isinstance(sched, pycvc.scheduler) else sched.sync_run + o = pycvc.execute_options() + o.env = env + o.max_memory = 4096 + sched.execute("(limit)", o) + run() + assert len(seen) == 2 and all(lim is not None and lim <= 4096 for lim in seen) + + +# ── resource_policy ───────────────────────────────────────────────────── + + +def _limits(steps=0, time_=0.0, memory=0, messages=0): + lim = pycvc.process_limits() + lim.max_steps, lim.max_time, lim.max_memory, lim.max_messages = steps, time_, memory, messages + return lim + + +def test_validate_limits(): + mode = pycvc.resource_policy.mode + assert [m.name for m in mode] == ["strict", "clamp", "warn"] + policy = pycvc.resource_policy() + assert policy.enforce is mode.clamp + r = pycvc.validate_limits(policy, _limits(1000, 60.0, 1024, 50)) + assert (r.max_steps, r.max_time, r.max_memory, r.max_messages) == (1000, 60.0, 1024, 50) + policy.max_steps_default = 500 + policy.max_time_default = 30.0 + assert (pycvc.validate_limits(policy, _limits()).max_steps, + pycvc.validate_limits(policy, _limits()).max_time) == (500, 30.0) + policy.max_steps_max = 100 + assert pycvc.validate_limits(policy, _limits(200)).max_steps == 100 # clamp + policy.enforce = mode.strict + _raises(RuntimeError, pycvc.validate_limits, policy, _limits(200)) + _raises(TypeError, setattr, policy, "enforce", 0) # an enum argument must be the enum + + +# ── intrinsics ────────────────────────────────────────────────────────── + + +def _intrinsic_env(app, sched): + ctx = pycvc.intrinsics_context() + ctx.sched = sched + ctx.root = pycvc.State.instance(app) + ctx.pid = 1 + env = _builtins() + pycvc.register_intrinsics(env, ctx) + return env, ctx + + +def test_register_intrinsics_over_a_context(): + app = pycvc.make_app() + sched = pycvc.scheduler() + env, ctx = _intrinsic_env(app, sched) + assert ctx.sched is sched and ctx.channels is None + for name in ("state-get", "state-set", "msg-send", "msg-recv", "self-pid", "spawn"): + assert isinstance(env.lookup(name), pycvc.value), name + o = pycvc.execute_options() + o.env = env + pid = sched.execute('(begin (state-set "py.x" "hello") (state-get "py.x"))', o) + assert sched.run()[pid] == "hello" + assert pycvc.state_get(app, "py.x") == "hello" + + +def test_registered_intrinsics_keep_their_context_alive(): + app = pycvc.make_app() + env, _ = _intrinsic_env(app, pycvc.scheduler()) # the context and scheduler go out of scope + gc.collect() + other = pycvc.scheduler() + o = pycvc.execute_options() + o.env = env + pid = other.execute('(begin (state-set "kept.x" "yes") (state-get "kept.x"))', o) + assert other.run()[pid] == "yes" + + +def test_apply_chroot_and_channels(): + app = pycvc.make_app() + sched = pycvc.scheduler() + ctx = pycvc.intrinsics_context() + ctx.sched = sched + pycvc.apply_chroot(ctx, pycvc.State.instance(app), "sandbox.user1") + assert ctx.root_path == "sandbox.user1" + env = _builtins() + pycvc.register_intrinsics(env, ctx) + o = pycvc.execute_options() + o.env = env + sched.execute('(state-set "x" "inside")', o) + sched.run() + assert pycvc.state_get(app, "sandbox.user1.x") == "inside" + + cp = pycvc.channel_policy() + cp.enforce = True + cp.declared = ["inbox"] + cp.global_ = ["g"] + ctx.channels = cp + assert ctx.channels is cp and list(ctx.channels.declared) == ["inbox"] + assert list(cp.global_) == ["g"] + + assert pycvc.resolve_channel_key("doc", "nav.done") == "doc.channels.nav.done" + assert pycvc.resolve_channel_key("doc", "#tick") == "#tick" + assert pycvc.resolve_channel_key("doc", "/g") == "g" + assert pycvc.resolve_channel_key("", "x") == "x" + fut = pycvc.make_future("ch.reply") + assert fut == {"__future__": "ch.reply"} + assert pycvc.future_channel_of(fut) == "ch.reply" + assert pycvc.future_channel_of(5) is None + + +if __name__ == "__main__": + tests = [v for k, v in sorted(globals().items()) if k.startswith("test_") and callable(v)] + for t in tests: + t() + print(" ok:", t.__name__) + print("pycvc state_exec evaluator parity tests: OK") diff --git a/docs/PYCVC_STATE_EXEC_PARITY.md b/docs/PYCVC_STATE_EXEC_PARITY.md index 39b477342..61334a690 100644 --- a/docs/PYCVC_STATE_EXEC_PARITY.md +++ b/docs/PYCVC_STATE_EXEC_PARITY.md @@ -49,19 +49,31 @@ approach agreed for closing the gap, and the plan. `None`, except for a value: `None` is nil, so `get_result`, `pop_pending_message` and `environment::lookup` raise `KeyError(pid / path / name)` instead. 8. **Callables cross as Python callables.** Where C++ takes a `native_fn` or a - `std::function` (`builtins::register_fn`, `execute_options::on_complete`), Python passes a - callable. It runs on the thread stepping the scheduler and takes the GIL for its call; a - Python exception becomes the C++ error a throwing callback would be. `on_complete` reads back - as the callable that was set. -9. **The GIL is released while programs run:** `scheduler.run`/`step` and - `async_scheduler.sync_run`/`sync_step`, so other Python threads keep going. The schedulers' + `std::function` (`builtins::register_fn`, the `on_complete` of `execute_options`, `process` + and the evaluators), Python passes a callable. It runs on the thread stepping the scheduler + or evaluator and takes the GIL for its call. A Python exception becomes the C++ error a + throwing callback would be. An `on_complete` member reads back as the callable that was set. +9. **The GIL is released while programs run:** `scheduler.run`/`step`, + `async_scheduler.sync_run`/`sync_step`, `stackless_evaluator.step`/`run`/`evaluate`/ + `evaluate_script` and `evaluator.evaluate`/`evaluate_script`. Other Python threads keep going, + and can call an evaluator's thread-safe `interrupt`, `pause` and `resume`. The schedulers' thread safety is the C++ one: while a scheduler runs, another thread may call only `async_scheduler.post_message`. +10. **References keep their owner alive.** A member or call that hands out a reference into its + object (`evaluator_state.stats`, `evaluator.stats()`, `process.state`, `Exec.scheduler()`) + returns a Python object that holds the owner. Every other reference return, such as a + struct's list or map member, comes back as a copy. +11. **Intrinsics keep their context alive.** C++'s `register_intrinsics(env, ctx)` requires the + caller to keep `ctx` valid for as long as `env` is used. In Python, every function it adds + holds the context, and the context holds the scheduler and channel policy assigned to it. + The context's `root` is a tree node, which lives as long as its app, so keep the app alive + as in C++. ## Inventory The audit (2026-10-17) found 97 public items in the 20 headers. Counted individually, with the 38 -`builtin_*` functions and the 7 `MIME_*` constants, it is 140. Status after PR 2: +`builtin_*` functions and the 7 `MIME_*` constants, it is 140. Items added to master since then +are listed too. Status after PR 3: | Header | Item | Python | Status | |---|---|---|---| @@ -78,24 +90,30 @@ The audit (2026-10-17) found 97 public items in the 20 headers. Counted individu | | `environment` (`bindings`, `outer`, `frozen`, `lookup`, `set`, `set_existing`, `extend`) | `pycvc.environment` | wrapped (PR 2), shared like `environment_ptr`. `bindings` reads and writes as a dict (a copy); `lookup` raises `KeyError` where C++ returns nullptr; `set` on a frozen scope (#660) raises `RuntimeError` | | | `closure`, `data_object`, `generator` | `pycvc.value` (opaque) | wrapped as opaque handles; no member access | | | `time_budget`, `eval_deadline_guard`, `eval_deadline_expired`, `thread_cpu_seconds` | none | exception: thread-local evaluator internals | +| | `kDefaultMaxAllocBytes`, `eval_alloc_limit`, `eval_alloc_check`, `alloc_bytes` (#660) | the same names under `pycvc.` | wrapped (PR 3). A Python function a program calls runs on the stepping thread, where the step's limit is armed, so it can check the limit as a builtin does | +| | `eval_alloc_guard` (#660) | none | exception: RAII that must nest last-in first-out, which a Python object's lifetime doesn't promise | +| | `detail::release_nested`, `detail::drop_owner` (#657) | none | exception: internals of `value_t`'s destructor | | state_value_codec.h | `encode_value`, `decode_value` | `pycvc.encode_value(State, v)`, `pycvc.decode_value(State)` | wrapped (PR 1) | -| | `encode_environment`/`decode_environment`, `encode_evaluator_state`/`decode_evaluator_state` | | PR 3, with the environment and evaluator types | +| | `encode_environment`/`decode_environment`, `encode_evaluator_state`/`decode_evaluator_state` | `pycvc.encode_environment(State, env)`, ... | wrapped (PR 3) | | scheduler_base.h | `execute_options`, `process_info`, `scheduler_stats` | `pycvc.execute_options`, `pycvc.process_info`, `pycvc.scheduler_stats` | wrapped (PR 2). `on_complete` is a Python callable, `signal_handlers` a dict, the `uint64_t` fields Python ints | | | `scheduling_policy` | `pycvc.scheduling_policy` | wrapped (PR 2), an `IntEnum` | -| | `scheduler_base` | none | exception: the abstract interface the intrinsics use; Python has the two concrete classes | -| scheduler.h | `scheduler` (about 45 members) | `pycvc.scheduler` | wrapped (PR 2). `run`/`step` release the GIL. `execute(str)` is the script overload, so a string can't be passed as a `value_t` expression (write it as the script `"\"text\""`). `id()`, `parser_settings()` return copies. Not wrapped: `mem_tracker()` (see memory_tracker.h), and `current_process()` / `queue_watch_event()`, which take the `process` struct (PR 3) | +| | `scheduler_base` | `pycvc.scheduler_base` | wrapped (PR 3) as the common base of the two classes (what `intrinsics_context.sched` holds). Its members are on the concrete classes | +| scheduler.h | `scheduler` (about 45 members) | `pycvc.scheduler` | wrapped (PR 2). `run`/`step` release the GIL. `execute(str)` is the script overload, so a string can't be passed as a `value_t` expression (write it as the script `"\"text\""`). `id()`, `parser_settings()` return copies. `current_process()` (shares the process), `queue_watch_event()` and `max_alloc_bytes()` (#660) are wrapped in PR 3. Not wrapped: `mem_tracker()` (see memory_tracker.h) | +| | `max_alloc_bytes_from_state(state&, id, base)` (#660) | `pycvc.max_alloc_bytes_from_state(State, …)` | wrapped (PR 3) | | async_scheduler.h | `async_scheduler` (about 55 members) | `pycvc.async_scheduler` | wrapped (PR 2), with the same notes as `scheduler`; `sync_run`/`sync_step` release the GIL, and `parser_defaults()` returns a copy. Coroutine `step()`/`run()` are an exception: Python gets `sync_step()`/`sync_run()` | | process.h | `process_status`, `to_string(process_status)`, `parse_process_status` | `pycvc.process_status` (an `IntEnum`), `pycvc.to_string`, `pycvc.parse_process_status` | wrapped (PR 2) | -| | `process` (about 35 fields), `make_process` | | PR 3, with `evaluator_state`, which a process holds | +| | `process`, `process::watch_event`, `process::watch_handler_entry`, `make_process` | `pycvc.process` (shared, as `process_ptr`), `pycvc.watch_event`, `pycvc.watch_handler_entry` (also `process.watch_event`, ...), `pycvc.make_process()` | wrapped (PR 3). `state` is a reference into the process; `inbox` reads as a list. The `steady_clock` time points (`create_time`, `last_run_start`, `wake_time`) are an exception: `elapsed_time()` and `accumulated_time` carry the same information | | builtins.h | `builtins::make_default_environment`, `builtins::register_fn` | `pycvc.builtins.make_default_environment()`, `pycvc.builtins.register_fn(env, name, fn)` | wrapped (PR 2). `fn` is a Python callable, or a `pycvc.value` holding a native fn | | | 38 `builtin_*` functions | none | exception: DSL builtins, called from DSL source | | | `percent_encode`, `percent_decode` | `pycvc.percent_encode`, `pycvc.percent_decode` | wrapped (PR 2) | -| stackless_evaluator.h, evaluator.h | `stackless_evaluator`, `evaluator`, `evaluator_state`, `evaluation_stats`, the evaluation exceptions | | PR 3 | +| stackless_evaluator.h, evaluator.h | `stackless_evaluator`, `evaluator`, `evaluator_state`, `eval_frame`, `eval_phase`, `evaluation_stats`, `evaluation_interrupted`, `evaluation_timeout` | the same names under `pycvc.` | wrapped (PR 3). `eval_phase` is an `IntEnum`; the two exceptions are `RuntimeError` subclasses. `on_complete` is a Python callable. `evaluator.user_macros()` returns a copy, a dict of name -> (params, body). `restrict_special_forms` takes any iterable of str, or None | +| | `return_exception` | none | exception: the internal unwinding of `(return ...)`, never seen outside an evaluation | | async_evaluator.h, async_stackless_evaluator.h | the async evaluators | none | exception: coroutine API | -| resource_policy.h | `resource_policy`, `process_limits`, `validate_limits` | | PR 3 | -| intrinsics.h | `register_intrinsics`, `intrinsics_context`, `apply_chroot`, `channel_policy` | `Exec` uses them internally | PR 3, with the intrinsics context's raw-pointer fields as `State`/scheduler references | -| | `document_scope`, `park_on_channel`, `make_future`, `future_channel_of` | none | exception: Ariadne/runtime internals (templates, raw process pointers) | -| stdlib.h | `stdlib_registry` | | after the stdlib is wired into the DSL (separate work: today only tests import it) | +| resource_policy.h | `resource_policy`, `process_limits`, `validate_limits` | `pycvc.resource_policy` (its `mode` is `pycvc.resource_policy.mode`, an `IntEnum`), `pycvc.process_limits`, `pycvc.validate_limits` | wrapped (PR 3) | +| intrinsics.h | `register_intrinsics`, `intrinsics_context`, `apply_chroot`, `channel_policy`, `resolve_channel_key`, `make_future`, `future_channel_of` | the same names under `pycvc.` | wrapped (PR 3). See approach item 11 for the context's lifetime. `channel_policy.global` is `global_` (a Python keyword). The context's `tracker`, `document` and `watches` aren't wrapped: `memory_tracker` is internal, `document_scope`'s slots are C++ templates, and `watches` is the intrinsics' own registry | +| | `document_scope`, `park_on_channel` | none | exception: Ariadne/runtime internals. `document_scope` is typed C++ template slots; `park_on_channel` takes raw process pointers and is valid only inside a running native function | +| stdlib.h | `stdlib_registry` | | next (PR 4). #660 loads the stdlib modules into every environment, so DSL programs already reach them | +| data_job.h (#658), heap.h (#667) | the deep-data jobs, the heap measurement, and the fields they add to `process_info`, `process`, `evaluator_state` and `evaluation_stats` | | after those PRs merge (PR 4); #658 wraps its own `data_options` | | exec_coordinator.h | `exec_coordinator` and its result types | none | deferred: it needs the cluster shard, membership and message-bus objects, which pycvc doesn't wrap | | memory_tracker.h | `memory_tracker` | none | exception: per-node byte accounting, internal to the schedulers | | generator.h, task.h | `generator`, `generator_next`, `task<>` | none | exception: coroutine and evaluator internals; a generator crosses as a `pycvc.value` | @@ -112,18 +130,21 @@ Each PR is stacked on the previous one and includes pytest checks against the C+ `values_equal`, `encode_value`, `decode_value`. - `bindings/pycvc/test_pycvc_state_exec.py`. - The pycvc README's outdated Status and Design sections are rewritten. -2. **PR 2 (this):** `environment` and `builtins`; `execute_options`, `process_info`, +2. **PR 2 (#671):** `environment` and `builtins`; `execute_options`, `process_info`, `scheduler_stats` and the enums; `scheduler` and `async_scheduler` as real classes, with their non-coroutine methods. `Exec.scheduler()` returns its scheduler. `bindings/pycvc/test_pycvc_state_exec_sched.py` repeats the C++ scheduler tests on both classes. -3. **PR 3:** the evaluators and `evaluator_state`, the `process` struct and the scheduler calls - that take it, the environment and evaluator-state codec, `resource_policy`, and - `register_intrinsics` / `intrinsics_context`. With `register_intrinsics`, a program submitted - to `Exec.scheduler()` can get the intrinsics `Exec.run` programs have; until then an - environment built in Python has only the builtins and what `register_fn` adds. +3. **PR 3 (this):** the evaluators, `evaluator_state`, `eval_frame` and `evaluation_stats`; the + `process` struct and the scheduler calls that take it; the environment and evaluator-state + codec; `resource_policy`; `register_intrinsics` and `intrinsics_context`, so an environment + built in Python can have the intrinsics `Exec.run` programs have; and master's + `max_alloc_bytes` additions (#660). `bindings/pycvc/test_pycvc_state_exec_eval.py` repeats the + C++ tests. +4. **PR 4:** `stdlib_registry`, and the API the deep-data (#658) and heap (#667) work adds, once + those land. -Known gaps after PR 1: +Known gaps: - Invalid UTF-8 in a DSL string raises when it crosses to Python, but no Python test produces such a string. Python can't create one, and the stdlib's `bytes.decode` isn't reachable from DSL code yet. From e3a01a3f1f95c77799a6bba1bca35ba894484549 Mon Sep 17 00:00:00 2001 From: Joe Rivera Date: Sat, 17 Oct 2026 23:53:09 -0500 Subject: [PATCH 16/17] pycvc: state_exec parity part 4 (data jobs, heap, stdlib registry, any-depth values) - The value conversion walks with an explicit stack instead of recursing, so a value nested any depth crosses both ways (the parser's max_depth default is going to 100000). A list or dict shared within a value converts to one object, both ways: sharing survives the trip, a shared structure converts in time proportional to its size, and a DSL list that holds itself becomes a Python list that holds itself. A Python value that holds itself still raises ValueError. - The deep-data jobs (#658, #665): data_job, the make_*_job factories, run_to_completion, run_data_job (a Python callable as `then`), call_function, is_compound, count_data_nodes, kDataNodeBytes. A job is handed to Python owned (std::unique_ptr); `who` is kept for good, as C++ holds the pointer. - The heap (#667): heap_options and its tree settings, measure_heap_share, make_heap_share_job, heap_remeasure_bytes, and the per-thread cycle collector; the heap fields on process and process_info, evaluator_state.heap_measured, stackless_evaluator.global_env(), eval_alloc_note. - stdlib_registry: standard() (a copy), shared_scope(), register_function (a Python callable or a pycvc.value native), list_*, import_module, lookup_qualified (KeyError where C++ returns nullptr). - SWIG_STD_MOVE moves on MSVC too (it reports __cplusplus 199711L without /Zc:__cplusplus), so a unique_ptr argument can be passed. test_pycvc_state_exec_data.py covers the new calls, and test_pycvc_state_exec.py the any-depth, sharing conversion. The parity doc and README are updated. --- bindings/pycvc/CMakeLists.txt | 1 + bindings/pycvc/README.md | 3 +- bindings/pycvc/pycvc.i | 498 +++++++++++++++---- bindings/pycvc/pycvc_value.h | 6 +- bindings/pycvc/test_pycvc_state_exec.py | 28 +- bindings/pycvc/test_pycvc_state_exec_data.py | 170 +++++++ docs/PYCVC_STATE_EXEC_PARITY.md | 23 +- 7 files changed, 613 insertions(+), 116 deletions(-) create mode 100644 bindings/pycvc/test_pycvc_state_exec_data.py diff --git a/bindings/pycvc/CMakeLists.txt b/bindings/pycvc/CMakeLists.txt index e2cfc52bd..f24179153 100644 --- a/bindings/pycvc/CMakeLists.txt +++ b/bindings/pycvc/CMakeLists.txt @@ -274,6 +274,7 @@ if(CVC_BUILD_PYCVC_CORE) pycvc_state_exec:test_pycvc_state_exec.py pycvc_state_exec_sched:test_pycvc_state_exec_sched.py pycvc_state_exec_eval:test_pycvc_state_exec_eval.py + pycvc_state_exec_data:test_pycvc_state_exec_data.py pycvc_world_clock:test_pycvc_world_clock.py pycvc_world_units:test_pycvc_world_units.py pycvc_dataprep:test_pycvc_dataprep.py diff --git a/bindings/pycvc/README.md b/bindings/pycvc/README.md index e6bb80260..f16a7697d 100644 --- a/bindings/pycvc/README.md +++ b/bindings/pycvc/README.md @@ -277,4 +277,5 @@ pycvc.register_intrinsics(env, ctx) # each function keeps ctx alive ``` Contract tests: `test_pycvc_exec.py`, `test_pycvc_state_exec.py`, -`test_pycvc_state_exec_sched.py`, `test_pycvc_state_exec_eval.py`. +`test_pycvc_state_exec_sched.py`, `test_pycvc_state_exec_eval.py`, +`test_pycvc_state_exec_data.py`. diff --git a/bindings/pycvc/pycvc.i b/bindings/pycvc/pycvc.i index 5de4c547d..60626c111 100644 --- a/bindings/pycvc/pycvc.i +++ b/bindings/pycvc/pycvc.i @@ -1456,6 +1456,7 @@ class AsyncStateObserver(state_observer): #include #include #include +#include #include #include #include @@ -1464,6 +1465,7 @@ class AsyncStateObserver(state_observer): #include #include #include +#include #include #include "pycvc_value.h" #include @@ -1477,10 +1479,18 @@ class AsyncStateObserver(state_observer): #include #include #include +#include #include #include #include +// SWIG passes by-value arguments with SWIG_STD_MOVE, which is std::move only when __cplusplus says +// C++11; MSVC reports 199711L without /Zc:__cplusplus, so there it copies, and a std::unique_ptr +// (a data job given to run_data_job) can't be copied. Move, as on GCC and Clang, for the wrapper +// functions, which come after this. +#undef SWIG_STD_MOVE +#define SWIG_STD_MOVE(OBJ) std::move(OBJ) + namespace pycvc { // The evaluators' on_complete parameter type, under a name SWIG can match a typemap on. using value_callback = std::function; @@ -1495,7 +1505,6 @@ using value_callback = std::function; namespace pycvc { namespace { using cvc::state_exec::value_t; -constexpr int kMaxValueDepth = 1000; swig_type_info *symbol_swig_type() { static swig_type_info *t = SWIG_TypeQuery("cvc::state_exec::symbol *"); @@ -1506,22 +1515,20 @@ swig_type_info *value_swig_type() { return t; } -bool too_deep(int depth) { - if (depth <= kMaxValueDepth) - return false; - PyErr_SetString(PyExc_ValueError, - "pycvc: value nests deeper than 1000 levels (or contains itself)"); - return true; -} - PyObject *text_to_py(const std::string &s) { return PyUnicode_DecodeUTF8(s.data(), static_cast(s.size()), "strict"); } -PyObject *to_py(const value_t &v, int depth) { +// The two directions walk a value with an explicit stack rather than recursing, so a value nested +// any depth converts (a parsed script may nest 100000 deep). A list or dict shared within a value +// converts once, to one object, so sharing survives the trip and a shared structure converts in +// time proportional to its size: a DSL list that holds itself becomes a Python list that holds +// itself. The other way, a Python value that holds itself raises ValueError, because only the +// evaluator's own containers are tracked for cycle collection (heap.h). + +// Everything but a list or dict: a new reference, or nullptr with an error set. +PyObject *leaf_to_py(const value_t &v) { using namespace cvc::state_exec; - if (too_deep(depth)) - return nullptr; if (v.is_nil()) Py_RETURN_NONE; if (const auto *b = std::get_if(&v.v)) @@ -1536,112 +1543,216 @@ PyObject *to_py(const value_t &v, int depth) { return PyBytes_FromStringAndSize(by->data.data(), static_cast(by->data.size())); if (const auto *sym = std::get_if(&v.v)) return SWIG_NewPointerObj(new symbol(*sym), symbol_swig_type(), SWIG_POINTER_OWN); - if (const auto *l = std::get_if(&v.v)) { - const Py_ssize_t n = *l ? static_cast((*l)->size()) : 0; - PyObject *out = PyList_New(n); + // closure / native_fn / data_object / generator: an opaque handle that converts back to the + // same value, so Python can hold it and pass it into the DSL again. + return SWIG_NewPointerObj(new value_t(v), value_swig_type(), SWIG_POINTER_OWN); +} + +PyObject *to_py(const value_t &root) { + using namespace cvc::state_exec; + struct frame { + const value_t *v; // a list or dict + PyObject *out; // its Python object, being filled + size_t next; // the next element to convert + size_t size; // the Python list's length + }; + std::vector stack; + std::unordered_map made; // container storage -> its Python object + // A new reference to `v` in Python. A list or dict not made yet is created empty and pushed to + // be filled; one already made (shared, or an enclosing one) is the same object. + auto convert = [&](const value_t &v) -> PyObject * { + const auto *l = std::get_if(&v.v); + const auto *d = std::get_if(&v.v); + if (l == nullptr && d == nullptr) + return leaf_to_py(v); + const void *storage = l ? static_cast(l->get()) : d->get(); + if (storage == nullptr) + return l ? PyList_New(0) : PyDict_New(); + if (const auto it = made.find(storage); it != made.end()) { + Py_INCREF(it->second); + return it->second; + } + const size_t size = l ? (*l)->size() : 0; + PyObject *out = l ? PyList_New(static_cast(size)) : PyDict_New(); if (out == nullptr) return nullptr; - for (Py_ssize_t k = 0; k < n; ++k) { - PyObject *item = to_py((**l)[static_cast(k)], depth + 1); + made.emplace(storage, out); + stack.push_back({&v, out, 0, size}); + return out; + }; + PyObject *result = convert(root); + if (result == nullptr) + return nullptr; + while (!stack.empty()) { + const size_t top = stack.size() - 1; // an index: convert() may grow the stack + const value_t &v = *stack[top].v; + PyObject *out = stack[top].out; + const size_t i = stack[top].next++; + if (const auto *l = std::get_if(&v.v)) { + if (i == stack[top].size) { + stack.pop_back(); + continue; + } + // The list can't change while the GIL is held, but a slot is filled with None rather than + // left empty should it be shorter than the Python list made for it. + PyObject *item = i < (*l)->size() ? convert((**l)[i]) : (Py_INCREF(Py_None), Py_None); if (item == nullptr) { - Py_DECREF(out); + Py_DECREF(result); return nullptr; } - PyList_SET_ITEM(out, k, item); // steals the new reference - } - return out; - } - if (const auto *dd = std::get_if(&v.v)) { - PyObject *out = PyDict_New(); - if (out == nullptr) - return nullptr; - if (*dd) { - for (const auto &kv : **dd) { - PyObject *key = text_to_py(kv.first); - PyObject *val = key ? to_py(kv.second, depth + 1) : nullptr; - const int rc = val ? PyDict_SetItem(out, key, val) : -1; - Py_XDECREF(key); - Py_XDECREF(val); - if (rc < 0) { - Py_DECREF(out); - return nullptr; - } + PyList_SET_ITEM(out, static_cast(i), item); // steals the reference + } else { + const auto &entries = **std::get_if(&v.v); + if (i >= entries.size()) { + stack.pop_back(); + continue; + } + PyObject *key = text_to_py(entries[i].first); + PyObject *val = key ? convert(entries[i].second) : nullptr; + const int rc = val ? PyDict_SetItem(out, key, val) : -1; + Py_XDECREF(key); + Py_XDECREF(val); + if (rc < 0) { + Py_DECREF(result); + return nullptr; } } - return out; } - // closure / native_fn / data_object / generator: an opaque handle that converts back to the - // same value, so Python can hold it and pass it into the DSL again. - return SWIG_NewPointerObj(new value_t(v), value_swig_type(), SWIG_POINTER_OWN); + return result; } -bool from_py(PyObject *o, value_t &out, int depth) { +// 1: `o` converted into `out`; 0: `o` is a list, tuple or dict, which from_py walks; -1: an error +// is set. +int leaf_from_py(PyObject *o, value_t &out) { using namespace cvc::state_exec; - if (too_deep(depth)) - return false; if (o == nullptr || o == Py_None) { out = value_t(); - return true; + return 1; } if (PyBool_Check(o)) { // before int: bool is an int subclass out = value_t(o == Py_True); - return true; + return 1; } if (PyLong_Check(o)) { int overflow = 0; const long long x = PyLong_AsLongLongAndOverflow(o, &overflow); if (overflow != 0) { PyErr_SetString(PyExc_OverflowError, "pycvc: int does not fit in 64 bits"); - return false; + return -1; } if (x == -1 && PyErr_Occurred()) - return false; + return -1; out = value_t(static_cast(x)); - return true; + return 1; } if (PyFloat_Check(o)) { const double d = PyFloat_AsDouble(o); if (d == -1.0 && PyErr_Occurred()) - return false; + return -1; out = value_t(d); - return true; + return 1; } if (PyUnicode_Check(o)) { Py_ssize_t n = 0; const char *s = PyUnicode_AsUTF8AndSize(o, &n); // fails (and raises) on lone surrogates if (s == nullptr) - return false; + return -1; out = value_t(std::string(s, static_cast(n))); - return true; + return 1; } if (PyBytes_Check(o)) { out = make_bytes(std::string(PyBytes_AS_STRING(o), static_cast(PyBytes_GET_SIZE(o)))); - return true; + return 1; } if (PyByteArray_Check(o)) { out = make_bytes( std::string(PyByteArray_AS_STRING(o), static_cast(PyByteArray_GET_SIZE(o)))); - return true; + return 1; } - if (PyList_Check(o) || PyTuple_Check(o)) { - const bool tup = PyTuple_Check(o); - const Py_ssize_t n = tup ? PyTuple_GET_SIZE(o) : PyList_GET_SIZE(o); - std::vector elems; - elems.reserve(static_cast(n)); - for (Py_ssize_t k = 0; k < n; ++k) { - value_t e; - if (!from_py(tup ? PyTuple_GET_ITEM(o, k) : PyList_GET_ITEM(o, k), e, depth + 1)) - return false; - elems.push_back(std::move(e)); + if (PyList_Check(o) || PyTuple_Check(o) || PyDict_Check(o)) + return 0; + void *p = nullptr; + if (SWIG_IsOK(SWIG_ConvertPtr(o, &p, symbol_swig_type(), 0)) && p != nullptr) { + out = value_t(*static_cast(p)); + return 1; + } + if (SWIG_IsOK(SWIG_ConvertPtr(o, &p, value_swig_type(), 0)) && p != nullptr) { + out = *static_cast(p); + return 1; + } + PyErr_Clear(); // a failed proxy lookup above may leave an attribute error behind + PyErr_Format(PyExc_TypeError, "pycvc: cannot convert %.200s to a state_exec value", + Py_TYPE(o)->tp_name); + return -1; +} + +bool from_py(PyObject *root, value_t &out) { + using namespace cvc::state_exec; + const int leaf = leaf_from_py(root, out); + if (leaf != 0) + return leaf > 0; + struct frame { + PyObject *o; // a list, tuple or dict + value_t v; // its value, being filled (shares its storage with the value handed out) + Py_ssize_t pos; + }; + std::vector stack; + std::unordered_map made; // container -> its value + std::unordered_set filling; // containers on the stack: met again, a cycle + // References on every container made, so none goes while its value is in `made` (a lookup of a + // proxy can run Python code). Dropped when this returns. + struct holds { + std::vector objects; + ~holds() { + for (PyObject *o : objects) + Py_DECREF(o); + } + } held; + // `o`'s value: one already made is shared; otherwise an empty list or dict, pushed to be filled. + auto container = [&](PyObject *o, value_t &v) -> bool { + if (filling.count(o) != 0) { + PyErr_SetString(PyExc_ValueError, "pycvc: value contains itself"); + return false; } - out = make_list(std::move(elems)); + if (const auto it = made.find(o); it != made.end()) { + v = it->second; + return true; + } + if (PyDict_Check(o)) { + v = make_dict(); + } else { + v = make_list(); + std::get(v.v)->reserve( + static_cast(PyTuple_Check(o) ? PyTuple_GET_SIZE(o) : PyList_GET_SIZE(o))); + } + Py_INCREF(o); + held.objects.push_back(o); + made.emplace(o, v); + filling.insert(o); + stack.push_back({o, v, 0}); return true; - } - if (PyDict_Check(o)) { - std::vector> entries; - PyObject *k = nullptr, *v = nullptr; - Py_ssize_t pos = 0; - while (PyDict_Next(o, &pos, &k, &v)) { + }; + // An element's value, into `v`. + auto element = [&](PyObject *item, value_t &v) -> bool { + Py_INCREF(item); // the container could drop it while a proxy lookup runs Python code + const int r = leaf_from_py(item, v); + const bool ok = r > 0 || (r == 0 && container(item, v)); + Py_DECREF(item); + return ok; + }; + if (!container(root, out)) + return false; + while (!stack.empty()) { + const size_t top = stack.size() - 1; // an index: container() may grow the stack + PyObject *o = stack[top].o; + if (PyDict_Check(o)) { + PyObject *k = nullptr; + PyObject *item = nullptr; + if (!PyDict_Next(o, &stack[top].pos, &k, &item)) { + filling.erase(o); + stack.pop_back(); + continue; + } if (!PyUnicode_Check(k)) { PyErr_Format(PyExc_TypeError, "pycvc: dict keys must be str, not %.200s", Py_TYPE(k)->tp_name); @@ -1651,38 +1762,38 @@ bool from_py(PyObject *o, value_t &out, int depth) { const char *ks = PyUnicode_AsUTF8AndSize(k, &n); if (ks == nullptr) return false; + std::string key(ks, static_cast(n)); value_t e; - if (!from_py(v, e, depth + 1)) + if (!element(item, e)) return false; - entries.emplace_back(std::string(ks, static_cast(n)), std::move(e)); + std::get(stack[top].v.v)->emplace_back(std::move(key), std::move(e)); + } else { + const bool tup = PyTuple_Check(o); + const Py_ssize_t i = stack[top].pos; + if (i >= (tup ? PyTuple_GET_SIZE(o) : PyList_GET_SIZE(o))) { + filling.erase(o); + stack.pop_back(); + continue; + } + stack[top].pos = i + 1; + value_t e; + if (!element(tup ? PyTuple_GET_ITEM(o, i) : PyList_GET_ITEM(o, i), e)) + return false; + std::get(stack[top].v.v)->push_back(std::move(e)); } - out = make_dict(std::move(entries)); - return true; } - void *p = nullptr; - if (SWIG_IsOK(SWIG_ConvertPtr(o, &p, symbol_swig_type(), 0)) && p != nullptr) { - out = value_t(*static_cast(p)); - return true; - } - if (SWIG_IsOK(SWIG_ConvertPtr(o, &p, value_swig_type(), 0)) && p != nullptr) { - out = *static_cast(p); - return true; - } - PyErr_Clear(); // a failed proxy lookup above may leave an attribute error behind - PyErr_Format(PyExc_TypeError, "pycvc: cannot convert %.200s to a state_exec value", - Py_TYPE(o)->tp_name); - return false; + return true; } } // namespace -PyObject *value_to_py(const cvc::state_exec::value_t &v) { return to_py(v, 0); } +PyObject *value_to_py(const cvc::state_exec::value_t &v) { return to_py(v); } PyObject *values_to_py_tuple(std::span values) { PyObject *t = PyTuple_New(static_cast(values.size())); if (t == nullptr) return nullptr; for (size_t i = 0; i < values.size(); ++i) { - PyObject *item = to_py(values[i], 0); + PyObject *item = to_py(values[i]); if (item == nullptr) { Py_DECREF(t); return nullptr; @@ -1692,7 +1803,7 @@ PyObject *values_to_py_tuple(std::span values) { return t; } -bool py_to_value(PyObject *o, cvc::state_exec::value_t &out) { return from_py(o, out, 0); } +bool py_to_value(PyObject *o, cvc::state_exec::value_t &out) { return from_py(o, out); } } // namespace pycvc %} @@ -2262,6 +2373,37 @@ bool py_to_macro_map(PyObject *o, macro_map &out) { %typemap(typecheck, precedence=SWIG_TYPECHECK_POINTER) pycvc::value_callback "$1 = ($input == Py_None || PyCallable_Check($input)) ? 1 : 0;" +// ── native functions from Python, and the names C++ keeps ── +%{ +#include +namespace pycvc { +namespace { +// A native_fn from `fn`: the native function a pycvc.value holds, or a Python callable wrapped by +// python_native_fn (which throws std::invalid_argument, prefixed with `who`, for anything else). +cvc::state_exec::native_fn native_from_py(const std::string &who, const std::string &name, + PyObject *fn) { + void *p = nullptr; + if (SWIG_IsOK(SWIG_ConvertPtr(fn, &p, value_swig_type(), 0)) && p != nullptr) { + const auto &held = *static_cast(p); + if (const auto *native = std::get_if(&held.v)) + return *native; + } + return python_native_fn(who, name, fn); +} + +// A `who` that lives for good. C++ keeps the const char* a job or call is given (for its errors, +// which can come after the call returns), where C++ passes a string literal; Python's string +// would go. Bounded by the distinct names used. +const char *stable_who(const std::string &who) { + static std::mutex mu; + static std::set names; // node-based: c_str() stays put + std::lock_guard lock(mu); + return names.insert(who).first->c_str(); +} +} // namespace +} // namespace pycvc +%} + // ── enums: pycvc.process_status and pycvc.scheduling_policy are enum.IntEnum classes ── // pycvc.process_status.killed is process_status::killed. An enum argument must be a member of its // IntEnum (strict, like values), which is also what lets to_string(process_status) and @@ -2565,6 +2707,10 @@ PYCVC_STATE_EXEC_NOGIL(cvc::state_exec::stackless_evaluator::evaluate) PYCVC_STATE_EXEC_NOGIL(cvc::state_exec::stackless_evaluator::evaluate_script) PYCVC_STATE_EXEC_NOGIL(cvc::state_exec::evaluator::evaluate) PYCVC_STATE_EXEC_NOGIL(cvc::state_exec::evaluator::evaluate_script) +// And the long walks: a job run to completion, a heap measurement, a cycle collection. +PYCVC_STATE_EXEC_NOGIL(cvc::state_exec::run_to_completion) +PYCVC_STATE_EXEC_NOGIL(cvc::state_exec::measure_heap_share) +PYCVC_STATE_EXEC_NOGIL(cvc::state_exec::collect_cycles) // Shared as in C++ (environment_ptr, process_ptr, ...). Struct members that are maps, lists or // values read and write as Python values (copies). @@ -2572,6 +2718,10 @@ PYCVC_STATE_EXEC_NOGIL(cvc::state_exec::evaluator::evaluate_script) %shared_ptr(cvc::state_exec::script_reader) %shared_ptr(cvc::state_exec::process) %shared_ptr(cvc::state_exec::intrinsics_context) +// A data job is owned (std::unique_ptr): Python owns one it is handed, and gives it up to +// run_data_job. +%include +%unique_ptr(cvc::state_exec::data_job) %naturalvar cvc::state_exec::value_tag; %naturalvar cvc::state_exec::environment::bindings; %naturalvar cvc::state_exec::execute_options::signal_handlers; @@ -2716,16 +2866,8 @@ public: static void register_fn(environment_ptr env, const std::string &name, PyObject *fn) { if (!env) throw std::invalid_argument("pycvc.builtins.register_fn: env is None"); - cvc::state_exec::value_t held; - void *p = nullptr; - if (SWIG_IsOK(SWIG_ConvertPtr(fn, &p, pycvc::value_swig_type(), 0)) && p != nullptr) - held = *static_cast(p); - if (const auto *native = std::get_if(&held.v)) { - cvc::state_exec::builtins::register_fn(env, name, *native); - return; - } cvc::state_exec::builtins::register_fn( - env, name, pycvc::python_native_fn("pycvc.builtins.register_fn", name, fn)); + env, name, pycvc::native_from_py("pycvc.builtins.register_fn", name, fn)); } } @@ -2813,8 +2955,18 @@ struct evaluator_state { }; %extend evaluator_state { evaluation_stats *_stats() { return &$self->stats; } + // A heap measurement's result while evaluation waits to resume (#667), or None. Read-only: it + // belongs to the measurement in flight. + PyObject *_heap_measured() const { + if (!$self->heap_measured) { + Py_INCREF(Py_None); + return Py_None; + } + return PyLong_FromUnsignedLongLong(static_cast(*$self->heap_measured)); + } %pythoncode %{ stats = property(_stats) + heap_measured = property(_heap_measured) %} } @@ -2850,6 +3002,7 @@ public: bool is_paused() const; void restrict_special_forms(std::shared_ptr > allowed); static const std::set &all_special_forms(); + environment_ptr global_env() const; // (#667) }; // Snapshot codec for an environment chain and for a whole evaluator state. @@ -2937,6 +3090,9 @@ struct process_info { double read_time; double eval_time; double data_time; + // The measured heap share (#667, heap.h), which max_memory covers too. + unsigned long long heap_memory; + unsigned long long heap_measurements; }; %extend process_info { %pythoncode %{ @@ -2977,6 +3133,7 @@ const size_t kDefaultMaxAllocBytes = 16777216; std::optional eval_alloc_limit(); void eval_alloc_check(size_t bytes, const char *who); size_t alloc_bytes(size_t count, size_t each); +void eval_alloc_note(size_t bytes); // (#667) report bytes allocated, for the heap's re-measures %pythonappend cvc::state_exec::process::_state %{ val._pycvc_owner = self %} @@ -3022,6 +3179,15 @@ struct process { unsigned long long message_count; unsigned long long message_bytes; int parent_pid; + // The heap share (#667, heap.h). + unsigned long long heap_bytes; + unsigned long long heap_alloc_bytes; + unsigned long long heap_steps; + unsigned long long heap_measurements; + size_t heap_check_steps; + bool heap_measuring; + bool heap_measure_due(unsigned long long state_bytes) const; + void release_state(); double elapsed_time() const; bool charges_cpu_time() const; void charge_cpu_time(double seconds); @@ -3246,7 +3412,137 @@ void apply_chroot(intrinsics_context &ctx, cvc::state &tree_root, const std::str std::string resolve_channel_key(const std::string &root_path, const std::string &channel); value_t make_future(const std::string &channel); std::optional future_channel_of(const value_t &v); + +// ── the heap (#667, heap.h) ── +// A process's heap share, and the per-thread reference-cycle collector, which a Python driver can +// run between scheduler runs on the thread that ran them. Not wrapped: heap_after_step (the +// schedulers' own per-step accounting) and track_captured_env / track_mutated (a value from +// Python is a fresh copy, so tracking it would mean nothing; the evaluator tracks its own). +struct heap_options { + size_t check_steps; +}; +heap_options heap_options_from_state(cvc::state &root, + const std::string &scheduler_id = std::string(), + const heap_options &base = heap_options()); +size_t measure_heap_share(const process &proc); +size_t heap_remeasure_bytes(size_t last_measured); +struct cycle_stats { + size_t candidates; + size_t examined; + size_t freed; +}; +cycle_stats collect_cycles(); +bool cycle_collection_due(); +void note_process_exit(); +size_t tracked_objects(); + +// ── deep-data jobs (#658, #665, data_job.h) ── +// A job runs a slice at a time: step(nodes) until it says it's done, then result(). The factories +// hand Python an owned job. From a Python function a program calls (it runs on the stepping +// thread), run_data_job and call_function work as for a C++ native: return what they return at +// once, and the evaluator finishes the call. Not wrapped: the call protocol for C++ job +// implementations (call_request, take_call, pending_call, call_returned, own_inputs) and +// detail::. make_concat_job, run_data_job and call_function are below, in pycvc's own wrappers. +%nodefaultctor data_job; +class data_job { +public: + virtual ~data_job(); + virtual bool step(size_t nodes) = 0; + virtual value_t result() = 0; + size_t nodes() const; +}; +const size_t kDataNodeBytes = 256; +bool is_compound(const value_t &v); +std::unique_ptr make_equal_job(value_t a, value_t b); +std::unique_ptr make_print_job(value_t v); +std::unique_ptr make_copy_job(value_t v); +std::unique_ptr make_substitute_job(value_t tmpl, + const std::unordered_map &subst); +std::unique_ptr make_heap_share_job(const process &proc); +value_t run_to_completion(data_job &job); +void count_data_nodes(size_t nodes); + +// ── stdlib_registry (stdlib.h) ── +// standard() returns a copy (C++ returns a const reference to the shared registry), so Python +// can't add to the one every environment shares. register_function takes a Python callable or a +// pycvc.value holding a native function, as builtins.register_fn does. lookup_qualified returns +// the function as a pycvc.value, or raises KeyError where C++ returns nullptr. +class stdlib_registry { +public: + stdlib_registry(); + static environment_ptr shared_scope(); + std::vector list_functions(const std::string &module) const; + std::vector list_modules() const; +}; +%extend stdlib_registry { + static cvc::state_exec::stdlib_registry standard() { + return cvc::state_exec::stdlib_registry::standard(); + } + void register_function(const std::string &module, const std::string &func_name, PyObject *fn) { + $self->register_function( + module, func_name, + pycvc::native_from_py("pycvc.stdlib_registry.register_function", func_name, fn)); + } + void import_module(const std::string &module, environment_ptr env, + const std::vector &specific_fns = std::vector()) { + if (!env) + throw std::invalid_argument("pycvc.stdlib_registry.import_module: env is None"); + $self->import_module(module, env, specific_fns); + } + value_t lookup_qualified(const std::string &qualified_name) const { + if (const cvc::state_exec::native_fn *fn = $self->lookup_qualified(qualified_name)) + return cvc::state_exec::value_t(*fn); + throw pycvc::no_value{qualified_name}; + } +} +} +} + +// make_concat_job, run_data_job and call_function: the C++ calls, except that `who` is kept for +// good (C++ holds the pointer for its errors), and run_data_job's `then` is a Python callable or +// None. +%{ +namespace pycvc { +std::unique_ptr +make_concat_job_kept(const std::vector &parts, const std::string &sep, + const std::string &who) { + return cvc::state_exec::make_concat_job(parts, sep, stable_who(who)); } + +cvc::state_exec::value_t run_data_job_then(std::unique_ptr job, + PyObject *then = nullptr) { + if (!job) + throw std::invalid_argument("pycvc.run_data_job: job is None"); + std::function after; + if (then != nullptr && then != Py_None) { + cvc::state_exec::native_fn fn = python_native_fn("pycvc.run_data_job", "then", then); + after = [fn](cvc::state_exec::value_t v) { + return fn(std::span(&v, 1)); + }; + } + return cvc::state_exec::run_data_job(std::move(job), std::move(after)); +} + +cvc::state_exec::value_t call_function_kept(const cvc::state_exec::value_t &fn, + const std::vector &args, + const std::string &who) { + return cvc::state_exec::call_function(fn, std::span(args), + stable_who(who)); +} +} // namespace pycvc +%} +%rename(make_concat_job) pycvc::make_concat_job_kept; +%rename(run_data_job) pycvc::run_data_job_then; +%rename(call_function) pycvc::call_function_kept; +namespace pycvc { +std::unique_ptr +make_concat_job_kept(const std::vector &parts, const std::string &sep, + const std::string &who); +cvc::state_exec::value_t run_data_job_then(std::unique_ptr job, + PyObject *then = nullptr); +cvc::state_exec::value_t call_function_kept(const cvc::state_exec::value_t &fn, + const std::vector &args, + const std::string &who); } // register_intrinsics(env, ctx): the C++ call, except that every function it adds keeps ctx (and diff --git a/bindings/pycvc/pycvc_value.h b/bindings/pycvc/pycvc_value.h index 5fb0fa844..f346fea8c 100644 --- a/bindings/pycvc/pycvc_value.h +++ b/bindings/pycvc/pycvc_value.h @@ -12,8 +12,10 @@ // - closure, native_fn, data_object, generator -> pycvc.value, an opaque handle that converts // back to the same value, so it can be passed into the DSL again. // -// Strict: an int outside 64 bits raises OverflowError, any other Python type raises TypeError, and -// a value nested deeper than 1000 levels (or a cyclic one) raises ValueError. Defined in the SWIG +// Strict: an int outside 64 bits raises OverflowError, and any other Python type raises TypeError. +// Any depth converts (the conversion doesn't recurse). A list or dict shared within a value +// converts to one shared object, both ways, so a DSL value that holds itself becomes a Python value +// that holds itself; a Python value that holds itself raises ValueError. Defined in the SWIG // wrapper (pycvc.i), which owns the pycvc.symbol / pycvc.value types. Call with the GIL held. #pragma once diff --git a/bindings/pycvc/test_pycvc_state_exec.py b/bindings/pycvc/test_pycvc_state_exec.py index 4c2da3dc6..ad4a6184e 100644 --- a/bindings/pycvc/test_pycvc_state_exec.py +++ b/bindings/pycvc/test_pycvc_state_exec.py @@ -104,13 +104,31 @@ def test_conversion_is_strict(): _raises(TypeError, pycvc.to_string, {1: 2}) # dict keys must be str _raises(OverflowError, pycvc.to_string, 2**70) _raises(TypeError, pycvc.to_string, {1, 2}) # a set has no state_exec equivalent - deep = [] - for _ in range(1100): - deep = [deep] - _raises(ValueError, pycvc.to_string, deep) loop = [] loop.append(loop) - _raises(ValueError, pycvc.to_string, loop) + _raises(ValueError, pycvc.to_string, loop) # only the evaluator's own cycles are collected + + +def test_conversion_takes_any_depth_and_keeps_sharing(): + # The conversion walks with an explicit stack, so a value nests any depth, both ways: here + # Python -> C++ (the quoted argument) -> Python (the result), 200000 deep. + deep = [] + for _ in range(200000): + deep = [deep] + ev = pycvc.stackless_evaluator(pycvc.builtins.make_default_environment()) + back = ev.evaluate([symbol("quote"), deep]) + depth = 0 + while back: + (back,) = back + depth += 1 + assert depth == 200000 + # A list shared within a value stays one object, both ways. + shared = [1, 2] + pair = ev.evaluate([symbol("quote"), [shared, shared]]) + assert pair[0] is pair[1] and pair[0] == [1, 2] + # A DSL list that holds itself comes out as a Python list that holds itself. + looped = ev.evaluate_script("(begin (set l (list 1)) (set-nth l 0 l) l)") + assert looped[0] is looped def test_registered_fns_get_symbols_and_strict_returns(): diff --git a/bindings/pycvc/test_pycvc_state_exec_data.py b/bindings/pycvc/test_pycvc_state_exec_data.py new file mode 100644 index 000000000..c8e964b68 --- /dev/null +++ b/bindings/pycvc/test_pycvc_state_exec_data.py @@ -0,0 +1,170 @@ +"""state_exec parity, part 4: deep-data jobs, the heap, and the stdlib registry. + +The deep-data jobs (data_job.h, #658/#665), the heap share and reference-cycle collector +(heap.h, #667) and stdlib_registry, under the C++ names. The value conversion itself has no +depth limit and keeps sharing (test_pycvc_state_exec.py). docs/PYCVC_STATE_EXEC_PARITY.md has +the inventory. +""" + +import pycvc +from pycvc import symbol + + +def _raises(exc, fn, *args): + try: + fn(*args) + except exc as e: + return e + raise AssertionError(f"{fn.__name__}{args!r} did not raise {exc.__name__}") + + +def _builtins(): + return pycvc.builtins.make_default_environment() + + +# ── deep-data jobs ────────────────────────────────────────────────────── + + +def test_a_job_runs_a_slice_at_a_time(): + job = pycvc.make_copy_job([1, [2, [3, [4]]]]) + assert isinstance(job, pycvc.data_job) + slices = 1 + while not job.step(1): + slices += 1 + assert slices > 1 and job.nodes() > 0 + assert job.result() == [1, [2, [3, [4]]]] + + +def test_the_job_factories(): + run = pycvc.run_to_completion + assert run(pycvc.make_equal_job([1, [2]], [1, [2]])) is True + assert run(pycvc.make_equal_job([1], [2])) is False + assert run(pycvc.make_print_job([symbol("a"), 1])) == pycvc.to_string([symbol("a"), 1]) + assert run(pycvc.make_concat_job(["a", 1, "b"], "-", "py-join")) == "a-1-b" + tmpl = [symbol("+"), symbol("x"), [symbol("x")]] + assert run(pycvc.make_substitute_job(tmpl, {"x": 41})) == [symbol("+"), 41, [41]] + assert pycvc.is_compound([1]) and not pycvc.is_compound([]) and not pycvc.is_compound(5) + assert pycvc.kDataNodeBytes == 256 + pycvc.count_data_nodes(3) # outside an evaluator step: nothing to count against + + +def test_run_data_job_outside_an_evaluator_runs_it_whole(): + assert pycvc.run_data_job(pycvc.make_copy_job([1, 2])) == [1, 2] + assert pycvc.run_data_job(pycvc.make_copy_job([1, 2]), len) == 2 # `then` maps the result + job = pycvc.make_copy_job([1]) + pycvc.run_data_job(job) # the job is given up to C++ + assert not job.thisown + + +def test_a_python_function_calls_dsl_functions(): + # call_function from a Python function a program calls: a native function is called + # there; a lambda runs as the process's own frames, and the call returns its result. + def apply_one(fn, arg): + return pycvc.call_function(fn, [arg], "py-apply") + + env = _builtins() + pycvc.builtins.register_fn(env, "apply-one", apply_one) + for sched in (pycvc.scheduler(), pycvc.async_scheduler()): + run = sched.run if isinstance(sched, pycvc.scheduler) else sched.sync_run + o = pycvc.execute_options() + o.env = env + native = sched.execute("(apply-one string.length \"four\")", o) + closure = sched.execute("(apply-one (lambda (x) (* x 2)) 21)", o) + results = run() + assert results[native] == 4 and results[closure] == 42 + # Outside a running program a lambda can't be called. + lam = pycvc.stackless_evaluator(_builtins()).evaluate_script("(lambda (x) x)") + e = _raises(RuntimeError, pycvc.call_function, lam, [1], "py-outside") + assert "py-outside" in str(e) + + +# ── the heap ──────────────────────────────────────────────────────────── + + +def test_heap_settings_and_fields(): + app = pycvc.make_app() + root = pycvc.State.instance(app) + assert pycvc.heap_options().check_steps == 4096 + root.child("state_exec.defaults.limits.heap_check_steps").value("100") + root.child("state_exec.schedulers.s1.limits.heap_check_steps").value("50") + assert pycvc.heap_options_from_state(root).check_steps == 100 + assert pycvc.heap_options_from_state(root, "s1").check_steps == 50 + assert pycvc.heap_remeasure_bytes(0) == 64 << 10 + assert pycvc.heap_remeasure_bytes(1 << 20) == 1 << 19 # half the last measurement + p = pycvc.make_process() + assert (p.heap_bytes, p.heap_measurements, p.heap_check_steps) == (0, 0, 4096) + assert not p.heap_measuring and not p.heap_measure_due(0) # no memory budget + assert isinstance(pycvc.measure_heap_share(p), int) + assert pycvc.run_to_completion(pycvc.make_heap_share_job(p)) == pycvc.measure_heap_share(p) + p.release_state() + assert p.state.heap_measured is None + pycvc.eval_alloc_note(10) # outside a step: nothing armed to note it + + +def test_a_budgeted_process_is_measured(): + for sched in (pycvc.scheduler(), pycvc.async_scheduler()): + run = sched.run if isinstance(sched, pycvc.scheduler) else sched.sync_run + o = pycvc.execute_options() + o.max_memory = 64 << 20 + pid = sched.execute("(begin (set big (collections.range 50000)) (length big))", o) + assert run()[pid] == 50000 + info = sched.get_process_info(pid) + assert info.heap_measurements >= 1 and info.heap_memory > 0 + + +def test_cycles_are_collected_on_this_thread(): + ev = pycvc.stackless_evaluator(_builtins()) + ev.evaluate_script( + "(begin (set i 0) (while (< i 300) (begin (let ((h nil)) (set h (lambda () h)))" + " (set i (+ i 1)))) 0)") + assert pycvc.tracked_objects() > 0 + stats = pycvc.collect_cycles() + assert isinstance(stats, pycvc.cycle_stats) and stats.freed > 0 + assert isinstance(pycvc.cycle_collection_due(), bool) + pycvc.note_process_exit() + + +def test_evaluator_global_env(): + env = _builtins() + ev = pycvc.stackless_evaluator(env) + assert ev.global_env().lookup("+") is not None + + +# ── stdlib_registry ───────────────────────────────────────────────────── + + +def test_the_standard_registry(): + reg = pycvc.stdlib_registry.standard() # a copy of the shared one + assert {"string", "bytes", "math", "collections"} <= set(reg.list_modules()) + assert "string.length" in reg.list_functions("string") + assert isinstance(reg.lookup_qualified("string.length"), pycvc.value) + assert _raises(KeyError, reg.lookup_qualified, "string.nope").args == ("string.nope",) + shared = pycvc.stdlib_registry.shared_scope() + assert shared.frozen and isinstance(shared.lookup("string.length"), pycvc.value) + # Adding to the copy leaves the shared registry alone. + reg.register_function("py", "py.twice", lambda x: 2 * x) + assert "py" in reg.list_modules() + assert "py" not in pycvc.stdlib_registry.standard().list_modules() + + +def test_a_registry_of_ones_own(): + reg = pycvc.stdlib_registry() + reg.register_function("py", "py.twice", lambda x: 2 * x) + reg.register_function("py", "py.len", reg.lookup_qualified("string.length")) # a native + env = pycvc.environment.extend(_builtins()) + reg.import_module("py", env) + ev = pycvc.stackless_evaluator(_builtins()) + assert ev.evaluate_script('(list (py.twice 21) (py.len "abc"))', env) == [42, 3] + only = pycvc.environment() + reg.import_module("math", only, ["math.sqrt"]) + assert list(only.bindings) == ["math.sqrt"] + _raises(RuntimeError, reg.import_module, "nope", only) + _raises(RuntimeError, reg.import_module, "py", None) + + +if __name__ == "__main__": + tests = [v for k, v in sorted(globals().items()) if k.startswith("test_") and callable(v)] + for t in tests: + t() + print(" ok:", t.__name__) + print("pycvc state_exec data/heap/stdlib parity tests: OK") diff --git a/docs/PYCVC_STATE_EXEC_PARITY.md b/docs/PYCVC_STATE_EXEC_PARITY.md index ca3ea3ef5..bf38765f1 100644 --- a/docs/PYCVC_STATE_EXEC_PARITY.md +++ b/docs/PYCVC_STATE_EXEC_PARITY.md @@ -29,7 +29,13 @@ approach agreed for closing the gap, and the plan. Conversion is strict: - an int outside 64 bits raises `OverflowError`; - a non-`str` dict key or any other Python type raises `TypeError`; - - a value nested deeper than 1000 levels, or one that contains itself, raises `ValueError`. + - a Python value that contains itself raises `ValueError`, since only the evaluator's own + containers are tracked for cycle collection. + + It takes any depth: it walks with an explicit stack, not recursion (PR 4), as the parser and + the evaluator do. A list or dict shared within a value converts to one object, both ways, so + sharing survives the trip, and a DSL list that holds itself comes out as a Python list that + holds itself. (PR 1's converter recursed and stopped at 1000 levels.) Before this, symbols became plain strings, closures became the text `""`, and any object became its `str()`. The old converter could also pass a null object into a list when a @@ -73,7 +79,7 @@ approach agreed for closing the gap, and the plan. The audit (2026-10-17) found 97 public items in the 20 headers. Counted individually, with the 38 `builtin_*` functions and the 7 `MIME_*` constants, it is 140. Items added to master since then -are listed too. Status after PR 3: +are listed too. Status after PR 4: | Header | Item | Python | Status | |---|---|---|---| @@ -84,7 +90,8 @@ are listed too. Status after PR 3: | | `script_reader` | `pycvc.script_reader` | wrapped (PR 1). `forms()`, `source()` and `options()` return copies | | | `parser` (`read`, `parse_expr`) | none | exception: it holds a `string_view` of its input, which Python can't keep alive. `script_reader` owns its source and gives the same slice-by-slice reading | | data_job.h (#658) | `data_options`, `data_options_from_state(state&, id, base)` | `pycvc.data_options`, `pycvc.data_options_from_state(State, …)` | wrapped (#658) | -| | the data jobs (`data_job`, `make_*_job`, `run_data_job`, ...) | | PR 4 | +| | `data_job` (`step`, `result`, `nodes`), `make_equal_job`, `make_print_job`, `make_concat_job`, `make_copy_job`, `make_substitute_job`, `run_to_completion`, `run_data_job`, `call_function`, `is_compound`, `count_data_nodes`, `kDataNodeBytes` (#658, #665) | the same names under `pycvc.` | wrapped (PR 4). A factory hands Python an owned job, which `run_data_job` takes over. `who` (C++ keeps the pointer for its errors) is kept for good. `run_data_job`'s `then` is a Python callable. From a Python function a program calls, `run_data_job` and `call_function` work as for a C++ native: return what they return at once | +| | the call protocol (`data_job::call_request`, `take_call`, `pending_call`, `call_returned`, `own_inputs`), `detail::deferred_data_job`, `detail::data_step_scope` | none | exception: for C++ job implementations and the evaluator's own bookkeeping; Python can't subclass a data job | | types.h | `symbol` | `pycvc.symbol` | wrapped (PR 1) | | | `value_t` | native types, or `pycvc.value` | wrapped (PR 1), see the table above | | | `to_string`, `values_equal` | `pycvc.to_string`, `pycvc.values_equal` | wrapped (PR 1) | @@ -116,8 +123,11 @@ are listed too. Status after PR 3: | resource_policy.h | `resource_policy`, `process_limits`, `validate_limits` | `pycvc.resource_policy` (its `mode` is `pycvc.resource_policy.mode`, an `IntEnum`), `pycvc.process_limits`, `pycvc.validate_limits` | wrapped (PR 3) | | intrinsics.h | `register_intrinsics`, `intrinsics_context`, `apply_chroot`, `channel_policy`, `resolve_channel_key`, `make_future`, `future_channel_of` | the same names under `pycvc.` | wrapped (PR 3). See approach item 11 for the context's lifetime. `channel_policy.global` is `global_` (a Python keyword). The context's `tracker`, `document` and `watches` aren't wrapped: `memory_tracker` is internal, `document_scope`'s slots are C++ templates, and `watches` is the intrinsics' own registry | | | `document_scope`, `park_on_channel` | none | exception: Ariadne/runtime internals. `document_scope` is typed C++ template slots; `park_on_channel` takes raw process pointers and is valid only inside a running native function | -| stdlib.h | `stdlib_registry` | | next (PR 4). #660 loads the stdlib modules into every environment, so DSL programs already reach them | -| heap.h (#667) | the heap measurement, and the fields it adds to `process_info` and `process` | | PR 4, after #667 merges | +| stdlib.h | `stdlib_registry` (`standard`, `shared_scope`, `register_function`, `list_functions`, `list_modules`, `import_module`, `lookup_qualified`) | `pycvc.stdlib_registry` | wrapped (PR 4). `standard()` returns a copy, so Python can't add to the shared registry; `register_function` takes a Python callable or a `pycvc.value` native; `lookup_qualified` raises `KeyError` where C++ returns nullptr | +| heap.h (#667) | `heap_options`, `heap_options_from_state`, `measure_heap_share`, `make_heap_share_job`, `heap_remeasure_bytes`, `cycle_stats`, `collect_cycles`, `cycle_collection_due`, `note_process_exit`, `tracked_objects` | the same names under `pycvc.` | wrapped (PR 4). The collector is per thread: run it on the thread that ran the programs. `measure_heap_share` and `collect_cycles` release the GIL | +| | `heap_after_step`, `track_captured_env`, `track_mutated` | none | exception: the schedulers' per-step accounting, and tracking that only means something for the evaluator's own containers (a value from Python is a fresh copy) | +| | #667's fields: `process.heap_*`, `heap_measure_due`, `release_state`; `process_info.heap_memory`/`heap_measurements`; `evaluator_state.heap_measured`; `stackless_evaluator::global_env()`; `eval_alloc_note` | the same names | wrapped (PR 4). `heap_measured` is read-only (the result of a measurement in flight) | +| | `eval_alloc_guard::allocated()` (#667) | none | exception: the guard isn't wrapped (see types.h) | | exec_coordinator.h | `exec_coordinator` and its result types | none | deferred: it needs the cluster shard, membership and message-bus objects, which pycvc doesn't wrap | | memory_tracker.h | `memory_tracker` | none | exception: per-node byte accounting, internal to the schedulers | | generator.h, task.h | `generator`, `generator_next`, `task<>` | none | exception: coroutine and evaluator internals; a generator crosses as a `pycvc.value` | @@ -145,8 +155,7 @@ Each PR is stacked on the previous one and includes pytest checks against the C+ built in Python can have the intrinsics `Exec.run` programs have; and master's `max_alloc_bytes` additions (#660). `bindings/pycvc/test_pycvc_state_exec_eval.py` repeats the C++ tests. -4. **PR 4:** `stdlib_registry`, and the API the deep-data (#658) and heap (#667) work adds, once - those land. +4. **PR 4 (this):** `stdlib_registry`, the deep-data jobs (#658, #665) and the heap (#667); and the value conversion without a depth limit (it walks with an explicit stack, as the parser and evaluator do, and keeps shared structure). `bindings/pycvc/test_pycvc_state_exec_data.py` covers them. Known gaps: - Invalid UTF-8 in a DSL string raises when it crosses to Python, but no Python test produces such From 56534ce3893d548ea9679944e9469afa5ef25621 Mon Sep 17 00:00:00 2001 From: Joe Rivera Date: Sun, 18 Oct 2026 16:03:12 -0500 Subject: [PATCH 17/17] pycvc: wrap #682/#683's call_function continuation, make_generator_job and the native nesting limits --- bindings/pycvc/pycvc.i | 37 +++++++++++++++++--- bindings/pycvc/test_pycvc_state_exec_data.py | 28 +++++++++++++++ docs/PYCVC_STATE_EXEC_PARITY.md | 4 +-- 3 files changed, 63 insertions(+), 6 deletions(-) diff --git a/bindings/pycvc/pycvc.i b/bindings/pycvc/pycvc.i index 1c5b9e064..1b32e66fb 100644 --- a/bindings/pycvc/pycvc.i +++ b/bindings/pycvc/pycvc.i @@ -3460,6 +3460,11 @@ public: size_t nodes() const; }; const size_t kDataNodeBytes = 256; +// The native-call nesting limits (#682): a level is refused past kMaxNativeNesting, or when less +// than kNativeStackReserve bytes of the thread's stack are left (128 KiB on Windows, 32 KiB +// elsewhere). detail::native_nesting, the RAII level, isn't wrapped. +const size_t kMaxNativeNesting = 256; +const size_t kNativeStackReserve = 131072; bool is_compound(const value_t &v); std::unique_ptr make_equal_job(value_t a, value_t b); std::unique_ptr make_print_job(value_t v); @@ -3533,15 +3538,37 @@ cvc::state_exec::value_t run_data_job_then(std::unique_ptr &args, - const std::string &who) { - return cvc::state_exec::call_function(fn, std::span(args), - stable_who(who)); + const std::string &who, PyObject *then = nullptr) { + const std::span argv(args); + if (then == nullptr || then == Py_None) + return cvc::state_exec::call_function(fn, argv, stable_who(who)); + // The overload with a continuation (#683): `then` runs on the call's value once the function + // has finished, however the evaluator ran it. + cvc::state_exec::native_fn after = python_native_fn("pycvc.call_function", "then", then); + return cvc::state_exec::call_function( + fn, argv, stable_who(who), [after](cvc::state_exec::value_t v) { + return after(std::span(&v, 1)); + }); +} + +// make_generator_job takes the generator as a pycvc.value (C++ takes its generator_ptr). +std::unique_ptr make_generator_job_from(const cvc::state_exec::value_t &gen, + bool collect = false) { + const auto *g = std::get_if(&gen.v); + if (g == nullptr || !*g) + throw std::invalid_argument("pycvc.make_generator_job: not a generator"); + return cvc::state_exec::make_generator_job(*g, collect); } } // namespace pycvc %} %rename(make_concat_job) pycvc::make_concat_job_kept; %rename(run_data_job) pycvc::run_data_job_then; %rename(call_function) pycvc::call_function_kept; +%rename(make_generator_job) pycvc::make_generator_job_from; +// One wrapper each, defaults filled in, rather than an overload per default (a value list has no +// overload typecheck). +%feature("compactdefaultargs") pycvc::call_function_kept; +%feature("compactdefaultargs") pycvc::make_generator_job_from; namespace pycvc { std::unique_ptr make_concat_job_kept(const std::vector &parts, const std::string &sep, @@ -3550,7 +3577,9 @@ cvc::state_exec::value_t run_data_job_then(std::unique_ptr &args, - const std::string &who); + const std::string &who, PyObject *then = nullptr); +std::unique_ptr make_generator_job_from(const cvc::state_exec::value_t &gen, + bool collect = false); } // register_intrinsics(env, ctx): the C++ call, except that every function it adds keeps ctx (and diff --git a/bindings/pycvc/test_pycvc_state_exec_data.py b/bindings/pycvc/test_pycvc_state_exec_data.py index c8e964b68..a94c89f40 100644 --- a/bindings/pycvc/test_pycvc_state_exec_data.py +++ b/bindings/pycvc/test_pycvc_state_exec_data.py @@ -78,6 +78,34 @@ def apply_one(fn, arg): assert "py-outside" in str(e) +def test_call_function_with_a_continuation(): + # The #683 overload: `then` gets the call's value once the function has run, however the + # evaluator ran it (a lambda runs as frames, so `then` comes after them). + def apply_then(fn, arg): + return pycvc.call_function(fn, [arg], "py-then", lambda v: v + 1) + + env = _builtins() + pycvc.builtins.register_fn(env, "apply-then", apply_then) + s = pycvc.scheduler() + o = pycvc.execute_options() + o.env = env + pid = s.execute("(apply-then (lambda (x) (* x 2)) 20)", o) + assert s.run()[pid] == 41 + + +def test_generator_jobs_and_nesting_limits(): + ev = pycvc.stackless_evaluator(_builtins()) + gen = ev.evaluate_script( + "(generator (lambda () (begin (yield 1) (yield 2) (yield 3))))") + assert isinstance(gen, pycvc.value) and gen.type_name() == "generator" + assert pycvc.run_to_completion(pycvc.make_generator_job(gen)) == [1] # the next value + assert pycvc.run_to_completion(pycvc.make_generator_job(gen, True)) == [2, 3] # the rest + assert pycvc.run_to_completion(pycvc.make_generator_job(gen)) == [] # exhausted + _raises(RuntimeError, pycvc.make_generator_job, 5) + assert pycvc.kMaxNativeNesting == 256 + assert pycvc.kNativeStackReserve in (128 << 10, 32 << 10) # Windows, elsewhere + + # ── the heap ──────────────────────────────────────────────────────────── diff --git a/docs/PYCVC_STATE_EXEC_PARITY.md b/docs/PYCVC_STATE_EXEC_PARITY.md index bf38765f1..2b051ea3e 100644 --- a/docs/PYCVC_STATE_EXEC_PARITY.md +++ b/docs/PYCVC_STATE_EXEC_PARITY.md @@ -90,8 +90,8 @@ are listed too. Status after PR 4: | | `script_reader` | `pycvc.script_reader` | wrapped (PR 1). `forms()`, `source()` and `options()` return copies | | | `parser` (`read`, `parse_expr`) | none | exception: it holds a `string_view` of its input, which Python can't keep alive. `script_reader` owns its source and gives the same slice-by-slice reading | | data_job.h (#658) | `data_options`, `data_options_from_state(state&, id, base)` | `pycvc.data_options`, `pycvc.data_options_from_state(State, …)` | wrapped (#658) | -| | `data_job` (`step`, `result`, `nodes`), `make_equal_job`, `make_print_job`, `make_concat_job`, `make_copy_job`, `make_substitute_job`, `run_to_completion`, `run_data_job`, `call_function`, `is_compound`, `count_data_nodes`, `kDataNodeBytes` (#658, #665) | the same names under `pycvc.` | wrapped (PR 4). A factory hands Python an owned job, which `run_data_job` takes over. `who` (C++ keeps the pointer for its errors) is kept for good. `run_data_job`'s `then` is a Python callable. From a Python function a program calls, `run_data_job` and `call_function` work as for a C++ native: return what they return at once | -| | the call protocol (`data_job::call_request`, `take_call`, `pending_call`, `call_returned`, `own_inputs`), `detail::deferred_data_job`, `detail::data_step_scope` | none | exception: for C++ job implementations and the evaluator's own bookkeeping; Python can't subclass a data job | +| | `data_job` (`step`, `result`, `nodes`), `make_equal_job`, `make_print_job`, `make_concat_job`, `make_copy_job`, `make_substitute_job`, `run_to_completion`, `run_data_job`, `call_function` (with or without #683's `then`), `make_generator_job` (#683; the generator is a `pycvc.value`), `is_compound`, `count_data_nodes`, `kDataNodeBytes`, `kMaxNativeNesting`, `kNativeStackReserve` (#682) (#658, #665) | the same names under `pycvc.` | wrapped (PR 4). A factory hands Python an owned job, which `run_data_job` takes over. `who` (C++ keeps the pointer for its errors) is kept for good. `run_data_job`'s `then` is a Python callable. From a Python function a program calls, `run_data_job` and `call_function` work as for a C++ native: return what they return at once | +| | the call protocol (`data_job::call_request`, `take_call`, `pending_call`, `call_returned`, `own_inputs`), `detail::deferred_data_job`, `detail::data_step_scope`, `detail::native_nesting` (#682) | none | exception: for C++ job implementations and the evaluator's own bookkeeping; Python can't subclass a data job | | types.h | `symbol` | `pycvc.symbol` | wrapped (PR 1) | | | `value_t` | native types, or `pycvc.value` | wrapped (PR 1), see the table above | | | `to_string`, `values_equal` | `pycvc.to_string`, `pycvc.values_equal` | wrapped (PR 1) |