pycvc: state_exec parity part 1 (values, parsing, parse_error, value codec) by transfix · Pull Request #661 · transfix/libcvc

transfix · 2026-10-17T20:56:48Z

Targets master. #653 and #658 have merged, and master is merged in (41411dc, keeping #658's data_options wrap), so the diff here is the parity work only. #671 (part 2) and #675 (part 3) are stacked on this PR.

Part 1 of bringing pycvc's state_exec API to the C++ surface, under the C++ names. docs/PYCVC_STATE_EXEC_PARITY.md has three parts:

the audit of all 20 state_exec headers: 97 public items, of which 3 were wrapped before this;
the agreed approach;
the PR plan, with every exception and its reason.

Approach (agreed)

Real types and functions, C++ names.
Tree arguments are pycvc.State. Where C++ takes cvc::state&, Python passes a pycvc.State; state_exec: read submitted scripts inside the process, a slice per step #653 already moved parse_options_from_state to this.
Values convert to native Python types, plus pycvc.symbol and an opaque pycvc.value. Conversion is strict.
parse_error keeps its type and position.
Exec stays as the Python-integration layer.

This PR

One strict value conversion (pycvc_value.h, defined in the SWIG wrapper), used by every API and by Exec:

`value_t`	Python
nil, bool, int64, double	`None`, `bool`, `int`, `float`
string	`str` (must be valid UTF-8)
bytes	`bytes` (`bytes` or `bytearray` in)
list	`list` (`list` or `tuple` in)
dict	`dict` (`str` keys in)
symbol	`pycvc.symbol`: the real struct, with `.name`, equality and hashing
closure, native fn, data object, generator	`pycvc.value`: the real `value_t`, opaque, and converts back unchanged

Strict:
- a non-str dict key or an unsupported type raises TypeError;
- an int beyond 64 bits raises OverflowError;
- nesting past 1000 levels, or a cycle, raises ValueError.
Replaces the lossy converter. Before, symbols became plain strings, closures the text "<closure>", and anything else str(obj).
Fixes two bugs in the old converter: it could put a null item into a list on invalid UTF-8, and it left an OverflowError pending on a big int.

New, under the C++ names:
- pycvc.parse and pycvc.parse_all;
- pycvc.parse_error: a RuntimeError subclass with .line and .col;
- pycvc.script_reader;
- pycvc.to_string and pycvc.values_equal;
- pycvc.encode_value(State, v) and pycvc.decode_value(State), the value snapshot codec.
Exec.run raises pycvc.parse_error for a syntax error whether the script was parsed at submit or read inside its process. In the second case it re-parses the killed process's script to recover the position, so it costs one extra parse, and only on failure.
parser isn't wrapped: it holds a string_view of its input, which Python can't keep alive. script_reader owns its source and gives the same slice-by-slice reading.
The pycvc README's Status and Design sections described a v0 facade. They now describe the direct-wrap design, and the state_exec section shows the new calls.

Behaviour changes for `Exec` users

A registered Python function now receives a symbol as pycvc.symbol (it was a str), and a closure as pycvc.value (it was "<closure>").
Returning an unsupported object, such as a set or a dict with non-str keys, is now a DSL error naming the problem. It used to be str(obj).

Tests

bindings/pycvc/test_pycvc_state_exec.py (13 tests, registered with CTest) covers each piece against the C++ behaviour.

Local results (Windows MSVC 14.38 Release, SWIG 4.4.1, Python 3.11, cvcpkg prefix): the new file passes, and so do pycvc's exec, async, coroutine, Ariadne, state, core, app, async-observer, threading, integration and proxy-hook suites. That's 87 tests, with one exception: test_pycvc.py::test_volume_rawiv_roundtrip fails with a Windows temp-file lock, and it fails the same way against the unmodified main-checkout build.

pycvc_gl: PR CI (ci.yml) doesn't build pycvc at all; only publish-cvcpkg.yml does, on version tags or a manual dispatch. So I built it locally on this branch (2a697c0).

Build: cvcGL, pycvc and pycvc_gl, with SDL and ImGui, VTK 9.5 from the cvcpkg prefix. The VTK-Python bridge is off (-DCVC_PYCVCGL_VTK_BRIDGE=OFF, since vtk-python isn't in the local prefix); that bridge is unrelated to this change.
SWIG: no warnings from the new state_exec section in either module.
Tests: all 12 pycvc_gl scripts pass. Some checks skip themselves for lack of an environment:
- test_pycvc_ari has no GL context here, so it skips;
- test_pycvc_vtk skips its bridge checks;
- test_pycvc_gl_imgui and test_pycvc_gl_viewport skip their offscreen-GL checks.
test_pycvc_state_exec, test_pycvc_exec and test_pycvc_state also pass on that build.

Next: PR 2 (environment and builtins, execute_options, process_info, the schedulers as real classes), then PR 3 (the evaluators, resource policy and intrinsics).

… P0) Declare the state_exec s-expression grammar in docs/STATE_EXEC_GRAMMAR.md (W3C EBNF, Unicode source rules, literal tables, and a table of every intentional change from the hand-written parser, tagged by phase P2-P4). The parser itself is unchanged; later phases build it from this grammar. state_exec_grammar_test tags each case with the phase it holds from: landed cases must hold, pending cases must still differ, the doc's EBNF must be closed, and every production needs a case. state_exec_corpus_test pins how the 597 DSL scripts the repo carries parse today (state_exec_corpus/, regenerated with gen_state_exec_corpus.py and CVC_UPDATE_GOLDENS=1). Also correct parser.h (no quasiquote) and the developer guide (false and #f read as nil, bytes literals exist).

Spec only: parse_options::max_depth on parse()/parse_all()/parser and execute_options::max_parse_depth for submitted scripts, both from P2. Records why the limit exists (the reader recurses on the native stack; the evaluators are stackless) and the corpus's real depths (max 13, median 2).

Spec only. max_depth resolves from state_exec.schedulers.<id>.parser.max_depth, then state_exec.defaults.parser.max_depth, then 256, matching scheduler::load_settings(); parse_options remains for callers without a tree. Drops the execute_options::max_parse_depth idea so the tree is the single source.

…grammar P1b) The parser now stops at parser.max_depth nested lists and quotes (default 256) with a parse_error, instead of recursing until the native stack overflows. Like the scheduler's other settings, the limit lives on the state tree: state_exec.schedulers.<id>.parser.max_depth, then state_exec.defaults.parser.max_depth, then 256; a missing, malformed or zero value is skipped. - parser: parse_options (max_depth) on parse()/parse_all()/parser, a depth guard on lists and quotes, and parse_options_from_state(). - scheduler: load_settings() resolves parser.* and publishes the effective value under state_exec.schedulers.<id>.parser.max_depth; execute() parses with it. - async_scheduler (the app's exec scheduler) reads the defaults key on every submit; Ariadne predicates read it from the app tree. - stackless evaluators: create_state(script, env, parse_options). The grammar test moves from a numeric kLandedPhase to named phases; the 257-deep case is P1b and holds now. Tests cover the default, a custom depth, quotes, 100000 open parens, the error position, tree resolution, bad values, and both schedulers.

…e read) Lists and quotes were parsed by native recursion, so input depth was bounded by the thread's stack. The parser now keeps open lists and quotes on a heap stack of frames and reads in a loop: native stack depth is constant however deeply the input nests (a test reads a million open parens and reports the unclosed list instead of crashing). The loop is a resumable read(out, budget): it reads up to `budget` tokens and returns datum / end / more, carrying open frames between calls, so a scheduler can parse a script a slice at a time. parse() and parse_all() call it with no budget. Output is unchanged, quirks included (a stray top-level `)` still reads as a symbol until P2): the conformance cases, the 597-script corpus and the evaluator suites all pass. parser.max_depth stays on the state tree; it now bounds parsed data for the code that still walks values recursively (list destruction, the snapshot codec, to_string) rather than the parser's own stack. parse_fib20_expr: about 3.4 us vs 3.1 us for the recursive parser (median of 6, MSVC Release, loaded machine).

The stackless parser's state is an input position plus a heap stack of open frames, so it can pause anywhere. New setting parser.step_tokens (on the state tree, resolved like parser.max_depth; default 0) lets a scheduler parse a submitted script inside the new process instead of at submit: each scheduler step reads N tokens, and the step that reaches the end sets up evaluation exactly as parsing up front would have. - Reading steps are process steps: they count against max_steps and max_time and interleave with other processes. - A syntax error surfaces like a runtime error: the async scheduler kills the process with it as the exit error; the sync scheduler throws it from step()/run(). With step_tokens = 0, execute() still throws. - Signals and watch events wait until the script is read (their handlers run on the evaluation stack, which doesn't exist yet). - The snapshot codec stores a mid-read process's source and settings and re-reads it after a restore. Adds parse_options::step_tokens, script_reader (owns its source and drives parser::read() a budget at a time) and evaluator_state::reading; stackless_evaluator::step() reads until the script is read. Tests cover the reader, the evaluator, the codec and both schedulers (results, syntax errors, max_steps, deferred signals).

…ens) app::exec_scheduler(), which runs Ariadne's actions and residents, now defaults parser.step_tokens to 1024, so it parses submitted scripts inside their processes. async_scheduler gains set_parser_defaults(): the built-in values under its watch root's state_exec.defaults.parser.* keys. A plain async_scheduler (pycvc's Exec) and the sync scheduler stay at 0, so they still report syntax errors at submit. Why 1024: every script the repo carries reads in one slice (the largest is 159 tokens, .ari scripts at most 24), so a normal action pays one extra step of its 5000; a slice costs about 80 us at the parser's measured ~80 ns/token, so a pathologically large script is spread over steps and frames instead of stalling a drain. Ariadne now says why a process died: a killed action's warning carries its exit error, and the per-frame sweep reports a killed resident (a syntax error in its body, a runtime error, its per-step cap) once, instead of letting it vanish silently.

…uler pycvc's Exec runs programs on a private async_scheduler that already watches the app's state tree, but its built-in parser defaults were plain parse_options{} while app::exec_scheduler() read scripts in 1024-token steps. Both now take async_scheduler::host_parser_defaults(), one definition (with the reasoning for 1024) for every libcvc host, and the app tree's state_exec.defaults.parser.* keys override both. Exec.run() still raises a RuntimeError with the parse error's message: from the killed process's exit error when the script is read in steps, or from execute() when the tree sets step_tokens to 0. Tests: test_pycvc_exec.py checks the parse error's message and that max_depth / step_tokens on the app tree configure Exec; the integration test pins the app scheduler to host_parser_defaults().

pycvc's state_exec surface was only Exec; the parser settings these PRs add had no Python counterpart, so the defaults weren't even observable. In pycvc's direct-wrap style (the real C++ types, no facade): - pycvc.parse_options: the real cvc::state_exec::parse_options struct (max_depth, step_tokens), C++ defaults included. - pycvc.async_scheduler.host_parser_defaults(): the real static, declared to SWIG on its own (async_scheduler isn't wrapped yet, so no constructor). - pycvc.parse_options_from_state(app, scheduler_id="", base=...): the C++ resolver, taking the app where C++ takes the root (pycvc's convention). - Exec.parser_defaults() / Exec.set_parser_defaults(): the same calls as the async_scheduler Exec wraps. test_pycvc_exec.py covers each against the C++ behaviour; the README gains a state_exec section. Built locally (SWIG 4.4.1, Python 3.11): exec, exec_async, exec_coro and ari tests, 26 passed.

Wrap the real cvc::state_exec::parse_options_from_state(state&, id, base) instead of a pycvc adapter that took the app: Python passes the tree root, pycvc.State.instance(app), exactly as C++ passes cvc::state&. This is the first decision of the pycvc/C++ parity approach (C++ cvc::state& parameters take a pycvc.State in Python).

…codec) Start bringing pycvc's state_exec API to the C++ surface under the C++ names (docs/PYCVC_STATE_EXEC_PARITY.md: the audit of all 20 state_exec headers, the agreed approach, and the PR plan). - One strict value_t <-> Python conversion (pycvc_value.h), used by every API and by Exec: native types for data, pycvc.symbol (the real struct) for symbols, and pycvc.value (the real value_t, opaque) for closures, native fns, data objects and generators, which convert back unchanged. Non-str dict keys and unsupported types raise TypeError, ints beyond 64 bits OverflowError, and nesting past 1000 levels or a cycle ValueError. This replaces the lossy converter (symbols became str, closures "<closure>", anything else str(obj)), which could also put a null item in a list on invalid UTF-8 and leave an OverflowError pending. - pycvc.parse / parse_all, pycvc.parse_error (a RuntimeError with .line and .col), pycvc.script_reader, to_string, values_equal, and the value codec encode_value / decode_value on a State node. - Exec.run raises pycvc.parse_error for a syntax error whether the script was parsed at submit or read inside the process (it re-parses a killed process's script to recover the position). - The pycvc README's Status/Design sections described a v0 facade; they now describe the direct-wrap design. test_pycvc_state_exec.py (13 tests) covers each against the C++ behaviour. Locally: those plus pycvc's exec, async, coro, ari, state and core suites pass (87), except test_volume_rawiv_roundtrip, which fails the same way on the unmodified build (a Windows temp-file lock).

…'s branch)

… data_options wrap)

…rap #658's data settings and budget fields on the schedulers

…o feat/pycvc-state-exec-parity-2

…-state-exec-parity-3

transfix added 11 commits October 16, 2026 23:58

This was referenced Oct 17, 2026

state_exec: read submitted scripts inside the process, a slice per step #653

Merged

pycvc: state_exec parity part 2 (environments, builtins, schedulers) #671

Merged

Merge master into feat/pycvc-state-exec-parity (#653 landed)

cd292e0

transfix changed the base branch from feat/state-exec-sliced-read to master October 18, 2026 01:39

Rerun CI on master (the merge push ran while the PR still targeted #653…

0c24a14

…'s branch)

transfix mentioned this pull request Oct 18, 2026

pycvc: state_exec parity part 3 (evaluators, processes, policy, intrinsics) #675

Merged

Merge master into feat/pycvc-state-exec-parity (#658 landed: keep its…

41411dc

… data_options wrap)

transfix added a commit that referenced this pull request Oct 18, 2026

Merge #661 (with master: #658) into feat/pycvc-state-exec-parity-2; w…

4f05226

…rap #658's data settings and budget fields on the schedulers

transfix mentioned this pull request Oct 18, 2026

state_exec: parser.max_depth 100000; the last recursive walks over parsed data #678

Merged

transfix merged commit 50c6d02 into master Oct 18, 2026
16 checks passed

transfix added a commit that referenced this pull request Oct 18, 2026

Merge master (#661 squash-merged as 50c6d02, plus #665/#667/#663) int…

8b18056

…o feat/pycvc-state-exec-parity-2

transfix added a commit that referenced this pull request Oct 18, 2026

Merge #671 (now on master, after #661's squash-merge) into feat/pycvc…

76e8c60

…-state-exec-parity-3

transfix mentioned this pull request Oct 18, 2026

pycvc: state_exec parity part 4 (data jobs, heap, stdlib registry, any-depth values) #679

Merged

Navigation Menu

Repository navigation

pycvc: state_exec parity part 1 (values, parsing, parse_error, value codec) - #661

pycvc: state_exec parity part 1 (values, parsing, parse_error, value codec)#661
transfix merged 14 commits into
masterfrom
feat/pycvc-state-exec-parity

transfix commented Oct 17, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

transfix commented Oct 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Approach (agreed)

This PR

Behaviour changes for Exec users

Tests

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

transfix commented Oct 17, 2026 •

edited

Loading

Behaviour changes for `Exec` users