pycvc: state_exec parity part 4 (data jobs, heap, stdlib registry, any-depth values)

[transfix]

Stacked on #675. The base is feat/pycvc-state-exec-parity-3; the diff here is part 4 only.

Part 4 of bringing pycvc's state_exec API to the C++ surface, under the C++ names. It wraps what #658, #665 and #667 added and stdlib_registry, and removes the value conversion's depth limit. After it, docs/PYCVC_STATE_EXEC_PARITY.md's inventory has every public item wrapped or listed as an exception with its reason.

The value conversion takes any depth

PR 1's converter recursed and stopped at 1000 levels with ValueError. Now it walks with an explicit stack, as the parser and the evaluator do. That matters because parser.max_depth's default is going to 100000; that change is the deep-data session's, agreed with it.

• Depth: a value nested any depth crosses both ways. A test sends a 200000-deep list from Python to C++ and back.
• Sharing: a list or dict shared within a value converts to one object, both ways. Sharing survives the trip, and a heavily shared structure converts in time proportional to its size (before, it was exponential).
• Cycles: a DSL list that holds itself comes out as a Python list that holds itself. A Python value that holds itself still raises ValueError, because only the evaluator's own containers are tracked for cycle collection.

New, under the C++ names

• Deep-data jobs (state_exec: deep-data work a slice per step, and the budget by kind #658, state_exec: the stdlib materializers run as sliced data steps, and predicates get them #665):
  • data_job (step, result, nodes), the make_*_job factories, run_to_completion and run_data_job (then is a Python callable), plus call_function, is_compound, count_data_nodes and kDataNodeBytes.
  • A factory hands Python an owned job (std::unique_ptr), which run_data_job takes over.
  • who is kept for good, because C++ holds the pointer for errors raised after the call.
  • From a Python function a program calls, call_function works as for a C++ native: it calls a native function directly, and a lambda runs as the process's own frames. So a Python function can call DSL lambdas; the tests do it on both schedulers.
• The heap (state_exec: max_memory covers the measured heap; collect reference cycles #667):
  • heap_options and heap_options_from_state (limits.heap_check_steps), measure_heap_share, make_heap_share_job and heap_remeasure_bytes;
  • the per-thread cycle collector: collect_cycles (with cycle_stats), cycle_collection_due, note_process_exit and tracked_objects;
  • the heap fields on process and process_info, process.heap_measure_due() and release_state();
  • evaluator_state.heap_measured (read-only), stackless_evaluator.global_env() and eval_alloc_note.
• stdlib_registry:
  • standard() returns a copy, so Python can't add to the registry every environment shares.
  • shared_scope() gives the frozen scope.
  • register_function takes a Python callable or a pycvc.value native.
  • list_functions, list_modules and import_module.
  • lookup_qualified raises KeyError where C++ returns nullptr.

Not wrapped, with reasons in the doc

• The data-job call protocol (call_request, take_call, pending_call, call_returned, own_inputs) and detail::: for C++ job implementations.
• heap_after_step: the schedulers' own per-step accounting.
• track_captured_env / track_mutated: a value from Python is a fresh copy, so tracking it would mean nothing.
• eval_alloc_guard::allocated(): the guard isn't wrapped.

Build note

SWIG passes by-value arguments with SWIG_STD_MOVE, which expands to std::move only when __cplusplus says C++11. MSVC reports 199711L unless built with /Zc:__cplusplus, so there the wrapper copied, and a std::unique_ptr can't be copied. pycvc.i now defines it as std::move for the wrapper functions, which is what GCC and Clang already do.

Tests

bindings/pycvc/test_pycvc_state_exec_data.py has 10 tests:

• a job run a slice at a time, and each factory;
• run_data_job outside an evaluator, and the job handed over;
• a Python function calling a native function and a lambda through call_function on both schedulers, and the error outside a running program;
• heap settings and fields; a process with a memory budget is measured;
• cycles collected on the thread that made them;
• global_env();
• the standard registry and one of your own.

test_pycvc_state_exec.py gains test_conversion_takes_any_depth_and_keeps_sharing.

Local results (Windows MSVC 14.38 Release, SWIG 4.4.1, Python 3.11, cvcpkg prefix), on master as of #677 plus #661's squash:

• all non-GL pycvc suites: 297 passed, 0 failed. The rawiv round-trip now passes, with volume: close the file after a successful rawiv write #663 on master.
• no SWIG warnings from pycvc.i;
• clang-format 18 clean, and ruff (E, F, W) clean.

Not run on this PR: pycvc_gl. Five of its scripts fail on master itself; see #675 for the control build. A separate fix is proposed.