diff --git a/bindings/pycvc/CMakeLists.txt b/bindings/pycvc/CMakeLists.txt index 6a2c364d5..396b2f413 100644 --- a/bindings/pycvc/CMakeLists.txt +++ b/bindings/pycvc/CMakeLists.txt @@ -271,6 +271,7 @@ if(CVC_BUILD_PYCVC_CORE) pycvc_exec:test_pycvc_exec.py pycvc_exec_async:test_pycvc_exec_async.py pycvc_exec_coro:test_pycvc_exec_coro.py + pycvc_state_exec:test_pycvc_state_exec.py pycvc_world_clock:test_pycvc_world_clock.py pycvc_world_units:test_pycvc_world_units.py pycvc_dataprep:test_pycvc_dataprep.py diff --git a/bindings/pycvc/README.md b/bindings/pycvc/README.md index 36e1ffc10..35c138f1b 100644 --- a/bindings/pycvc/README.md +++ b/bindings/pycvc/README.md @@ -12,20 +12,24 @@ Python package that imports `pycvc`. ## Status -- **v0 (this): `Geometry`** — build meshes/polylines incrementally or in - bulk, set per-vertex colors, compute normals, and read/write files - (`.off`, CVC-raw, …) via libcvc's `geometry_file_io`. -- Planned: `Volume` (cvc::volume + SDF), `State` (cvc::state tree), and - numpy fast-paths; then a `vr3` module (exposed by VolumeRover3's embedded - interpreter) that turns a `pycvc.Geometry` into a live scene node. +pycvc covers geometry and volumes (with numpy views), the state tree (`State`), +images, models, world units and clocks, LOD, state_exec (below), and, in the +`pycvc_gl` module, cvcGL scenes, Ariadne and Dear ImGui HUDs. Each area has a +contract test (`test_pycvc_*.py`). ## Design -`pycvc_geometry.h` is a SWIG-safe facade: it *forward-declares* -`cvc::geometry` behind a `shared_ptr`, so SWIG only ever parses -`std::string`/`std::vector`/primitive signatures — never libcvc's heavy -headers. Only the `.cpp` includes libcvc. C++ host apps reach the wrapped -`cvc::geometry` via `native()` (which is `%ignore`d on the Python side). +pycvc wraps the **real** libcvc types under their C++ names; there is no +parallel facade layer (`pycvc.i`). `pycvc.volume` is a `cvc::volume`, `pycvc.State` +is a `cvc::state`, `pycvc.parse` is `cvc::state_exec::parse`. The libcvc headers +are `%include`d and curated with `%ignore` for members that can't cross. A thin +`%extend` layer adds Python conveniences such as numpy views. + +Every constructor or operation that needs an app takes it explicitly +(`pycvc.volume(app)`, `pycvc.State.instance(app)`). There is no module-global +"current app". Where C++ takes a `cvc::state&`, Python passes a `pycvc.State`. +The goal is for the Python API to match the C++ API. For state_exec the audit +and plan are in `docs/PYCVC_STATE_EXEC_PARITY.md`. ## Build @@ -213,6 +217,23 @@ pycvc.data_options_from_state(root, "", ex.data_defaults()) # what Exec `Exec` starts from the same defaults as the app's exec scheduler (Ariadne), and the same `state_exec.defaults.parser.*` and `state_exec.defaults.data.*` keys -override both. A syntax error -raises a `RuntimeError` with the parse error's message. Contract test: -`test_pycvc_exec.py`. +override both. A syntax error raises `pycvc.parse_error`, a `RuntimeError` with +`.line` and `.col`. + +The rest of state_exec is the C++ API under the C++ names +(`docs/PYCVC_STATE_EXEC_PARITY.md` lists what is wrapped and what comes next): + +```python +from pycvc import symbol +pycvc.parse("(f 'x \"s\")") # [symbol('f'), [symbol('quote'), symbol('x')], 's'] +pycvc.parse_all("1 2") # [1, 2] +pycvc.to_string([symbol("a"), 1]) # '(a 1)' +pycvc.encode_value(root.child("snap"), {"k": [1, 2]}) +pycvc.decode_value(root.child("snap")) # {'k': [1, 2]} +``` + +Values convert to native Python types, with `pycvc.symbol` for symbols and +`pycvc.value` as an opaque handle for closures and other non-data values. Such +a handle converts back to the same value when passed into the DSL. Conversion +is strict: non-`str` dict keys, ints outside 64 bits and unsupported types +raise. Contract tests: `test_pycvc_exec.py`, `test_pycvc_state_exec.py`. diff --git a/bindings/pycvc/pycvc.i b/bindings/pycvc/pycvc.i index 9d8a1338b..bdd4098d5 100644 --- a/bindings/pycvc/pycvc.i +++ b/bindings/pycvc/pycvc.i @@ -1443,23 +1443,371 @@ class AsyncStateObserver(state_observer): self._workers = [] %} -// ── state_exec parser settings (docs/STATE_EXEC_GRAMMAR.md, "Parser settings") ── -// DIRECT WRAP, under the C++ names: the real cvc::state_exec::parse_options -// struct, parse_options_from_state (which takes a State, the tree root, as in -// C++: pycvc.State.instance(app)) and async_scheduler::host_parser_defaults(). -// Only these members are declared to SWIG; the rest of parser.h / -// async_scheduler.h (value_t, the parser, the scheduler itself) is not wrapped -// yet, so async_scheduler has no Python constructor. Exec's parser_defaults / -// set_parser_defaults are in pycvc_exec.h. -// The deep-data settings (data_options, data_options_from_state, "Deep data on the -// executor") are wrapped the same way. +// ── state_exec (docs/PYCVC_STATE_EXEC_PARITY.md) ───────────────────────── +// DIRECT WRAP of cvc::state_exec under the C++ names. Where C++ takes a +// cvc::state&, Python passes a State (State.instance(app) is the root). Only +// the declarations below are visible to SWIG; the parity doc lists what is +// wrapped, what comes next, and what is deliberately not wrapped (the +// coroutine API, the string_view-based parser class, ...). The deep-data +// settings (data_options, data_options_from_state, "Deep data on the +// executor") are wrapped the same way as the parser settings. %{ #include #include #include +#include +#include +#include "pycvc_value.h" +#include +#include +#include +#include +#include %} + +// ── values: the one value_t <-> Python conversion (pycvc_value.h) ────── +// Defined here because it creates and recognises the pycvc.symbol and +// pycvc.value proxies, which need the SWIG runtime; pycvc_exec.cpp calls it +// through pycvc_value.h. +%{ +namespace pycvc { +namespace { +using cvc::state_exec::value_t; +constexpr int kMaxValueDepth = 1000; + +swig_type_info *symbol_swig_type() { + static swig_type_info *t = SWIG_TypeQuery("cvc::state_exec::symbol *"); + return t; +} +swig_type_info *value_swig_type() { + static swig_type_info *t = SWIG_TypeQuery("cvc::state_exec::value_tag *"); + return t; +} + +bool too_deep(int depth) { + if (depth <= kMaxValueDepth) + return false; + PyErr_SetString(PyExc_ValueError, + "pycvc: value nests deeper than 1000 levels (or contains itself)"); + return true; +} + +PyObject *text_to_py(const std::string &s) { + return PyUnicode_DecodeUTF8(s.data(), static_cast(s.size()), "strict"); +} + +PyObject *to_py(const value_t &v, int depth) { + using namespace cvc::state_exec; + if (too_deep(depth)) + return nullptr; + if (v.is_nil()) + Py_RETURN_NONE; + if (const auto *b = std::get_if(&v.v)) + return PyBool_FromLong(*b); + if (const auto *i = std::get_if(&v.v)) + return PyLong_FromLongLong(static_cast(*i)); + if (const auto *d = std::get_if(&v.v)) + return PyFloat_FromDouble(*d); + if (const auto *s = std::get_if(&v.v)) + return text_to_py(*s); + if (const auto *by = std::get_if(&v.v)) + return PyBytes_FromStringAndSize(by->data.data(), static_cast(by->data.size())); + if (const auto *sym = std::get_if(&v.v)) + return SWIG_NewPointerObj(new symbol(*sym), symbol_swig_type(), SWIG_POINTER_OWN); + if (const auto *l = std::get_if(&v.v)) { + const Py_ssize_t n = *l ? static_cast((*l)->size()) : 0; + PyObject *out = PyList_New(n); + if (out == nullptr) + return nullptr; + for (Py_ssize_t k = 0; k < n; ++k) { + PyObject *item = to_py((**l)[static_cast(k)], depth + 1); + if (item == nullptr) { + Py_DECREF(out); + return nullptr; + } + PyList_SET_ITEM(out, k, item); // steals the new reference + } + return out; + } + if (const auto *dd = std::get_if(&v.v)) { + PyObject *out = PyDict_New(); + if (out == nullptr) + return nullptr; + if (*dd) { + for (const auto &kv : **dd) { + PyObject *key = text_to_py(kv.first); + PyObject *val = key ? to_py(kv.second, depth + 1) : nullptr; + const int rc = val ? PyDict_SetItem(out, key, val) : -1; + Py_XDECREF(key); + Py_XDECREF(val); + if (rc < 0) { + Py_DECREF(out); + return nullptr; + } + } + } + return out; + } + // closure / native_fn / data_object / generator: an opaque handle that converts back to the + // same value, so Python can hold it and pass it into the DSL again. + return SWIG_NewPointerObj(new value_t(v), value_swig_type(), SWIG_POINTER_OWN); +} + +bool from_py(PyObject *o, value_t &out, int depth) { + using namespace cvc::state_exec; + if (too_deep(depth)) + return false; + if (o == nullptr || o == Py_None) { + out = value_t(); + return true; + } + if (PyBool_Check(o)) { // before int: bool is an int subclass + out = value_t(o == Py_True); + return true; + } + if (PyLong_Check(o)) { + int overflow = 0; + const long long x = PyLong_AsLongLongAndOverflow(o, &overflow); + if (overflow != 0) { + PyErr_SetString(PyExc_OverflowError, "pycvc: int does not fit in 64 bits"); + return false; + } + if (x == -1 && PyErr_Occurred()) + return false; + out = value_t(static_cast(x)); + return true; + } + if (PyFloat_Check(o)) { + const double d = PyFloat_AsDouble(o); + if (d == -1.0 && PyErr_Occurred()) + return false; + out = value_t(d); + return true; + } + if (PyUnicode_Check(o)) { + Py_ssize_t n = 0; + const char *s = PyUnicode_AsUTF8AndSize(o, &n); // fails (and raises) on lone surrogates + if (s == nullptr) + return false; + out = value_t(std::string(s, static_cast(n))); + return true; + } + if (PyBytes_Check(o)) { + out = make_bytes(std::string(PyBytes_AS_STRING(o), static_cast(PyBytes_GET_SIZE(o)))); + return true; + } + if (PyByteArray_Check(o)) { + out = make_bytes( + std::string(PyByteArray_AS_STRING(o), static_cast(PyByteArray_GET_SIZE(o)))); + return true; + } + if (PyList_Check(o) || PyTuple_Check(o)) { + const bool tup = PyTuple_Check(o); + const Py_ssize_t n = tup ? PyTuple_GET_SIZE(o) : PyList_GET_SIZE(o); + std::vector elems; + elems.reserve(static_cast(n)); + for (Py_ssize_t k = 0; k < n; ++k) { + value_t e; + if (!from_py(tup ? PyTuple_GET_ITEM(o, k) : PyList_GET_ITEM(o, k), e, depth + 1)) + return false; + elems.push_back(std::move(e)); + } + out = make_list(std::move(elems)); + return true; + } + if (PyDict_Check(o)) { + std::vector> entries; + PyObject *k = nullptr, *v = nullptr; + Py_ssize_t pos = 0; + while (PyDict_Next(o, &pos, &k, &v)) { + if (!PyUnicode_Check(k)) { + PyErr_Format(PyExc_TypeError, "pycvc: dict keys must be str, not %.200s", + Py_TYPE(k)->tp_name); + return false; + } + Py_ssize_t n = 0; + const char *ks = PyUnicode_AsUTF8AndSize(k, &n); + if (ks == nullptr) + return false; + value_t e; + if (!from_py(v, e, depth + 1)) + return false; + entries.emplace_back(std::string(ks, static_cast(n)), std::move(e)); + } + out = make_dict(std::move(entries)); + return true; + } + void *p = nullptr; + if (SWIG_IsOK(SWIG_ConvertPtr(o, &p, symbol_swig_type(), 0)) && p != nullptr) { + out = value_t(*static_cast(p)); + return true; + } + if (SWIG_IsOK(SWIG_ConvertPtr(o, &p, value_swig_type(), 0)) && p != nullptr) { + out = *static_cast(p); + return true; + } + PyErr_Clear(); // a failed proxy lookup above may leave an attribute error behind + PyErr_Format(PyExc_TypeError, "pycvc: cannot convert %.200s to a state_exec value", + Py_TYPE(o)->tp_name); + return false; +} +} // namespace + +PyObject *value_to_py(const cvc::state_exec::value_t &v) { return to_py(v, 0); } + +PyObject *values_to_py_tuple(std::span values) { + PyObject *t = PyTuple_New(static_cast(values.size())); + if (t == nullptr) + return nullptr; + for (size_t i = 0; i < values.size(); ++i) { + PyObject *item = to_py(values[i], 0); + if (item == nullptr) { + Py_DECREF(t); + return nullptr; + } + PyTuple_SET_ITEM(t, static_cast(i), item); + } + return t; +} + +bool py_to_value(PyObject *o, cvc::state_exec::value_t &out) { return from_py(o, out, 0); } +} // namespace pycvc +%} + +%typemap(in) cvc::state_exec::value_tag (cvc::state_exec::value_tag temp) %{ + if (!pycvc::py_to_value($input, temp)) + SWIG_fail; + $1 = temp; +%} +%typemap(in) const cvc::state_exec::value_tag & (cvc::state_exec::value_tag temp) %{ + if (!pycvc::py_to_value($input, temp)) + SWIG_fail; + $1 = &temp; +%} +%typemap(typecheck, precedence=SWIG_TYPECHECK_POINTER) cvc::state_exec::value_tag, + const cvc::state_exec::value_tag & "$1 = 1;" +%typemap(out) cvc::state_exec::value_tag %{ + $result = pycvc::value_to_py(static_cast($1)); + if (!$result) + SWIG_fail; +%} +%typemap(out) std::vector %{ + { // a block, so the wrapper's later `goto fail` doesn't jump over this initialization + const std::vector &pycvc_vals = $1; + $result = PyList_New(static_cast(pycvc_vals.size())); + if (!$result) + SWIG_fail; + for (size_t i = 0; i < pycvc_vals.size(); ++i) { + PyObject *item = pycvc::value_to_py(pycvc_vals[i]); + if (!item) { + Py_DECREF($result); + $result = nullptr; + SWIG_fail; + } + PyList_SET_ITEM($result, static_cast(i), item); + } + } +%} + +// ── parse_error: pycvc.parse_error, a RuntimeError with .line and .col ── +%{ +static PyObject *pycvc_parse_error_type = nullptr; + +static void pycvc_raise_parse_error(const cvc::state_exec::parse_error &e) { + PyObject *inst = PyObject_CallFunction(pycvc_parse_error_type, "s", e.what()); + if (inst == nullptr) + return; + PyObject *line = PyLong_FromSize_t(e.line()); + PyObject *col = PyLong_FromSize_t(e.col()); + if (line != nullptr) + PyObject_SetAttrString(inst, "line", line); + if (col != nullptr) + PyObject_SetAttrString(inst, "col", col); + Py_XDECREF(line); + Py_XDECREF(col); + PyErr_SetObject(pycvc_parse_error_type, inst); + Py_DECREF(inst); +} +%} +%init %{ + pycvc_parse_error_type = PyErr_NewException("pycvc.parse_error", PyExc_RuntimeError, NULL); + if (pycvc_parse_error_type != NULL) + PyDict_SetItemString(d, "parse_error", pycvc_parse_error_type); +%} +%pythoncode %{ +parse_error = _pycvc.parse_error +%} + +// For the state_exec declarations (and Exec, below): the default mapping, with +// parse_error first so it raises pycvc.parse_error. Scoped to this section, so +// the modules that %import pycvc.i keep PYCVC_DEFAULT_EXCEPTION. +%exception { + try { + $action + } catch (const cvc::state_exec::parse_error &e) { + pycvc_raise_parse_error(e); + SWIG_fail; + } catch (const cvc::exception &e) { + SWIG_exception(SWIG_RuntimeError, e.what()); + } catch (const std::exception &e) { + SWIG_exception(SWIG_RuntimeError, e.what()); + } catch (...) { + SWIG_exception(SWIG_RuntimeError, "pycvc: C++ exception (see libcvc)"); + } +} + namespace cvc { namespace state_exec { + +// symbol: the real struct (name), so a symbol stays a symbol in Python. +struct symbol { + std::string name; +}; +%extend symbol { + symbol(const std::string &name) { return new cvc::state_exec::symbol{name}; } + %pythoncode %{ + def __eq__(self, other): + return isinstance(other, symbol) and self.name == other.name + def __ne__(self, other): + return not self.__eq__(other) + def __hash__(self): + return hash(("pycvc.symbol", self.name)) + def __repr__(self): + return "symbol(%r)" % (self.name,) + %} +} + +// value: the real value_t, as an opaque handle. Plain data converts to native +// Python types; a closure, native fn, data object or generator arrives as one of +// these and converts back to the same value. Not constructible from Python. +%rename(value) value_tag; +%nodefaultctor value_tag; +struct value_tag { + bool is_nil() const; + bool is_truthy() const; + std::string type_name() const; +}; +%extend value_tag { + %pythoncode %{ + def __repr__(self): + return "" % (self.type_name(), to_string(self)) + def __eq__(self, other): + try: + return values_equal(self, other) + except (TypeError, ValueError, OverflowError): + return NotImplemented + def __ne__(self, other): + r = self.__eq__(other) + return r if r is NotImplemented else not r + __hash__ = None + %} +} +typedef value_tag value_t; + +std::string to_string(const value_t &val); +bool values_equal(const value_t &a, const value_t &b); + +// Parser (docs/STATE_EXEC_GRAMMAR.md). struct parse_options { size_t max_depth; // lists and quotes that may nest (default 256) size_t step_tokens; // 0: parse at submit; N: read N tokens per process step @@ -1473,6 +1821,26 @@ parse_options parse_options_from_state(cvc::state &root, data_options data_options_from_state(cvc::state &root, const std::string &scheduler_id = std::string(), const data_options &base = data_options()); +value_t parse(const std::string &input, const parse_options &opts = parse_options()); +std::vector parse_all(const std::string &input, + const parse_options &opts = parse_options()); + +// script_reader owns its source, so it is the Python way to read a script a +// slice at a time (parser views a string_view and isn't wrapped). forms(), +// source() and options() return copies here; C++ returns references. +class script_reader { +public: + script_reader(std::string source, const parse_options &opts); + bool step(size_t budget); + std::vector forms(); + std::string source() const; + parse_options options() const; +}; + +// Snapshot codec for one value: write it under a State node, read it back. +void encode_value(cvc::state &node, const value_t &val); +value_t decode_value(cvc::state &node); + %nodefaultctor async_scheduler; %nodefaultdtor async_scheduler; class async_scheduler { @@ -1489,3 +1857,5 @@ public: // SWIG's raw-passthrough type (the callable crosses uninspected; the .cpp // INCREFs and manages its lifetime). %include "pycvc_exec.h" + +PYCVC_DEFAULT_EXCEPTION diff --git a/bindings/pycvc/pycvc_exec.cpp b/bindings/pycvc/pycvc_exec.cpp index 474842084..e0d4bf0fc 100644 --- a/bindings/pycvc/pycvc_exec.cpp +++ b/bindings/pycvc/pycvc_exec.cpp @@ -6,6 +6,8 @@ #include "pycvc_exec.h" +#include "pycvc_value.h" // the shared value_t <-> Python conversion + #include #include #include @@ -30,101 +32,6 @@ namespace pycvc { namespace { using namespace cvc::state_exec; -// ── value_t <-> Python marshaling ─────────────────────────────────────── -// Convert a DSL value to a NEW Python reference. Scalars + list/dict cross; -// opaque DSL types (symbol/closure/native_fn/data/generator) degrade to their -// printed form so a Python fn still receives something sensible. -PyObject *value_to_py(const value_t &v) { - if (v.is_nil()) - Py_RETURN_NONE; - if (const auto *b = std::get_if(&v.v)) - return PyBool_FromLong(*b); - if (const auto *i = std::get_if(&v.v)) - return PyLong_FromLongLong(static_cast(*i)); - if (const auto *d = std::get_if(&v.v)) - return PyFloat_FromDouble(*d); - if (const auto *s = std::get_if(&v.v)) - return PyUnicode_FromStringAndSize(s->data(), static_cast(s->size())); - if (const auto *by = std::get_if(&v.v)) - return PyBytes_FromStringAndSize(by->data.data(), static_cast(by->data.size())); - if (const auto *l = std::get_if(&v.v)) { - const auto &vec = **l; - PyObject *out = PyList_New(static_cast(vec.size())); - for (Py_ssize_t k = 0; k < static_cast(vec.size()); ++k) - PyList_SET_ITEM(out, k, value_to_py(vec[k])); // steals the new ref - return out; - } - if (const auto *dd = std::get_if(&v.v)) { - const auto &pairs = **dd; - PyObject *out = PyDict_New(); - for (const auto &kv : pairs) { - PyObject *pv = value_to_py(kv.second); - PyDict_SetItemString(out, kv.first.c_str(), pv); - Py_DECREF(pv); - } - return out; - } - // Fallback: the DSL's printed form. - std::string s = to_string(v); - return PyUnicode_FromStringAndSize(s.data(), static_cast(s.size())); -} - -// Convert a Python object (borrowed) to a DSL value. bool BEFORE int (bool is a -// subclass of int in Python). dict order is preserved. -value_t py_to_value(PyObject *o) { - if (o == nullptr || o == Py_None) - return value_t(std::monostate{}); - if (PyBool_Check(o)) - return value_t(o == Py_True); - if (PyLong_Check(o)) - return value_t(static_cast(PyLong_AsLongLong(o))); - if (PyFloat_Check(o)) - return value_t(PyFloat_AsDouble(o)); - if (PyUnicode_Check(o)) { - Py_ssize_t n = 0; - const char *s = PyUnicode_AsUTF8AndSize(o, &n); - return value_t(std::string(s ? s : "", s ? static_cast(n) : 0)); - } - // Python `bytes` -> a first-class binary value (bytes_value), the counterpart - // of the text `str` path above. Lets a Python DSL fn receive/return a raw - // binary msg-send/msg-recv payload without lossy UTF-8 coercion. - if (PyBytes_Check(o)) { - char *buf = nullptr; - Py_ssize_t n = 0; - PyBytes_AsStringAndSize(o, &buf, &n); - return make_bytes(std::string(buf ? buf : "", buf ? static_cast(n) : 0)); - } - if (PyList_Check(o) || PyTuple_Check(o)) { - bool tup = PyTuple_Check(o); - Py_ssize_t n = tup ? PyTuple_Size(o) : PyList_Size(o); - std::vector elems; - elems.reserve(static_cast(n)); - for (Py_ssize_t k = 0; k < n; ++k) - elems.push_back(py_to_value(tup ? PyTuple_GetItem(o, k) : PyList_GetItem(o, k))); - return value_t(std::make_shared>(std::move(elems))); - } - if (PyDict_Check(o)) { - auto pairs = std::make_shared>>(); - PyObject *k = nullptr, *v = nullptr; - Py_ssize_t pos = 0; - while (PyDict_Next(o, &pos, &k, &v)) { - PyObject *ks = PyObject_Str(k); - Py_ssize_t n = 0; - const char *s = ks ? PyUnicode_AsUTF8AndSize(ks, &n) : ""; - pairs->emplace_back(std::string(s ? s : "", s ? static_cast(n) : 0), py_to_value(v)); - Py_XDECREF(ks); - } - return value_t(dict_ptr(std::move(pairs))); - } - // Anything else: its str(). - PyObject *s = PyObject_Str(o); - Py_ssize_t n = 0; - const char *cs = s ? PyUnicode_AsUTF8AndSize(s, &n) : ""; - value_t out(std::string(cs ? cs : "", cs ? static_cast(n) : 0)); - Py_XDECREF(s); - return out; -} - std::string fetch_py_error() { if (!PyErr_Occurred()) return "unknown error"; @@ -189,18 +96,23 @@ std::shared_ptr own_callable(PyObject *callable) { value_t call_python(const std::shared_ptr &holder, std::span args, const std::string &name) { PyGILState_STATE gil = PyGILState_Ensure(); - PyObject *pyargs = PyTuple_New(static_cast(args.size())); - for (Py_ssize_t i = 0; i < static_cast(args.size()); ++i) - PyTuple_SET_ITEM(pyargs, i, value_to_py(args[i])); // steals the new ref - PyObject *res = PyObject_CallObject(holder.get(), pyargs); - Py_DECREF(pyargs); + PyObject *pyargs = values_to_py_tuple(args); // nullptr (error set) if an arg can't convert + PyObject *res = pyargs ? PyObject_CallObject(holder.get(), pyargs) : nullptr; + Py_XDECREF(pyargs); if (res == nullptr) { std::string msg = fetch_py_error(); PyGILState_Release(gil); throw std::runtime_error("python DSL fn '" + name + "': " + msg); } - value_t out = py_to_value(res); + value_t out; + const bool converted = py_to_value(res, out); // strict: e.g. a set or non-str dict key fails Py_DECREF(res); + if (!converted) { + std::string msg = fetch_py_error(); + PyGILState_Release(gil); + throw std::runtime_error("python DSL fn '" + name + + "' returned an unconvertible value: " + msg); + } PyGILState_Release(gil); return out; } @@ -287,7 +199,10 @@ void drive_coros(PyObject *loop, std::vector> } else if (exc == Py_None) { Py_DECREF(exc); PyObject *res = PyObject_CallMethod(task, "result", nullptr); - payload = res ? py_to_value(res) : async_error_dict("python coro: " + fetch_py_error()); + value_t converted; + payload = res && py_to_value(res, converted) + ? converted + : async_error_dict("python coro: " + fetch_py_error()); Py_XDECREF(res); } else { PyObject *s = PyObject_Str(exc); @@ -503,11 +418,9 @@ void Exec::register_async_coro(const std::string &name, PyObject *coro_fn) { const std::string done = resolve_channel_key(root, chan); // The verb runs during sync_run (GIL released by the pump) — re-acquire it to touch Python. PyGILState_STATE gil = PyGILState_Ensure(); - PyObject *pyargs = PyTuple_New(static_cast(args.size())); - for (Py_ssize_t i = 0; i < static_cast(args.size()); ++i) - PyTuple_SET_ITEM(pyargs, i, value_to_py(args[i])); - PyObject *coro = PyObject_CallObject(holder.get(), pyargs); - Py_DECREF(pyargs); + PyObject *pyargs = values_to_py_tuple(args); // nullptr (error set) on an unconvertible arg + PyObject *coro = pyargs ? PyObject_CallObject(holder.get(), pyargs) : nullptr; + Py_XDECREF(pyargs); PyObject *task = nullptr; if (coro != nullptr) { task = PyObject_CallMethod(impl->loop, "create_task", "O", coro); @@ -577,6 +490,7 @@ std::string Exec::run(const std::string &src) { auto idle_deadline = clock::now() + std::chrono::duration(kIdleTimeoutSecs); std::optional result; std::optional error; // set -> throw AFTER the pool is quiesced + bool killed = false; // the scheduler killed the process (error has its exit error) try { while (true) { sched.drain_ingress(); // deliver any worker post_message FIRST, so a parked receiver readies @@ -608,6 +522,7 @@ std::string Exec::run(const std::string &src) { error = (ex && !ex->empty()) ? *ex : ("pycvc.Exec.run: program killed (pid " + std::to_string(pid) + ")"); + killed = true; break; } // Fast-path only on OUR OWN pid's runnability — has_runnable() scans siblings too, so gating @@ -642,6 +557,12 @@ std::string Exec::run(const std::string &src) { // to Python, so a worker can never post to / be destroyed against a torn-down scheduler. inflight.wait_idle(); sched.reap(pid); // drop the finished/killed process from the private scheduler (no leak) + if (killed) { + // A script read inside its process (parser.step_tokens) reports a syntax error only as the + // exit error's text. Re-parse it, so a syntax error raises pycvc.parse_error with its position + // either way, as it does when the script is parsed at submit. + (void)parse_all(src, parse_options_from_state(*impl_->root, {}, sched.parser_defaults())); + } if (error) throw std::runtime_error(*error); if (!result.has_value()) diff --git a/bindings/pycvc/pycvc_value.h b/bindings/pycvc/pycvc_value.h new file mode 100644 index 000000000..c9f4bf28a --- /dev/null +++ b/bindings/pycvc/pycvc_value.h @@ -0,0 +1,38 @@ +// pycvc_value.h — the one conversion between state_exec values (cvc::state_exec::value_t) and +// Python, used by every pycvc API that takes or returns a value (pycvc.parse, encode_value, +// Exec's registered functions, ...). docs/PYCVC_STATE_EXEC_PARITY.md has the full table. +// +// value_t -> Python (and back): +// - nil, bool, int64, double -> None, bool, int, float. +// - string -> str. The text must be valid UTF-8; invalid text raises UnicodeDecodeError. +// - bytes_value -> bytes (bytes or bytearray coming in). +// - list -> list (list or tuple coming in). +// - dict -> dict (keys must be str coming in; others raise TypeError). +// - symbol -> pycvc.symbol. +// - closure, native_fn, data_object, generator -> pycvc.value, an opaque handle that converts +// back to the same value, so it can be passed into the DSL again. +// +// Strict: an int outside 64 bits raises OverflowError, any other Python type raises TypeError, and +// a value nested deeper than 1000 levels (or a cyclic one) raises ValueError. Defined in the SWIG +// wrapper (pycvc.i), which owns the pycvc.symbol / pycvc.value types. Call with the GIL held. +#pragma once + +#include +#include + +struct _object; +typedef _object PyObject; + +namespace pycvc { + +/// A new reference, or nullptr with a Python exception set. +PyObject *value_to_py(const cvc::state_exec::value_t &v); + +/// A new tuple of the converted values (a DSL call's arguments), or nullptr with a Python +/// exception set. +PyObject *values_to_py_tuple(std::span values); + +/// Convert `o` into `out`. Returns false, with a Python exception set, when it can't. +bool py_to_value(PyObject *o, cvc::state_exec::value_t &out); + +} // namespace pycvc diff --git a/bindings/pycvc/test_pycvc_state_exec.py b/bindings/pycvc/test_pycvc_state_exec.py new file mode 100644 index 000000000..4c2da3dc6 --- /dev/null +++ b/bindings/pycvc/test_pycvc_state_exec.py @@ -0,0 +1,183 @@ +"""state_exec parity, part 1: parsing, values, parse_error and the snapshot codec. + +pycvc wraps cvc::state_exec under the C++ names; docs/PYCVC_STATE_EXEC_PARITY.md has +the approach and the full C++ vs Python inventory. Values convert to native Python +types, with pycvc.symbol for symbols and the opaque pycvc.value for closures and +other non-data values. +""" + +import pycvc +from pycvc import symbol + + +def _raises(exc, fn, *args): + try: + fn(*args) + except exc as e: + return e + raise AssertionError(f"{fn.__name__}{args!r} did not raise {exc.__name__}") + + +# ── parse / parse_all ─────────────────────────────────────────────────── + + +def test_parse_returns_python_values(): + assert pycvc.parse("42") == 42 + assert pycvc.parse("-2.5") == -2.5 + assert pycvc.parse('"café"') == "café" + assert pycvc.parse("#t") is True + assert pycvc.parse("nil") is None + assert pycvc.parse("") is None # nil, as in C++ + assert pycvc.parse('b"\\x00\\xff"') == b"\x00\xff" + assert pycvc.parse('(define x "s")') == [symbol("define"), symbol("x"), "s"] + assert pycvc.parse("'x") == [symbol("quote"), symbol("x")] + + +def test_parse_all_returns_every_form(): + assert pycvc.parse_all("1 (2 3) ; c\n x") == [1, [2, 3], symbol("x")] + assert pycvc.parse_all(" ; nothing\n") == [] + + +def test_parse_options_apply(): + o = pycvc.parse_options() + o.max_depth = 2 + assert pycvc.parse("((1))", o) == [[1]] + _raises(pycvc.parse_error, pycvc.parse, "(((1)))", o) + + +# ── parse_error ───────────────────────────────────────────────────────── + + +def test_parse_error_carries_its_position(): + o = pycvc.parse_options() + o.max_depth = 2 + e = _raises(pycvc.parse_error, pycvc.parse, "(a\n ((b)))", o) + assert isinstance(e, RuntimeError) # existing `except RuntimeError` still works + assert (e.line, e.col) == (2, 4) + assert "nesting deeper than 2" in str(e) + e = _raises(pycvc.parse_error, pycvc.parse_all, '(a "unterminated') + assert "unterminated string" in str(e) + + +def test_exec_run_raises_parse_error_either_way(): + ex = pycvc.Exec(pycvc.make_app()) + # Exec reads a script inside its process by default (step_tokens 1024)... + e = _raises(pycvc.parse_error, ex.run, "(begin\n (set x") + assert "unclosed parenthesis" in str(e) and e.line == 2 + # ...or parses it at submit (step_tokens 0): the same error, at the same place. + o = pycvc.parse_options() + o.step_tokens = 0 + ex.set_parser_defaults(o) + e2 = _raises(pycvc.parse_error, ex.run, "(begin\n (set x") + assert (str(e2), e2.line, e2.col) == (str(e), e.line, e.col) + + +# ── symbol ────────────────────────────────────────────────────────────── + + +def test_symbol_is_the_cpp_struct(): + s = symbol("x") + assert s.name == "x" + assert s == symbol("x") and s != symbol("y") + assert s != "x" # a symbol is not a string + assert hash(s) == hash(symbol("x")) + assert repr(s) == "symbol('x')" + assert {symbol("a"): 1}[symbol("a")] == 1 + + +# ── to_string / values_equal ──────────────────────────────────────────── + + +def test_to_string_and_values_equal(): + assert pycvc.to_string([symbol("a"), 1, "s", None]) == '(a 1 "s" nil)' + assert pycvc.to_string(b"\x00A") == 'b"\\x00A"' + assert pycvc.to_string((1, 2)) == "(1 2)" # a tuple converts as a list + assert pycvc.to_string(bytearray(b"ab")) == 'b"ab"' + assert pycvc.values_equal([1, {"k": "v"}], [1, {"k": "v"}]) + assert not pycvc.values_equal([1], [2]) + + +# ── strict conversion ─────────────────────────────────────────────────── + + +def test_conversion_is_strict(): + _raises(TypeError, pycvc.to_string, {1: 2}) # dict keys must be str + _raises(OverflowError, pycvc.to_string, 2**70) + _raises(TypeError, pycvc.to_string, {1, 2}) # a set has no state_exec equivalent + deep = [] + for _ in range(1100): + deep = [deep] + _raises(ValueError, pycvc.to_string, deep) + loop = [] + loop.append(loop) + _raises(ValueError, pycvc.to_string, loop) + + +def test_registered_fns_get_symbols_and_strict_returns(): + ex = pycvc.Exec(pycvc.make_app()) + seen = [] + ex.register_fn("see", lambda x: seen.append(x) or x) + assert ex.run("(see 'abc)") == "abc" + assert seen[0] == symbol("abc") # was the plain string "abc" + ex.register_fn("bad", lambda: {1, 2}) + e = _raises(RuntimeError, ex.run, "(bad)") + assert "cannot convert set" in str(e), str(e) + + +# ── opaque values ─────────────────────────────────────────────────────── + + +def test_closures_cross_as_opaque_values_and_come_back(): + ex = pycvc.Exec(pycvc.make_app()) + kept = [] + ex.register_fn("keep", lambda f: kept.append(f) or 1) + ex.register_fn("give", lambda: kept[0]) + ex.run("(keep (lambda (x) (+ x 1)))") + f = kept[0] + assert isinstance(f, pycvc.value) # was the text "" + assert f.type_name() == "closure" + assert not f.is_nil() and f.is_truthy() + assert "closure" in repr(f) + assert ex.run("(begin (set g (give)) (g 41))") == "42" + + +# ── snapshot codec ────────────────────────────────────────────────────── + + +def test_encode_and_decode_value_on_a_state_node(): + app = pycvc.make_app() + node = pycvc.State.instance(app).child("snap.v") + original = { + "name": "café", + "items": [1, 2.5, None, True, symbol("s")], + "blob": b"\x00\x01\xff", + } + pycvc.encode_value(node, original) + assert pycvc.decode_value(node) == original + + +# ── script_reader ─────────────────────────────────────────────────────── + + +def test_script_reader_reads_in_slices(): + r = pycvc.script_reader('(a 1) (b "two")', pycvc.parse_options()) + steps = 1 + while not r.step(1): + steps += 1 + assert r.forms() == [[symbol("a"), 1], [symbol("b"), "two"]] + assert steps > 3 + assert r.source() == '(a 1) (b "two")' + assert r.options().max_depth == 256 + + +def test_script_reader_raises_parse_error(): + r = pycvc.script_reader("(a", pycvc.parse_options()) + _raises(pycvc.parse_error, r.step, 100) + + +if __name__ == "__main__": + tests = [v for k, v in sorted(globals().items()) if k.startswith("test_") and callable(v)] + for t in tests: + t() + print(" ok:", t.__name__) + print("pycvc state_exec parity tests: OK") diff --git a/docs/PYCVC_STATE_EXEC_PARITY.md b/docs/PYCVC_STATE_EXEC_PARITY.md new file mode 100644 index 000000000..6e9ce0813 --- /dev/null +++ b/docs/PYCVC_STATE_EXEC_PARITY.md @@ -0,0 +1,107 @@ +# pycvc ↔ C++ API parity for state_exec + +Goal: pycvc's state_exec API is the C++ API, under the same names, so Python code reads like the +C++ it drives. pycvc already wraps libcvc this way: `pycvc.i` wraps the real types and adds no +facade layer. This doc records the audit of `inc/cvc/state/state_exec/*.h` against pycvc, the +approach agreed for closing the gap, and the plan. + +## Approach + +1. **Real types and functions, C++ names.** SWIG wraps the actual declarations: + `cvc::state_exec::parse` is `pycvc.parse`, `symbol` is `pycvc.symbol`, and so on. Members + that can't cross are listed below as exceptions, with the reason. +2. **Tree arguments are `State`s.** Where C++ takes a `cvc::state&`, Python passes a + `pycvc.State` (`pycvc.State.instance(app)` is the root, `root.child("a.b")` a node). The + older `state_set(app, …)` functions stay as a convenience layer. +3. **Values convert to native Python types**, with two small types for what Python lacks + (`bindings/pycvc/pycvc_value.h`): + + | `value_t` | Python | + |---|---| + | nil, bool, int64, double | `None`, `bool`, `int`, `float` | + | string | `str`. The text must be valid UTF-8, or `UnicodeDecodeError` is raised | + | `bytes_value` | `bytes`; `bytes` or `bytearray` coming in | + | list | `list`; a `list` or `tuple` coming in | + | dict | `dict`; keys must be `str` coming in | + | symbol | `pycvc.symbol`: the real struct, with `.name`, equality and hashing | + | closure, native_fn, data_object, generator | `pycvc.value`: the real `value_t` as an opaque handle that converts back to the same value | + + Conversion is strict: + - an int outside 64 bits raises `OverflowError`; + - a non-`str` dict key or any other Python type raises `TypeError`; + - a value nested deeper than 1000 levels, or one that contains itself, raises `ValueError`. + + Before this, symbols became plain strings, closures became the text `""`, and any + object became its `str()`. The old converter could also pass a null object into a list when a + string wasn't valid UTF-8, and it left an `OverflowError` pending on a big int. +4. **Errors keep their type.** `parse_error` is `pycvc.parse_error`, a `RuntimeError` subclass + with `.line` and `.col`. Other C++ exceptions stay `RuntimeError`. +5. **`Exec` stays** as the Python-integration layer on top of `async_scheduler`. It handles the + GIL around Python callbacks, an asyncio loop for coroutines, and compute-pool offload. It uses + the same value conversion as everything else. `Exec.run` raises `pycvc.parse_error` for a + syntax error whether the script was parsed at submit or read inside the process. + +## Inventory + +The audit (2026-10-17) found 97 public items in the 20 headers. Counted individually, with the 38 +`builtin_*` functions and the 7 `MIME_*` constants, it is 140. Status after PR 1: + +| Header | Item | Python | Status | +|---|---|---|---| +| parser.h | `parse_options` | `pycvc.parse_options` | wrapped (#653) | +| | `parse_options_from_state(state&, id, base)` | `pycvc.parse_options_from_state(State, …)` | wrapped (#653) | +| | `parse`, `parse_all` | `pycvc.parse`, `pycvc.parse_all` | wrapped (PR 1) | +| | `parse_error` (line, col) | `pycvc.parse_error` | wrapped (PR 1) | +| | `script_reader` | `pycvc.script_reader` | wrapped (PR 1). `forms()`, `source()` and `options()` return copies | +| | `parser` (`read`, `parse_expr`) | none | exception: it holds a `string_view` of its input, which Python can't keep alive. `script_reader` owns its source and gives the same slice-by-slice reading | +| types.h | `symbol` | `pycvc.symbol` | wrapped (PR 1) | +| | `value_t` | native types, or `pycvc.value` | wrapped (PR 1), see the table above | +| | `to_string`, `values_equal` | `pycvc.to_string`, `pycvc.values_equal` | wrapped (PR 1) | +| | `make_list`, `make_dict`, `make_bytes`, `nil_value`, `true_value`, `false_value` | `list`, `dict`, `bytes`, `None`, `True`, `False` | exception: the native Python values are the equivalents | +| | `environment` | | PR 2 | +| | `closure`, `data_object`, `generator` | `pycvc.value` (opaque) | wrapped as opaque handles; no member access | +| | `time_budget`, `eval_deadline_guard`, `eval_deadline_expired`, `thread_cpu_seconds` | none | exception: thread-local evaluator internals | +| state_value_codec.h | `encode_value`, `decode_value` | `pycvc.encode_value(State, v)`, `pycvc.decode_value(State)` | wrapped (PR 1) | +| | `encode_environment`/`decode_environment`, `encode_evaluator_state`/`decode_evaluator_state` | | PR 3, with the environment and evaluator types | +| scheduler_base.h | `execute_options`, `process_info`, `scheduler_stats`, `scheduling_policy` | | PR 2 | +| scheduler.h | `scheduler` (about 45 members) | | PR 2 | +| async_scheduler.h | `async_scheduler` (about 55 members) | `pycvc.async_scheduler.host_parser_defaults()` only | PR 2. Coroutine `step()`/`run()` are an exception: Python gets the existing `sync_step()`/`sync_run()` | +| process.h | `process_status`, `to_string(process_status)`, `parse_process_status` | | PR 2 | +| | `process` (about 35 fields), `make_process` | | PR 2, read-only where safe | +| builtins.h | `builtins::make_default_environment`, `builtins::register_fn` | `Exec.register_fn` | PR 2. `register_fn` takes a Python callable | +| | 38 `builtin_*` functions | none | exception: DSL builtins, called from DSL source | +| | `percent_encode`, `percent_decode` | | PR 2 | +| stackless_evaluator.h, evaluator.h | `stackless_evaluator`, `evaluator`, `evaluator_state`, `evaluation_stats`, the evaluation exceptions | | PR 3 | +| async_evaluator.h, async_stackless_evaluator.h | the async evaluators | none | exception: coroutine API | +| resource_policy.h | `resource_policy`, `process_limits`, `validate_limits` | | PR 3 | +| intrinsics.h | `register_intrinsics`, `intrinsics_context`, `apply_chroot`, `channel_policy` | `Exec` uses them internally | PR 3, with the intrinsics context's raw-pointer fields as `State`/scheduler references | +| | `document_scope`, `park_on_channel`, `make_future`, `future_channel_of` | none | exception: Ariadne/runtime internals (templates, raw process pointers) | +| stdlib.h | `stdlib_registry` | | after the stdlib is wired into the DSL (separate work: today only tests import it) | +| exec_coordinator.h | `exec_coordinator` and its result types | none | deferred: it needs the cluster shard, membership and message-bus objects, which pycvc doesn't wrap | +| memory_tracker.h | `memory_tracker` | none | exception: per-node byte accounting, internal to the schedulers | +| generator.h, task.h | `generator`, `generator_next`, `task<>` | none | exception: coroutine and evaluator internals; a generator crosses as a `pycvc.value` | +| utf8.h | `decode`, `count`, `byte_offset`, `encode`, `is_valid` | none | exception: Python's `str` already counts and indexes code points | + +## Plan + +Each PR is stacked on the previous one and includes pytest checks against the C++ behaviour. + +1. **PR 1 (this):** values, parsing and the value codec. + - `pycvc_value.h`: the strict shared converter, used by `Exec` too. + - `pycvc.symbol` and `pycvc.value`. + - `parse`, `parse_all`, `parse_error` with position, `script_reader`, `to_string`, + `values_equal`, `encode_value`, `decode_value`. + - `bindings/pycvc/test_pycvc_state_exec.py`. + - The pycvc README's outdated Status and Design sections are rewritten. +2. **PR 2:** `environment` and `builtins`; `execute_options`, `process_info`, `scheduler_stats` and + the enums; `scheduler` and `async_scheduler` as real classes, with their non-coroutine + methods. `Exec` exposes its scheduler. +3. **PR 3:** the evaluators and `evaluator_state`, the environment and evaluator-state codec, + `resource_policy`, and `register_intrinsics` / `intrinsics_context`. + +Known gaps after PR 1: +- Invalid UTF-8 in a DSL string raises when it crosses to Python, but no Python test produces such + a string. Python can't create one, and the stdlib's `bytes.decode` isn't reachable from DSL code + yet. +- `Exec.run` finds a syntax error in a script read inside its process by parsing it again after + the process is killed. That costs one extra parse, and only on failure.