state_exec: state-json and state-from-json, a slice per step

docs/STATE_EXEC_DEVELOPER_GUIDE.md

@@ -562,8 +562,19 @@ register_intrinsics(env, &ictx);
 ;; Store/retrieve binary data objects
 (state-data-set "myapp.blob" some-value)
 (state-data-get "myapp.blob")
+
+;; A subtree's values as JSON, and back: the document cvc::state::json() writes, keyed by paths
+;; relative to the node ("" for the node itself). Both run as sliced data steps.
+(state-json "myapp.config")   ;; => "{\n    \"timeout\": \"30\"\n}\n"
+(state-from-json "myapp.copy" (state-json "myapp.config"))
 ```
 
+`state-json` returns nil for a missing node and reads through a transparent link, as
+`state-get` does. `state-from-json` writes through a writable transparent link, as `state-set`
+does. It sets every non-empty string, number, `true`, `false` and `null` at its key, and returns
+nil. It checks the whole document before it sets anything, so a malformed document changes
+nothing. Its errors name the line, as in `state-from-json: line 3: expected ':'`.
+
 ### State Expiry
 
 ```lisp
@@ -1447,8 +1458,16 @@ process with it as the exit error; the sync scheduler's `run()` throws it:
 
 These functions do their work as a data job (`data_job.h`; "Deep data on the executor" in
 `docs/STATE_EXEC_GRAMMAR.md`): `string.split`, `string.join`, `string.replace`,
-`string.encode`, `bytes.slice`, `bytes.concat`, `bytes.decode` and every `collections.*`
-function.
+`string.encode`, `bytes.slice`, `bytes.concat`, `bytes.decode`, every `collections.*`
+function, and the intrinsics `state-json` and `state-from-json`.
+
+`state-json` and `state-from-json` run `cvc::state`'s own writer and reader
+(`cvc/state/state_json.h`), the code behind `json()`, `json(j)`, `save()` and `restore()`, so
+they produce and accept the same documents. The writer's unit is a node visited. The reader's
+is a value scanned, and it scans each value twice: once to check the document, once to set it.
+Both also count one unit per 256 bytes. The writer holds the nodes it has yet to visit, so a
+node another process removes between slices is still written, as it was when its parent was
+read. The document is held to the allocation limit as it grows.
 
 - **Slices.** A slice is `data.step_nodes` nodes: one per element visited, and one per 256
   bytes (`kDataNodeBytes`) scanned or copied. With `data.step_nodes` = N the call does one

docs/STATE_EXEC_GRAMMAR.md

@@ -226,8 +226,8 @@ then `state_exec.defaults.data.step_nodes`, then the scheduler's built-in defaul
 A node is an element visited: a list element or a dict entry walked, or an element freed. With
 `step_nodes` = N:
 
-- **Native calls.** `=`, `!=`, `str`, `str-concat`, `print` and `state-data-get` hand their walk
-  to the evaluator (`run_data_job`). It does one slice inside the call's step; if the walk isn't
+- **Native calls.** `=`, `!=`, `str`, `str-concat`, `print`, `state-data-get`, `state-json` and
+  `state-from-json` hand their walk to the evaluator (`run_data_job`). It does one slice inside the call's step; if the walk isn't
   done, the call's frame waits and the following steps are **data steps** that each run N more
   nodes. The step that finishes it completes the call. A walk under N nodes finishes in the
   call's own step, so ordinary programs take no extra steps.

inc/cvc/state/state.h

@@ -485,6 +485,12 @@ class state : public boost::enable_shared_from_this<state> {
   void ptree(const boost::property_tree::ptree &);
 
   // returns a json version of the property map
+  //
+  // The document is written and read by state_json_writer / state_json_reader
+  // (cvc/state/state_json.h), whose walks state_exec's state-json and state-from-json run a
+  // slice per scheduler step. The text is what write_json(ptree()) writes, and the reader keeps
+  // read_json()'s grammar, but its container stack is on the heap, so a deeply nested document
+  // costs no native stack. A malformed document sets nothing.
   std::string json();
 
   // sets this property tree based on a json
@@ -859,6 +865,8 @@ class state : public boost::enable_shared_from_this<state> {
   static state_ptr instancePtr(app &ctx);
 
 private:
+  friend class state_subtree_cursor; // snapshots a node's value and children under its lock
+
   state(const state &);
 };
 

inc/cvc/state/state_json.h

@@ -0,0 +1,151 @@
+/*
+  Copyright 2026 The University of Texas at Austin
+
+  This file is part of libcvc.
+
+  libcvc is free software; you can redistribute it and/or
+  modify it under the terms of the GNU Lesser General Public
+  License version 2.1 as published by the Free Software Foundation.
+*/
+
+// A state subtree's JSON document (state::json(), state::save()) and loading one
+// (state::json(j), state::restore()), as walks that can stop after any number of nodes and go
+// on later. cvc::state runs them to completion; state_exec's state-json and state-from-json run
+// them a slice per scheduler step. One implementation, so the two cannot drift.
+//
+// The document is one level: an entry per node, keyed by its path relative to the node it
+// describes ("" for that node), in pre-order with children in name order, holding the node's
+// value, or "" for a node below it with neither a value nor children. Values only, not data().
+// It is written exactly as boost::property_tree::write_json() writes that one-level tree (the
+// format json() had), and read as read_json() followed by ptree(pt) would read it.
+
+#ifndef CVC_STATE_STATE_JSON_H
+#define CVC_STATE_STATE_JSON_H
+
+#include <cstddef>
+#include <cvc/state/state.h>
+#include <string>
+#include <string_view>
+#include <utility>
+#include <vector>
+
+namespace cvc {
+
+// Pre-order over a subtree, a node per next(). A heap stack of levels, not a call per level.
+// Each node's value and children are snapshotted under its own lock, released before going
+// down, and the children are held by owning pointer, so a node removed from the tree between two
+// calls is still visited (as it was when its parent was snapshotted) and none dangles. Changes
+// made between calls show in the nodes visited after them.
+class state_subtree_cursor {
+public:
+  // Walk `start`'s subtree. The cursor keeps `start` alive while it walks.
+  explicit state_subtree_cursor(state &start);
+
+  enum class visit {
+    listed,   // the node belongs in the document: it has a value, or it is childless below start
+    unlisted, // it does not: an empty node with children, or an empty start
+    done      // no nodes are left
+  };
+
+  // Visit the next node: its path relative to start ("" for start) and its value.
+  visit next(std::string &path, std::string &value);
+
+  // Nodes visited so far.
+  std::size_t visited() const { return visited_; }
+
+private:
+  struct level {
+    std::vector<std::pair<state::state_ptr, std::string>> kids; // with their relative paths
+    std::size_t next = 0;
+  };
+  // Snapshot `n`'s value and children, whose paths are `rel`'s plus their names.
+  level open(state &n, const std::string &rel, std::string &value);
+
+  state::state_ptr keep_;
+  state *start_;
+  bool started_ = false;
+  std::vector<level> stack_;
+  std::size_t visited_ = 0;
+};
+
+// The document of a subtree, written a slice at a time.
+class state_json_writer {
+public:
+  explicit state_json_writer(state &start) : cursor_(start) {}
+
+  // Visit at most `nodes` more nodes (0: all that remain). True once the document is complete.
+  bool step(std::size_t nodes);
+
+  std::size_t nodes() const { return cursor_.visited(); }
+  // The text so far (all of it once step() has returned true).
+  const std::string &text() const { return out_; }
+  std::string take() { return std::move(out_); }
+
+private:
+  state_subtree_cursor cursor_;
+  std::string out_ = "{\n";
+  std::string path_, value_;
+  bool any_ = false, done_ = false;
+};
+
+// Load a document into a node, a slice at a time, as state::json(j) does. Every string, number,
+// true, false and null at any depth of the document is a value; one that is not empty is set
+// at its key relative to the target ("" inside an array: the target itself), in document order.
+// Numbers keep their text; true, false and null are those words. Objects and arrays only nest.
+// The whole document is checked first, so a malformed one sets nothing; errors are
+// boost::property_tree::json_parser_error, as read_json() throws them.
+class state_json_reader {
+public:
+  // `doc` is borrowed: it must outlive the reader, or be passed again to rebind().
+  state_json_reader(state &target, std::string_view doc, std::string filename = {});
+
+  // The same text at a new address (the reader keeps only offsets into it).
+  void rebind(std::string_view doc) { doc_ = doc; }
+  std::string_view doc() const { return doc_; }
+
+  // Scan at most `values` more values (0: all that remain); each is scanned twice, once to
+  // check and once to set. True once the document is loaded.
+  bool step(std::size_t values);
+
+  // Values scanned so far (both passes), and bytes.
+  std::size_t values() const { return values_; }
+  std::size_t bytes() const { return bytes_; }
+  // Values set so far.
+  std::size_t applied() const { return applied_; }
+
+private:
+  enum class want { start, value, key_or_end, value_or_end, key, comma_or_end, end, done };
+  struct frame {
+    bool array;
+    std::string key; // an object's current key
+  };
+
+  [[noreturn]] void fail(const char *msg) const;
+  bool at_end() const { return pos_ >= doc_.size(); }
+  char peek() const { return doc_[pos_]; }
+  void advance();
+  bool have(char c);
+  void expect(char c, const char *msg);
+  char need(const char *msg) const;
+  void skip_ws();
+  void scan_string(std::string &out);
+  void scan_escape(std::string &out);
+  unsigned scan_hex_quad();
+  void scan_scalar(std::string &out);
+  void emit(const std::string &text);
+  want after_value() const { return stack_.empty() ? want::end : want::comma_or_end; }
+
+  state::state_ptr keep_;
+  state *target_;
+  std::string_view doc_;
+  std::string filename_;
+  bool applying_ = false; // second pass
+  want want_ = want::start;
+  std::vector<frame> stack_;
+  std::size_t pos_ = 0, line_ = 1, scanned_ = 0;
+  std::size_t values_ = 0, bytes_ = 0, applied_ = 0;
+};
+
+} // namespace cvc
+
+#endif // CVC_STATE_STATE_JSON_H

src/cvc/CMakeLists.txt

@@ -76,6 +76,7 @@ set(INCLUDE_FILES
   ../../inc/cvc/utility/utility.h
   ../../inc/cvc/state/state_object.h
   ../../inc/cvc/state/state_list.h
+  ../../inc/cvc/state/state_json.h
   ../../inc/cvc/state/state_eviction_store.h
   ../../inc/cvc/state/state_memory_manager.h
   ../../inc/cvc/state/state_exec/types.h
@@ -200,6 +201,7 @@ set(SOURCE_FILES
   processing/smoothing.cpp
   volume/spider_io.cpp
   state/state.cpp
+  state/state_json.cpp
   state/state_list.cpp
   state/state_eviction_store.cpp
   state/state_memory_manager.cpp

src/cvc/ariadne/ariadne.cpp

@@ -421,9 +421,10 @@ class ReactiveEngine {
         "is-float", "is-string", "is-null", "is-list", "type-of", "not", "and", "or",
         // bounded compound construction (by-reference; not the doubling materializers) + access
         "list", "cons", "car", "cdr", "nth", "length", "dict", "get-attr",
-        // side-effect-free state readers (state-data-get deep-copies; no aliasing, no callables)
-        "state-get", "state-exists", "state-children", "state-data-get", "state-root-path",
-        "state-has-expiry", "state-is-expired",
+        // side-effect-free state readers (state-data-get deep-copies; no aliasing, no callables;
+        // state-json writes its document as sliced data steps, under the allocation limit)
+        "state-get", "state-exists", "state-children", "state-data-get", "state-json",
+        "state-root-path", "state-has-expiry", "state-is-expired",
         // the stdlib: the pure, bounded functions (a scalar, or a result no bigger than its
         // argument), and the materializers, which do their work as sliced data steps (each
         // eval slices, see data_step_nodes_) under the per-call allocation limit. A function

src/cvc/state/state.cpp

@@ -36,8 +36,11 @@
 #include <cvc/core/exception.h>
 #include <cvc/state/state.h>
 #include <cvc/state/state_cluster_shard.h>
+#include <cvc/state/state_json.h>
 #include <cvc/state/state_message.h>
 #include <cvc/utility/utility.h>
+#include <fstream>
+#include <iterator>
 #include <map>
 #include <memory>
 #include <set>
@@ -680,44 +683,16 @@ boost::property_tree::ptree state::ptree() {
   // holding the node's value -- or "" for a node other than this one with neither a value nor
   // children. For the root, relative paths are full names, as they always were. boost's ptree and
   // its JSON writer and parser recurse once per nesting level, so with one level, building this,
-  // writing it as JSON, reading it back, copying and destroying it don't recurse once per state
-  // level. A heap stack of levels; each node's value and children are snapshotted under its own
-  // lock, released before going down, and a child's path is its parent's, a separator and its
-  // name, carried down.
-  struct level {
-    std::vector<std::pair<state_ptr, std::string>> kids; // with their relative paths
-    std::size_t next = 0;
-  };
-  const char *const fn = BOOST_CURRENT_FUNCTION;
-  const auto open = [&pt, fn, this](state &n, const std::string &n_rel, bool top) {
-    level l;
-    std::string v;
-    {
-      boost::mutex::scoped_lock lock(n._mutex);
-      v = n._value;
-      l.kids.reserve(n._children.size());
-      for (const child_map::value_type &kv : n._children)
-        if (kv.second)
-          l.kids.emplace_back(kv.second, n_rel.empty() ? kv.first : n_rel + SEPARATOR + kv.first);
-    }
-    _ctx.log(6, boost::str(boost::format("%s :: %s = %s\n") % fn % n_rel % v));
-    if (!v.empty())
-      pt.push_back(property_tree::ptree::value_type(n_rel, property_tree::ptree(v)));
-    else if (!top && l.kids.empty())
-      pt.push_back(property_tree::ptree::value_type(n_rel, property_tree::ptree()));
-    return l;
-  };
-  std::vector<level> stack;
-  stack.push_back(open(*this, std::string(), true));
-  while (!stack.empty()) {
-    level &l = stack.back();
-    if (l.next == l.kids.size()) {
-      stack.pop_back();
-      continue;
-    }
-    std::pair<state_ptr, std::string> kid = std::move(l.kids[l.next++]);
-    boost::this_thread::interruption_point();
-    stack.push_back(open(*kid.first, kid.second, false)); // invalidates `l`
+  // copying and destroying it don't recurse once per state level. The walk is
+  // state_subtree_cursor's, which json() and state_exec's state-json share.
+  state_subtree_cursor cursor(*this);
+  std::string path, value;
+  for (;;) {
+    const state_subtree_cursor::visit v = cursor.next(path, value);
+    if (v == state_subtree_cursor::visit::done)
+      break;
+    if (v == state_subtree_cursor::visit::listed)
+      pt.push_back(property_tree::ptree::value_type(path, property_tree::ptree(value)));
   }
   return pt;
 }
@@ -772,7 +747,13 @@ void state::ptree(const boost::property_tree::ptree &pt) { apply_state_ptree(*th
 //  Returns a json string version of the property map.
 // ---- Change History ----
 // 01/12/2014 -- Joe R. -- Creation.
-std::string state::json() { return cvc::json(ptree()); }
+// The text write_json(ptree()) writes, from state_json_writer (state_exec's state-json runs the
+// same writer a slice at a time).
+std::string state::json() {
+  state_json_writer w(*this);
+  w.step(0);
+  return w.take();
+}
 
 // -----------
 // state::json
@@ -781,7 +762,13 @@ std::string state::json() { return cvc::json(ptree()); }
 //  Sets this state object and its children based on an incoming json.
 // ---- Change History ----
 // 01/13/2014 -- Joe R. -- Creation.
-void state::json(const std::string &j) { ptree(cvc::json(j)); }
+// What read_json() and ptree(pt) did, from state_json_reader (state_exec's state-from-json runs
+// the same reader a slice at a time): the document is checked whole first, so a malformed one
+// sets nothing, and its nesting costs no native stack.
+void state::json(const std::string &j) {
+  state_json_reader r(*this, j);
+  r.step(0);
+}
 
 // -----------
 // state::save
@@ -791,7 +778,19 @@ void state::json(const std::string &j) { ptree(cvc::json(j)); }
 // ---- Change History ----
 // 03/16/2012 -- Joe R. -- Creation.
 // 01/12/2014 -- Joe R. -- Forcing json.
-void state::save(const std::string &filename) { write_json(filename, ptree()); }
+// The file write_json(filename, ptree()) wrote: the same text, through a text-mode stream, and
+// the same errors.
+void state::save(const std::string &filename) {
+  using boost::property_tree::json_parser::json_parser_error;
+  state_json_writer w(*this);
+  w.step(0);
+  std::ofstream out(filename.c_str());
+  if (!out)
+    BOOST_PROPERTY_TREE_THROW(json_parser_error("cannot open file", filename, 0));
+  out << w.text() << std::flush;
+  if (!out.good())
+    BOOST_PROPERTY_TREE_THROW(json_parser_error("write error", filename, 0));
+}
 
 // --------------
 // state::restore
@@ -801,11 +800,16 @@ void state::save(const std::string &filename) { write_json(filename, ptree()); }
 // ---- Change History ----
 // 03/16/2012 -- Joe R. -- Creation.
 // 01/12/2014 -- Joe R. -- Forcing json.
+// read_json(filename, pt) then ptree(pt): a text-mode stream, the same errors, and the reader
+// json(j) uses.
 void state::restore(const std::string &filename) {
-  using namespace boost;
-  property_tree::ptree pt;
-  read_json(filename, pt);
-  ptree(pt);
+  using boost::property_tree::json_parser::json_parser_error;
+  std::ifstream in(filename.c_str());
+  if (!in)
+    BOOST_PROPERTY_TREE_THROW(json_parser_error("cannot open file", filename, 0));
+  const std::string doc((std::istreambuf_iterator<char>(in)), std::istreambuf_iterator<char>());
+  state_json_reader r(*this, doc, filename);
+  r.step(0);
 }
 
 // ---------------