[transfix]

Follows #664.

Why

#664 removed the per-level recursion from most of cvc::state, but listed three walks it did not cover:

• ptree()/json()/save() and their loader (ptree(pt), json(j), restore());
• traverse();
• reset(true).

Each still used a native frame per tree level, so a deep tree overflows a small stack (wasm's 64 KiB). Two of them were also wrong in other ways:

• traverse() visited a node once per ancestor in the walk. It called traverse() on every descendant children() listed, and each of those did the same. A node d levels down a chain was visited 2^(d-1) times: a 20-deep chain made 1,048,576 visits for 21 nodes (12.6 s). state_cluster_shard::snapshot() walks with traverse(), so its snapshots listed (and encoded) entries that many times.
• reset(true) deadlocked against ordinary slots. It held every ancestor's lock while each descendant's touch signaled. A slot that looks its node up from the root takes those locks. The sync adapter's valueChanged slots do exactly that, so root("a").reset() with an adapter attached and children under a hangs on a non-recursive mutex.

What

• ptree(), json(), save(): one level. The document has an entry per node, keyed by its full path, in pre-order (children in name order). Each entry holds the node's value, or "" for a childless empty node below the start.

  • Those are exactly the leaves of the nested document these used to write, in the same order. The keys were already full paths, so nesting added nothing.

  • boost's ptree, write_json and read_json all recurse once per nesting level. Only making the walk iterative would still have left json() overflowing; the nested form could not be made deep-safe without a JSON codec of our own.

  • No consumer in the repo reads the nested shape:

    • pycvc state_json, ariadne save_state/restore_state and state_list feed it back to the loader;
    • cvc-cli state json prints it, and xmlrpc cvcstate_get_json returns it.

    That printed or returned output is the visible change: it is now one flat object.

  • Each node's value and children are read once under its own lock. Before, the walk held every ancestor's lock while listing the subtree below it.

• The loader walks every nesting level with a heap stack, so documents written in the old nested format still load (a test loads one).
• traverse(): each node once.
  • The order is pre-order: the start node, then each descendant whose full name matches re (all of them when it is empty).
  • A visited node's traverseEnter fires before func sees it, and its traverseExit fires after the nodes under it. With a pattern, the walk still goes under nodes that do not match, as before.
  • It walks held state_ptrs instead of finding each node again by name from the root. That lookup recreated nodes removed meanwhile.
  • A node's children are read after func has seen it, as children() used to be.
• reset(true): same order, no locks held while signaling.
  • A node is cleared before the nodes under it and touched after them, one child's whole subtree before the next child's. ResetClearsTopDownAndTouchesBottomUp pins that order and passes on state: walk the tree and its links without recursing once per level #664's code too.
  • Each node is cleared and its children snapshotted under its own lock, which is released before going down. Each level holds its node, so a touch's ancestors stay alive.
• reset(false) swaps the child map out under the lock and destroys it after the lock is released, so the subtree's destroyed() slots no longer run under the node's lock.

Measurements (MSVC 14.38 Release, Windows)

Deepest operation that passes, before (#664's code) → after. A 64 KiB or 256 KiB stack is a thread with that stack; 1 MiB is the main thread.

Operation	64 KiB	256 KiB	1 MiB
json() + save(), then json(j) + restore() into a fresh app (test capped at 1000)	41 → 1000 ✔	222 → 1000 ✔	938 → 1000 ✔
reset(true, false) of a chain	133 → 100000 ✔	693 → 100000 ✔	2911 → 100000 ✔
reset() with a listener on the root (capped at 3000)	127 → 3000 ✔	686 → 3000 ✔	2906 → 3000 ✔
traverse() of a chain (capped at 3000)	see below → 3000 ✔	→ 3000 ✔	→ 3000 ✔

• traverse() before never got deep enough to overflow, because its cost was exponential. Visits for a chain d deep, before: 1024 at d = 10 (19 ms), 65,536 at 16 (0.86 s), 1,048,576 at 20 (12.6 s). After: d + 1 visits, and 3000 deep takes 0.27 s on a 64 KiB stack.
• The JSON round trip also passes 3000 deep on 64 KiB. It takes about 13 s, which is why the test stops at 1000. A document keyed by full paths is quadratic in depth: at 3000, building the ptree took 47 ms, writing 36 ms, parsing 158 ms, and loading 2.6 s. Loading resolves each key from the root.
• ResetSignalsWithoutTheAncestorsLocks deadlocks on state: walk the tree and its links without recursing once per level #664's code. Its watchdog fires after 60 s. On this branch it passes in under 1 ms.

Tests

• New state_deep_tree_test cases (8; the file now has 18). The deep ones run on a 64 KiB thread:
  • a 1000-deep json()/save() → json(j)/restore() round trip;
  • the flat format, and loading a document in the old nested format;
  • traverse() order, enter/exit nesting and the regex filter;
  • traverse() 3000 deep, with visit and enter/exit counts;
  • reset(true, false) 100000 deep, and reset() 3000 deep with a root listener (leaf first, then upward);
  • the reset order;
  • reset's slots and reset(false)'s destroyed() slots taking the ancestors' locks.
• StateSnapshotProtocolTest.SnapshotListsEachNodeOnce: before, a.b.c was listed twice.
• CVC_STATE_DEEP_DEPTH / CVC_STATE_DEEP_STACK_KIB work as in state: walk the tree and its links without recursing once per level #664.
• All 174 test executables ran locally on this branch, rebased on master (3998 ctest cases). Failures:
  • image_test ReadsCmykInputAsRgb and cvc_cli_test Vol2Img/Img2Vol: ImageMagick has no decode delegate here, as on master.
  • StateClusterMembership.FailureDetectionDead and .HeartbeatRevivesSuspectPeer failed once while the pycvc scripts ran alongside. They passed in the run before the rebase and 5 of 5 times alone. These are heartbeat timing tests, and the membership code uses none of these walks.
• pycvc (PR CI doesn't build it): all 22 scripts, run from the build tree with the deps' DLL directory registered. 20 pass, including test_pycvc_state.py's state_json round trip. The other two:
  • test_pycvc.py fails on the .rawiv file lock that volume: close the file after a successful rawiv write #663 fixes.
  • test_pycvc_lod.py passes every check, then crashes at interpreter exit. state: walk the tree and its links without recursing once per level #664's build does the same (3 of 3 runs on each).
• clang-format 18 is clean, and check_test_targets.py is OK (174 executables).

Found, not fixed

• state_list::erase()/insert() copy an element's subtree with dst.ptree(src.ptree()). ptree()'s keys are full paths from the root, but ptree(pt) applies them relative to dst. So the copy lands under dst.<src's full path>, and dst's own children are not updated. This is the same before and after this PR. Only scalar list elements are tested today. The same asymmetry applies to xmlrpc cvcstate_set_json(path, …) and ariadne restore_state() on a node other than the root.
• Loading resolves every full-path key from the root, so it is quadratic in depth. That is fine for real trees. A resolver that reuses the previous key's prefix would make it linear.