Skip to content

Bound Julia INTERTHREAD subscriptions; declare Julia and Python interthread in interface.yml and goby interface - #488

Merged
tsaubergine merged 4 commits into
3.0from
claude/eloquent-mendel-qtcwli
Nov 20, 2026
Merged

tsaubergine merged 4 commits into
3.0from
claude/eloquent-mendel-qtcwli

Conversation

@tsaubergine

@tsaubergine tsaubergine commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Resolves #485 and #484.

Julia INTERTHREAD subscriptions get a bounded queue each, with an overflow policy, and publishers no longer wait with the registry lock held. interface.yml can declare interthread for Julia and Python, and can name the task or thread that makes each outer-layer call. The binding enforces both at run time, and goby interface draws each task or thread with its calls and validates its subscriptions.

As agreed while planning, two parts of the issues are not in this PR:

Changes

Julia delivery (#485)

  • Per-subscription queues. Goby.subscribe(...; capacity = 50, overflow = :block) on Goby.INTERTHREAD. overflow is one of:
    • :block: wait for room.
    • :drop_oldest: replace the oldest queued message.
    • :drop_newest: discard the new message.
    • :error: deliver to the other subscribers, then throw Goby.InterthreadOverflowError.
  • No waiting under the registry lock. A publisher offers the message to every subscriber, then waits only on full :block ones. A full subscription now delays only its own group's publishers; before, one full mailbox stalled every INTERTHREAD call in the process.
  • Self-deadlock is an error. Publishing to your own full :block subscription throws Goby.InterthreadDeadlockError instead of hanging.
  • Fair service. Each task has one inbox, holding its mailbox (loop ticks, C++ traffic) and its subscription queues, served in turn so a busy group can't starve a quiet one.
  • Goby.interthread_stats() reports group, task, capacity, policy, and messages queued, delivered and dropped. The first drop on a subscription logs a warning.
  • New src/share/Goby.jl/src/Interthread.jl and InterthreadContract.jl; GobyMultiThread.jl moves onto the inboxes.

interthread: in interface.yml (#484)

  • New keys on interthread entries:
    • thread: a Julia task module's name, or a Python Thread subclass's or application class's name.
    • scheme: NATIVE: any value of the bound language.
    • capacity and overflow: subscriptions, Julia only. The Python generator refuses them.
  • Schema, shared-format README and conformance corpus updated.
  • Enforced contract. Both generators put the declaration in the generated module (Goby.interthread_contract(...) in Julia, goby.interthread_contract([...]) in Python). Once declared, an undeclared group, a message of another type, or a call from another thread raises InterfaceContractError. Type names are resolved at startup, and a declared subscription that nothing made is reported.
  • Manifest. Both generators write <target>_julia_interthread.yml or <target>_python_interthread.yml. goby_add_julia_app() and goby_add_python_app() record it as the target's GOBY_INTERTHREAD_MANIFEST property. They also now call generate_interfaces() when the project exports interface files, as goby_add_app() does.
  • GobyJulia.cmake and GobyPython.cmake set GOBY_JULIA_HAS_INTERTHREAD_CONTRACT and GOBY_PYTHON_HAS_INTERTHREAD_CONTRACT, so downstream projects can tell the feature is there.

thread: on interprocess and intermodule entries

The generated C++ makes every outer-layer call on the tasks' or threads' behalf, so goby_clang_tool used to draw them all on the C++ class. In the Julia multithread example, that made an application publishing and subscribing to one group draw a loop on itself.

  • An interprocess or intermodule entry may now give thread. Several threads may declare the same call; without thread, any thread may make it.
  • The binding allows the call from that thread only. Generated as Goby.outer_layer_contract(...) and goby.outer_layer_contract([...]), for the calls that give a thread. In Julia the check runs where the call is made, before a task hands it to the C++ task, which is skipped since it only republishes. Calls are matched on the message name without its scope, as the C++ glue matches them.
  • The generated C++ handles each call once, however many threads declare it, with the strongest necessity any of them gives it.
  • goby_clang_tool draws the call on that thread. The manifest carries the entries as outer_threads, and -gen draws each matching glue call on the thread named. A call is matched by its group's runtime string or its C++ expression, the name interface.yml uses. The latter needed the group variable's qualified name, which -gen now keeps for this.

Interface tools (#484)

  • goby_clang_tool -gen -interthread-manifest <file> checks the manifest, adds its interthread section to <target>_interface.yml, and applies outer_threads. goby_export_interface() passes it from the target property.
  • goby interface reads julia_interthread and python_interthread: each task or thread becomes a thread of the application, so visualize (Graphviz and D2) draws it and validate checks its required subscriptions.
    • A Julia entry connects by group alone, as Goby.jl delivers.
    • A Python entry connects by group and type, or by group alone when it subscribes to object.
  • list and explain show a Julia subscription's capacity and overflow, via a new delivery field on InterfaceEndpoint.
  • New src/middleware/tool/interface/language_interthread.h, shared by both tools.

Behaviour change

  • The Julia generator no longer writes C++ for interthread:. Previously it generated calls to interthread(), which SingleThreadApplication, the base of every Julia example, doesn't have.
  • INTERTHREAD in a Julia application run as one task (Goby.run(app)) is now an error. Before, a publication went to the C++ glue and a subscription was never delivered.
  • A Julia task that subscribed twice to a group used to receive each message twice per callback. It now receives it once per callback.
  • A Python application whose interface.yml declares interthread is now held to it. One that declared some interthread groups only to get constants, and used others undeclared, will now raise.
  • An entry without thread beside another for the same call is now rejected, since "any thread" already covers it. Two entries naming different threads are allowed.

Docs

  • doc250_languages.md: a "Declaring interthread" section with outer-layer thread, and the Julia queues, overflow table and slow-consumer pattern.
  • doc260_interfaces.md: outline bullets for -interthread-manifest and outer_threads.
  • Goby.jl/README.md, src/python/README.md and share/interface/README.md updated to match.
  • release.md: entries under a new 3.8.0 heading.

Testing

  • goby_test_julia_interthread (new; Julia only, no gobyd):
    • Each overflow policy against a stalled consumer.
    • A blocked publisher doesn't hold up another group.
    • Self-publish deadlock, double subscription, round-robin service and stats.
    • Contract checks: undeclared group, wrong type, wrong task, conflicting keyword, incompatible callback, unresolvable name, and incompatible pub/sub types.
    • Outer-layer thread checks.
  • goby_test_julia_app and goby_test_python_app: their interface.yml declares interthread, and the Julia one also an interprocess thread. The tests check the generated contract, the outer-layer check and the manifest, including outer_threads. The Julia test also checks the single-task error.
  • Python unit tests:
    • Generator: the contract, the manifest, the single C++ branch with the strongest necessity, and the generated outer_layer_contract.
    • Runtime: enforcement, subclass matching, type resolution, unmade-subscription reporting, and outer-layer thread checks.
  • goby_test_interface_conformance: both generators agree on the extended corpus. thread_on_interprocess.yml moved from invalid to valid as thread_on_outer_layers.yml, and duplicate_threaded_publication.yml was added.
  • goby_test_middleware_clang_tool_gen:
    • Merges a Julia and a Python manifest, and refuses a malformed one.
    • Draws a glue call on the task outer_threads names when the manifest gives the group by its C++ expression (groups::plain, runtime dynamic_test::plain).
    • Refuses outer_threads on interthread.
  • goby_test_middleware_interface_tool: a new deployment with a Julia and a Python application, with expected .dot and .d2 output and validate, list and explain checks.
  • End to end: goby_clang_tool -gen -interthread-manifest on the real Julia and Python test-app glue produces the merged interface files, with the Julia tx publication drawn on Main.
  • Full suite (run before the thread: change): 62 of 63 pass locally. goby_test_intervehicle_driver_restart timed out in its Iridium restart sequence; it includes none of the changed code. All tests touched by the thread: change pass after it.
  • D2 output passes check_d2.awk and matches the expected file, but I didn't render it with d2 (not available here).

The examples are in GobySoft/goby3-examples#62.


🤖 Generated with Claude Code

https://claude.ai/code/session_01TCmHErFgxXnJfoD35VZF2d

…ed in interface.yml

Each Julia INTERTHREAD subscription now has its own queue, sized by
capacity, with an overflow policy (:block, :drop_oldest, :drop_newest,
:error). Publishers no longer wait with the subscription registry held,
so one full subscription stalls only its own group's publishers rather
than every INTERTHREAD call. A task serves its subscriptions and mailbox
in turn, and Goby.interthread_stats() reports what each has queued,
delivered and dropped (#485).

interface.yml may declare interthread, with thread, scheme NATIVE for
values of the bound language, and (Julia only) capacity and overflow.
The Julia and Python generators turn it into a contract that every
interthread publication and subscription is held to, and into a
manifest, <target>_<language>_interthread.yml, for goby_clang_tool
(#484). The Julia generator no longer writes C++ for interthread, which
Goby.jl carries between tasks.

INTERTHREAD in a Julia application run as one task is now an error
rather than silently undelivered, and a task that subscribed twice to a
group no longer receives each message twice per callback.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01TCmHErFgxXnJfoD35VZF2d
goby_clang_tool -gen -interthread-manifest adds the julia_interthread or
python_interthread section a binding's generator writes to the
application's interface file; goby_export_interface() passes it from the
target's GOBY_INTERTHREAD_MANIFEST property, which goby_add_julia_app()
and goby_add_python_app() set. goby interface reads each task or thread
as a thread of the application: visualize draws them, validate checks
their required subscriptions, and list and explain show a Julia
subscription's capacity and overflow (#484).

A Julia subscription is met by group alone, as Goby.jl delivers; a
Python one by group and type, or by group alone when it takes object.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01TCmHErFgxXnJfoD35VZF2d
An interprocess or intermodule entry in interface.yml may now name the
Julia task or Python thread that makes the call. The binding allows the
call from that thread only, checked where it is made (in Julia, before a
task hands it to the C++ task). The generated C++ handles the call once
however many threads declare it, with the strongest necessity any gives
it.

The generators add these entries to the manifest as outer_threads, and
goby_clang_tool -gen draws each matching glue call on the thread named,
rather than on the C++ class that makes every call on the threads'
behalf. A call is matched by its group's runtime string or its C++
expression, which is how interface.yml names it.

InterthreadContractError becomes InterfaceContractError in both
languages, since it now covers outer-layer calls too.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01TCmHErFgxXnJfoD35VZF2d
@tsaubergine
tsaubergine merged commit ed8b018 into 3.0 Nov 20, 2026
9 checks passed
@tsaubergine
tsaubergine deleted the claude/eloquent-mendel-qtcwli branch November 20, 2026 05:24
tsaubergine added a commit to GobySoft/goby3-examples that referenced this pull request Nov 20, 2026
… examples (#62)

- julia/slow_consumer (new): a fast producer, a slow consumer whose queue capacity and overflow the configuration chooses, and a monitor; reports InterthreadQueueStats on groups::julia_queue_stats, checked by a goby test harness.
- julia/basic_multithread and python/basic_multithread declare their interthread traffic and each interprocess call's thread in interface.yml, and use the generated group constants.
- Deployments, figures, validate and launch_check tests for the three; goby_export_interface() passes the interthread manifest to goby_clang_tool.
- Requires Goby 3.8 (GobySoft/goby3#488); skipped otherwise.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01TCmHErFgxXnJfoD35VZF2d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add configurable backpressure and callback isolation for Julia INTERTHREAD subscriptions

2 participants