Repository navigation
Bound Julia INTERTHREAD subscriptions; declare Julia and Python interthread in interface.yml and goby interface - #488
Merged
Conversation
…ed in interface.yml Each Julia INTERTHREAD subscription now has its own queue, sized by capacity, with an overflow policy (:block, :drop_oldest, :drop_newest, :error). Publishers no longer wait with the subscription registry held, so one full subscription stalls only its own group's publishers rather than every INTERTHREAD call. A task serves its subscriptions and mailbox in turn, and Goby.interthread_stats() reports what each has queued, delivered and dropped (#485). interface.yml may declare interthread, with thread, scheme NATIVE for values of the bound language, and (Julia only) capacity and overflow. The Julia and Python generators turn it into a contract that every interthread publication and subscription is held to, and into a manifest, <target>_<language>_interthread.yml, for goby_clang_tool (#484). The Julia generator no longer writes C++ for interthread, which Goby.jl carries between tasks. INTERTHREAD in a Julia application run as one task is now an error rather than silently undelivered, and a task that subscribed twice to a group no longer receives each message twice per callback. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01TCmHErFgxXnJfoD35VZF2d
goby_clang_tool -gen -interthread-manifest adds the julia_interthread or python_interthread section a binding's generator writes to the application's interface file; goby_export_interface() passes it from the target's GOBY_INTERTHREAD_MANIFEST property, which goby_add_julia_app() and goby_add_python_app() set. goby interface reads each task or thread as a thread of the application: visualize draws them, validate checks their required subscriptions, and list and explain show a Julia subscription's capacity and overflow (#484). A Julia subscription is met by group alone, as Goby.jl delivers; a Python one by group and type, or by group alone when it takes object. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01TCmHErFgxXnJfoD35VZF2d
Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01TCmHErFgxXnJfoD35VZF2d
An interprocess or intermodule entry in interface.yml may now name the Julia task or Python thread that makes the call. The binding allows the call from that thread only, checked where it is made (in Julia, before a task hands it to the C++ task). The generated C++ handles the call once however many threads declare it, with the strongest necessity any gives it. The generators add these entries to the manifest as outer_threads, and goby_clang_tool -gen draws each matching glue call on the thread named, rather than on the C++ class that makes every call on the threads' behalf. A call is matched by its group's runtime string or its C++ expression, which is how interface.yml names it. InterthreadContractError becomes InterfaceContractError in both languages, since it now covers outer-layer calls too. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01TCmHErFgxXnJfoD35VZF2d
tsaubergine
added a commit
to GobySoft/goby3-examples
that referenced
this pull request
Nov 20, 2026
… examples (#62) - julia/slow_consumer (new): a fast producer, a slow consumer whose queue capacity and overflow the configuration chooses, and a monitor; reports InterthreadQueueStats on groups::julia_queue_stats, checked by a goby test harness. - julia/basic_multithread and python/basic_multithread declare their interthread traffic and each interprocess call's thread in interface.yml, and use the generated group constants. - Deployments, figures, validate and launch_check tests for the three; goby_export_interface() passes the interthread manifest to goby_clang_tool. - Requires Goby 3.8 (GobySoft/goby3#488); skipped otherwise. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01TCmHErFgxXnJfoD35VZF2d
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves #485 and #484.
Julia INTERTHREAD subscriptions get a bounded queue each, with an overflow policy, and publishers no longer wait with the registry lock held.
interface.ymlcan declare interthread for Julia and Python, and can name the task or thread that makes each outer-layer call. The binding enforces both at run time, andgoby interfacedraws each task or thread with its calls and validates its subscriptions.As agreed while planning, two parts of the issues are not in this PR:
INTERTHREADsubscriptions #485): a slow consumer goes in its own task module instead. Noted on Add configurable backpressure and callback isolation for JuliaINTERTHREADsubscriptions #485.goby interface(goby_clang_tool) to emit JuliaINTERTHREADinterface contracts #484, Add configurable backpressure and callback isolation for JuliaINTERTHREADsubscriptions #485): moved to Consider a request/reply model to complement publish/subscribe at all layers #486, to be designed for all layers.Changes
Julia delivery (#485)
Goby.subscribe(...; capacity = 50, overflow = :block)onGoby.INTERTHREAD.overflowis one of::block: wait for room.:drop_oldest: replace the oldest queued message.:drop_newest: discard the new message.:error: deliver to the other subscribers, then throwGoby.InterthreadOverflowError.:blockones. A full subscription now delays only its own group's publishers; before, one full mailbox stalled every INTERTHREAD call in the process.:blocksubscription throwsGoby.InterthreadDeadlockErrorinstead of hanging.Goby.interthread_stats()reports group, task, capacity, policy, and messages queued, delivered and dropped. The first drop on a subscription logs a warning.src/share/Goby.jl/src/Interthread.jlandInterthreadContract.jl;GobyMultiThread.jlmoves onto the inboxes.interthread:ininterface.yml(#484)thread: a Julia task module's name, or a PythonThreadsubclass's or application class's name.scheme: NATIVE: any value of the bound language.capacityandoverflow: subscriptions, Julia only. The Python generator refuses them.Goby.interthread_contract(...)in Julia,goby.interthread_contract([...])in Python). Once declared, an undeclared group, a message of another type, or a call from another thread raisesInterfaceContractError. Type names are resolved at startup, and a declared subscription that nothing made is reported.<target>_julia_interthread.ymlor<target>_python_interthread.yml.goby_add_julia_app()andgoby_add_python_app()record it as the target'sGOBY_INTERTHREAD_MANIFESTproperty. They also now callgenerate_interfaces()when the project exports interface files, asgoby_add_app()does.GobyJulia.cmakeandGobyPython.cmakesetGOBY_JULIA_HAS_INTERTHREAD_CONTRACTandGOBY_PYTHON_HAS_INTERTHREAD_CONTRACT, so downstream projects can tell the feature is there.thread:on interprocess and intermodule entriesThe generated C++ makes every outer-layer call on the tasks' or threads' behalf, so
goby_clang_toolused to draw them all on the C++ class. In the Julia multithread example, that made an application publishing and subscribing to one group draw a loop on itself.interprocessorintermoduleentry may now givethread. Several threads may declare the same call; withoutthread, any thread may make it.Goby.outer_layer_contract(...)andgoby.outer_layer_contract([...]), for the calls that give a thread. In Julia the check runs where the call is made, before a task hands it to the C++ task, which is skipped since it only republishes. Calls are matched on the message name without its scope, as the C++ glue matches them.goby_clang_tooldraws the call on that thread. The manifest carries the entries asouter_threads, and-gendraws each matching glue call on the thread named. A call is matched by its group's runtime string or its C++ expression, the nameinterface.ymluses. The latter needed the group variable's qualified name, which-gennow keeps for this.Interface tools (#484)
goby_clang_tool -gen -interthread-manifest <file>checks the manifest, adds its interthread section to<target>_interface.yml, and appliesouter_threads.goby_export_interface()passes it from the target property.goby interfacereadsjulia_interthreadandpython_interthread: each task or thread becomes a thread of the application, sovisualize(Graphviz and D2) draws it andvalidatechecks its required subscriptions.object.listandexplainshow a Julia subscription's capacity and overflow, via a newdeliveryfield onInterfaceEndpoint.src/middleware/tool/interface/language_interthread.h, shared by both tools.Behaviour change
interthread:. Previously it generated calls tointerthread(), whichSingleThreadApplication, the base of every Julia example, doesn't have.Goby.run(app)) is now an error. Before, a publication went to the C++ glue and a subscription was never delivered.interface.ymldeclaresinterthreadis now held to it. One that declared some interthread groups only to get constants, and used others undeclared, will now raise.threadbeside another for the same call is now rejected, since "any thread" already covers it. Two entries naming different threads are allowed.Docs
doc250_languages.md: a "Declaring interthread" section with outer-layerthread, and the Julia queues, overflow table and slow-consumer pattern.doc260_interfaces.md: outline bullets for-interthread-manifestandouter_threads.Goby.jl/README.md,src/python/README.mdandshare/interface/README.mdupdated to match.release.md: entries under a new 3.8.0 heading.Testing
goby_test_julia_interthread(new; Julia only, no gobyd):goby_test_julia_appandgoby_test_python_app: theirinterface.ymldeclares interthread, and the Julia one also an interprocessthread. The tests check the generated contract, the outer-layer check and the manifest, includingouter_threads. The Julia test also checks the single-task error.outer_layer_contract.goby_test_interface_conformance: both generators agree on the extended corpus.thread_on_interprocess.ymlmoved from invalid to valid asthread_on_outer_layers.yml, andduplicate_threaded_publication.ymlwas added.goby_test_middleware_clang_tool_gen:outer_threadsnames when the manifest gives the group by its C++ expression (groups::plain, runtimedynamic_test::plain).outer_threadson interthread.goby_test_middleware_interface_tool: a new deployment with a Julia and a Python application, with expected.dotand.d2output andvalidate,listandexplainchecks.goby_clang_tool -gen -interthread-manifeston the real Julia and Python test-app glue produces the merged interface files, with the Juliatxpublication drawn onMain.thread:change): 62 of 63 pass locally.goby_test_intervehicle_driver_restarttimed out in its Iridium restart sequence; it includes none of the changed code. All tests touched by thethread:change pass after it.check_d2.awkand matches the expected file, but I didn't render it withd2(not available here).The examples are in GobySoft/goby3-examples#62.
🤖 Generated with Claude Code
https://claude.ai/code/session_01TCmHErFgxXnJfoD35VZF2d