Skip to content

[runtime][python] Release per-action Pemja objects - #1049

Draft
joeyutong wants to merge 2 commits into
apache:mainfrom
joeyutong:codex/fix-pemja-action-references
Draft

[runtime][python] Release per-action Pemja objects#1049
joeyutong wants to merge 2 commits into
apache:mainfrom
joeyutong:codex/fix-pemja-action-references

Conversation

@joeyutong

@joeyutong joeyutong commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

Linked issue: #1048

Purpose of change

PythonActionExecutor repeatedly creates Java-side Pemja PyObject handles while a Python Action is running. Those handles own native Python references and were previously dropped without calling close().

This change:

  • closes the temporary Python Event and coroutine wrappers after each Action invocation;
  • keeps a pending coroutine alive through the independent interpreter-global reference established by interpreter.set(...);
  • closes the temporary wrapper returned by each interpreter.get(...) poll;
  • deletes the interpreter-global coroutine reference after the coroutine completes;
  • returns only the awaitable completion state from Python instead of materializing ignored yielded/returned values across JNI.

The last point avoids a recursive ownership problem: an Action output may contain arbitrary nested Python objects, while Java never consumes the output. Actions communicate with the runtime through emitted Events, so the narrow boundary is a boolean completion state.

This is complementary to #944: that PR handles executor-lifetime objects during attempt cleanup, while this PR handles objects created repeatedly during the lifetime of an open executor.

Tests

  • PythonActionExecutorTest: 11 passed, covering synchronous completion, async storage, Action failure, set failure, pending/completed polling, and polling failure.
  • test_function.py: 27 passed, including yielded and returned values being reduced to completion state.
  • ./tools/lint.sh -c: passed for Python lint and all Java Spotless modules.
  • Local Flink A/B: 12 fresh single-slot jobs covering six ownership boundaries in fixed/leak variants, 100 records with a 1 MiB payload each. Every fixed variant ended with zero tracked live objects and zero retained payload. Each leak variant retained exactly 100 affected objects or globals; payload-bearing variants retained 100 MiB. Full details are in [Draft] [Bug] PythonActionExecutor leaks per-action Pemja PyObject references #1048.

API

No user-facing API changes. PythonActionExecutor and call_python_awaitable are internal runtime bridge helpers. The helper now returns bool rather than (bool, output) because the output was not consumed. PyObject.close() failures continue through the existing ActionTask.invoke(...) throws Exception path.

Documentation

  • doc-needed
  • doc-not-needed
  • doc-included

Was this patch authored or co-authored using generative AI tooling?

  • Yes
  • No

Generated-by: Codex (GPT-5)

@github-actions github-actions Bot added doc-not-needed Your PR changes do not impact docs fixVersion/0.4.0 priority/major Default priority of the PR or issue. labels Aug 25, 2026
joeyutong and others added 2 commits August 26, 2026 17:02
Close action-scoped PyObject handles after interpreter ownership is established and remove completed awaitables from interpreter globals.

Generated-by: Codex (GPT-5)
Co-Authored-By: Claude Code <[email protected]>
AI-Model: gpt-5
AI-Contributed/Feature: 37/37
AI-Contributed/UT: 155/155
Return only the awaitable completion state across Pemja because Action yielded and returned values are not consumed by the runtime.

Co-Authored-By: Claude Code <[email protected]>

AI-Model: gpt-5
AI-Contributed/Feature: 29/29
AI-Contributed/UT: 17/17
@joeyutong
joeyutong force-pushed the codex/fix-pemja-action-references branch from fb3666e to 2114dc0 Compare August 26, 2026 09:08
@github-actions github-actions Bot added doc-not-needed Your PR changes do not impact docs and removed doc-not-needed Your PR changes do not impact docs labels Aug 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

doc-not-needed Your PR changes do not impact docs fixVersion/0.4.0 priority/major Default priority of the PR or issue.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant