[runtime][python] Manage Pemja references across resource bridges

[joeyutong]

Linked issue: #1057

Purpose of change

Pemja values crossing the Java/Python bridge may carry native Python references. The current resource paths do not share one ownership rule, so temporary values, nested results, long-lived resources, and partial initialization failures can retain references differently.

This change introduces PythonObjectScope as the common ownership abstraction:

• an operation scope recursively owns PyObject handles reachable through maps, iterables, and object arrays, deduplicates them by identity, and releases them in reverse order;
• release(...) makes ownership transfer to a long-lived wrapper explicit;
• values that must outlive a bridge call are materialized into detached Java primitives, lists, and maps before the raw references are closed;
• resource creation and MCP discovery use transactional rollback, closing both Python lifecycle state and native handles before any failed resource can remain registered.

The abstraction is applied consistently to chat and embedding models, vector stores, MCP servers/tools/prompts, Mem0 call-scoped values, Python-backed FunctionTool calls, and Java/Python resource-cache initialization.

This complements #944, which owns executor-lifetime roots and the shutdown close chain, and #1049, which owns per-Action execution values. This PR covers the remaining operation-level resource bridges and initialization rollback paths.

Tests

• mvn -pl runtime -am -DskipITs test
  • plan: 294 tests, 1 skipped
  • runtime: 722 tests, 0 failures/errors
• Final rollback-path changes: focused PythonResourceProviderTest,ResourceCacheTest
• bash tools/ut.sh -p: 935 passed, 13 skipped, 165 deselected
• ./tools/lint.sh -c
• bash tools/check-license.sh
• git diff --check

Local Flink 2.2 A/B validation ran six jobs, each with a fresh TaskManager, one probe subtask, Pemja 0.5.7, and 200 measured operations:

• chat: fixed 0 live bridge values; leak variant retained 600 messages, 200 tools, 200 MiB, and increased RSS by 160.3 MiB;
• vector: fixed 0 live bridge values; leak variant retained 600 documents, 200 normalized handles, about 100 MiB, and increased RSS by 99.1 MiB;
• resource recreation: fixed 0 live resources with 200 logical closes; leak variant retained 200 resources and 50 MiB with no close calls.

The deterministic live-reference, close-count, and reachable-payload measurements are the primary evidence; RSS is supporting evidence because JVM/CPython startup and native allocator behavior affect short runs.

API

This adds an @Internal ownership utility and a default materialization hook to the existing Python resource adapter. It does not change user-facing resource contracts, and the default method is binary compatible.

Documentation

• doc-needed
• doc-not-needed
• doc-included

Was this patch authored or co-authored using generative AI tooling?

• Yes
• No

Generated-by: Codex (GPT-5)