The native error slot is never cleared, so a call that fails without writing a message reports the previous one.
+
+
+
+
+
backgroundhow two threads come to share one object
+
+
+The object is shared, but the error slot is not: each thread has its own. That is why one thread's marker cannot clear another thread's pending error.
+
+
+
+
+
beforemain: the slot is only ever read
+
+
+The second failure inherits the first failure's message, and its exception type.
+
+
+
+
+
afterreading consumes: a marker is written back
+
+
+Reading the slot marks it. A failure that writes nothing now reads back "no error" instead of a stale message.
+
+
+
+
+
+
+
Notes
+
Both threads reach the same object because both hold a reference to it; the GIL keeps them from running Python at the same instant but is handed away entirely during a native call — the GIL figure on page 20 shows what it does and does not cover.
+
The slot is thread-local and sticky. Python cannot empty it — the library exposes no call for that — so the branch writes a known value in instead, produced by asking the library to free address 2, which it can never be tracking. That free fails predictably and leaves a message the wrapper recognises.
+
The exact text is learned at import rather than hardcoded, so it matches the build actually loaded. If the learned text does not contain 0x2, the mechanism switches itself off and the library behaves as it did on main.
+
Why it matters beyond a wrong message. When a consuming call fails, the wrapper decides who owns the pointer by reading this slot. A stale UntrackedPointer: message makes it conclude the pointer is still Python's, and the object is kept alive holding memory the native side already freed.
+
+c2pa.py:986 _NO_ERROR_MARKER_ADDR
+c2pa.py:1013-1037 _read_native_error — marks on both exit paths
+c2pa.py:1428-1457 _learn_no_error_marker_text
+main:696-716 _read_native_error — "Peeks: the error stays in the native slot"
+tests test_stale_error_not_misattributed_after_preset_error, test_reading_the_native_error_consumes_it
+
Not a lock protecting data from other threads — a marked stretch of time on one thread, between a call returning and its error being read. Any free inside it destroys the message.
+
+
+
+
+
backgroundshared objects, per-thread windows
+
+
+The section is thread-local because the error slot it protects is. A window open on thread A must not gate thread B's frees, or the two would block each other.
+
+
+
+
A Python object lives on the process heap — one region of memory shared by every thread. A thread is not a container that holds objects; it is a separate execution position, with its own call stack, walking through that same shared memory.
+
+
So nothing switches threads and nothing is handed over. In the probe behind this figure, a Reader created on MainThread and used from worker-1 stayed at the same address the whole time, with the same id(). Two threads simply looked at the same place.
+
+
What is shared is the name. An ordinary closure is enough:
+
+
r = Reader("image/jpeg", io.BytesIO(img))
+
+def worker():
+ return r.json() # closure captures r
+
+threading.Thread(target=worker).start()
+
+
No serialisation, no copy, no transfer step. Compare multiprocessing, where a genuinely separate heap forces objects to be pickled across — there id(r) would differ and mutations would not be visible. Threads have no such boundary, and that is exactly what separates them from processes.
+
+
+
+
The glossary sentence that follows is about executing bytecode: it says nothing about which objects a thread may reach. The GIL keeps MainThread and worker-1 from running Python instructions in the same instant. It does not stop them both holding a reference to one Reader, and it does not stop one calling close() while the other is mid-call.
+
+
+
backgroundwhat the GIL is, and why it does not save you here
+
+
+Two independent reasons the GIL does not prevent these bugs: it is given up entirely during a native call, and it was never a guarantee that a check and the use that follows it happen as one step.
+
+
+
+
+
beforemain: a finalizer's free lands in the window
+
+
+A free of an untracked pointer writes its own complaint into the same slot. CPython runs finalizers at any bytecode boundary, so this needs no threads at all.
+
+
+
+
+
afterthe window is marked; frees inside it are queued
+
+
+The free is postponed, not skipped. Without the pending list it would never happen at all, because nothing else would touch that object again.
+
+
+
+
+
+
+
Notes
+
Objects do not belong to threads. If they did, a close() on thread B could not reach thread A's handle, and most of this branch would be unnecessary.
+
Why the window exists. A C interface cannot raise an exception, so failure arrives in two pieces: a return value, and a message fetched by a separate c2pa_error() call. Between the two, arbitrary Python runs.
+
Thread-local because the native slot is: one thread's section must not gate another's frees. Depth-counted because native calls nest — and _read_native_error is itself one. A boolean would be cleared by the innermost exit while an outer classification was still reading.
+
The drain never hides your exception. It keeps only the first failure and returns it rather than raising; the bare raise re-raises whatever the body threw. A cleanup problem is logged, never substituted for the error you were reporting.
+
+c2pa.py:1040-1101 _native_section, _in_native_section, _register_for_section_flush
+c2pa.py:473-474 registration · c2pa.py:517-521 re-registration
+c2pa.py:1071-1081 _drain — swaps the list, isolates each failure
+main none of these symbols exist
+tests test_native_section_defers_unrelated_finalizer_free, test_section_drain_error_does_not_mask_the_body_error
+
A close() frees a handle another thread has already passed into a native call.
+
+
+
+
+
backgroundhow thread B gets hold of thread A's reader
+
+
+No handover happens. Both threads simply hold the same reference, so both may call methods on it at any time.
+
+
+
+
+
beforemain: close frees immediately
+
+
+The validity check passed before the free. Nothing re-checks it, and the fault happens inside native code with no Python traceback.
+
+
+
+
+
aftercalls are counted; the close is recorded and deferred
+
+
+The object is unusable from the moment close is called, but the memory outlives the call that is using it.
+
+
+
+
+
+
+
Notes
+
Both threads reach the same object because both hold a reference to it; the GIL keeps them from running Python at the same instant but is handed away entirely during a native call — the GIL figure on page 20 shows what it does and does not cover.
+
Why not just hold a lock. These native calls run caller-supplied stream callbacks, which can call back into this API, possibly from a new thread. A lock held across the call would deadlock against that re-entry. So the lock is held only long enough to change a counter, never across the call itself — page 70 shows the case that makes this unavoidable.
+
Two independent reasons defer a teardown — this object's own call being in flight, and the native section. The recorded flag merges with and, so a "close without freeing" can never be upgraded to a free by a later caller who does not know the pointer already moved.
+
+c2pa.py:265-273 the new per-resource state
+c2pa.py:346-363 _native_call — counts, does not lock across the call
+c2pa.py:464-476 _teardown — the deferring branch
+main:264-267 __init__ was three assignments; main:330-337 _teardown freed at once
+tests test_close_inside_callback_defers_free, test_deferred_consume_is_not_upgraded_to_free
+
A borrowing call validates its pointer once. A consuming call on another thread then hands that same pointer to the library to be freed.
+
+
+
+
+
backgroundone builder, two threads, two kinds of call
+
+
+Both calls act on the same native handle. Nothing in Python prevents the second from starting while the first is still running.
+
+
+
+
+
beforemain: no entry guard, the consume proceeds
+
+
+The registry check that normally catches a freed pointer already passed, at the top of thread A's call.
+
+
+
+
+
afterthe consume is refused while a borrow is in flight
+
+
+Refused rather than queued: waiting would mean waiting on caller-supplied callbacks of unbounded duration.
+
+
+
+
+
+
+
Notes
+
Both threads reach the same object because both hold a reference to it; the GIL keeps them from running Python at the same instant but is handed away entirely during a native call — the GIL figure on page 20 shows what it does and does not cover.
+
Why deferring does not work here. A racing close() can be postponed because Python performs that free. A consuming call's free happens inside the library, during the call, as part of taking ownership. Python neither schedules it nor can postpone it.
+
Two checks cover the two orderings. _ensure_not_borrowed() catches a borrow already running. Marking the object CLOSED before releasing the lock catches one arriving afterwards, because every borrowing call re-checks validity under that same lock.
+
A separate counter tracks mutating calls, so two mutations cannot overlap and a read cannot run during one.
+
+c2pa.py:310-320 _ensure_not_borrowed · c2pa.py:322-330 _ensure_no_mutating_call
+c2pa.py:722-730 _begin_consume — check, then mark CLOSED under the lock
+main:501-510 _consume_and_swap called straight through, no guard
+tests test_consume_during_foreign_borrow_raises, test_unborrowed_consume_proceeds
+
What gets freed is a function pointer Python created, which the library calls through while signing.
+
+
+
+
+
backgroundthe context is shared, and so is its callback
+
+
+The trampoline is kept alive by one attribute on the shared Context. Either thread can drop the last reference to it.
+
+
+
+
+
beforemain: close drops the callback, unguarded
+
+
+Native holds the trampoline's address but no reference to it, so ordinary Python reference counting can free it mid-call.
+
+
+
+
+
afterthe context is held in flight for the duration of the sign
+
+
+The same guard also refuses a sign that starts on an already-closed context, instead of signing without the signer.
+
+
+
+
+
+
+
Notes
+
Both threads reach the same object because both hold a reference to it; the GIL keeps them from running Python at the same instant but is handed away entirely during a native call — the GIL figure on page 20 shows what it does and does not cover.
+
What a trampoline is. To let native code call a Python function, ctypes builds a small object native can call like a C function. It is an ordinary Python object with ordinary reference counting, and nothing on the native side holds a reference to it. Keeping it alive as long as native might call it is the caller's job; here the Context holds that reference.
+
The quiet failure. If the callback is already gone when the sign begins, native signs without calling it and reports success. You get a file that looks signed and is not. The guard's validity check turns that into an exception.
+
The guard is duck-typed, so a caller-supplied context implementing only the published contract still works — at the cost of no in-flight protection. A test exists to ensure the built-in Context never falls into that unprotected branch.
+
+c2pa.py:4362-4372 the guarded context-sign
+c2pa.py:1945-1957 _context_guard — duck-typed on _native_call
+main:1722-1724 _release dropped the callback unconditionally
+tests test_context_sign_after_close_raises_rather_than_skipping_signer, test_built_in_context_still_gets_in_flight_protection
+
Every other page shows two threads doing something deliberate. This one shows a third thread that never touched the object and frees it anyway.
+
+
+
+
+
backgrounda free is not always something someone asked for
+
+
+Reference counting destroys an object wherever its count reaches zero. Thread C may be a worker that only returned a value; the free still runs on its stack.
+
+
+
+
+
beforemain: that free is immediate, and lands wherever C happens to be
+
+
+The object being freed and the thread doing the freeing are unrelated. C is damaged by work it never asked for.
+
+
+
+
+
afterthe gate is C's own state, not the object's
+
+
+The check is on the freeing thread's state, never on the object's. That is the only thing that works when the freeing thread is arbitrary.
+
+
+
+
+
+
+
Notes
+
The freeing thread is arbitrary. It is tempting to assume whoever frees the object is one of the threads using it. A pool worker that merely returned a value can be the one running c2pa_free, so the design cannot rely on that assumption anywhere.
+
This is what forces the deferral gate to be thread-local. If the section were a property of the resource, thread C's finalizer would consult the wrong object's state entirely — the reader being freed is not the one C was working with.
+
The same applies to _released: several threads can reach the same teardown at once, so idempotency has to key on a flag, not on the lifecycle state, which the deferred path sets while the free is still owed.
+
+c2pa.py:439-477 _teardown — the gate is _in_native_section(), a thread property
+c2pa.py:479-502 _finish_teardown — idempotent via _released
+c2pa.py:1040 _native_section_state = threading.local()
+tests test_third_thread_gc_of_dropped_reference_frees_exactly_once (200 resources, 4 workers,
+ asserts every handle freed exactly once), test_json_racing_finalizer_does_not_crash,
+ test_cross_thread_create_and_close_frees_exactly_once
+
This is the scenario that rules out the fix everyone reaches for first. A per-object lock deadlocks here, and so does a reentrant one.
+
+
+
+
+
backgroundwhat a stream callback is allowed to do
+
+
+The library gives up control to user code in the middle of its own operation, and cannot bound what that code does.
+
+
+
+
+
beforethe obvious fix: hold the object's lock across the call
+
+
+Reentrancy solves the same-thread case only. Here the blocked party is a second thread, so an RLock blocks it exactly as a plain lock would.
+
+
+
+
+
aftercount the call instead of locking across it
+
+
+The counter provides the same guarantee as the lock without being something another thread can wait on.
+
+
+
+
+
+
+
Notes
+
A per-object lock does not work. It is the natural first answer to the races page 30 describes, and this is the case that rules it out. The test's own docstring puts it plainly: "A lock held across construction deadlocks here, whether it is global or per-object."
+
Note what is not the problem. This is not re-entrancy — that case is real and an RLock handles it. Here the callback does not take the lock itself; it waits for a different thread that needs it. No lock design survives that, because the deadlock is between two threads with a cycle through user code the library never sees.
+
test_stream_callback_blocking_on_other_thread_does_not_deadlock builds it exactly: a readinto that starts a helper touching the same reader, joins it with a ten-second timeout, and records a failure if the helper is still alive. It runs the construction five times over.
+
+c2pa.py:346-363 _native_call — lock only around the counter, never across the call
+c2pa.py:332-344 _lock — "Never hold this across a native call that drives stream callbacks"
+c2pa.py:3303-3307 with_fragment — the same reasoning, via a non-blocking acquire
+tests test_stream_callback_blocking_on_other_thread_does_not_deadlock,
+ test_stream_callback_reentering_api_does_not_deadlock, test_concurrent_storm_terminates
+
Seven problems in the Python C2PA wrapper, one page each: a diagram of what went wrong before, a diagram of what the fix does, and a short note underneath.
+
+
Most of these corrupt native memory rather than raise an exception, so the visible symptom is a crash somewhere unrelated, a hang, or output that is quietly wrong. On main, ManagedResource.__init__ is three assignments: no lock, no record of calls in progress, no deferred cleanup.
Code references are to src/c2pa/c2pa.py; the “before” quotes are from git show main:src/c2pa/c2pa.py.
+
+
+
+
diff --git a/demo/style.css b/demo/style.css
new file mode 100644
index 00000000..1f9727af
--- /dev/null
+++ b/demo/style.css
@@ -0,0 +1,354 @@
+:root {
+ color-scheme: light dark;
+ --bg: #fbfaf8;
+ --fg: #1c1a17;
+ --muted: #5d5750;
+ --rule: #ddd7cf;
+ --card: #ffffff;
+ --code-bg: #f4f1ec;
+ --accent: #b3261e;
+ --accent-soft: rgba(179, 38, 30, 0.12);
+ --ok: #1c6b4a;
+ --ok-soft: rgba(28, 107, 74, 0.12);
+}
+
+@media (prefers-color-scheme: dark) {
+ :root:not([data-theme="light"]) {
+ --bg: #16151a;
+ --fg: #eae7e2;
+ --muted: #a49e97;
+ --rule: #35323a;
+ --card: #1e1d23;
+ --code-bg: #232228;
+ --accent: #ff8f84;
+ --accent-soft: rgba(255, 143, 132, 0.16);
+ --ok: #6cc79b;
+ --ok-soft: rgba(108, 199, 155, 0.16);
+ }
+}
+
+:root[data-theme="dark"] {
+ --bg: #16151a;
+ --fg: #eae7e2;
+ --muted: #a49e97;
+ --rule: #35323a;
+ --card: #1e1d23;
+ --code-bg: #232228;
+ --accent: #ff8f84;
+ --accent-soft: rgba(255, 143, 132, 0.16);
+ --ok: #6cc79b;
+ --ok-soft: rgba(108, 199, 155, 0.16);
+}
+
+* { box-sizing: border-box; }
+
+body {
+ margin: 0;
+ background: var(--bg);
+ color: var(--fg);
+ font: 16px/1.65 -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif;
+ -webkit-font-smoothing: antialiased;
+}
+
+.wrap {
+ max-width: 46rem;
+ margin: 0 auto;
+ padding: 3rem 1.25rem 6rem;
+}
+
+.crumb {
+ font-size: 0.8rem;
+ color: var(--muted);
+ margin-bottom: 2rem;
+ letter-spacing: 0.02em;
+}
+
+.crumb a { color: var(--muted); }
+
+h1 {
+ font-size: 1.95rem;
+ line-height: 1.2;
+ margin: 0 0 0.4rem;
+ letter-spacing: -0.02em;
+}
+
+.standfirst {
+ font-size: 1.08rem;
+ color: var(--muted);
+ margin: 0 0 2.6rem;
+ line-height: 1.55;
+}
+
+h2 {
+ font-size: 1.18rem;
+ margin: 3rem 0 0.9rem;
+ padding-top: 1.4rem;
+ border-top: 1px solid var(--rule);
+ letter-spacing: -0.01em;
+}
+
+h3 {
+ font-size: 1rem;
+ margin: 2rem 0 0.6rem;
+}
+
+p { margin: 0 0 1rem; }
+
+a { color: inherit; text-decoration-color: var(--rule); text-underline-offset: 2px; }
+a:hover { text-decoration-color: currentColor; }
+
+code {
+ font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;
+ font-size: 0.87em;
+ background: var(--code-bg);
+ padding: 0.1em 0.34em;
+ border-radius: 3px;
+}
+
+pre {
+ background: var(--code-bg);
+ border: 1px solid var(--rule);
+ border-radius: 6px;
+ padding: 0.9rem 1rem;
+ overflow-x: auto;
+ margin: 0 0 1rem;
+}
+
+pre code { background: none; padding: 0; font-size: 0.8rem; line-height: 1.55; }
+
+.filename {
+ font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;
+ font-size: 0.74rem;
+ color: var(--muted);
+ margin-bottom: 0.35rem;
+ letter-spacing: 0.01em;
+}
+
+figure { margin: 2rem 0; }
+
+figure svg {
+ display: block;
+ width: 100%;
+ max-width: 100%;
+ height: auto;
+ color: var(--fg);
+}
+
+figcaption {
+ font-size: 0.85rem;
+ color: var(--muted);
+ margin-top: 0.85rem;
+ line-height: 1.5;
+}
+
+ol, ul { margin: 0 0 1rem; padding-left: 1.4rem; }
+li { margin-bottom: 0.5rem; }
+
+.steps { counter-reset: step; list-style: none; padding-left: 0; }
+
+.steps li {
+ counter-increment: step;
+ position: relative;
+ padding-left: 2.1rem;
+ margin-bottom: 0.8rem;
+}
+
+.steps li::before {
+ content: counter(step);
+ position: absolute;
+ left: 0;
+ top: 0.08rem;
+ width: 1.45rem;
+ height: 1.45rem;
+ border-radius: 50%;
+ background: var(--code-bg);
+ border: 1px solid var(--rule);
+ color: var(--muted);
+ font-size: 0.76rem;
+ font-weight: 600;
+ display: flex;
+ align-items: center;
+ justify-content: center;
+}
+
+.steps li.bad::before {
+ background: var(--accent-soft);
+ border-color: var(--accent);
+ color: var(--accent);
+}
+
+.note {
+ border-left: 3px solid var(--rule);
+ padding: 0.15rem 0 0.15rem 1rem;
+ margin: 1.5rem 0;
+ color: var(--muted);
+ font-size: 0.94rem;
+}
+
+.note.warn { border-left-color: var(--accent); }
+.note strong { color: var(--fg); }
+
+.tests { list-style: none; padding-left: 0; }
+
+.tests li {
+ padding: 0.6rem 0;
+ border-bottom: 1px solid var(--rule);
+ font-size: 0.93rem;
+}
+
+.tests li:first-child { border-top: 1px solid var(--rule); }
+
+.tests .tname {
+ font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;
+ font-size: 0.8rem;
+ display: block;
+ margin-bottom: 0.15rem;
+}
+
+.tests .twhat { color: var(--muted); font-size: 0.88rem; }
+
+table { border-collapse: collapse; width: 100%; font-size: 0.9rem; margin: 0 0 1rem; }
+th, td { text-align: left; padding: 0.55rem 0.7rem 0.55rem 0; border-bottom: 1px solid var(--rule); vertical-align: top; }
+th { font-size: 0.78rem; text-transform: uppercase; letter-spacing: 0.06em; color: var(--muted); font-weight: 600; }
+
+.scroll { overflow-x: auto; }
+
+.pagenav {
+ display: flex;
+ justify-content: space-between;
+ gap: 1rem;
+ margin-top: 4rem;
+ padding-top: 1.4rem;
+ border-top: 1px solid var(--rule);
+ font-size: 0.9rem;
+}
+
+.pagenav a { color: var(--muted); }
+.pagenav a:hover { color: var(--fg); }
+
+/* index */
+.cards { display: grid; gap: 0; margin-top: 2rem; }
+
+.card {
+ display: block;
+ padding: 1.3rem 0;
+ border-top: 1px solid var(--rule);
+ text-decoration: none;
+ color: inherit;
+}
+
+.card:last-child { border-bottom: 1px solid var(--rule); }
+.card:hover .card-title { text-decoration: underline; text-underline-offset: 3px; }
+
+.card-num {
+ font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;
+ font-size: 0.74rem;
+ color: var(--muted);
+}
+
+.card-title { font-size: 1.05rem; font-weight: 600; margin: 0.2rem 0 0.35rem; }
+.card-desc { font-size: 0.92rem; color: var(--muted); margin: 0; line-height: 1.55; }
+
+.tag {
+ display: inline-block;
+ font-size: 0.7rem;
+ letter-spacing: 0.04em;
+ text-transform: uppercase;
+ padding: 0.15rem 0.45rem;
+ border-radius: 3px;
+ border: 1px solid var(--rule);
+ color: var(--muted);
+ margin-left: 0.5rem;
+ vertical-align: 0.1rem;
+}
+
+.tag.crash { color: var(--accent); border-color: var(--accent); background: var(--accent-soft); }
+
+/* diagram-first page format */
+.diagrams { margin: 2.5rem 0 0; }
+
+.panel { margin: 0 0 2.6rem; }
+
+.panel-label {
+ display: flex;
+ align-items: baseline;
+ gap: 0.6rem;
+ margin-bottom: 0.7rem;
+}
+
+.panel-tag {
+ font-size: 0.7rem;
+ letter-spacing: 0.08em;
+ text-transform: uppercase;
+ font-weight: 700;
+ padding: 0.18rem 0.5rem;
+ border-radius: 3px;
+}
+
+.panel-tag.before { color: var(--accent); background: var(--accent-soft); }
+.panel-tag.after { color: var(--ok); background: var(--ok-soft); }
+
+.panel-claim { font-size: 0.95rem; color: var(--muted); }
+
+.panel figure { margin: 0; }
+.panel figcaption { margin-top: 0.6rem; }
+
+.footnote {
+ margin-top: 3rem;
+ padding-top: 1.4rem;
+ border-top: 1px solid var(--rule);
+ font-size: 0.92rem;
+ color: var(--muted);
+}
+
+.footnote p { margin: 0 0 0.7rem; }
+.footnote strong { color: var(--fg); }
+.footnote code { font-size: 0.85em; }
+
+.refs {
+ margin-top: 1.2rem;
+ font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;
+ font-size: 0.74rem;
+ color: var(--muted);
+ line-height: 1.9;
+}
+
+.panel-tag { color: var(--muted); background: var(--code-bg); }
+
+.panel-note {
+ margin-top: 1.1rem;
+ font-size: 0.92rem;
+ color: var(--muted);
+ line-height: 1.6;
+}
+
+.panel-note p { margin: 0 0 0.8rem; }
+.panel-note p:last-child { margin-bottom: 0; }
+.panel-note strong { color: var(--fg); }
+
+.panel-note pre {
+ margin: 0.9rem 0;
+ background: var(--code-bg);
+}
+
+.bridge {
+ margin: 0 0 2.6rem;
+ padding-left: 1rem;
+ border-left: 3px solid var(--rule);
+ font-size: 0.94rem;
+ color: var(--muted);
+ line-height: 1.6;
+}
+
+.panel-tag.also { color: var(--muted); background: var(--code-bg); }
+
+.footnote h2 {
+ font-size: 0.78rem;
+ text-transform: uppercase;
+ letter-spacing: 0.08em;
+ color: var(--muted);
+ margin: 0 0 0.9rem;
+ padding: 0;
+ border: 0;
+ font-weight: 600;
+}
diff --git a/docs/native-resources-management.md b/docs/native-resources-management.md
index 9d4de693..20a669c9 100644
--- a/docs/native-resources-management.md
+++ b/docs/native-resources-management.md
@@ -99,7 +99,7 @@ Python owns and frees two kinds of things: the **single current native handle**
Therefore, the managed resources have the following principles:
-- Each `ManagedResource` holds exactly one `_handle`. `_swap_handle()` replaces it with the pointer a consuming call returned and does not free the old value, since the native side took it (see [Consume-and-swap](#consume-and-swap)).
+- Each `ManagedResource` holds exactly one `_handle`. `_consume_and_swap()` replaces it with the pointer a consuming call returned and does not free the old value, since the native side took it (see [Consume-and-swap](#consume-and-swap)).
- `_teardown(free_handle=False)`, `_consume_no_replacement()`, and `_consume_into()` all close or advance the object without calling `c2pa_free`, because ownership moved to the native side.
- Only a few sites free a live handle, and most free a pointer this layer still provably owns: normal teardown (`_teardown(free_handle=True)`); the create-then-validate path, which frees a freshly created pointer if activation fails; and the constructors that free a raw pointer when wrapping it raises, since no instance took ownership (`Signer.from_info`, `Signer.from_callback`, `Builder.from_archive`). The exception is `_release_handle()`, a *guarded* free used only when ownership is unknown (a consuming call failed without setting an error, or a Python exception was raised before the native side reported anything): if the native side already took the pointer, its address is no longer in the registry and `c2pa_free` is a `-1` no-op, so the free touches no memory. No path frees a pointer known to have been consumed and reallocated (see [Why an ownership-taken failure does not free](#why-an-ownership-taken-failure-does-not-free)).
- `_release()` drops stream wrappers, callbacks, and caches before the native pointer is freed (see [Subclass-specific cleanup with `_release()`](#subclass-specific-cleanup)).
@@ -112,7 +112,7 @@ Each risk and its mechanism:
| Hazard | Covered by | How |
| --- | --- | --- |
-| Freeing a pointer a consuming call already took (single flow) | `_swap_handle` / `_teardown(free_handle=False)` triage | The consumed pointer is abandoned, never freed. The retained-vs-consumed decision reads the native error tag (`UntrackedPointer:` / `WrongPointerType:` / `NullParameter:` / `InvalidBufferSize:` mean not taken). |
+| Freeing a pointer a consuming call already took (single flow) | `_consume_and_swap` / `_teardown(free_handle=False)` triage | The consumed pointer is abandoned, never freed. The retained-vs-consumed decision reads the native error tag (`UntrackedPointer:` / `WrongPointerType:` / `NullParameter:` / `InvalidBufferSize:` mean not taken). |
| A forked child freeing a pointer its parent owns | PID stamp (`record_owner_pid` / `is_foreign_process`) | Cleanup in a process that did not allocate the pointer nulls the handle and marks `CLOSED` without freeing (see [Fork safety](#fork-safety)). |
| Two **threads** racing a `close()` against an in-flight native call on the same object, where the allocator recycles a just-freed address | `_op_lock` / `_native_call()` / `_pending_teardown` | A close arriving while a native call is in flight is recorded rather than applied. The last caller to leave `_native_call()` performs the deferred free (see [Locking and in-flight tracking](#locking-and-in-flight-tracking)). |
@@ -277,7 +277,7 @@ The mark is provisional. `_abort_consume()` restores the previous state when the
`_raise_consume_failure()` performs that restore, on the pre-consume branch only. The reservation is held until the branch is known. `_read_native_error()` is itself a native call and releases the GIL, so a resource restored to `ACTIVE` before the error is classified is visible as usable to another thread while the native side may already own its handle.
-`_consume_and_swap()` is excluded. `_swap_handle()` requires the resource to stay `ACTIVE` and the object remains usable with its replacement pointer, so there is no `CLOSED` mark to make and no check. Its callers (`Reader.with_fragment`, `Builder.with_archive`) pass streams whose callbacks re-enter this API, so they hold their own `_native_call()`. `Reader.with_fragment()` additionally serializes itself with a lock of its own, described in [`Reader.with_fragment()`](#readerwith_fragment).
+`_consume_and_swap()` is excluded. `_consume_and_swap()` requires the resource to stay `ACTIVE` and the object remains usable with its replacement pointer, so there is no `CLOSED` mark to make and no check. Its callers (`Reader.with_fragment`, `Builder.with_archive`) pass streams whose callbacks re-enter this API, so they hold their own `_native_call()`. `Reader.with_fragment()` additionally serializes itself with a lock of its own, described in [`Reader.with_fragment()`](#readerwith_fragment).
### Context lifetime during a context-sign
@@ -299,7 +299,7 @@ A sign cannot start once the Context is closed, and raises `C2paError` instead.
| **Cleanup is idempotent** | Calling `close()` (or exiting a `with` block) multiple times is safe; after the first successful cleanup, further calls do nothing. |
| **Cleanup never raises (ordinary errors)** | The cleanup path catches and logs `Exception`, never re-raising it. `_release()` runs inside `_safe_release()`, which logs and swallows; the `c2pa_free` call has its own handler; and `_cleanup_resources()` wraps both. The original exception from the `with` block (if any) is never masked. **Asynchronous interrupts are the deliberate exception.** The cleanup handlers catch `Exception`, which excludes the `BaseException` signals the interpreter raises to unwind a process (a cancellation request or an exit in progress). Those propagate through cleanup untouched, and the remaining free may not run. Such a signal means the process is being torn down and its address space, native allocations included, is about to be reclaimed as a whole. Catching it would suppress a shutdown the caller asked for in order to complete a free that is about to become irrelevant, so the handlers stay scoped to `Exception`. |
| **State transitions are one-way** | Lifecycle moves only from UNINITIALIZED to ACTIVE to CLOSED. A closed resource cannot be reactivated. |
-| **Transitions go through helper methods** | Subclasses call `_activate()`, `_swap_handle()` or `_teardown()` and never assign `_handle` or `_lifecycle_state` directly. `_activate()` and `_swap_handle()` validate before mutating, so an object cannot end up active with a null handle. |
+| **Transitions go through helper methods** | Subclasses call `_activate()`, `_consume_and_swap()` or `_teardown()` and never assign `_handle` or `_lifecycle_state` directly. `_activate()` and `_consume_and_swap()` validate before mutating, so an object cannot end up active with a null handle. |
| **Ownership transfer is safe** | When a pointer is transferred elsewhere (e.g. via `_teardown(free_handle=False)`), the object stops managing it and does not call `c2pa_free` on it. |
| **Public methods validate lifecycle state** | Every public method that uses the handle calls `_ensure_valid_state()` before doing so; closed or invalid state yields `C2paError` instead of undefined behavior or crashes. The exceptions touch no handle: `is_valid` reports the state rather than requiring it, and the `get_supported_mime_types` classmethods query the library itself. |
@@ -339,7 +339,7 @@ stateDiagram-v2
[*] --> UNINITIALIZED : __init__()
UNINITIALIZED --> ACTIVE : _activate(handle)
UNINITIALIZED --> CLOSED : close() before activation
- ACTIVE --> ACTIVE : _swap_handle(new_handle)
+ ACTIVE --> ACTIVE : _consume_and_swap(new_handle)
ACTIVE --> CLOSED : close() / __exit__ / __del__ / _teardown()
```
@@ -354,7 +354,7 @@ Each transition has one method that performs it, and subclasses must go through
| Method | Transition | What it enforces |
| --- | --- | --- |
| `_activate(handle)` | UNINITIALIZED to ACTIVE | Rejects a null handle, and refuses to run on an already-activated resource. A rejected activation leaves the object exactly as it was. |
-| `_swap_handle(new_handle)` | ACTIVE to ACTIVE | Requires the resource to already be active and the replacement to be non-null. Used when an FFI call consumed the old handle and returned a new one. |
+| `_consume_and_swap(new_handle)` | ACTIVE to ACTIVE | Requires the resource to already be active and the replacement to be non-null. Used when an FFI call consumed the old handle and returned a new one. |
| `_teardown(free_handle=False)` | ACTIVE to CLOSED | Drops the handle without freeing it, for when ownership passed to the native side (e.g. `Signer` into `Context`). Runs `_release()` first, so subclass cleanup still happens. Unlike the other two it enforces no precondition on the current state: it closes whatever it is given. |
| `_release_handle()` | ACTIVE to CLOSED | Frees the handle (guarded, via `_teardown(free_handle=True)`) and closes the object. Same post-state as the consumed teardown. A resource that is already non-ACTIVE takes the other branch, which clears the handle without freeing it; the reserved consume paths call `_teardown()` directly for that reason. |
@@ -590,7 +590,7 @@ On success the object stays `ACTIVE` because the Python-side object is still val
One `with_fragment()` call does two things:
-1. The FFI call consumes the Reader's current handle and returns a replacement, which `_swap_handle()` stores.
+1. The FFI call consumes the Reader's current handle and returns a replacement, which `_consume_and_swap()` stores.
2. The Reader updates its own Python-side fields: the `Stream` wrappers it owns and the manifest caches. Both still describe the consumed handle.
`_fragment_streams` holds the `Stream` wrapper for the current fragment. Each call replaces that list rather than appending to it, closing the previous wrapper immediately. The native reader never reads a superseded fragment back, and each open wrapper pins a native stream, its callbacks, and the caller's buffer.
@@ -637,7 +637,7 @@ self._consume_and_swap(
Reader._ERROR_MESSAGES['fragment_error'])
```
-The call is passed as a lambda because the helper supplies the handle and, on success, replaces it via `_swap_handle()`.
+The call is passed as a lambda because the helper supplies the handle and, on success, replaces it via `_consume_and_swap()`.
The helper exists because a failed return can be ambiguous. The native functions run in phases: it validates the **borrowed pointer** (passed in without transferring ownership; the caller still owns it unless the callee explicitly takes it over), then takes ownership, then does the work. A failure in the first phase and a failure after the second come back to Python as the same value (a null pointer, or a non-zero status), but they leave ownership in opposite places.
@@ -668,7 +668,7 @@ Three consume helpers share this triage; they differ only in what the FFI call r
| Helper | Success return | Success action |
| --- | --- | --- |
-| `_consume_and_swap()` | a replacement pointer | `_swap_handle()`, resource stays `ACTIVE` |
+| `_consume_and_swap()` | a replacement pointer | installs the replacement, resource stays `ACTIVE` |
| `_consume_no_replacement()` | a status code (`0` = ok) | `_teardown(free_handle=False)`, resource `CLOSED` |
| `_consume_into()` | a *different* object's pointer | `_teardown(free_handle=False)`, the pointer returned for the caller to own |
@@ -785,7 +785,7 @@ different situation when writing a new subclass:
| A Python instance needs to wrap a handle a native call already returned, without creating a new one | `_wrap_native_handle(handle)` (classmethod) |
| Ordinary teardown (`close()`, `__del__`) | Neither: these already route through `_cleanup_resources()` and `_teardown()`. Nothing outside `ManagedResource` itself calls `_teardown()` directly. |
-`_activate()` and `_swap_handle()` are two low-level primitives this
+`_activate()` and `_consume_and_swap()` are two low-level primitives this
situation table builds on.
## Implementing a subclass of `ManagedResource`
@@ -852,7 +852,7 @@ class NativeResource(ManagedResource):
- `_init_attrs()` called after an FFI call that can raise leaves `_release()` accessing attributes that do not exist yet when that call fails, crashing with `AttributeError`. It belongs immediately after `super().__init__()`, before anything that can fail.
-- Assigning `self._handle` or `self._lifecycle_state` directly bypasses the checks that make the lifecycle safe. `_activate()` refuses a null handle and refuses to run on an already-active object; `_swap_handle()` requires the resource to be active and the replacement non-null. Direct assignment gives up both, and the resulting bugs (an ACTIVE object with a null handle, or a silently discarded pointer) surface far from their cause.
+- Assigning `self._handle` or `self._lifecycle_state` directly bypasses the checks that make the lifecycle safe. `_activate()` refuses a null handle and refuses to run on an already-active object; `_consume_and_swap()` requires the resource to be active and the replacement non-null. Direct assignment gives up both, and the resulting bugs (an ACTIVE object with a null handle, or a silently discarded pointer) surface far from their cause.
- A `_release()` that raises has its exception silently swallowed by `_cleanup_resources()`, visible only in the logs. A small lifecycle for managed resources would let `_release()` check whether they need releasing; the actual release call wrapped in try/except is a fallback for unexpected failures.
diff --git a/src/c2pa/c2pa.py b/src/c2pa/c2pa.py
index 6c697862..8d3fb24c 100644
--- a/src/c2pa/c2pa.py
+++ b/src/c2pa/c2pa.py
@@ -235,9 +235,9 @@ class ManagedResource:
- Call `_activate(handle)` once the native pointer is created and
validated, which takes ownership of it and marks the resource active.
Never assign `self._handle` or `self._lifecycle_state` directly.
- - Call `_swap_handle(new_handle)` instead when an FFI call consumed the
- current handle and returned a replacement (the success side of
- `_consume_and_swap`).
+ - Call `_consume_and_swap(ffi_call, message)` when an FFI call consumes
+ the current handle and returns a replacement: reserve the handle,
+ run the call, setup the new handle.
- Call `_teardown(free_handle=False)` when an FFI call took ownership of
the handle without returning a replacement: the new owner frees it,
so this does not.
@@ -268,33 +268,33 @@ def __init__(self):
self._handle = None
self._op_lock = threading.RLock()
self._inflight = 0
+ self._mut_inflight = 0
self._pending_teardown = None
+ self._teardown_lock = threading.Lock()
self._released = False
record_owner_pid(self)
- def _lock(self):
- """Return this resource's operation lock.
+ def _live_op_lock(self):
+ """Return this resource's operation lock, for mutual exclusion.
- Reentrant because it is possible to run a finalizer at any bytecode
- boundary, including inside a region this thread has already locked,
- and because a consuming call tears the handle down from inside the
- locked region.
+ Reentrant: a finalizer can run at any bytecode boundary, including
+ inside a region this thread already locked, and a consuming call
+ tears the handle down from inside the locked region.
Falls back to a fresh lock when the attribute is missing.
Never hold this across a native call that drives stream callbacks
(construction, resource_to_stream, the Builder stream methods,
- signing). Those calls release the Global Interpreter Lock (GIL)
+ signing).
+ Those calls release the Global Interpreter Lock (GIL)
and re-enter caller-supplied Python code, which may call back into
this API on another thread.
- Only calls that touch no callbacks are serialized here.
-
- Raises in a forked child rather than returning the lock.
- A child inherits this lock in whatever state it had at fork(),
- and a thread holding it does not exist in the child to release it,
- so acquiring it there waits and waits and waits.
- The child's copy is unusable for the same reason a closed resource is,
- and reports the same error.
+
+ Raises in a forked child instead of returning the lock: a child
+ inherits it in whatever state it had at fork(), and no thread in
+ the child exists to release it, so acquiring there hangs forever.
+ The child's copy is as unusable as a closed resource, so it
+ reports the same error.
"""
if is_foreign_process(self):
raise C2paError(f"{type(self).__name__} is closed")
@@ -308,6 +308,25 @@ def _lock(self):
pass
return lock
+ def _live_teardown_lock(self):
+ """Lock to protect teardowns.
+
+ Held only for plain attribute updates, never across a native call or
+ an acquisition of the operation lock, so it can be taken either alone
+ or inside the operation lock without an ordering cycle.
+
+ Falls back to a fresh lock when the attribute is missing.
+ """
+ lock = getattr(self, '_teardown_lock', None)
+ if lock is None:
+ lock = threading.Lock()
+ try:
+ self._teardown_lock = lock
+ lock = self._teardown_lock
+ except Exception:
+ pass
+ return lock
+
def _ensure_not_borrowed(self):
"""Raise if a native call is in flight on this handle.
@@ -320,38 +339,76 @@ def _ensure_not_borrowed(self):
f"{name} is in use by another operation and "
f"cannot be consumed")
+ def _ensure_no_mutating_call(self):
+ """Raise if a mutating native call is in flight on this handle.
+
+ Raises:
+ C2paError: when a mutating native call is in progress.
+ """
+ if getattr(self, '_mut_inflight', 0) > 0:
+ raise C2paError(
+ f"{type(self).__name__} is running a mutating operation")
+
@contextlib.contextmanager
- def _native_call(self):
- """Hold the handle valid across a native call that goes back
- and forth to native layers.
+ def _guarded_op(self, *, refuse_mut=True):
+ """Hold this resource's operation lock its duration,
+ and mark this thread as inside a native-error section.
- Calls that pass a Stream to the native library run caller-supplied
- callbacks, so the lock cannot be held across them. Instead the call
- is counted as in flight, and a teardown arriving meanwhile records
- its intent rather than freeing. The last caller out performs the free.
+ Note: Ordering is important and as the native section opens first
+ for the native call and closes last.
+
+ Never hold this across a native call that drives stream callbacks.
+ Those calls release the Global Interpreter Lock
+ and re-enter caller-supplied code, which may call back into this API
+ on another thread.
+ """
+ with _native_section():
+ try:
+ with self._live_op_lock():
+ if refuse_mut:
+ self._ensure_no_mutating_call()
+ yield
+ finally:
+ self._maybe_flush_pending()
- The resource is marked closed as soon as the teardown is recorded, so
- a caller that closed it cannot keep using it while the free is
- pending.
+ @contextlib.contextmanager
+ def _native_call(self):
+ """Hold the handle valid across a native call that runs
+ caller-supplied stream callbacks, so _live_op_lock() can't be held.
+ Count the call as in-flight/in-progress.
+ A free intent (teardown) is registered and the last caller frees.
+ A free intent marks the resource as closed, preventing further use.
"""
- with self._lock():
+ with self._live_op_lock():
self._ensure_valid_state()
self._inflight = getattr(self, '_inflight', 0) + 1
try:
- yield
+ with _native_section():
+ yield
finally:
- with self._lock():
+ with self._live_op_lock():
self._inflight -= 1
- pending = (self._pending_teardown
- if self._inflight == 0 else None)
- if pending is not None:
- self._pending_teardown = None
- # Released the lock before the free:
- # _teardown takes it again, and keeping the two acquisitions
- # separate means the counter update is never held across
- # the release work.
- if pending is not None:
- self._teardown(pending)
+ self._maybe_flush_pending()
+
+ @contextlib.contextmanager
+ def _exclusive_native_call(self):
+ """Exclusively marks this handle as being mutated.
+ A free intent (teardown) is registered and the last caller frees.
+ A free intent marks the resource as closed, preventing further use.
+ """
+ with self._live_op_lock():
+ self._ensure_valid_state()
+ self._ensure_no_mutating_call()
+ self._mut_inflight = getattr(self, '_mut_inflight', 0) + 1
+ self._inflight = getattr(self, '_inflight', 0) + 1
+ try:
+ with _native_section():
+ yield
+ finally:
+ with self._live_op_lock():
+ self._mut_inflight -= 1
+ self._inflight -= 1
+ self._maybe_flush_pending()
@staticmethod
def _free_native_ptr(ptr):
@@ -371,8 +428,10 @@ def _free_native_ptr(ptr):
result = _lib.c2pa_free(ptr)
if result != 0:
logger.debug(
- "c2pa_free returned %s for an untracked pointer ",
+ "c2pa_free returned %s for an untracked pointer",
result)
+ # Reset error slot.
+ _write_no_error_marker()
return result
def _ensure_valid_state(self):
@@ -409,73 +468,155 @@ def _teardown(self, free_handle: bool):
"""Close the object: run _release, optionally free the handle, null it.
free_handle=False (consumed) frees nothing, the new owner needs to free.
- Holds the operation lock so the free cannot happen between another
- thread's state check and its use of the handle in a native call.
+ The frees run under an operation lock.
+ Deferred when any gate is blocking:
+ - this resource's own handle is in flight in a native call
+ - this thread is inside a native-error section for some call
+ - someone else holds the operation lock (free intent gets queued)
The forked-child case is handled before the lock is taken, because
- _lock() raises in a child: this path has to finish rather than report
- an error, so it cannot rely on acquiring.
+ _live_op_lock() raises in a child: this path has to finish rather
+ than report an error, so it cannot rely on acquiring.
"""
if is_foreign_process(self):
- # The parent owns the handle and frees its own copy. Mark this one
- # closed and drop the pointer so the child cannot use or free it.
self._handle = None
self._lifecycle_state = LifecycleState.CLOSED
return
- with self._lock():
+ if getattr(self, '_released', False):
+ return
+ self._record_pending_intent(free_handle)
+
+ lock = self._live_op_lock()
+ if not lock.acquire(blocking=False):
+ self._close_lifecycle()
+ _register_for_section_flush(self)
+ return
+
+ try:
if getattr(self, '_released', False):
- # A racing close()/__del__ already ran the release branch
- # under this lock.
- # Idempotent: nothing left to release or free.
- # Keyed on the release having happened, not on CLOSED: the
- # deferred path below sets CLOSED without releasing, and still
- # owes a release performed by _native_call()'s finally.
+ # Checks released as it recorded possible free intents.
return
- if getattr(self, '_inflight', 0) > 0:
- # A native call is running that re-enters calling non-native
- # code and is still using this handle.
- # Record the intent and whichever caller leaves
- # _native_call last performs the free.
- # Mark the resource closed now so it cannot be used
- # while the free is pending.
- #
- # free_handle=False records that a consuming call handed
- # ownership to the native library. Ownership does not come
- # back, so a later teardown cannot restore the right to free:
- # the recorded value only ever moves True -> False, never the
- # reverse. Without this, a _teardown(True) arriving second
- # (from _release_handle, whose state check is read outside
- # this lock and can go stale) frees a pointer native owns.
- if self._pending_teardown is None:
- self._pending_teardown = free_handle
- else:
- self._pending_teardown = (
- self._pending_teardown and free_handle)
- self._lifecycle_state = LifecycleState.CLOSED
+ if getattr(self, '_inflight', 0) > 0 or _in_native_section():
+ # Closes the resource so it can't be used anymore.
+ # Records also pending actual frees.
+ self._close_lifecycle()
+ if _in_native_section():
+ _register_for_section_flush(self)
return
- self._released = True
+ with self._live_teardown_lock():
+ pending = getattr(self, '_pending_teardown', None)
+ if pending is not None:
+ free_handle = pending and free_handle
+ self._pending_teardown = None
+ self._finish_teardown(free_handle)
+ finally:
+ lock.release()
+
+ def _record_pending_intent(self, free_handle: bool):
+ """Queue a teardown intent, leaving the resource usable until
+ the intent runs.
+ A queued consume wins over a free, since native already owns a
+ consumed handle: freeing it again corrupts memory, where a missed
+ free only leaks.
+ """
+ with self._live_teardown_lock():
+ pending = getattr(self, '_pending_teardown', None)
+ if pending is None:
+ self._pending_teardown = free_handle
+ else:
+ self._pending_teardown = pending and free_handle
+
+ def _close_lifecycle(self):
+ """Close the resource so it can no longer be used."""
+ with self._live_teardown_lock():
self._lifecycle_state = LifecycleState.CLOSED
- self._safe_release()
- handle, self._handle = self._handle, None
- if free_handle and handle:
- try:
- # Subclasses may override the deallocator.
- type(self)._free_native_ptr(handle)
- except Exception:
- logger.error("Failed to free native %s resources",
- type(self).__name__, exc_info=True)
+ def _record_pending_teardown(self, free_handle: bool):
+ """Record a teardown intent and queue it.
+ Also closes the resource, and a queued consume wins
+ over a (new) teardown request.
+ """
+ self._record_pending_intent(free_handle)
+ self._close_lifecycle()
- def _release_handle(self):
- """Free this handle, then close the object. Used only where ownership is
- unknown (a guarded free is a real free if ours, a no-op if not).
+ def _finish_teardown(self, free_handle: bool):
+ """Once teardown can run, runs the actual release.
+ Steps: release, null the handle, free if requested.
"""
- if self._lifecycle_state != LifecycleState.ACTIVE:
+ if is_foreign_process(self):
self._handle = None
self._lifecycle_state = LifecycleState.CLOSED
return
+
+ if getattr(self, '_released', False):
+ # Already done by another caller (concurrent caller).
+ return
+
+ self._released = True
+ self._lifecycle_state = LifecycleState.CLOSED
+ self._safe_release()
+
+ handle, self._handle = self._handle, None
+ if free_handle and handle:
+ try:
+ ManagedResource._free_native_ptr(handle)
+ except Exception:
+ logger.error("Failed to free native %s resources",
+ type(self).__name__, exc_info=True)
+
+ def _has_pending_teardown(self) -> bool:
+ """Check if a teardown request is waiting for the resource."""
+ return getattr(self, '_pending_teardown', None) is not None
+
+ def _flush_pending_pass(self):
+ """Attempt to run pending teardowns.
+ """
+
+ with self._live_op_lock():
+ if getattr(self, '_pending_teardown', None) is None:
+ return
+ if getattr(self, '_inflight', 0) > 0:
+ return
+ if _in_native_section():
+ _register_for_section_flush(self)
+ return
+ with self._live_teardown_lock():
+ free_handle = self._pending_teardown
+ self._pending_teardown = None
+ self._finish_teardown(free_handle)
+
+ def _maybe_flush_pending(self):
+ """Recheck if a teardown can run after something
+ that blocked it cleared.
+ """
+ if is_foreign_process(self):
+ return
+
+ self._flush_pending_pass()
+ if self._has_pending_teardown() and not getattr(
+ self, '_released', False):
+ self._flush_pending_pass()
+
+ def _release_handle(self):
+ """Free this handle and close the object, unless a queued teardown
+ already owns the free. Used only where ownership is unknown
+ (a guarded free is a real free if ours, a no-op if not).
+ Nulling the handle under a queued teardown would leave it nothing
+ to free.
+ """
+ with self._live_op_lock():
+ owned_elsewhere = getattr(
+ self, '_pending_teardown', None) is not None
+ if not owned_elsewhere and (
+ self._lifecycle_state != LifecycleState.ACTIVE):
+ self._handle = None
+ self._lifecycle_state = LifecycleState.CLOSED
+ owned_elsewhere = True
+ if owned_elsewhere:
+ self._maybe_flush_pending()
+ return
self._teardown(free_handle=True)
def _activate(self, handle):
@@ -503,22 +644,21 @@ def _activate(self, handle):
def _create_and_activate(self, ffi_call, error_message, *,
check=lambda r: not r):
- """Obtain a fresh native pointer, validate it, and take ownership.
- On any failure before ownership transfers, the pointer is freed
- and the error re-raised.
+ """Get a new pointer/handle, validate, take ownership.
Args:
ffi_call: Zero-arg callable returning a fresh native pointer.
error_message: Message for the C2paError raised on failure.
- check: Predicate marking a result invalid
- (default: a falsy pointer).
+ check: Lambda determining result invalidity.
Raises:
- C2paError: If the pointer fails validation; it is freed first.
+ C2paError: If the pointer fails the validation step.
"""
- ptr = ffi_call()
+ ptr = None
try:
- _check_ffi_operation_result(ptr, error_message, check=check)
+ with _native_section():
+ ptr = ffi_call()
+ _check_ffi_operation_result(ptr, error_message, check=check)
self._activate(ptr)
except Exception:
if ptr:
@@ -526,40 +666,30 @@ def _create_and_activate(self, ffi_call, error_message, *,
raise
return ptr
- def _swap_handle(self, new_handle):
- """Replace the handle after an FFI call consumed the old one and
- returned a replacement.
- A null return from such a call is ambiguous (the callee may have
- failed validation before taking ownership, or failed the operation
- after), so callers must not call this with a null replacement.
- Requires the resource to be active.
-
- Args:
- new_handle: Non-null native pointer returned by the FFI call
-
- Raises:
- C2paError: If the resource is not ACTIVE or new_handle is null
- """
- name = type(self).__name__
- if self._lifecycle_state != LifecycleState.ACTIVE:
- raise C2paError(
- f"{name}: cannot swap the handle of a resource that is not "
- f"active ({self._lifecycle_state.name})")
- if not new_handle:
- raise C2paError(f"{name}: cannot swap in a null handle")
-
- self._handle = new_handle
-
# Errors set by native lib, hinting at the cause of the error
# These errors here means the pointer got somehow rejected by the lib,
# so it is still ours to deal with.
_PRE_CONSUME_ERROR_TAGS = (
"UntrackedPointer:",
"WrongPointerType:",
- "NullParameter:",
- "InvalidBufferSize:",
)
+ # An error tag starts the message or follows this one wrapper.
+ _NATIVE_ERROR_WRAPPER = "Other: "
+
+ @staticmethod
+ def _is_pre_consume_rejection(error: str) -> bool:
+ """True when native rejected the handle before taking ownership.
+
+ Anchored, not a substring search: native quotes caller text verbatim,
+ so a tag mid-message describes the caller's input.
+ """
+ body = error
+ if body.startswith(ManagedResource._NATIVE_ERROR_WRAPPER):
+ body = body[len(ManagedResource._NATIVE_ERROR_WRAPPER):]
+ return any(body.startswith(tag)
+ for tag in ManagedResource._PRE_CONSUME_ERROR_TAGS)
+
def _invoke_consume(self, ffi_call, error_message, *, reserved=False):
"""Run an FFI call that consumes this handle, returning its raw result.
@@ -582,6 +712,8 @@ def _invoke_consume(self, ffi_call, error_message, *, reserved=False):
ctypes.ArgumentError: If marshalling failed; handle untouched.
C2paError: If the call raised any other exception.
"""
+ # Same thread that makes the call, same thread-local slot.
+ _write_no_error_marker()
try:
return ffi_call(self._handle)
except ctypes.ArgumentError:
@@ -590,8 +722,7 @@ def _invoke_consume(self, ffi_call, error_message, *, reserved=False):
raise
except Exception as e:
if reserved:
- # A reservation leaves the resource CLOSED with the handle set,
- # which _release_handle() nulls without freeing.
+ # Resource left close (handle set).
self._teardown(free_handle=True)
else:
self._release_handle()
@@ -601,16 +732,12 @@ def _raise_consume_failure(self, error_message, previous_state=None):
"""Raise the error from an FFI handler consuming call.
The native error is read before any free so a free's own
- pointer-tracking error cannot overwrite it: the native error slot is
- sticky and thread-local and the SDK does not clear it before the call,
- so this trusts that the failing native path set its own error.
-
- That ordering is required:
- c2pa_free on a handle the registry no longer tracks returns -1 and
- overwrites the slot with its own "Other: UntrackedPointer: 0x..."
- message. Freeing first would therefore replace the real failure
- with another one and, because that substitute carries a pre-consume
- tag, invert the retain/consume decision made below.
+ pointer-tracking error cannot overwrite it.
+ The native error slot is sticky and thread-local,
+ and the native SDK does not clear it before the call.
+ _invoke_consume marks the slot as carrying no error right before a
+ consuming call, so a failure that sets no error of its own reads back
+ as no error rather than as a stale one left by an earlier call.
A caller that reserved the handle with _begin_consume() passes
previous_state and stays reserved until this classification finishes.
@@ -631,8 +758,7 @@ def _raise_consume_failure(self, error_message, previous_state=None):
"""
error = _read_native_error()
if error:
- if any(tag in error
- for tag in ManagedResource._PRE_CONSUME_ERROR_TAGS):
+ if ManagedResource._is_pre_consume_rejection(error):
logger.warning(
"%s: native call rejected the handle before taking "
"ownership (%s); handle retained",
@@ -649,7 +775,8 @@ def _raise_consume_failure(self, error_message, previous_state=None):
self._teardown(free_handle=False)
_raise_typed_c2pa_error(error)
- # No error in the slot: ownership is unknown, so free defensively.
+ # No error of its own: ownership is unknown, so free defensively.
+ # c2pa_free returns -1 for an address native already reclaimed.
# A reservation leaves the resource CLOSED with the handle set,
# which _release_handle() nulls without freeing.
if previous_state is not None:
@@ -660,60 +787,102 @@ def _raise_consume_failure(self, error_message, previous_state=None):
def _begin_consume(self):
"""Reserve this handle for a consuming call, or raise.
+ This is the initiation of an exclusive borrow.
+
+ Marks the resource as closed, stopping other borrows.
+ After this, the call is considered in-flight.
+ The caller owns the matching decrement.
Returns:
The lifecycle state to restore if the call turns out not to have
consumed the handle.
Raises:
- C2paError: If a native call is in flight on this resource.
+ C2paError: Unusable resource or native call in progress.
"""
- with self._lock():
+ with self._live_op_lock():
# A consumed or closed resource has no handle left to hand over;
# without this the call would pass a null pointer to native.
self._ensure_valid_state()
self._ensure_not_borrowed()
previous = self._lifecycle_state
self._lifecycle_state = LifecycleState.CLOSED
+ self._inflight = getattr(self, '_inflight', 0) + 1
return previous
def _abort_consume(self, previous_state):
"""Undo _begin_consume() after a call that did not take the handle.
- A pre-consume rejection leaves the handle ours,
- so the resource has to become usable again.
+ A pre-consume tag usually means the handle is still ours, so the
+ resource becomes usable again. The tag can also name another tracked
+ argument, which this does not distinguish.
+
+ A deferred free still happens when the section drains, so a resource
+ with a queued teardown stays closed.
"""
- with self._lock():
+ with self._live_op_lock():
+ if self._pending_teardown is not None:
+ return
if self._lifecycle_state == LifecycleState.CLOSED and self._handle:
self._lifecycle_state = previous_state
def _consume_and_swap(self, ffi_call, error_message):
- """Run an FFI call that consumes this handle and returns a replacement.
- On success the native lib consumed the handle and returned a new one,
- which we swap in. A null return is a failure.
-
- Unlike the consuming teardown paths this neither refuses a borrowed
- handle nor pre-marks the resource CLOSED: _swap_handle() requires it to
- stay ACTIVE, and the object remains usable afterwards with its new
- pointer.
+ """Run an FFI call consuming the handle, reserving it.
+ A replacement handle will be swapping in on success
+ (a returned null value is a failure).
"""
- new_ptr = self._invoke_consume(ffi_call, error_message)
- if new_ptr:
- try:
- self._swap_handle(new_ptr)
- except Exception:
- # _swap_handle refuses a resource a concurrent close() left
- # CLOSED. Native consumed the old pointer and returned this
- # one, so nothing else holds it.
- try:
- ManagedResource._free_native_ptr(new_ptr)
- except Exception:
- logger.error(
- "Failed to free the replacement %s handle",
- type(self).__name__, exc_info=True)
- raise
- return
- self._raise_consume_failure(error_message)
+
+ previous_state = self._begin_consume()
+ try:
+ with _native_section():
+ new_ptr = self._invoke_consume(
+ ffi_call, error_message, reserved=True)
+ if new_ptr:
+ with self._live_op_lock():
+ self._handle = new_ptr
+ if self._pending_teardown is None:
+ self._lifecycle_state = previous_state
+ return
+ self._raise_consume_failure(error_message, previous_state)
+ except BaseException:
+ self._abort_consume(previous_state)
+ raise
+ finally:
+ # Decrement to handle parallel potential in-flight consumers.
+ with self._live_op_lock():
+ self._inflight -= 1
+ self._maybe_flush_pending()
+
+ def _consume_reserved(self, ffi_call, error_message, *, succeeded):
+ """Run a reserved consuming call and mark the handle consumed on
+ success.
+
+ Args:
+ succeeded: Reads the call's raw result and returns whether it
+ succeeded. Each entry point has its own convention: a status
+ code, or a replacement pointer.
+
+ Returns:
+ The call's raw result, for callers that hand it on.
+ """
+ previous_state = self._begin_consume()
+ try:
+ with _native_section():
+ result = self._invoke_consume(
+ ffi_call, error_message, reserved=True)
+ if succeeded(result):
+ self._teardown(free_handle=False)
+ return result
+ self._raise_consume_failure(error_message, previous_state)
+ except BaseException:
+ self._abort_consume(previous_state)
+ raise
+ finally:
+ # Same order as _consume_and_swap: drop _inflight under the
+ # lock, then flush.
+ with self._live_op_lock():
+ self._inflight -= 1
+ self._maybe_flush_pending()
def _consume_no_replacement(self, ffi_call, error_message):
"""Run an FFI call that consumes this handle on success, when the native
@@ -721,17 +890,9 @@ def _consume_no_replacement(self, ffi_call, error_message):
handle. A non-zero status is a failure routed to
_raise_consume_failure.
"""
- previous_state = self._begin_consume()
- try:
- result = self._invoke_consume(
- ffi_call, error_message, reserved=True)
- except Exception:
- self._abort_consume(previous_state)
- raise
- if result == 0:
- self._teardown(free_handle=False)
- return
- self._raise_consume_failure(error_message, previous_state)
+ self._consume_reserved(
+ ffi_call, error_message,
+ succeeded=lambda status: status == 0)
def _consume_into(self, ffi_call, error_message):
"""Run an FFI call that consumes this handle and returns a *different*
@@ -739,17 +900,10 @@ def _consume_into(self, ffi_call, error_message):
and the new pointer is returned for the caller to own. A null return is
a failure routed to _raise_consume_failure.
"""
- previous_state = self._begin_consume()
- try:
- result = self._invoke_consume(
- ffi_call, error_message, reserved=True)
- except Exception:
- self._abort_consume(previous_state)
- raise
- if result:
- self._teardown(free_handle=False)
- return result
- self._raise_consume_failure(error_message, previous_state)
+ # A null pointer is falsy.
+ return self._consume_reserved(
+ ffi_call, error_message,
+ succeeded=lambda pointer: bool(pointer))
@classmethod
def _wrap_native_handle(cls, handle):
@@ -790,10 +944,8 @@ def _cleanup_resources(self):
if hasattr(self, '_lifecycle_state'):
self._lifecycle_state = LifecycleState.CLOSED
return
- if (
- hasattr(self, '_lifecycle_state')
- and self._lifecycle_state != LifecycleState.CLOSED
- ):
+ if hasattr(self, '_lifecycle_state'):
+ # Closes here must defer to the teardown checks.
self._teardown(free_handle=True)
except Exception:
pass
@@ -911,25 +1063,123 @@ class C2paStream(ctypes.Structure):
]
+# Address passed to c2pa_free to plant a marker in the native error slot.
+# 2 is not an allocatable address.
+_NO_ERROR_MARKER_ADDR = 2
+
+# Exact text the native lib writes for a failed free of _NO_ERROR_MARKER_ADDR.
+_NO_ERROR_MARKER_TEXT = None
+
+
+def _write_no_error_marker():
+ """A c2pa_free of an address the registry does not track writes
+ an expected error message learned at import into the
+ thread-local error slot and returns -1.
+
+ This marker mechanism exists to distinguish a consuming call that
+ failed without setting its own error from a stale message left
+ by an earlier call on the same thread.
+
+ No-op when the marker text could not be learned at import.
+ """
+ if _NO_ERROR_MARKER_TEXT is None:
+ return
+ _lib.c2pa_free(_NO_ERROR_MARKER_ADDR)
+
+
+def _is_no_error_marker(message: str) -> bool:
+ """True for the marker meaning "no current error of our own"."""
+ return message == _NO_ERROR_MARKER_TEXT
+
+
def _read_native_error() -> Optional[str]:
"""Read the last error from the native library, or None if unset.
- Peeks: the error stays in the native slot,
- until the next error overwrites it.
-
- With no error set the native side still returns an owned pointer to an
- empty string, so the pointer alone does not tell us whether there is an
- error. Only a non-empty message counts as one; the empty string still
- has to be freed.
+ The slot is marked as carrying no error before returning, so a
+ given error is reported once, by the caller that observes it. The native
+ slot is thread-local and sticky, so a message left in place stays readable
+ indefinitely and is available to be reported again by a later,
+ unrelated call that failed without setting an error of its own
+ (or a missing clear of an error slot).
"""
error = _lib.c2pa_error()
if not error:
+ # NULL means the message could not be rendered, not that the slot
+ # is empty, so it still has to be marked.
+ _write_no_error_marker()
return None
try:
message = ctypes.string_at(error).decode('utf-8')
finally:
_lib.c2pa_string_free(error)
- return message or None
+
+ _write_no_error_marker()
+ if not message or _is_no_error_marker(message):
+ return None
+ return message
+
+
+_native_section_state = threading.local()
+
+
+def _in_native_section() -> bool:
+ """True while this thread is between an FFI call and reading back the
+ native error it may have set (see _native_section())."""
+ return getattr(_native_section_state, 'depth', 0) > 0
+
+
+def _register_for_section_flush(resource):
+ """Record that `resource`'s teardown was deferred only because this
+ thread's native-error section was open."""
+ pending = getattr(_native_section_state, 'pending_resources', None)
+ if pending is not None:
+ pending.append(resource)
+
+
+@contextlib.contextmanager
+def _native_section():
+ """Mark this thread as inside a section where a native call's result is
+ about to be read back: an error-slot check, or a consuming call's
+ success/failure classification.
+
+ Reentrant: a nested native call on the same thread nests.
+ """
+ state = _native_section_state
+ depth = getattr(state, 'depth', 0)
+ state.depth = depth + 1
+ if depth == 0:
+ state.pending_resources = []
+
+ def _drain():
+ """Flush every deferred resource. Returns the first error raised."""
+ pending, state.pending_resources = state.pending_resources, []
+ first_error = None
+ for resource in pending:
+ try:
+ resource._maybe_flush_pending()
+ except BaseException as e: # noqa: BLE001
+ if first_error is None:
+ first_error = e
+ return first_error
+
+ try:
+ yield
+ except BaseException:
+ state.depth -= 1
+ if state.depth == 0:
+ drain_error = _drain()
+ if drain_error is not None:
+ logger.error(
+ "Deferred teardown failed while unwinding: %s",
+ drain_error)
+ raise
+ else:
+ state.depth -= 1
+ if state.depth == 0:
+ drain_error = _drain()
+ if drain_error is not None:
+ logger.error(
+ "Deferred teardown failed: %s", drain_error)
class C2paSignerInfo(ctypes.Structure):
@@ -969,6 +1219,7 @@ def __init__(self, alg, sign_cert, private_key, ta_url):
alg = alg_str
elif isinstance(alg, str):
# String to bytes, as requested by native lib
+ _check_cstr_arg("alg", alg)
alg = alg.encode('utf-8')
elif isinstance(alg, bytes):
# In bytes already
@@ -986,6 +1237,7 @@ def __init__(self, alg, sign_cert, private_key, ta_url):
pass
elif isinstance(ta_url, str):
# String to bytes, as requested by native lib
+ _check_cstr_arg("ta_url", ta_url)
ta_url = ta_url.encode('utf-8')
elif isinstance(ta_url, bytes):
# In bytes already
@@ -1254,6 +1506,41 @@ def _setup_function(func, argtypes, restype=None):
)
_setup_function(_lib.c2pa_free, [ctypes.c_void_p], ctypes.c_int)
+
+def _learn_no_error_marker_text():
+ """Plant the marker once and read back the exact text the native lib
+ produces for it, so equality checks match this build of the lib.
+
+ Runs on the importing thread; the text is a format constant, so the
+ learned value holds for every thread.
+
+ No-op/None if the marker couldn't be learned.
+ """
+ _lib.c2pa_free(_NO_ERROR_MARKER_ADDR)
+ raw = _lib.c2pa_error()
+ if not raw:
+ logger.warning(
+ "c2pa: could not find out error marker")
+ return None
+ try:
+ text = ctypes.string_at(raw).decode('utf-8')
+ finally:
+ _lib.c2pa_string_free(raw)
+ if not text:
+ logger.warning(
+ "c2pa: error-slot marker not set, some errors may be stale")
+ return None
+ marker_hex = hex(_NO_ERROR_MARKER_ADDR)
+ if marker_hex not in text:
+ logger.warning(
+ "c2pa: error-slot marker %s unclear, some errors may be stale",
+ marker_hex)
+ return None
+ return text
+
+
+_NO_ERROR_MARKER_TEXT = _learn_no_error_marker_text()
+
_setup_function(
_lib.c2pa_context_builder_set_signer,
[ctypes.POINTER(C2paContextBuilder), ctypes.POINTER(C2paSigner)],
@@ -1467,12 +1754,63 @@ def _convert_to_py_string(value) -> str:
# Ignore clean up issues
pass
except (ctypes.ArgumentError, TypeError, ValueError, OSError):
- # Invalid pointer type or value
+ # Invalid pointer type or value, gracefully handled by native lib.
+ try:
+ _lib.c2pa_string_free(value)
+ except Exception:
+ pass
return ""
return py_string
+def _check_cstr_arg(name: str, value) -> None:
+ """Reject a string argument the native layer would refuse.
+ Checking here keeps the rejection on this side of the boundary,
+ where the handle is known to be untouched.
+
+ Raises:
+ C2paError: With same message the native layer would have produced.
+ """
+ if value is None:
+ raise C2paError(f"NullParameter: {name}")
+
+ embedded_nul = (
+ '\x00' in value if isinstance(value, str) else b'\x00' in value)
+ if embedded_nul:
+ # ctypes truncates at the first NUL.
+ raise C2paError(f"NullParameter: {name} contains a null byte")
+
+
+def _check_handle_arg(name: str, handle) -> None:
+ """Reject a null handle argument.
+ Note: registry membership is not observable from Python,
+ so a tracked-but-invalid pointer still reaches native.
+
+ Raises:
+ C2paError: With same message the native layer would have produced.
+ """
+ if not handle:
+ raise C2paError(f"NullParameter: {name}")
+
+
+def _check_bytes_arg(name: str, buffer) -> None:
+ """Reject a byte buffer the native layer would refuse.
+
+ Native rejects a null pointer and any size outside 1..=isize::MAX.
+ An empty buffer reaches it as size 0.
+ Checking here keeps the rejection on this side of the boundary,
+ where the handle is known to be untouched.
+
+ Raises:
+ C2paError: With same message the native layer would have produced.
+ """
+ if buffer is None:
+ raise C2paError(f"NullParameter: {name}")
+ if len(buffer) == 0:
+ raise C2paError(f"InvalidBufferSize: 0 for '{name}'")
+
+
def _raise_typed_c2pa_error(error_str: str) -> None:
"""Parse an error string and raise the appropriate typed C2paError.
@@ -1670,16 +2008,19 @@ def load_settings(settings: Union[str, dict], format: str = "json") -> None:
raise C2paError(f"Failed to serialize settings to JSON: {e}")
try:
+ _check_cstr_arg("settings", settings_str)
+ _check_cstr_arg("format", format)
settings_bytes = settings_str.encode('utf-8')
format_bytes = format.encode('utf-8')
except (AttributeError, UnicodeEncodeError) as e:
raise C2paError(f"Failed to encode settings to UTF-8: {e}")
- result = _lib.c2pa_load_settings(settings_bytes, format_bytes)
- _check_ffi_operation_result(
- result,
- "Error loading settings",
- check=lambda r: r != 0)
+ with _native_section():
+ result = _lib.c2pa_load_settings(settings_bytes, format_bytes)
+ _check_ffi_operation_result(
+ result,
+ "Error loading settings",
+ check=lambda r: r != 0)
@contextlib.contextmanager
@@ -1687,11 +2028,8 @@ def _context_guard(context):
"""Hold a caller-supplied context valid across a native call.
ContextProvider requires only is_valid and execution_context.
- A provider that also manages a native handle,
- such as the built-in Context, offers _native_call,
- which counts the call in flight so a concurrent close() records
- its intent and defers the free until the call returns. A provider
- implementing just the two required properties runs without that guard.
+ _native_call may also be implemented on other handlers, and
+ will leverage managed resources capabilities accordingly.
"""
native_call = getattr(context, "_native_call", None)
if native_call is None:
@@ -1793,7 +2131,7 @@ def set(self, path: str, value: str) -> 'Settings':
path_bytes = _to_utf8_bytes(path, "settings path")
value_bytes = _to_utf8_bytes(value, "settings value")
- with self._lock():
+ with self._guarded_op():
self._ensure_valid_state()
_check_ffi_operation_result(
@@ -1819,7 +2157,7 @@ def update(
"""
data_bytes = _to_utf8_bytes(data, "settings data")
- with self._lock():
+ with self._guarded_op():
self._ensure_valid_state()
_check_ffi_operation_result(
@@ -1936,11 +2274,13 @@ def __init__(
# a successful build consumes it, so close() is then a no-op.
with self._NativeBuilder() as nb:
if settings is not None:
- _check_ffi_operation_result(
- _lib.c2pa_context_builder_set_settings(
- nb._handle, settings._c_settings),
- "Failed to set settings on Context",
- check=lambda r: r != 0)
+ # Count in-progress reads.
+ with nb._guarded_op(), settings._native_call():
+ _check_ffi_operation_result(
+ _lib.c2pa_context_builder_set_settings(
+ nb._handle, settings._c_settings),
+ "Failed to set settings on Context",
+ check=lambda r: r != 0)
if signer is not None:
# No in-flight guard around the hand-off: the consume
@@ -1950,10 +2290,9 @@ def __init__(
# also makes the consume refuse to start while another
# thread is borrowing the handle to sign with.
#
- # Pin the callback first: a rejected signer is retained,
- # not closed and leaked, and _release() nulls _callback_cb
- # once the signer is torn down.
+ # Retain a rejected signer for later teardown.
self._signer_callback_cb = signer._callback_cb
+ _check_handle_arg('builder', nb._handle)
signer._consume_no_replacement(
lambda h: _lib.c2pa_context_builder_set_signer(
nb._handle, h),
@@ -2289,7 +2628,9 @@ def __del__(self):
if is_foreign_process(self):
return
lock = getattr(self, '_close_lock', None)
- with lock if lock is not None else contextlib.nullcontext():
+ if lock is not None and not lock.acquire(blocking=False):
+ return
+ try:
# Only cleanup if not already closed and we have a valid stream
if hasattr(self, '_closed') and not self._closed:
stream = self._stream
@@ -2304,6 +2645,9 @@ def __del__(self):
self._stream = None
self._closed = True
self._initialized = False
+ finally:
+ if lock is not None:
+ lock.release()
except Exception:
# Destructors must not raise exceptions
pass
@@ -2316,7 +2660,7 @@ def close(self):
Errors during cleanup are logged but not raised to ensure cleanup.
Multiple calls to close() are handled gracefully.
"""
- # Checked before the lock, as _lock() and __del__ do:
+ # Checked before the lock, as _live_op_lock() and __del__ do:
# a child inherits _close_lock in whatever state it had at fork(),
# and the thread holding it does not exist there to release it.
if is_foreign_process(self):
@@ -2893,13 +3237,16 @@ def _init_from_context(self, context, format_or_path,
context.execution_context),
Reader._ERROR_MESSAGES['reader_error'])
+ _check_cstr_arg('format', format_arg)
+ _check_handle_arg('stream', self._own_stream._stream)
if manifest_data is not None:
+ _check_bytes_arg('manifest_data', manifest_data)
manifest_array = (
ctypes.c_ubyte *
len(manifest_data)).from_buffer_copy(manifest_data)
# Consume current reader,
# with manifest data and stream (C FFI pattern),
- # to create a new one (switch out)
+ # to switch it out using _consume_and_swap.
self._consume_and_swap(
lambda handle: (
_lib.c2pa_reader_with_manifest_data_and_stream(
@@ -2935,7 +3282,6 @@ def _init_attrs(self):
self._fragment_streams = []
# Serializes with_fragment against itself.
- # Held across the native call, unlike _op_lock. Only with_fragment takes it.
self._fragment_lock = threading.RLock()
# Caches for manifest JSON string and parsed data.
@@ -2990,7 +3336,7 @@ def _get_cached_manifest_data(self) -> Optional[dict]:
"""
# Locked so the cache fields can't be read and written
# across concurrent handle swaps.
- with self._lock():
+ with self._guarded_op():
if self._manifest_data_cache is None:
if self._manifest_json_str_cache is None:
self._manifest_json_str_cache = self.json()
@@ -3000,10 +3346,9 @@ def _get_cached_manifest_data(self) -> Optional[dict]:
self._manifest_json_str_cache
)
except json.JSONDecodeError:
- # Reset cache to reattempt read, possibly
+ # Next call should retry the read.
self._manifest_data_cache = None
self._manifest_json_str_cache = None
- # Failed to parse manifest JSON
return None
return self._manifest_data_cache
@@ -3029,27 +3374,19 @@ def with_fragment(self, format: Optional[str], stream,
C2paError: If there was an error processing the fragment.
On failure the native call may already have consumed the
underlying object, in which case this Reader is closed and
- cannot be retried: create a new one instead of reusing this
- instance.
+ cannot be retried: create a new one.
C2paError: If another thread is inside this method on the same
- Reader. This one leaves the Reader untouched, so the call can
- be retried once that thread returns.
+ Reader, or another native call is in flight on it.
"""
format_arg = _format_ffi_arg(_encode_format(format, "Reader"))
# A forked child cannot wait on a lock no surviving thread will
- # release, so it reports the same error _lock() does.
+ # release, so it reports the same error _live_op_lock() does.
if is_foreign_process(self):
raise C2paError(f"{type(self).__name__} is closed")
# The native call and the ownership transfer are one unit.
- # Taken without blocking because the call drives caller-supplied stream
- # callbacks: a second thread, including one a callback starts, would
- # otherwise wait here for a native call that is itself waiting on that
- # callback to return.
- #
- # Reentrant, so the thread already inside this region passes through
- # and re-enters the native call, which rejects the handle it consumed.
+ # Reentrant so a thread already here can continue.
if not self._fragment_lock.acquire(blocking=False):
raise C2paError(
f"{type(self).__name__} is already processing a fragment "
@@ -3057,25 +3394,27 @@ def with_fragment(self, format: Optional[str], stream,
try:
# The native reader keeps reading through both streams.
main_obj = Stream(stream)
- frag_obj = Stream(fragment_stream)
+ frag_obj = None
try:
- with self._native_call():
- self._consume_and_swap(
- lambda handle: _lib.c2pa_reader_with_fragment(
- handle,
- format_arg,
- main_obj._stream,
- frag_obj._stream,
- ),
- Reader._ERROR_MESSAGES['fragment_error'])
+ frag_obj = Stream(fragment_stream)
+ _check_cstr_arg('format', format_arg)
+ _check_handle_arg('stream', main_obj._stream)
+ _check_handle_arg('fragment', frag_obj._stream)
+ self._consume_and_swap(
+ lambda handle: _lib.c2pa_reader_with_fragment(
+ handle,
+ format_arg,
+ main_obj._stream,
+ frag_obj._stream,
+ ),
+ Reader._ERROR_MESSAGES['fragment_error'])
except Exception:
main_obj.close()
- frag_obj.close()
+ if frag_obj is not None:
+ frag_obj.close()
raise
- # Locked so a concurrent close() cannot run _release()
- # between the check and the field swap.
- with self._lock():
+ with self._guarded_op(refuse_mut=False):
try:
self._ensure_valid_state()
except Exception:
@@ -3123,11 +3462,9 @@ def json(self) -> str:
C2paError: If there was an error getting the JSON
"""
- # Lock due to checks on native handles.
- with self._lock():
+ with self._guarded_op():
self._ensure_valid_state()
- # Return cached result if available
if self._manifest_json_str_cache is not None:
return self._manifest_json_str_cache
@@ -3135,7 +3472,6 @@ def json(self) -> str:
_check_ffi_operation_result(
result, "Error during manifest parsing in Reader")
- # Cache the result and return it
self._manifest_json_str_cache = _convert_to_py_string(result)
return self._manifest_json_str_cache
@@ -3155,7 +3491,7 @@ def detailed_json(self) -> str:
the Reader has been closed.
"""
- with self._lock():
+ with self._guarded_op():
self._ensure_valid_state()
result = _lib.c2pa_reader_detailed_json(self._handle)
@@ -3178,7 +3514,7 @@ def crjson(self) -> str:
call returns null.
"""
- with self._lock():
+ with self._guarded_op():
self._ensure_valid_state()
result = _lib.c2pa_reader_crjson(self._handle)
@@ -3300,8 +3636,9 @@ def resource_to_stream(self, uri: str, stream: Any) -> int:
Raises:
C2paError: If there was an error writing the resource to stream
"""
+ _check_cstr_arg("uri", uri)
uri_str = uri.encode('utf-8')
- with self._native_call(), Stream(stream) as stream_obj:
+ with self._exclusive_native_call(), Stream(stream) as stream_obj:
result = _lib.c2pa_reader_resource_to_stream(
self._handle, uri_str, stream_obj._stream)
@@ -3322,7 +3659,7 @@ def is_embedded(self) -> bool:
Raises:
C2paError: If there was an error checking the embedded status
"""
- with self._lock():
+ with self._guarded_op():
self._ensure_valid_state()
result = _lib.c2pa_reader_is_embedded(self._handle)
@@ -3340,18 +3677,16 @@ def get_remote_url(self) -> Optional[str]:
Raises:
C2paError: If there was an error getting the remote URL
"""
- with self._lock():
+ with self._guarded_op():
self._ensure_valid_state()
result = _lib.c2pa_reader_remote_url(self._handle)
if result is None:
- # No remote URL set (manifest is embedded)
+ # No remote URL set (manifest is embedded).
return None
- # Convert the C string to Python string
- url_str = _convert_to_py_string(result)
- return url_str
+ return _convert_to_py_string(result)
class Signer(ManagedResource):
@@ -3379,10 +3714,12 @@ def from_info(cls, signer_info: C2paSignerInfo) -> 'Signer':
Raises:
C2paError: If there was an error creating the signer
"""
- signer_ptr = _lib.c2pa_signer_from_info(ctypes.byref(signer_info))
+ with _native_section():
+ signer_ptr = _lib.c2pa_signer_from_info(ctypes.byref(signer_info))
- _check_ffi_operation_result(
- signer_ptr, "Failed to create signer from configured signer_info")
+ _check_ffi_operation_result(
+ signer_ptr,
+ "Failed to create signer from configured signer_info")
try:
return cls(signer_ptr)
@@ -3505,16 +3842,17 @@ def wrapped_callback(
callback_cb = SignerCallback(wrapped_callback)
# Create the signer with the wrapped callback
- signer_ptr = _lib.c2pa_signer_create(
- None,
- callback_cb,
- alg,
- certs_bytes,
- tsa_url_bytes
- )
+ with _native_section():
+ signer_ptr = _lib.c2pa_signer_create(
+ None,
+ callback_cb,
+ alg,
+ certs_bytes,
+ tsa_url_bytes
+ )
- _check_ffi_operation_result(signer_ptr,
- "Failed to create signer")
+ _check_ffi_operation_result(signer_ptr,
+ "Failed to create signer")
try:
# Create and return the signer instance with the callback
@@ -3569,7 +3907,7 @@ def reserve_size(self) -> int:
Raises:
C2paError: If there was an error getting the size
"""
- with self._lock():
+ with self._guarded_op():
self._ensure_valid_state()
result = _lib.c2pa_signer_reserve_size(self._handle)
@@ -3676,11 +4014,11 @@ def from_archive(
stream_obj = Stream(stream)
try:
- handle = _lib.c2pa_builder_from_archive(stream_obj._stream)
+ with _native_section():
+ handle = _lib.c2pa_builder_from_archive(stream_obj._stream)
- _check_ffi_operation_result(handle,
- "Failed to create builder from archive"
- )
+ _check_ffi_operation_result(
+ handle, "Failed to create builder from archive")
try:
# A builder from an archive here carries no context.
@@ -3759,6 +4097,8 @@ def _init_from_context(self, context, json_str):
context.execution_context),
Builder._ERROR_MESSAGES['builder_error'])
+ _check_cstr_arg('manifest_json', json_str)
+ # _consume_and_swap reserves the handle.
self._consume_and_swap(
lambda handle: _lib.c2pa_builder_with_definition(
handle, json_str),
@@ -3781,7 +4121,7 @@ def set_no_embed(self):
into the asset when signing.
This is useful when creating cloud or sidecar manifests.
"""
- with self._lock():
+ with self._guarded_op():
self._ensure_valid_state()
_lib.c2pa_builder_set_no_embed(self._handle)
@@ -3799,7 +4139,7 @@ def set_remote_url(self, remote_url: str):
"""
url_bytes = _to_utf8_bytes(remote_url, "remote URL")
- with self._lock():
+ with self._guarded_op():
self._ensure_valid_state()
result = _lib.c2pa_builder_set_remote_url(self._handle, url_bytes)
@@ -3835,7 +4175,7 @@ def set_intent(
Raises:
C2paError: If there was an error setting the intent
"""
- with self._lock():
+ with self._guarded_op():
self._ensure_valid_state()
result = _lib.c2pa_builder_set_intent(
@@ -3861,7 +4201,7 @@ def add_resource(self, uri: str, stream: Any):
C2paError: If there was an error adding the resource
"""
uri_bytes = _to_utf8_bytes(uri, "resource URI")
- with self._native_call(), Stream(stream) as stream_obj:
+ with self._exclusive_native_call(), Stream(stream) as stream_obj:
result = _lib.c2pa_builder_add_resource(
self._handle, uri_bytes, stream_obj._stream)
@@ -3922,7 +4262,7 @@ def add_ingredient_from_stream(
ingredient_str = _to_utf8_bytes(ingredient_json, "ingredient JSON")
format_str = _to_utf8_bytes(format, "ingredient format")
- with self._native_call(), Stream(source) as source_stream:
+ with self._exclusive_native_call(), Stream(source) as source_stream:
result = (
_lib.c2pa_builder_add_ingredient_from_stream(
self._handle,
@@ -3951,7 +4291,7 @@ def add_action(self, action_json: Union[str, dict]) -> None:
"""
action_str = _to_utf8_bytes(action_json, "action JSON")
- with self._lock():
+ with self._guarded_op():
self._ensure_valid_state()
result = _lib.c2pa_builder_add_action(self._handle, action_str)
@@ -3971,7 +4311,7 @@ def to_archive(self, stream: Any) -> None:
Raises:
C2paError: If there was an error writing the archive
"""
- with self._native_call(), Stream(stream) as stream_obj:
+ with self._exclusive_native_call(), Stream(stream) as stream_obj:
result = _lib.c2pa_builder_to_archive(
self._handle, stream_obj._stream)
@@ -3996,7 +4336,7 @@ def write_ingredient_archive(self, ingredient_id: str, stream: Any) -> None:
ingredient_id_str = _to_utf8_bytes(ingredient_id, "ingredient_id")
- with self._native_call(), Stream(stream) as stream_obj:
+ with self._exclusive_native_call(), Stream(stream) as stream_obj:
result = _lib.c2pa_builder_write_ingredient_archive(
self._handle, ingredient_id_str, stream_obj._stream)
@@ -4016,7 +4356,7 @@ def add_ingredient_from_archive(self, stream: Any) -> None:
Raises:
C2paError: If there was an error reading the archive
"""
- with self._native_call(), Stream(stream) as stream_obj:
+ with self._exclusive_native_call(), Stream(stream) as stream_obj:
result = _lib.c2pa_builder_add_ingredient_from_archive(
self._handle, stream_obj._stream)
@@ -4041,11 +4381,13 @@ def with_archive(self, stream: Any) -> 'Builder':
C2paError: If there was an error loading the archive. On failure
the native call may already have consumed the underlying
object, in which case this Builder is closed and cannot be
- retried: create a new one instead of reusing this instance.
+ retried: create a new one.
+ C2paError: If another native call is in flight on this Builder.
"""
self._ensure_valid_state()
- with self._native_call(), Stream(stream) as stream_obj:
+ with Stream(stream) as stream_obj:
+ _check_handle_arg('stream', stream_obj._stream)
self._consume_and_swap(
lambda handle: _lib.c2pa_builder_with_archive(
handle, stream_obj._stream),
@@ -4093,15 +4435,9 @@ def _sign_internal(
manifest_bytes_ptr = ctypes.POINTER(ctypes.c_ubyte)()
try:
- # _native_call covers the signing call only.
- # The close() below is deliberately outside it,
- # so the deferred teardown it records is performed
- # on the way out rather than being deferred forever.
- with self._native_call():
+ # Signing needs short guard sections (a Signer can be used in parallel).
+ with self._exclusive_native_call():
if signer is not None:
- # Signer needs its own in-flight guard.
- # Entered inside self's guard so concurrent signs
- # sharing objects (Signers) acquire in one order.
with signer._native_call():
result = _lib.c2pa_builder_sign(
self._handle,
@@ -4113,14 +4449,7 @@ def _sign_internal(
)
else:
# The Context pins the consumed signer's callback, which
- # native invokes during this call.
- # Its in-flight guard defers a close() arriving on another
- # thread, the same way the signer branch above defers one
- # for a borrowed Signer.
- #
- # Entered inside self's guard, matching the Builder to
- # Signer order, so the two acquisitions are always
- # taken in one direction.
+ # native invokes during this call
with _context_guard(self._context):
result = _lib.c2pa_builder_sign_context(
self._handle,
@@ -4129,18 +4458,20 @@ def _sign_internal(
dest_stream._stream,
ctypes.byref(manifest_bytes_ptr),
)
- # Sign borrows the Builder without taking ownership.
- # Closing here ensures resources clean up,
- # and single use/single sign done by a Builder.
- self.close()
except Exception as e:
self.close()
raise C2paError(f"Error during signing: {e}") from e
- _check_ffi_operation_result(
- result,
- "Error during signing",
- check=lambda r: r < 0)
+ try:
+ # _native_call already closed, so close() can free.
+ with _native_section():
+ _check_ffi_operation_result(
+ result,
+ "Error during signing",
+ check=lambda r: r < 0)
+ finally:
+ # Single use for a Builder, once signed, close.
+ self.close()
# Capture the manifest bytes if available
manifest_bytes = b""
@@ -4366,25 +4697,30 @@ def format_embeddable(format: str, manifest_bytes: bytes) -> tuple[int, bytes]:
Raises:
C2paError: If there was an error converting the manifest
"""
+ _check_cstr_arg("format", format)
format_str = format.encode('utf-8')
manifest_array = (ctypes.c_ubyte * len(manifest_bytes)).from_buffer_copy(
manifest_bytes
)
result_bytes_ptr = ctypes.POINTER(ctypes.c_ubyte)()
- result = _lib.c2pa_format_embeddable(
- format_str,
- manifest_array,
- len(manifest_bytes),
- ctypes.byref(result_bytes_ptr)
- )
+ with _native_section():
+ result = _lib.c2pa_format_embeddable(
+ format_str,
+ manifest_array,
+ len(manifest_bytes),
+ ctypes.byref(result_bytes_ptr)
+ )
- _check_ffi_operation_result(
- result,
- "Failed to format embeddable manifest",
- check=lambda r: r < 0)
+ _check_ffi_operation_result(
+ result,
+ "Failed to format embeddable manifest",
+ check=lambda r: r < 0)
size = result
+ if not result_bytes_ptr:
+ raise C2paError(
+ "Failed to format embeddable manifest: no data returned")
try:
result_bytes = ctypes.string_at(result_bytes_ptr, size)
except Exception as e:
@@ -4498,20 +4834,22 @@ def ed25519_sign(data: bytes, private_key: str) -> bytes:
# Encode private key to bytes
try:
+ _check_cstr_arg("private_key", private_key)
key_bytes = private_key.encode('utf-8')
except UnicodeError as e:
raise C2paError.Encoding(
f"Invalid UTF-8 characters in private key: {str(e)}")
# Perform the signing operation
- signature_ptr = _lib.c2pa_ed25519_sign(
- data_array,
- data_size,
- key_bytes
- )
+ with _native_section():
+ signature_ptr = _lib.c2pa_ed25519_sign(
+ data_array,
+ data_size,
+ key_bytes
+ )
- _check_ffi_operation_result(signature_ptr,
- "Failed to sign data with Ed25519")
+ _check_ffi_operation_result(signature_ptr,
+ "Failed to sign data with Ed25519")
try:
# Ed25519 signatures are always 64 bytes
diff --git a/tests/perf/baseline.json b/tests/perf/baseline.json
index c151efe5..5d7017ee 100644
--- a/tests/perf/baseline.json
+++ b/tests/perf/baseline.json
@@ -2,299 +2,304 @@
"_meta": {
"memray_version": "1.19.3",
"python_version": "3.12.13",
- "c2pa_native_version": "c2pa-v0.90.0",
+ "c2pa_native_version": "c2pa-v0.90.16",
"iterations": 200,
"perf_env": "python-3.12-slim",
"arch": "aarch64"
},
"reader_jpeg_legacy": {
- "peak_bytes": 3851610,
- "leaked_bytes": 3351823,
- "total_allocations": 1362322
+ "peak_bytes": 3912724,
+ "leaked_bytes": 3414162,
+ "total_allocations": 1324293
},
"reader_jpeg_with_context": {
- "peak_bytes": 3845367,
- "leaked_bytes": 3345097,
- "total_allocations": 1349879
+ "peak_bytes": 3907256,
+ "leaked_bytes": 3407478,
+ "total_allocations": 1333897
},
"reader_manifest_data_context": {
- "peak_bytes": 7636730,
- "leaked_bytes": 3468040,
- "total_allocations": 1147359
+ "peak_bytes": 7692972,
+ "leaked_bytes": 3524955,
+ "total_allocations": 1132827
},
"reader_mp4": {
- "peak_bytes": 4222601,
- "leaked_bytes": 3345724,
- "total_allocations": 4095915
+ "peak_bytes": 4272788,
+ "leaked_bytes": 3406355,
+ "total_allocations": 4018933
},
"reader_wav": {
- "peak_bytes": 4523095,
- "leaked_bytes": 3355666,
- "total_allocations": 742391
+ "peak_bytes": 4573253,
+ "leaked_bytes": 3416313,
+ "total_allocations": 773409
},
"builder_sign_jpeg_legacy": {
- "peak_bytes": 7785129,
- "leaked_bytes": 3468507,
- "total_allocations": 1041412
+ "peak_bytes": 7844930,
+ "leaked_bytes": 3530986,
+ "total_allocations": 1046607
},
"builder_sign_jpeg_with_context": {
- "peak_bytes": 7779538,
- "leaked_bytes": 3463042,
- "total_allocations": 1027485
+ "peak_bytes": 7839456,
+ "leaked_bytes": 3524460,
+ "total_allocations": 1058202
},
"builder_sign_png_legacy": {
- "peak_bytes": 8023081,
- "leaked_bytes": 3468300,
- "total_allocations": 3883115
+ "peak_bytes": 8082891,
+ "leaked_bytes": 3530892,
+ "total_allocations": 3888499
},
"builder_sign_png_with_context": {
- "peak_bytes": 8017008,
- "leaked_bytes": 3462829,
- "total_allocations": 3869515
+ "peak_bytes": 8077349,
+ "leaked_bytes": 3524774,
+ "total_allocations": 3900456
},
"builder_sign_jpeg_parallel_split_pool": {
- "peak_bytes": 45854797,
- "leaked_bytes": 3840928,
- "total_allocations": 1035646
+ "peak_bytes": 45936892,
+ "leaked_bytes": 3893837,
+ "total_allocations": 1062520
},
"builder_sign_jpeg_parallel_split_barrier": {
- "peak_bytes": 45844809,
- "leaked_bytes": 3861014,
- "total_allocations": 1037741
+ "peak_bytes": 45905378,
+ "leaked_bytes": 3892593,
+ "total_allocations": 1061149
},
"builder_sign_png_parallel_split_pool": {
- "peak_bytes": 46586728,
- "leaked_bytes": 3868054,
- "total_allocations": 3877696
+ "peak_bytes": 46673225,
+ "leaked_bytes": 3929128,
+ "total_allocations": 3904507
},
"builder_sign_png_parallel_split_barrier": {
- "peak_bytes": 46082548,
- "leaked_bytes": 3879161,
- "total_allocations": 3879780
+ "peak_bytes": 46143013,
+ "leaked_bytes": 3910964,
+ "total_allocations": 3903125
},
"builder_sign_gif": {
- "peak_bytes": 14635465,
- "leaked_bytes": 3461270,
- "total_allocations": 17017654
+ "peak_bytes": 14696646,
+ "leaked_bytes": 3524513,
+ "total_allocations": 17048351
},
"builder_sign_heic": {
- "peak_bytes": 4698434,
- "leaked_bytes": 3469086,
- "total_allocations": 1563419
+ "peak_bytes": 4759642,
+ "leaked_bytes": 3532315,
+ "total_allocations": 1582361
},
"builder_sign_m4a": {
- "peak_bytes": 18833496,
- "leaked_bytes": 3469085,
- "total_allocations": 5194205
+ "peak_bytes": 18895243,
+ "leaked_bytes": 3532373,
+ "total_allocations": 5213365
},
"builder_sign_webp": {
- "peak_bytes": 8991237,
- "leaked_bytes": 3461271,
- "total_allocations": 916145
+ "peak_bytes": 9052463,
+ "leaked_bytes": 3524559,
+ "total_allocations": 950737
},
"builder_sign_avi": {
- "peak_bytes": 7130933,
- "leaked_bytes": 3461270,
- "total_allocations": 89982012
+ "peak_bytes": 7192106,
+ "leaked_bytes": 3524502,
+ "total_allocations": 90011891
},
"builder_sign_mp4": {
- "peak_bytes": 6245379,
- "leaked_bytes": 3469085,
- "total_allocations": 3788717
+ "peak_bytes": 6306630,
+ "leaked_bytes": 3532325,
+ "total_allocations": 3805992
},
"builder_sign_tiff": {
- "peak_bytes": 13213169,
- "leaked_bytes": 3461271,
- "total_allocations": 10862700
+ "peak_bytes": 13274395,
+ "leaked_bytes": 3524559,
+ "total_allocations": 10898003
},
"builder_sign_jpeg_parent_of": {
- "peak_bytes": 14265295,
- "leaked_bytes": 3461665,
- "total_allocations": 2506107
+ "peak_bytes": 14324563,
+ "leaked_bytes": 3525074,
+ "total_allocations": 2495210
},
"builder_sign_jpeg_component_of": {
- "peak_bytes": 14266996,
- "leaked_bytes": 3462012,
- "total_allocations": 2551180
+ "peak_bytes": 14325966,
+ "leaked_bytes": 3524803,
+ "total_allocations": 2538825
},
"builder_sign_jpeg_parent_and_component": {
- "peak_bytes": 14665241,
- "leaked_bytes": 3614613,
- "total_allocations": 4523960
+ "peak_bytes": 14605703,
+ "leaked_bytes": 3610907,
+ "total_allocations": 4464450
},
"builder_sign_jpeg_parent_and_component_mixed_mime": {
- "peak_bytes": 14568780,
- "leaked_bytes": 3462718,
- "total_allocations": 5517180
+ "peak_bytes": 14627596,
+ "leaked_bytes": 3525478,
+ "total_allocations": 5516963
},
"builder_sign_jpeg_two_components_same_mime": {
- "peak_bytes": 14559274,
- "leaked_bytes": 3564233,
- "total_allocations": 4497379
+ "peak_bytes": 14602589,
+ "leaked_bytes": 3610897,
+ "total_allocations": 4436718
},
"builder_sign_jpeg_two_components_mixed_mime": {
- "peak_bytes": 14564839,
- "leaked_bytes": 3461873,
- "total_allocations": 5490592
+ "peak_bytes": 14624276,
+ "leaked_bytes": 3525287,
+ "total_allocations": 5489138
},
"builder_sign_jpeg_archive_roundtrip": {
- "peak_bytes": 14297571,
- "leaked_bytes": 3481212,
- "total_allocations": 3467149
+ "peak_bytes": 14356429,
+ "leaked_bytes": 3545507,
+ "total_allocations": 3432412
},
"builder_from_archive_roundtrip": {
- "peak_bytes": 14297349,
- "leaked_bytes": 3480475,
- "total_allocations": 3101030
+ "peak_bytes": 14353258,
+ "leaked_bytes": 3542427,
+ "total_allocations": 3024501
},
"builder_with_archive_swap": {
- "peak_bytes": 3681081,
- "leaked_bytes": 3350198,
- "total_allocations": 704373
+ "peak_bytes": 3753525,
+ "leaked_bytes": 3421374,
+ "total_allocations": 744039
},
"reader_with_fragment_swap": {
- "peak_bytes": 3778159,
- "leaked_bytes": 3353205,
- "total_allocations": 3787587
+ "peak_bytes": 3839453,
+ "leaked_bytes": 3414104,
+ "total_allocations": 3806570
},
"with_fragment_pre_consume_rejection": {
- "peak_bytes": 3778057,
- "leaked_bytes": 3354795,
- "total_allocations": 2094004
+ "peak_bytes": 3841126,
+ "leaked_bytes": 3417826,
+ "total_allocations": 2128201
},
"with_archive_post_consume_failure": {
- "peak_bytes": 3350600,
- "leaked_bytes": 3308056,
- "total_allocations": 175290
+ "peak_bytes": 3423615,
+ "leaked_bytes": 3380332,
+ "total_allocations": 208578
},
"with_fragment_marshalling_error": {
- "peak_bytes": 3708068,
- "leaked_bytes": 3352335,
- "total_allocations": 2077090
+ "peak_bytes": 3767911,
+ "leaked_bytes": 3413267,
+ "total_allocations": 2094661
},
"with_fragment_mixed_outcomes": {
- "peak_bytes": 3779175,
- "leaked_bytes": 3356294,
- "total_allocations": 2656787
+ "peak_bytes": 3840297,
+ "leaked_bytes": 3417238,
+ "total_allocations": 2686269
},
"builder_to_archive_with_ingredient": {
- "peak_bytes": 14069232,
- "leaked_bytes": 3337316,
- "total_allocations": 1830896
+ "peak_bytes": 14142488,
+ "leaked_bytes": 3409388,
+ "total_allocations": 1790801
},
"builder_sign_jpeg_archive_roundtrip_ingredient_in_archive": {
- "peak_bytes": 14287046,
- "leaked_bytes": 3481977,
- "total_allocations": 5879957
+ "peak_bytes": 14345054,
+ "leaked_bytes": 3543673,
+ "total_allocations": 5769615
},
"builder_write_ingredient_archive": {
- "peak_bytes": 14069289,
- "leaked_bytes": 3337377,
- "total_allocations": 1805304
+ "peak_bytes": 14142437,
+ "leaked_bytes": 3409341,
+ "total_allocations": 1767419
},
"builder_sign_jpeg_add_ingredient_from_archive": {
- "peak_bytes": 14133742,
- "leaked_bytes": 3480831,
- "total_allocations": 3415920
+ "peak_bytes": 14207929,
+ "leaked_bytes": 3544945,
+ "total_allocations": 3383511
},
"builder_ingredient_archive_roundtrip": {
- "peak_bytes": 14284443,
- "leaked_bytes": 3480809,
- "total_allocations": 5132060
+ "peak_bytes": 14345163,
+ "leaked_bytes": 3545508,
+ "total_allocations": 5061203
},
"builder_sign_jpeg_two_ingredient_archives": {
- "peak_bytes": 14134560,
- "leaked_bytes": 3481604,
- "total_allocations": 4215728
+ "peak_bytes": 14208534,
+ "leaked_bytes": 3545923,
+ "total_allocations": 4185124
},
"reader_error_no_manifest": {
- "peak_bytes": 3564471,
- "leaked_bytes": 3323629,
- "total_allocations": 276175
+ "peak_bytes": 3622191,
+ "leaked_bytes": 3383889,
+ "total_allocations": 291735
},
"builder_error_invalid_manifest": {
- "peak_bytes": 3352053,
- "leaked_bytes": 3297079,
- "total_allocations": 113926
+ "peak_bytes": 3421406,
+ "leaked_bytes": 3365544,
+ "total_allocations": 126199
},
"reader_string_apis": {
- "peak_bytes": 3978113,
- "leaked_bytes": 3346111,
- "total_allocations": 2287335
+ "peak_bytes": 4039136,
+ "leaked_bytes": 3407512,
+ "total_allocations": 2238705
},
"signer_construction": {
- "peak_bytes": 3350893,
- "leaked_bytes": 3288137,
- "total_allocations": 153245
+ "peak_bytes": 3421644,
+ "leaked_bytes": 3358098,
+ "total_allocations": 159717
},
"builder_from_context_construction": {
- "peak_bytes": 3350600,
- "leaked_bytes": 3288582,
- "total_allocations": 112688
+ "peak_bytes": 3423152,
+ "leaked_bytes": 3360708,
+ "total_allocations": 146014
},
"fork_reader_collect": {
- "peak_bytes": 3850530,
- "leaked_bytes": 3353063,
- "total_allocations": 1328122
+ "peak_bytes": 3911936,
+ "leaked_bytes": 3413740,
+ "total_allocations": 1284294
},
"fork_contended_mutex": {
- "peak_bytes": 7679019,
- "leaked_bytes": 3482128,
- "total_allocations": 67472694
+ "peak_bytes": 7700288,
+ "leaked_bytes": 3510073,
+ "total_allocations": 66621221
},
"fork_thread_local_orphan": {
- "peak_bytes": 3936170,
- "leaked_bytes": 3439733,
- "total_allocations": 1381055
+ "peak_bytes": 4073730,
+ "leaked_bytes": 3581333,
+ "total_allocations": 1339630
},
"fork_gc_cycle": {
- "peak_bytes": 3850434,
- "leaked_bytes": 3353160,
- "total_allocations": 1332098
+ "peak_bytes": 3913068,
+ "leaked_bytes": 3414664,
+ "total_allocations": 1289268
},
"fork_parent_frees_after_fork": {
- "peak_bytes": 5447584,
- "leaked_bytes": 3350400,
- "total_allocations": 24829257
+ "peak_bytes": 5602620,
+ "leaked_bytes": 3423572,
+ "total_allocations": 23965279
},
"fork_child_closes_then_parent_frees": {
- "peak_bytes": 5446620,
- "leaked_bytes": 3350407,
- "total_allocations": 24829254
+ "peak_bytes": 5603711,
+ "leaked_bytes": 3424393,
+ "total_allocations": 23965271
},
"fork_child_sys_exit": {
- "peak_bytes": 3850546,
- "leaked_bytes": 3353234,
- "total_allocations": 1335925
+ "peak_bytes": 3911952,
+ "leaked_bytes": 3413956,
+ "total_allocations": 1301497
},
"fork_stream_cleanup": {
- "peak_bytes": 3464063,
- "leaked_bytes": 3291969,
- "total_allocations": 105340
+ "peak_bytes": 3532824,
+ "leaked_bytes": 3361106,
+ "total_allocations": 110397
},
"fork_swap_cleanup": {
- "peak_bytes": 3681171,
- "leaked_bytes": 3350696,
- "total_allocations": 714376
+ "peak_bytes": 3753679,
+ "leaked_bytes": 3421936,
+ "total_allocations": 754042
},
"fork_contended_mutex_swap": {
- "peak_bytes": 7302379,
- "leaked_bytes": 3475147,
- "total_allocations": 35948516
+ "peak_bytes": 7360409,
+ "leaked_bytes": 3525035,
+ "total_allocations": 37359891
},
"fork_contended_mutex_wrap": {
- "peak_bytes": 7288748,
- "leaked_bytes": 3463411,
- "total_allocations": 34847186
+ "peak_bytes": 7140341,
+ "leaked_bytes": 3522965,
+ "total_allocations": 34204380
},
"fork_consumed_signer": {
- "peak_bytes": 3350894,
- "leaked_bytes": 3288906,
- "total_allocations": 175055
+ "peak_bytes": 3421645,
+ "leaked_bytes": 3359803,
+ "total_allocations": 206540
},
"swap_chain_churn": {
- "peak_bytes": 3681161,
- "leaked_bytes": 3350287,
- "total_allocations": 672537
+ "peak_bytes": 3753669,
+ "leaked_bytes": 3421527,
+ "total_allocations": 679964
+ },
+ "deferred_teardown_flush_queue": {
+ "peak_bytes": 4103247,
+ "leaked_bytes": 3412690,
+ "total_allocations": 2448668
}
}
\ No newline at end of file
diff --git a/tests/perf/scenarios.py b/tests/perf/scenarios.py
index 23300aed..87dd512e 100644
--- a/tests/perf/scenarios.py
+++ b/tests/perf/scenarios.py
@@ -587,8 +587,8 @@ def scenario_reader_with_fragment_pre_consume_rejection(
# Fail loudly: without these the scenario still runs when the
# ownership logic regresses, and a rejection that stops being
# recognised looks identical to a pass.
- if not any(tag in str(e) for tag in
- c2pa_module.ManagedResource._PRE_CONSUME_ERROR_TAGS):
+ if not c2pa_module.ManagedResource._is_pre_consume_rejection(
+ str(e)):
raise AssertionError(
f"expected a pre-consume rejection, got: {e}") from e
if reader._handle is None:
@@ -1297,6 +1297,40 @@ def scenario_swap_chain_churn(iterations: int = 100) -> None:
context.close()
+def scenario_deferred_teardown_flush_queue(iterations: int = 100) -> None:
+ """Close resources from inside an open native-error section, so their
+ teardowns defer onto one pending list and are drained together when the
+ section closes.
+
+ Two resources per iteration rather than one: a single-element queue cannot
+ show a resource stranded behind its predecessor.
+ """
+ signed_bytes = SIGNED_JPEG.read_bytes()
+ real_free = c2pa_module.ManagedResource._free_native_ptr
+ for _ in _iterate(iterations):
+ first = Reader("image/jpeg", io.BytesIO(signed_bytes))
+ second = Reader("image/jpeg", io.BytesIO(signed_bytes))
+
+ freed = []
+ c2pa_module.ManagedResource._free_native_ptr = staticmethod(
+ lambda ptr: (freed.append(ptr), real_free(ptr))[1])
+ try:
+ with c2pa_module._native_section():
+ first.close()
+ second.close()
+ # Fail loudly: a free here means the teardown was not deferred.
+ if freed:
+ raise AssertionError(
+ "teardown inside a section freed immediately "
+ "instead of deferring")
+ if len(freed) != 2:
+ raise AssertionError(
+ f"drain freed {len(freed)} of 2 deferred handles; "
+ f"the rest leak")
+ finally:
+ c2pa_module.ManagedResource._free_native_ptr = real_free
+
+
def scenario_fork_swap_cleanup(iterations: int = 100) -> None:
"""Fork safety benchmark scenario:
the handle a Builder owns at fork time came from with_archive(), which
@@ -1403,6 +1437,7 @@ def scenario_fork_stream_cleanup(iterations: int = 100) -> None:
"fork_contended_mutex_wrap": scenario_fork_contended_mutex_wrap,
"fork_consumed_signer": scenario_fork_consumed_signer,
"swap_chain_churn": scenario_swap_chain_churn,
+ "deferred_teardown_flush_queue": scenario_deferred_teardown_flush_queue,
}
diff --git a/tests/test_unit_tests.py b/tests/test_unit_tests.py
index 2c1fb42e..92ecaa67 100644
--- a/tests/test_unit_tests.py
+++ b/tests/test_unit_tests.py
@@ -31,6 +31,7 @@
import shutil
import ctypes
import threading
+import concurrent.futures
# Suppress deprecation warnings
warnings.simplefilter("ignore", category=DeprecationWarning)
@@ -51,6 +52,15 @@
ALTERNATIVE_INGREDIENT_TEST_FILE = os.path.join(FIXTURES_DIR, "cloud.jpg")
+def _fail_with_native_error(tag_bytes):
+ """Build a mock FFI callable that sets a native error and returns None.
+ """
+ def _mock(*args):
+ c2pa_module._lib.c2pa_error_set_last(tag_bytes)
+ return None
+ return _mock
+
+
def load_test_settings_json():
"""
Load default (legacy) trust configuration test settings from a
@@ -1362,7 +1372,6 @@ def test_sign_and_read_is_not_embedded(self):
# Direct the Builder not to embed the manifest into the asset
builder.set_no_embed()
-
with open(temp_file_path, "wb") as temp_file:
manifest_data = builder.sign(
signer, "image/jpeg", file, temp_file)
@@ -7821,7 +7830,7 @@ def test_callbacks_return_minus_one_after_stream_collected(self):
class TestManagedResourceLifecycle(unittest.TestCase):
- """Lifecycle primitives (_activate, _swap_handle, _wrap_native_handle),
+ """Lifecycle primitives (_activate, _consume_and_swap, _wrap_native_handle),
the _owner_pid stamp that governs which process may free a handle, and
the ownership hand-offs between Python and the native library.
@@ -7966,41 +7975,47 @@ def test_activate_does_not_mutate_on_rejection(self):
"rejected activation replaced the handle")
self.assertEqual(res._lifecycle_state, LifecycleState.ACTIVE)
- def test_swap_handle_does_not_free_consumed_handle(self):
+ def test_consume_and_swap_does_not_free_consumed_handle(self):
res = self._FakeHandleResource()
res._activate(0xAAA1)
- res._swap_handle(0xAAA2)
+ res._consume_and_swap(lambda h: 0xAAA2, "swap: {}")
# The FFI already owns and frees the old pointer.
self.assertEqual(self.freed, [])
self.assertEqual(res._handle, 0xAAA2)
+ self.assertEqual(res._lifecycle_state, LifecycleState.ACTIVE)
res.close()
self.assertEqual(self.freed, [0xAAA2])
- def test_swap_handle_requires_active_resource(self):
+ def test_consume_and_swap_requires_active_resource(self):
uninitialized = self._FakeHandleResource()
with self.assertRaises(Error) as ctx:
- uninitialized._swap_handle(0x1)
- self.assertIn("not active", str(ctx.exception))
+ uninitialized._consume_and_swap(lambda h: 0x1, "swap: {}")
+ self.assertIn("not properly initialized", str(ctx.exception))
closed = self._FakeHandleResource()
closed._activate(0x2)
closed.close()
- with self.assertRaises(Error):
- closed._swap_handle(0x3)
+ self.freed.clear()
+ with self.assertRaises(Error) as ctx:
+ closed._consume_and_swap(lambda h: 0x3, "swap: {}")
+ self.assertIn("closed", str(ctx.exception))
+ self.assertEqual(self.freed, [])
- def test_swap_handle_rejects_null_replacement(self):
+ def test_null_replacement_is_a_failure_that_frees_the_handle(self):
+ """A null return with no native error leaves ownership unknown,
+ so the handle is freed defensively and the resource closed."""
res = self._FakeHandleResource()
res._activate(0x7777)
- with self.assertRaises(Error) as ctx:
- res._swap_handle(None)
+ with self.assertRaises(Error):
+ res._consume_and_swap(lambda h: None, "swap: {}")
- self.assertIn("null handle", str(ctx.exception))
- self.assertEqual(res._handle, 0x7777)
- self.assertEqual(res._lifecycle_state, LifecycleState.ACTIVE)
+ self.assertEqual(self.freed, [0x7777])
+ self.assertIsNone(res._handle)
+ self.assertEqual(res._lifecycle_state, LifecycleState.CLOSED)
def test_wrap_native_handle_bypasses_init(self):
seen = []
@@ -8055,7 +8070,7 @@ def test_every_construction_path_records_owner_pid(self):
# A swap keeps the original stamp:
# the replacement handle was allocated by the same process
# that created the object.
- wrapped._swap_handle(0xA3)
+ wrapped._consume_and_swap(lambda h: 0xA3, "swap: {}")
self.assertEqual(wrapped._owner_pid, pid)
def test_foreign_child_skips_free_for_wrapped_and_swapped(self):
@@ -8065,7 +8080,7 @@ def test_foreign_child_skips_free_for_wrapped_and_swapped(self):
swapped = self._FakeHandleResource()
swapped._activate(0xC2)
- swapped._swap_handle(0xC3)
+ swapped._consume_and_swap(lambda h: 0xC3, "swap: {}")
swapped._owner_pid = os.getpid() + 1
swapped.close()
@@ -8091,7 +8106,7 @@ def test_owning_process_frees_wrapped_and_swapped_exactly_once(self):
swapped = self._FakeHandleResource()
swapped._activate(0xC5)
- swapped._swap_handle(0xC6)
+ swapped._consume_and_swap(lambda h: 0xC6, "swap: {}")
swapped.close()
# 0xC5 was consumed by the test FFI swap.
@@ -8277,12 +8292,11 @@ def test_construction_failure_leaves_nothing_to_free(self):
c2pa_module._lib.c2pa_builder_from_json = real_json
def test_context_build_null_return_frees_builder(self):
- # Set a pre-consume tag in the error slot to mock a pointer rejection.
+ # Mock a pointer rejection.
settings = Settings()
- c2pa_module._lib.c2pa_error_set_last(
- b"UntrackedPointer: mocked pre-consume rejection")
real_build = c2pa_module._lib.c2pa_context_builder_build
- c2pa_module._lib.c2pa_context_builder_build = lambda ptr: None
+ c2pa_module._lib.c2pa_context_builder_build = _fail_with_native_error(
+ b"UntrackedPointer: mocked pre-consume rejection")
try:
with self.assertRaises(Error):
Context(settings=settings)
@@ -8341,6 +8355,180 @@ def test_consume_no_replacement_marks_consumed_on_other_error(self):
self.assertIsNone(res._handle)
self.assertEqual(res._lifecycle_state, LifecycleState.CLOSED)
+ def test_invoke_consume_success_does_not_consult_error_slot(self):
+ """A successful consuming call must not read the error slot at all:
+ only a failure inspects it."""
+ res = self._FakeHandleResource()
+ res._activate(0xCAFE)
+
+ res._consume_no_replacement(lambda h: 0, "set failed: {}")
+
+ self.assertIsNone(c2pa_module._read_native_error())
+
+ def test_consume_no_replacement_retains_on_tag_set_by_the_call_itself(self):
+ """Only a *stale* tag left over from before the call is the
+ thing being defended against."""
+ res = self._FakeHandleResource()
+ res._activate(0xCAFE)
+
+ def fake_call(handle):
+ c2pa_module._lib.c2pa_error_set_last(
+ b"UntrackedPointer: rejected by the call itself")
+ return -1
+
+ with self.assertRaises(Error):
+ res._consume_no_replacement(fake_call, "set failed: {}")
+
+ # Rejected before ownership transferred: handle retained.
+ self.assertEqual(res._handle, 0xCAFE)
+ self.assertEqual(res._lifecycle_state, LifecycleState.ACTIVE)
+ self.assertEqual(self.freed, [])
+ res.close()
+ self.assertEqual(self.freed, [0xCAFE])
+
+ def test_native_section_defers_unrelated_finalizer_free(self):
+ """A finalizer for a completely unrelated resource firing mid
+ native-call must not free immediately.
+ """
+ victim = self._FakeHandleResource()
+ victim._activate(0xCAFE)
+ bystander = self._FakeHandleResource()
+ bystander._activate(0xB00B)
+
+ def polluting_free(ptr):
+ self.freed.append(ptr)
+ # Freeing and untracked/ pointer writes its own error into the
+ # same thread-local slot.
+ c2pa_module._lib.c2pa_error_set_last(
+ "Other: UntrackedPointer: {:#x}".format(ptr).encode())
+ return -1
+ ManagedResource._free_native_ptr = staticmethod(polluting_free)
+
+ def ffi_call(handle):
+ nonlocal bystander
+ del bystander # last reference dropped: __del__ fires right here
+ return None # the real call failed but set no error of its own
+
+ # A bare section: the consume needs the error section, but not a
+ # borrow on its own handle. _ensure_not_borrowed
+ # refuses a consume nested in a _native_call() on the same resource.
+ with c2pa_module._native_section():
+ with self.assertRaises(Error):
+ victim._consume_no_replacement(ffi_call, "op failed: {}")
+
+ self.assertIsNone(
+ victim._handle,
+ "victim was wrongly retained")
+ self.assertEqual(victim._lifecycle_state, LifecycleState.CLOSED)
+ # The bystander's free is deferred to the section close, so it
+ # runs after the consuming call, before the victim's free.
+ self.assertEqual(self.freed, [0xB00B, 0xCAFE],
+ "deferred free did not run once, before victim's")
+
+ def test_teardown_deferred_by_own_inflight_and_section_together(self):
+ """A resource blocked by its own handle being in-flight,
+ and a wholly separate native-error section is also open on this thread
+ must not free until both clear, and must free exactly once."""
+ res = self._FakeHandleResource()
+ res._activate(0xCAFE)
+
+ call_cm = res._native_call()
+ call_cm.__enter__()
+ try:
+ section_cm = c2pa_module._native_section()
+ section_cm.__enter__()
+ try:
+ res.close()
+ self.assertEqual(res._lifecycle_state, LifecycleState.CLOSED)
+ self.assertEqual(self.freed, [],
+ "freed while still in flight")
+ finally:
+ section_cm.__exit__(None, None, None)
+ # The independent section closed, but res's own in-flight
+ # guard is still up: still not freed.
+ self.assertEqual(self.freed, [],
+ "flushed while the in-flight guard still held")
+ finally:
+ call_cm.__exit__(None, None, None)
+ # Both gates clear only once native_call's own exit drops inflight
+ # to 0, which is what should trigger the free.
+ self.assertEqual(self.freed, [0xCAFE])
+
+ def test_nested_native_sections_flush_only_at_outermost_close(self):
+ """A native-error section opened inside another, already-open one
+ on the same thread must not flush anything until the outermost
+ one closes."""
+ res = self._FakeHandleResource()
+ res._activate(0xCAFE)
+
+ outer = c2pa_module._native_section()
+ outer.__enter__()
+ try:
+ inner = c2pa_module._native_section()
+ inner.__enter__()
+ try:
+ res.close()
+ self.assertEqual(self.freed, [])
+ finally:
+ inner.__exit__(None, None, None)
+ # Inner closed, outer is still open: still deferred.
+ self.assertEqual(self.freed, [],
+ "inner section flushed before the outer closed")
+ finally:
+ outer.__exit__(None, None, None)
+ self.assertEqual(self.freed, [0xCAFE])
+
+ def test_native_section_flush_isolates_exceptions(self):
+ """One deferred free raising during a section's flush must not
+ stop the rest of that flush from running."""
+ good = self._FakeHandleResource()
+ good._activate(0xC0FFEE)
+ bad = self._FakeHandleResource()
+ bad._activate(0xBAD)
+
+ def flaky_free(ptr):
+ if ptr == 0xBAD:
+ raise RuntimeError("simulated free failure")
+ self.freed.append(ptr)
+ return 0
+ ManagedResource._free_native_ptr = staticmethod(flaky_free)
+
+ with self.assertLogs('c2pa', level='ERROR') as captured:
+ with c2pa_module._native_section():
+ bad.close()
+ good.close()
+
+ self.assertEqual(self.freed, [0xC0FFEE],
+ "a failing deferred free stopped the rest")
+ self.assertTrue(
+ any('Failed to free native' in line
+ for line in captured.output),
+ "the failing deferred free was not logged: "
+ "{}".format(captured.output))
+
+ def test_stale_error_not_misattributed_after_preset_error(self):
+ """A stale tag left by an earlier, unrelated call on this thread
+ must not be read as this call's own error."""
+ # A stale tag from an earlier, unrelated call.
+ c2pa_module._lib.c2pa_error_set_last(
+ b"Other: UntrackedPointer: 0xdeadbeef")
+
+ res = self._FakeHandleResource()
+ res._activate(0xCAFE)
+
+ # Fails without setting any error of its own.
+ # The marker written inside _invoke_consume must have cleared
+ # the stale tag, so this routes to the "no error of our own" branch.
+ with self.assertRaises(Error):
+ res._consume_no_replacement(lambda h: -1, "op failed: {}")
+
+ # A misattributed stale tag would have matched
+ # _PRE_CONSUME_ERROR_TAGS and left the resource ACTIVE.
+ self.assertIsNone(res._handle)
+ self.assertEqual(res._lifecycle_state, LifecycleState.CLOSED)
+ self.assertEqual(self.freed, [0xCAFE],
+ "unknown ownership must free, not drop the handle")
+
class TestManagedResourceObjects(TestContextAPIs):
"""Tests native resource handling management when managed manually.
@@ -8602,9 +8790,9 @@ def test_builder_with_archive_null_return_marks_consumed(self):
# Mimic a non-tag error: native took ownership then failed and dropped
# the value itself, so the handle is marked consumed, not freed.
- c2pa_module._lib.c2pa_error_set_last(b"Other: mocked test error")
real_call = c2pa_module._lib.c2pa_builder_with_archive
- c2pa_module._lib.c2pa_builder_with_archive = lambda b, s: None
+ c2pa_module._lib.c2pa_builder_with_archive = _fail_with_native_error(
+ b"Other: mocked test error")
# Instrument before the failure...
freed = self._instrument_frees()
@@ -8638,11 +8826,9 @@ def test_reader_with_fragment_null_return_marks_consumed(self):
# Mimic a non-tag error: native took ownership then failed and dropped
# the value itself, so the handle is marked consumed, not freed.
- c2pa_module._lib.c2pa_error_set_last(b"Other: mocked test error")
-
real_call = c2pa_module._lib.c2pa_reader_with_fragment
- c2pa_module._lib.c2pa_reader_with_fragment = (
- lambda r, f, s, frag: None)
+ c2pa_module._lib.c2pa_reader_with_fragment = _fail_with_native_error(
+ b"Other: mocked test error")
# Instrument before failure so any free would be counted.
freed = self._instrument_frees()
@@ -8710,11 +8896,11 @@ def _raise(*_args):
@staticmethod
def _is_pre_consume_rejection(error_message):
- """True if this native error means ownership never transferred."""
+ """True if this native error means ownership never transferred.
+ """
if not error_message:
return False
- return any(tag in error_message
- for tag in ManagedResource._PRE_CONSUME_ERROR_TAGS)
+ return ManagedResource._is_pre_consume_rejection(error_message)
def _stale_reader_handle(self):
"""A freed, untracked pointer, captured before close() nulls it.
@@ -8740,31 +8926,6 @@ def _untracked_reader_handle():
return (ctypes.cast(buf, ctypes.POINTER(c2pa_module.C2paReader)),
buf)
- def test_with_fragment_pre_consume_rejection_keeps_handle(self):
- # Rejected before native lib took ownership,
- # so nothing was consumed and the handle is still ours.
- init_path = os.path.join(FIXTURES_DIR, "dashinit.mp4")
- fragment_path = os.path.join(FIXTURES_DIR, "dash1.m4s")
- with open(init_path, "rb") as init:
- reader = Reader("video/mp4", init)
- real_handle = reader._handle
-
- reader._handle = self._stale_reader_handle()
- try:
- with open(init_path, "rb") as init, \
- open(fragment_path, "rb") as frag:
- with self.assertRaises(Error) as caught:
- reader.with_fragment("video/mp4", init, frag)
- finally:
- reader._handle = real_handle
-
- self.assertIn("UntrackedPointer", str(caught.exception))
- # Ownership never transferred, so the resource stays usable.
- self.assertIsNotNone(reader._handle)
- self.assertEqual(reader._lifecycle_state, LifecycleState.ACTIVE)
- self.assertTrue(reader.json())
- reader.close()
-
def test_with_fragment_pre_consume_rejection_does_not_leak(self):
# A handle dropped on this path leaks one reader per call.
init_path = os.path.join(FIXTURES_DIR, "dashinit.mp4")
@@ -8805,88 +8966,116 @@ def _reader_from_context(self):
"Failed to create reader: {}")
return reader
- def test_null_parameter_rejection_retains_the_handle(self):
- """A null argument is rejected before the reader is untracked.
- Ownership never transferred, so the handle is still ours to free.
- Treating it as consumed leaks one reader per call.
+ def test_preflight_rejects_before_the_consuming_call(self):
+ """A bad argument must be refused before the handle reaches native.
+
+ Native validates arguments and takes ownership in an order that
+ differs between versions, so a rejection that reaches native leaves
+ ownership ambiguous. Refusing here keeps the handle unambiguously
+ ours.
"""
reader = self._reader_from_context()
- handle = reader._handle
- freed = self._instrument_frees()
+ called = []
with self.assertRaises(Error) as caught:
- with reader._native_call():
- reader._consume_and_swap(
- lambda h: c2pa_module._lib.c2pa_reader_with_stream(
- h, b"image/jpeg", None),
- "Failed to configure reader: {}")
-
- self.assertIn("NullParameter", str(caught.exception))
- self.assertIsNotNone(reader._handle, "the retained handle was dropped")
- self.assertEqual(reader._lifecycle_state, LifecycleState.ACTIVE)
+ reader._consume_and_swap(
+ lambda h: (called.append(h),
+ c2pa_module._check_bytes_arg(
+ 'manifest_data', b''))[1],
+ "Failed: {}")
- reader.close()
+ self.assertIn("InvalidBufferSize", str(caught.exception))
self.assertEqual(
- self._free_count(freed, handle), 1,
- "a handle the native side never took was leaked")
+ len(called), 1,
+ "the guard should raise inside the call, before native runs")
- def test_invalid_buffer_size_rejection_retains_the_handle(self):
- """A zero-length manifest buffer is rejected before the untrack..
- """
+ def test_preflight_rejection_frees_the_handle_exactly_once(self):
+ """The handle is still ours after a preflight rejection, so it is
+ freed rather than abandoned."""
+ freed = self._instrument_frees()
reader = self._reader_from_context()
handle = reader._handle
- freed = self._instrument_frees()
- empty = (ctypes.c_ubyte * 4)()
- with Stream(io.BytesIO(b"abc")) as stream_obj:
- with self.assertRaises(Error) as caught:
- with reader._native_call():
- reader._consume_and_swap(
- lambda h: (
- c2pa_module._lib
- .c2pa_reader_with_manifest_data_and_stream(
- h, b"image/jpeg", stream_obj._stream,
- empty, 0)
- ),
- "Failed to configure reader: {}")
-
- self.assertIn("InvalidBufferSize", str(caught.exception))
- self.assertIsNotNone(reader._handle, "the retained handle was dropped")
- self.assertEqual(reader._lifecycle_state, LifecycleState.ACTIVE)
+ with self.assertRaises(Error):
+ reader._consume_and_swap(
+ lambda h: c2pa_module._check_bytes_arg(
+ 'manifest_data', b''),
+ "Failed: {}")
reader.close()
self.assertEqual(
self._free_count(freed, handle), 1,
- "a handle the native side never took was leaked")
+ "a preflight-rejected handle must be freed exactly once")
+
+ def test_reader_with_empty_manifest_data_never_calls_native(self):
+ """End-to-end: the guard is wired into the public path, not just
+ available as a helper."""
+ context = Context()
+ self.addCleanup(context.close)
+ with open(os.path.join(FIXTURES_DIR,
+ DEFAULT_TEST_FILE_NAME), "rb") as image:
+ image_bytes = image.read()
- def test_repeated_rejections_do_not_accumulate_handles(self):
- """Every rejected call must give its handle back, not just the first.
- """
- handles = []
freed = self._instrument_frees()
- for _ in range(10):
- reader = self._reader_from_context()
- handles.append(reader._handle)
- with self.assertRaises(Error):
- with reader._native_call():
- reader._consume_and_swap(
- lambda h: c2pa_module._lib.c2pa_reader_with_stream(
- h, b"image/jpeg", None),
- "Failed to configure reader: {}")
- reader.close()
+ with self.assertRaises(Error) as caught:
+ Reader("image/jpeg", io.BytesIO(image_bytes),
+ manifest_data=b"", context=context)
- leaked = [h for h in handles if self._free_count(freed, h) == 0]
+ # The guard raises before the FFI call, so the reader handle is still
+ # the binding's to free: exactly one free, and no abandoned handle.
+ self.assertIn("InvalidBufferSize", str(caught.exception))
self.assertEqual(
- leaked, [], f"{len(leaked)} of {len(handles)} handles leaked")
+ len(freed), 1,
+ "a preflight-rejected reader handle must be reclaimed, not leaked")
- def test_repeated_with_fragment_does_not_accumulate_streams(self):
- """Repeated calls on one Reader must not pile up fragment streams.
+ def test_check_cstr_arg_rejects_none_and_embedded_nul(self):
+ """Both cases would reach native as something other than the caller
+ passed: None as a null pointer, an embedded NUL as a short string."""
+ with self.assertRaises(Error) as none_case:
+ c2pa_module._check_cstr_arg('format', None)
+ self.assertIn("NullParameter", str(none_case.exception))
+
+ with self.assertRaises(Error) as nul_case:
+ c2pa_module._check_cstr_arg('format', "image/\x00jpeg")
+ self.assertIn("null byte", str(nul_case.exception))
+
+ c2pa_module._check_cstr_arg('format', "image/jpeg")
+ c2pa_module._check_cstr_arg('format', b"")
- Each retained wrapper pins a native C2paStream, four ctypes callback
- trampolines and the caller's buffer, so an unbounded list grows the
- process by tens of megabytes over a long-lived Reader. Every other
- fragment test builds a fresh Reader per call, which never accumulates.
+ def test_load_settings_rejects_embedded_nul(self):
+ with self.assertRaises(Error) as caught:
+ load_settings('{"a": 1}', format="json\x00")
+ self.assertIn("null byte", str(caught.exception))
+
+ def test_format_embeddable_null_out_pointer_raises_not_crashes(self):
+ real = c2pa_module._lib.c2pa_format_embeddable
+ c2pa_module._lib.c2pa_format_embeddable = (
+ lambda fmt, data, size, out: 128)
+ try:
+ with self.assertRaises(Error) as caught:
+ format_embeddable("image/jpeg", b"junk")
+ finally:
+ c2pa_module._lib.c2pa_format_embeddable = real
+ self.assertIn("no data returned", str(caught.exception))
+
+ def test_check_bytes_arg_rejects_none_and_empty(self):
+ for bad in (None, b""):
+ with self.assertRaises(Error):
+ c2pa_module._check_bytes_arg('manifest_data', bad)
+
+ c2pa_module._check_bytes_arg('manifest_data', b"x")
+
+ def test_check_handle_arg_rejects_null(self):
+ """A null handle is a NullParameter on both native versions."""
+ with self.assertRaises(Error):
+ c2pa_module._check_handle_arg('stream', None)
+
+ c2pa_module._check_handle_arg(
+ 'stream', ctypes.cast(1, ctypes.c_void_p))
+
+ def test_repeated_with_fragment_does_not_accumulate_streams(self):
+ """Repeated with_fragment Reader calls should not accumulate streams.
"""
init_path = os.path.join(FIXTURES_DIR, "dashinit.mp4")
fragment_path = os.path.join(FIXTURES_DIR, "dash1.m4s")
@@ -8911,7 +9100,7 @@ def test_repeated_with_fragment_does_not_accumulate_streams(self):
all(s.closed for s in superseded[:-1]),
"a superseded fragment stream was dropped without being closed")
- # The reader still works on the fragment it currently holds.
+ # The reader still works on the fragment it holds.
self.assertTrue(reader.json())
def test_with_archive_post_consume_failure_consumes_handle(self):
@@ -8971,10 +9160,9 @@ def test_unknown_failure_drops_handle_without_freeing(self):
consumed_handle = reader._handle
# Simulate an error being set
- c2pa_module._lib.c2pa_error_set_last(b"Other: mocked test error")
real_call = c2pa_module._lib.c2pa_reader_with_fragment
- c2pa_module._lib.c2pa_reader_with_fragment = (
- lambda r, f, s, frag: None)
+ c2pa_module._lib.c2pa_reader_with_fragment = _fail_with_native_error(
+ b"Other: mocked test error")
try:
with open(init_path, "rb") as init, \
open(fragment_path, "rb") as frag:
@@ -9024,9 +9212,8 @@ def test_pre_consume_tags_still_match_the_native_wording(self):
message = str(caught.exception)
self.assertTrue(
self._is_pre_consume_rejection(message),
- f"the native rejection wording changed and no longer matches "
- f"_PRE_CONSUME_ERROR_TAGS; ownership will be misjudged: "
- f"{message!r}")
+ f"rejection wording does not match _PRE_CONSUME_ERROR_TAGS, "
+ f"so ownership will be misjudged: {message!r}")
reader.close()
def test_stale_handle_is_actually_rejected_every_time(self):
@@ -9077,7 +9264,7 @@ def test_perf_scenario_bogus_handle_is_rejected(self):
self.assertTrue(
self._is_pre_consume_rejection(str(caught.exception)),
- "the perf scenarios' bogus handle is no longer rejected, so "
+ "the perf bogus handle was not rejected, so "
"with_fragment_pre_consume_rejection measures nothing")
# Handle kept, so the reader still works and frees normally.
self.assertEqual(reader._lifecycle_state, LifecycleState.ACTIVE)
@@ -9085,17 +9272,15 @@ def test_perf_scenario_bogus_handle_is_rejected(self):
reader.close()
def test_every_null_return_sets_its_own_error(self):
- # Reading the slot without clearing it is only sound because every
- # null return sets an error. Check each path reports its own.
+ # Each null-returning path must report the error it set itself, never
+ # one left behind by an earlier call.
init_path = os.path.join(FIXTURES_DIR, "dashinit.mp4")
fragment_path = os.path.join(FIXTURES_DIR, "dash1.m4s")
- # Leave a recognisable error behind, so anything stale shows up.
- try:
- Reader("image/jpeg", io.BytesIO(b"not an image")).json()
- except Error:
- pass
- self.assertIn("NotSupported", c2pa_module._read_native_error() or "")
+ # Set a recognizable error, so anything stale is caught by the
+ # assertNotIn checks.
+ c2pa_module._lib.c2pa_error_set_last(
+ b"NotSupported: planted by the test")
# Pre-consume rejection: reports UntrackedPointer, not NotSupported.
with open(init_path, "rb") as init:
@@ -9165,21 +9350,19 @@ def worker():
self.assertEqual(problems, [],
"ownership was misjudged under concurrency")
- def test_reading_the_native_error_does_not_empty_the_slot(self):
- # c2pa_error() peeks, so nothing Python can call empties the slot.
- # _consume_and_swap depends on this.
- try:
- Reader("image/jpeg", io.BytesIO(b"not an image")).json()
- except Error:
- pass
+ def test_reading_the_native_error_consumes_it(self):
+ # c2pa_error() itself peeks, so _read_native_error marks the slot as
+ # carrying no error once it has read one.
+ # An error belongs to the caller that observes it;
+ # leaving it readable lets a later, unrelated failure report it as its own.
+ c2pa_module._lib.c2pa_error_set_last(b"Io: read me exactly once")
first = c2pa_module._read_native_error()
self.assertTrue(first, "expected a native error to have been set")
- self.assertEqual(
- c2pa_module._read_native_error(), first,
- "reading emptied the native slot; the comments in "
- "_consume_and_swap about a persistent error are now wrong")
+ self.assertIsNone(
+ c2pa_module._read_native_error(),
+ "the native error stayed readable after being reported once")
def test_read_native_error_returns_none_for_an_empty_message(self):
# c2pa_error() returns an owned pointer to "" when no error is set,
@@ -9197,22 +9380,30 @@ def test_read_native_error_returns_none_for_an_empty_message(self):
finally:
c2pa_module._lib.c2pa_error = original
- def test_mocked_null_without_error_is_a_known_limitation(self):
- # A null with no error of its own is the case that breaks: the slot
- # still holds whatever came before. No native path does this, so it
- # is pinned here rather than defended in _consume_and_swap.
+ def test_null_return_with_no_native_error_is_treated_as_consumed(self):
+ # A null with no error of its own is the case that breaks without
+ # the marker:
+ # the slot still held whatever an unrelated, earlier call on this same
+ # (pooled) thread left behind, and a stale UntrackedPointer/
+ # WrongPointerType tag would make this call believe it still owned a
+ # handle the native side already dropped.
init_path = os.path.join(FIXTURES_DIR, "dashinit.mp4")
fragment_path = os.path.join(FIXTURES_DIR, "dash1.m4s")
+ # A stale, unrelated tag left by a prior call on this thread.
c2pa_module._lib.c2pa_error_set_last(
b"UntrackedPointer: 0xdeadbeef")
with open(init_path, "rb") as init:
reader = Reader("video/mp4", init)
+ consumed_handle = reader._handle
real_call = c2pa_module._lib.c2pa_reader_with_fragment
+ # The fake native call sets no error of its own,
+ # the marker planted by _invoke_consume is left in the slot.
c2pa_module._lib.c2pa_reader_with_fragment = (
lambda r, f, s, frag: None)
+ freed = self._instrument_frees()
try:
with open(init_path, "rb") as init, \
open(fragment_path, "rb") as frag:
@@ -9220,16 +9411,14 @@ def test_mocked_null_without_error_is_a_known_limitation(self):
reader.with_fragment("video/mp4", init, frag)
finally:
c2pa_module._lib.c2pa_reader_with_fragment = real_call
- # Nothing clears the slot, so a planted tag would follow other
- # tests around and change how their failures are classified.
- c2pa_module._lib.c2pa_error_set_last(
- b"Other: cleared by test teardown")
- # The stale tag wins, so the handle is kept. Safe here (the mock
- # consumed nothing), and the reader is still usable.
- self.assertIsNotNone(reader._handle)
- self.assertEqual(reader._lifecycle_state, LifecycleState.ACTIVE)
- reader.close()
+ # The marker survived, not the stale tag.
+ self.assertIsNone(reader._handle)
+ self.assertEqual(reader._lifecycle_state, LifecycleState.CLOSED)
+ # Ownership is unknown, so the handle is freed once. c2pa_free
+ # returns -1 if native had already taken the value.
+ self.assertEqual(self._free_count(freed, consumed_handle), 1,
+ "unknown-ownership handle was not freed once")
# Backfilling a pointer minted by a direct FFI call. Builder.from_archive
# is the only production caller of _wrap_native_handle, so these are the
@@ -9378,10 +9567,9 @@ def test_consumed_reader_closes_backing_file(self):
self.assertFalse(backing_file.closed)
# Simulate an error being set
- c2pa_module._lib.c2pa_error_set_last(b"Other: mocked test error")
real_call = c2pa_module._lib.c2pa_reader_with_fragment
- c2pa_module._lib.c2pa_reader_with_fragment = (
- lambda r, f, s, frag: None)
+ c2pa_module._lib.c2pa_reader_with_fragment = _fail_with_native_error(
+ b"Other: mocked test error")
try:
with open(DEFAULT_TEST_FILE, "rb") as main, \
open(DEFAULT_TEST_FILE, "rb") as frag:
@@ -9400,9 +9588,9 @@ def test_consumed_builder_releases_context(self):
archive = self._make_archive()
# Simulate an error being set
- c2pa_module._lib.c2pa_error_set_last(b"Other: mocked test error")
real_call = c2pa_module._lib.c2pa_builder_with_archive
- c2pa_module._lib.c2pa_builder_with_archive = lambda b, s: None
+ c2pa_module._lib.c2pa_builder_with_archive = _fail_with_native_error(
+ b"Other: mocked test error")
try:
with self.assertRaises(Error):
builder.with_archive(archive)
@@ -9449,10 +9637,9 @@ def test_consumed_reader_clears_caches(self):
self.assertIsNotNone(reader._manifest_json_str_cache)
# Simulate an error being set
- c2pa_module._lib.c2pa_error_set_last(b"Other: mocked test error")
real_call = c2pa_module._lib.c2pa_reader_with_fragment
- c2pa_module._lib.c2pa_reader_with_fragment = (
- lambda r, f, s, frag: None)
+ c2pa_module._lib.c2pa_reader_with_fragment = _fail_with_native_error(
+ b"Other: mocked test error")
try:
with open(DEFAULT_TEST_FILE, "rb") as main, \
open(DEFAULT_TEST_FILE, "rb") as frag:
@@ -9524,6 +9711,36 @@ def _boom(*args):
self.assertIs(ctx.exception.__cause__, sentinel,
"signing error dropped the original exception")
+ def test_sign_reports_the_native_error_it_set(self):
+ """sign() reads its error in a later section than the call itself.
+ The signing call runs inside one _native_call() block and the result
+ check runs in a separate _native_section() afterwards, so anything
+ that marks the slot as carrying no error on section exit would discard
+ the real message between the two.
+ """
+ builder = Builder(self.test_manifest)
+ signer = self._ctx_make_signer()
+ self.addCleanup(signer.close)
+
+ real_sign = c2pa_module._lib.c2pa_builder_sign
+
+ def _fail(*args):
+ c2pa_module._lib.c2pa_error_set_last(
+ b"Signature: native signing refused")
+ return -1
+
+ c2pa_module._lib.c2pa_builder_sign = _fail
+ try:
+ with self.assertRaises(Error) as ctx:
+ builder.sign(signer, "image/jpeg",
+ io.BytesIO(b"x"), io.BytesIO())
+ finally:
+ c2pa_module._lib.c2pa_builder_sign = real_sign
+
+ self.assertIn("native signing refused", str(ctx.exception),
+ "the native signing error was lost before it was read")
+ self.assertIsInstance(ctx.exception, Error.Signature)
+
class TestErrorPlumbing(unittest.TestCase):
"""Covers the error helpers themselves, which had no direct tests."""
@@ -9552,18 +9769,37 @@ def test_unmapped_tag_falls_back_to_base_error(self):
# Base class only: no subclass should claim an unknown tag.
self.assertIs(type(ctx.exception), Error)
- def test_pre_consume_tag_match_is_substring_not_prefix(self):
- """The tags arrive mid-string, so the match must stay a substring one.
+ def test_pre_consume_tag_match_skips_the_one_wrapper(self):
+ """A tag reaches the classifier behind at most one "Other: " wrapper.
+ The match is anchored after that wrapper, not a substring search.
+ """
+ classify = ManagedResource._is_pre_consume_rejection
+
+ self.assertTrue(classify("Other: UntrackedPointer: 0xdeadb000"))
+ self.assertTrue(classify("UntrackedPointer: 0xdeadb000"))
+ self.assertTrue(classify("Other: WrongPointerType: 0xdeadb000"))
- Guards the triage in _raise_consume_failure against being "cleaned up"
- into error.startswith(tag), which would match nothing and silently
- turn every retained handle into a consumed one.
+ def test_stream_release_preserves_a_pending_error(self):
+ """Releasing a Stream must not clear an error set by another call.
+
+ __del__ runs at any bytecode boundary, including between an FFI call
+ and its error read, so anything that clears the slot here reports the
+ caller's failure as "Unknown error".
"""
- wire_error = "Other: UntrackedPointer: 0xdeadb000"
- tags = ManagedResource._PRE_CONSUME_ERROR_TAGS
+ for label, dispose in (
+ ("close", lambda st: st.close()),
+ ("__del__", lambda st: st.__del__()),
+ ):
+ with self.subTest(dispose=label):
+ stream = c2pa_module.Stream(io.BytesIO(b"payload"))
+ self._set_native_error("Io: the failure the caller wants")
- self.assertTrue(any(tag in wire_error for tag in tags))
- self.assertFalse(any(wire_error.startswith(tag) for tag in tags))
+ dispose(stream)
+
+ self.assertEqual(
+ c2pa_module._read_native_error(),
+ "Io: the failure the caller wants",
+ "releasing a Stream swallowed a pending native error")
def test_check_ffi_operation_result_raises_with_native_message(self):
self._set_native_error("Io: disk exploded")
@@ -9655,6 +9891,340 @@ def test_supported_mime_types_reports_the_native_message(self):
c2pa_module._get_supported_mime_types(lambda count: None, None)
self.assertIn("mime lookup failed", str(ctx.exception))
+ def test_reading_an_error_does_not_leave_it_readable(self):
+ """An error is reportable once, by the reader that observes it.
+ """
+ self._set_native_error("Io: read me once")
+
+ self.assertEqual(
+ c2pa_module._read_native_error(), "Io: read me once")
+ self.assertIsNone(
+ c2pa_module._read_native_error(),
+ "the same native error was reported a second time")
+
+ def test_handled_error_does_not_survive_later_operations(self):
+ """A caught failure must not leave its error in-place
+ (tests the slot is cleaned up).
+ """
+ with self.assertRaises(Error):
+ Reader("image/jpeg", io.BytesIO(b"not an image"))
+
+ for _ in range(20):
+ c2pa_module.Stream(io.BytesIO(b"x"))
+
+ self.assertIsNone(
+ c2pa_module._read_native_error(),
+ "a handled error was still resident after 20 successful calls")
+
+ def test_later_failure_does_not_inherit_a_handled_errors_type(self):
+ """A failure with no error of its own must not see an older one.
+ """
+ with self.assertRaises(Error) as first:
+ Reader("image/jpeg", io.BytesIO(b"not an image"))
+ self.assertIsInstance(first.exception, Error.NotSupported)
+
+ with self.assertRaises(Error) as second:
+ c2pa_module._check_ffi_operation_result(
+ None, "Later unrelated failure: {}")
+
+ self.assertNotIsInstance(
+ second.exception, Error.NotSupported,
+ "the later failure inherited the handled error's type")
+ self.assertIn("Unknown error", str(second.exception))
+ self.assertNotIn(
+ "type is unsupported", str(second.exception),
+ "the later failure reported the handled error's message")
+
+ def test_the_no_error_marker_never_reaches_a_caller(self):
+ """The marker is internal, not a message for users."""
+ marker = c2pa_module._NO_ERROR_MARKER_TEXT
+
+ c2pa_module._write_no_error_marker()
+ self.assertIsNone(
+ c2pa_module._read_native_error(),
+ "the marker was reported as if it were a native error")
+
+ c2pa_module._write_no_error_marker()
+ with self.assertRaises(Error) as ctx:
+ c2pa_module._check_ffi_operation_result(None, "fallback: {}")
+ self.assertNotIn(marker, str(ctx.exception))
+ self.assertIn("Unknown error", str(ctx.exception))
+
+ def test_write_no_error_marker_writes_the_learned_text(self):
+ c2pa_module._write_no_error_marker()
+ raw = c2pa_module._lib.c2pa_error()
+ try:
+ text = ctypes.string_at(raw).decode('utf-8')
+ finally:
+ c2pa_module._lib.c2pa_string_free(raw)
+ self.assertEqual(text, c2pa_module._NO_ERROR_MARKER_TEXT)
+
+ def test_read_native_error_maps_the_marker_to_none(self):
+ c2pa_module._write_no_error_marker()
+ self.assertIsNone(c2pa_module._read_native_error())
+
+ def test_read_native_error_marks_the_slot_when_the_pointer_is_null(self):
+ """A NULL from c2pa_error must still leave the slot marked.
+
+ c2pa_error returns NULL when the stored message cannot be rendered as
+ a C string. The message stays in the thread-local slot, which is
+ sticky, so returning without planting the marker leaves that message
+ readable by the next call that fails without setting an error of its
+ own, which then reports it as its own failure.
+ """
+ c2pa_module._lib.c2pa_error_set_last(b"Io: unreadable original")
+
+ original = c2pa_module._lib.c2pa_error
+ try:
+ c2pa_module._lib.c2pa_error = lambda: None
+ self.assertIsNone(
+ c2pa_module._read_native_error(),
+ "a NULL pointer must read as no error")
+ finally:
+ c2pa_module._lib.c2pa_error = original
+
+ self.assertIsNone(
+ c2pa_module._read_native_error(),
+ "the NULL branch left the message in the slot instead of "
+ "planting the marker")
+
+ def test_a_failure_after_a_null_read_does_not_inherit_the_old_message(self):
+ """The message surviving a NULL read must not become someone's error."""
+ c2pa_module._lib.c2pa_error_set_last(b"Io: belongs to an earlier call")
+
+ original = c2pa_module._lib.c2pa_error
+ try:
+ c2pa_module._lib.c2pa_error = lambda: None
+ c2pa_module._read_native_error()
+ finally:
+ c2pa_module._lib.c2pa_error = original
+
+ with self.assertRaises(Error) as ctx:
+ c2pa_module._check_ffi_operation_result(
+ None, "Later unrelated failure: {}")
+
+ self.assertNotIn(
+ "belongs to an earlier call", str(ctx.exception),
+ "a later failure reported a message left by an earlier call")
+ self.assertIn("Unknown error", str(ctx.exception))
+
+ def test_every_real_rejection_wording_is_classified_as_pre_consume(self):
+ """Every tag arrives bare or behind the "Other: " wrapper."""
+ wrapper = c2pa_module.ManagedResource._NATIVE_ERROR_WRAPPER
+ classify = c2pa_module.ManagedResource._is_pre_consume_rejection
+
+ for tag in c2pa_module.ManagedResource._PRE_CONSUME_ERROR_TAGS:
+ bare = f"{tag} some detail"
+ wrapped = f"{wrapper}{tag} some detail"
+ self.assertTrue(
+ classify(bare),
+ f"a bare {tag} rejection was read as a consumed handle")
+ self.assertTrue(
+ classify(wrapped),
+ f"a wrapped {tag} rejection was read as a consumed handle")
+
+ def test_caller_text_quoting_a_tag_is_not_a_rejection(self):
+ """A tag inside the message body describes the caller's input.
+
+ Native errors quote caller-supplied strings verbatim: a JSON parse
+ failure repeats the offending value, an Io failure names the path.
+ Reading one of those as a pre-consume rejection hands the resource back
+ as usable after native may already own and have dropped its handle.
+ """
+ classify = c2pa_module.ManagedResource._is_pre_consume_rejection
+
+ forged = (
+ 'Json: invalid type: string "NullParameter: x", expected a '
+ 'sequence at line 1 column 43',
+ 'Json: invalid type: string "WrongPointerType: y", expected a '
+ 'sequence at line 1 column 46',
+ "Io: cannot open /tmp/UntrackedPointer: 0xdead.jpg",
+ "Other: manifest text mentions InvalidBufferSize: in passing",
+ )
+ for message in forged:
+ self.assertFalse(
+ classify(message),
+ f"caller text was read as a pointer rejection: {message!r}")
+
+ def test_caller_text_quoting_a_tag_reaches_the_error_slot(self):
+ """test_caller_text_quoting_a_tag_is_not_a_rejection forges this
+ wording; the library really produces it.
+ """
+ c2pa_module._lib.c2pa_builder_from_json(
+ b'{"claim_generator_info": "NullParameter: injected"}')
+ message = c2pa_module._read_native_error()
+
+ self.assertIn(
+ "NullParameter:", message,
+ "caller text did not reach the error slot verbatim: the "
+ "forged wording is stale")
+ self.assertFalse(
+ c2pa_module.ManagedResource._is_pre_consume_rejection(message),
+ f"a caller-supplied string forged a pointer rejection: {message!r}")
+
+ def test_a_failing_flush_does_not_strand_the_rest_of_the_queue(self):
+ """One resource raising must not skip the resources queued behind it.
+ """
+ flushed = []
+
+ class Recorder:
+ def __init__(self, name, raises=None):
+ self.name = name
+ self.raises = raises
+
+ def _maybe_flush_pending(self):
+ if self.raises is not None:
+ raise self.raises
+ flushed.append(self.name)
+
+ first = Recorder("first")
+ middle = Recorder("middle", raises=KeyboardInterrupt())
+ last = Recorder("last")
+
+ with self.assertLogs("c2pa", level="ERROR"):
+ with c2pa_module._native_section():
+ for resource in (first, middle, last):
+ c2pa_module._register_for_section_flush(resource)
+
+ self.assertEqual(
+ flushed, ["first", "last"],
+ "a resource queued behind a failing one was never flushed, "
+ "so its handle leaks")
+
+ def test_a_failing_flush_logs_the_first_exception(self):
+ """Failures on drain should be logged."""
+ flushed = []
+
+ class Recorder:
+ def __init__(self, name, raises=None):
+ self.name = name
+ self.raises = raises
+
+ def _maybe_flush_pending(self):
+ if self.raises is not None:
+ raise self.raises
+ flushed.append(self.name)
+
+ with self.assertLogs("c2pa", level="ERROR") as captured:
+ with c2pa_module._native_section():
+ for resource in (
+ Recorder("boom", raises=RuntimeError("first failure")),
+ Recorder("survivor"),
+ Recorder("later", raises=RuntimeError("second failure"))):
+ c2pa_module._register_for_section_flush(resource)
+
+ self.assertTrue(
+ any("first failure" in message for message in captured.output))
+ self.assertEqual(
+ flushed, ["survivor"],
+ "a resource between two failing ones was never flushed")
+
+ def test_runtime_does_not_call_error_set_last(self):
+ """The marker mechanism must not depend on c2pa_error_set_last,
+ so this module loads against native builds that lack it."""
+ for fn in (c2pa_module.ManagedResource._invoke_consume,
+ c2pa_module._read_native_error,
+ c2pa_module._write_no_error_marker):
+ self.assertNotIn(
+ 'c2pa_error_set_last', inspect.getsource(fn))
+
+
+class TestMarkerOutlivesPointerConsumptionSemantics(unittest.TestCase):
+ """The marker is needed for reasons independent of pointer ownership.
+
+ The native error slot is sticky and thread-local, so failure paths
+ that carry no still need to tell an error this call set from an
+ earlier, unrelated call left behind.
+ """
+
+ def setUp(self):
+ # Leave no message from an earlier test in this thread's slot.
+ c2pa_module._write_no_error_marker()
+
+ def test_non_consuming_failure_does_not_inherit_a_read_error(self):
+ c2pa_module._lib.c2pa_error_set_last(b"Signature: earlier task")
+ # The rightful owner reports it, which re-marks the slot.
+ self.assertEqual(
+ c2pa_module._read_native_error(), "Signature: earlier task")
+
+ # A later, unrelated failure that sets no error of its own must
+ # report its own fallback, not the planted Signature message.
+ with self.assertRaises(Error) as ctx:
+ c2pa_module._check_ffi_operation_result(
+ 0, "later op failed: {}", check=lambda r: r == 0)
+
+ self.assertNotIn("earlier task", str(ctx.exception))
+ self.assertIn("Unknown error", str(ctx.exception))
+ self.assertNotIsInstance(ctx.exception, Error.Signature)
+
+ def test_settings_set_failure_reports_its_own_error(self):
+ settings = Settings()
+ self.addCleanup(settings.close)
+
+ c2pa_module._lib.c2pa_error_set_last(b"Signature: earlier task")
+ self.assertEqual(
+ c2pa_module._read_native_error(), "Signature: earlier task")
+
+ with self.assertRaises(Error) as ctx:
+ settings.set("builder.thumbnail.enabled", "not-a-json-value")
+
+ self.assertNotIn("earlier task", str(ctx.exception))
+
+ def test_marker_is_per_thread_across_pooled_reuse(self):
+ """The slot is thread-local, so a pooled worker must not hand one
+ task's error to the next task that runs on it."""
+ def failing_task():
+ c2pa_module._lib.c2pa_error_set_last(b"Io: first task")
+ return c2pa_module._read_native_error()
+
+ def quiet_task():
+ # Sets no error; must not see the previous task's message.
+ return c2pa_module._read_native_error()
+
+ # One worker guarantees both tasks run on the same OS thread.
+ with concurrent.futures.ThreadPoolExecutor(max_workers=1) as pool:
+ self.assertEqual(pool.submit(failing_task).result(),
+ "Io: first task")
+ self.assertIsNone(
+ pool.submit(quiet_task).result(),
+ "a pooled thread carried an error across unrelated tasks")
+
+ def test_one_thread_marker_does_not_clear_another_threads_error(self):
+ """Marking on one thread must leave another thread's pending error
+ readable: the slot is per thread, and so is the marker."""
+ set_on_worker = threading.Event()
+ marked_on_main = threading.Event()
+ seen = {}
+
+ def worker():
+ c2pa_module._lib.c2pa_error_set_last(b"Io: worker error")
+ set_on_worker.set()
+ self.assertTrue(marked_on_main.wait(5))
+ seen["worker"] = c2pa_module._read_native_error()
+
+ thread = threading.Thread(target=worker, daemon=True)
+ thread.start()
+ self.assertTrue(set_on_worker.wait(5))
+
+ c2pa_module._write_no_error_marker()
+ marked_on_main.set()
+ thread.join(5)
+
+ self.assertEqual(seen.get("worker"), "Io: worker error")
+
+ def test_marker_path_is_reached_without_any_consuming_call(self):
+ """The non-consuming path reaches the marker through _read_native_error,
+ never through _invoke_consume."""
+ self.assertIn("_read_native_error",
+ inspect.getsource(
+ c2pa_module._check_ffi_operation_result))
+ self.assertNotIn("_invoke_consume",
+ inspect.getsource(
+ c2pa_module._check_ffi_operation_result))
+ # _read_native_error is what re-marks the slot after every read.
+ self.assertIn("_write_no_error_marker",
+ inspect.getsource(c2pa_module._read_native_error))
+
class TestErrorsStillRaiseAfterCleanup(unittest.TestCase):
"""Each surface that lost a _clear_error_state() call still reports."""
@@ -9721,7 +10291,7 @@ def test_marshalling_error_retains_the_handle(self):
An ArgumentError means the call never reached native, so the handle is
untouched and must NOT be freed. Without this, a zero-free assertion
- could pass simply because the counter never fires.
+ could pass because the counter never fires.
"""
def bad_marshal(handle):
raise ctypes.ArgumentError("marshalling failed")
@@ -9736,28 +10306,6 @@ def bad_marshal(handle):
self.assertIsNotNone(resource._handle)
self.assertEqual(resource._lifecycle_state, LifecycleState.ACTIVE)
- def test_pre_consume_rejection_restores_the_resource(self):
- """A handle native rejected before taking ownership stays usable.
-
- The reservation is held until _raise_consume_failure classifies the
- error, so no other thread sees the resource as ACTIVE while its
- ownership is still undetermined.
- """
- resource = Settings()
- self.freed.clear()
- real_read = c2pa_module._read_native_error
- c2pa_module._read_native_error = (
- lambda: "Other: UntrackedPointer: 0x1234")
- try:
- with self.assertRaises(Error):
- resource._consume_no_replacement(lambda h: 1, "consume: {}")
- finally:
- c2pa_module._read_native_error = real_read
-
- self.assertEqual(resource._lifecycle_state, LifecycleState.ACTIVE)
- self.assertIsNotNone(resource._handle)
- self.assertEqual(self.freed, [])
-
def test_post_consume_failure_keeps_the_resource_closed(self):
"""An error without a pre-consume tag means native took ownership.
@@ -9796,27 +10344,26 @@ def test_failure_without_a_native_error_frees_the_handle(self):
"an unknown-ownership failure dropped the handle without freeing")
self.assertIsNone(resource._handle)
- def test_rejected_replacement_is_freed(self):
- """A replacement _swap_handle refuses must not be left unowned.
-
- Native consumed the old pointer and returned this one, so nothing else
- holds it.
+ def test_close_called_during_parallel_call(self):
+ """Parallel closes handling.
"""
resource = Settings()
spare = Settings()
replacement = spare._handle
- # Detach so only the code under test can free it.
+ # Only test should be able to free.
spare._handle = None
spare._lifecycle_state = LifecycleState.CLOSED
self.freed.clear()
- # A close() arriving mid-call leaves the resource CLOSED.
- resource._lifecycle_state = LifecycleState.CLOSED
+ def close_then_swap(handle):
+ resource.close()
+ return replacement
- with self.assertRaises(Error):
- resource._consume_and_swap(lambda h: replacement, "swap: {}")
+ resource._consume_and_swap(close_then_swap, "swap: {}")
self.assertIn(replacement, self.freed)
+ self.assertIsNone(resource._handle)
+ self.assertEqual(resource._lifecycle_state, LifecycleState.CLOSED)
class TestContextProviderContract(unittest.TestCase):
@@ -9872,8 +10419,7 @@ def test_built_in_context_still_gets_in_flight_protection(self):
class TestLockOrderStaticAnalysis(unittest.TestCase):
- """Static analysis over the source, not runtime behavior:
- no threads are spawned here.
+ """Static analysis over the source: no threads are spawned here.
"""
def test_no_conflicting_lock_acquisition_order(self):
@@ -9920,13 +10466,19 @@ def lock_name_for_with(item):
and any(ctx.attr in attrs
for attrs in lock_attrs_by_class.values())):
return ctx.attr
- # with self._lock(): returns _op_lock itself.
+ # with self._guarded_op(): returns _op_lock itself, and the
+ # accessors return the lock they are named for.
+ lock_by_method = {
+ "_guarded_op": "_op_lock",
+ "_live_op_lock": "_op_lock",
+ "_live_teardown_lock": "_teardown_lock",
+ }
if (isinstance(ctx, ast.Call)
and isinstance(ctx.func, ast.Attribute)
- and ctx.func.attr == "_lock"
+ and ctx.func.attr in lock_by_method
and isinstance(ctx.func.value, ast.Name)
and ctx.func.value.id == "self"):
- return "_op_lock"
+ return lock_by_method[ctx.func.attr]
return None
def lock_name_for_acquire(node):
diff --git a/tests/test_unit_tests_threaded.py b/tests/test_unit_tests_threaded.py
index 20537e46..457c2a55 100644
--- a/tests/test_unit_tests_threaded.py
+++ b/tests/test_unit_tests_threaded.py
@@ -34,7 +34,7 @@
from c2pa import Builder, C2paError as Error, Reader, C2paSigningAlg as SigningAlg, C2paSignerInfo, Signer, sdk_version # noqa: E501
from c2pa import Context, Settings
-from c2pa.c2pa import ManagedResource, Stream, LifecycleState
+from c2pa.c2pa import ManagedResource, Stream, LifecycleState, _native_section
import c2pa.c2pa as c2pa_module
from c2pa.lib import is_foreign_process, record_owner_pid
@@ -213,7 +213,7 @@ def _foreign_reader_with_lock_held(self, fragment_lock=False):
def hold_the_lock():
held = (reader._fragment_lock if fragment_lock
- else reader._lock())
+ else reader._guarded_op())
with held:
holding.set()
release.wait(30)
@@ -228,6 +228,51 @@ def hold_the_lock():
reader._owner_pid = os.getpid() + 1
return reader
+ def _foreign_stream_with_close_lock_held(self):
+ """A Stream in the state a forked child inherits: _close_lock held by
+ a thread that does not exist in the child, and a foreign owner PID.
+ """
+ stream = Stream(io.BytesIO(b"payload"))
+ holding = threading.Event()
+ release = threading.Event()
+
+ def hold_the_lock():
+ with stream._close_lock:
+ holding.set()
+ release.wait(30)
+
+ holder = threading.Thread(target=hold_the_lock, daemon=True)
+ holder.start()
+ self.assertTrue(holding.wait(self._TIMEOUT),
+ "helper thread never acquired _close_lock")
+ # Cleanups run last-registered-first, so this one runs after
+ # release.set and holder.join.
+ self.addCleanup(self._reclaim_foreign_stream, stream)
+ self.addCleanup(holder.join, self._TIMEOUT)
+ self.addCleanup(release.set)
+
+ stream._owner_pid = os.getpid() + 1
+ return stream
+
+ def _reclaim_foreign_stream(self, stream):
+ """Release a stream the foreign-process path left tracked."""
+ stream._owner_pid = os.getpid()
+ stream._closed = False
+ stream.close()
+
+ def test_stream_close_completes_with_close_lock_held(self):
+ """close() must take the foreign-process path without acquiring
+ _close_lock, which no surviving thread would release."""
+ stream = self._foreign_stream_with_close_lock_held()
+
+ outcome = self._run_with_timeout(stream.close)
+
+ self.assertEqual(outcome, "ok",
+ "close() blocked on the inherited _close_lock")
+ self.assertTrue(stream._closed,
+ "close() returned without marking the stream closed")
+ self.assertFalse(stream._initialized)
+
def _run_with_timeout(self, operation):
"""Run operation on a worker; return 'ok', the exception, or None if it
was still running when the timeout expired."""
@@ -335,7 +380,7 @@ def test_parent_copy_unaffected(self):
class TestReaderWithFragmentConcurrency(unittest.TestCase):
"""with_fragment's native call and its stream-ownership transfer
- must must not interleave with another with_fragment on the same Reader.
+ must not interleave with another with_fragment on the same Reader.
"""
def setUp(self):
@@ -359,17 +404,15 @@ def test_close_during_with_fragment_does_not_double_close_stream(self):
entered_gap = threading.Event()
release_gap = threading.Event()
- real_native_call = reader._native_call
+ real_consume_and_swap = reader._consume_and_swap
- @contextlib.contextmanager
- def gated_native_call():
- with real_native_call():
- yield
+ def gated_consume_and_swap(ffi_call, error_message):
+ real_consume_and_swap(ffi_call, error_message)
# Pauses in with_fragment's window before it reassigns _own_stream/_fragment_streams.
entered_gap.set()
release_gap.wait(5)
- reader._native_call = gated_native_call
+ reader._consume_and_swap = gated_consume_and_swap
result = {}
@@ -388,7 +431,8 @@ def run_with_fragment():
entered_gap.wait(5),
"with_fragment never reached the post-native-call gap")
- # close() must win the race cleanly, not leave with_fragment hung, crashed, or silently successful.
+ # close() must win the race, and with_fragment must not hang,
+ # crash, or succeed without signalling.
reader.close()
release_gap.set()
worker.join(5)
@@ -432,11 +476,11 @@ def test_read_during_swap_never_serves_the_previous_handles_manifest(self):
# Populates the cache with the soon to be replaced handle.
self.assertEqual(reader.json(), before)
- real_lock = reader._lock
+ real_lock = reader._guarded_op
at_gap = threading.Event()
leave_gap = threading.Event()
# _native_call takes this lock before the swap does,
- # so park on the acquisition that actually performed the swap.
+ # so park on the acquisition that performed the swap.
swapped = []
class GatedLock:
@@ -458,7 +502,7 @@ def __exit__(self, exc_type, exc_val, exc_tb):
return result
swapped.append(reader._own_stream)
- reader._lock = lambda: GatedLock(real_lock())
+ reader._guarded_op = lambda **kw: GatedLock(real_lock(**kw))
served = {}
@@ -496,7 +540,7 @@ def read_in_gap():
"json() must not be served a manifest cached from the "
"handle with_fragment already replaced")
finally:
- reader._lock = real_lock
+ reader._guarded_op = real_lock
reader.close()
def test_manifest_accessors_stay_consistent_while_fragments_advance(self):
@@ -510,6 +554,7 @@ def test_manifest_accessors_stay_consistent_while_fragments_advance(self):
stop = threading.Event()
unexpected = []
served = []
+ swaps = []
def read_manifest():
while not stop.is_set():
@@ -525,10 +570,12 @@ def advance():
while not stop.is_set():
try:
self._advance(reader)
+ swaps.append(None)
except Error:
pass
except BaseException as e: # noqa: BLE001 - asserted below
unexpected.append(repr(e))
+ time.sleep(0.001)
workers = ([threading.Thread(target=read_manifest, daemon=True)
for _ in range(3)]
@@ -547,6 +594,9 @@ def advance():
"a manifest accessor or fragment advance hung")
self.assertEqual(unexpected, [])
self.assertTrue(served, "no manifest was ever read")
+ self.assertGreater(
+ len(swaps), 1,
+ "fragments did not advance during the run")
self.assertTrue(
set(served) <= valid,
"a manifest was served that matches neither the pre- nor the "
@@ -558,21 +608,19 @@ def test_interleaved_with_fragment_leaves_reader_consistent(self):
reader = Reader("video/mp4", io.BytesIO(self.init_bytes))
# Parks one call between its native call
- # and its native handle bookkeeping.
- real_native_call = reader._native_call
+ # and its stream bookkeeping.
+ real_consume_and_swap = reader._consume_and_swap
in_gap = threading.Event()
contended = threading.Event()
leave_gap = threading.Event()
- @contextlib.contextmanager
- def gated_native_call():
- with real_native_call():
- yield
+ def gated_consume_and_swap(ffi_call, error_message):
+ real_consume_and_swap(ffi_call, error_message)
if not in_gap.is_set():
in_gap.set()
leave_gap.wait(10)
- reader._native_call = gated_native_call
+ reader._consume_and_swap = gated_consume_and_swap
class ContentionReportingLock:
"""Flags when a caller finds the lock it wraps already held.
@@ -669,7 +717,7 @@ def second():
wrapper._closed,
"reader retained a released stream wrapper")
finally:
- reader._native_call = real_native_call
+ reader._consume_and_swap = real_consume_and_swap
reader._fragment_lock = real_fragment_lock
reader.close()
@@ -3474,7 +3522,7 @@ def seek(self, offset, whence=0):
"the lock the running call holds")
self.assertIsInstance(
state["result"], Error,
- "the re-entrant call must be refused, not silently interleaved")
+ "the re-entrant call must be refused, not interleaved")
def test_same_thread_reentry_does_not_corrupt_the_reader(self):
"""_fragment_lock is reentrant, so a callback calling with_fragment
@@ -3593,66 +3641,6 @@ def hold_then_reenter():
self.assertTrue(stream._closed)
-@unittest.skipUnless(hasattr(os, "fork"), "requires fork()")
-class TestStreamCloseAfterFork(unittest.TestCase):
- """A forked child must not wait on a lock no surviving thread will
- release.
- """
-
- def test_close_in_child_does_not_block_on_an_inherited_lock(self):
- stream = Stream(io.BytesIO(b"payload"))
-
- holding = threading.Event()
- release = threading.Event()
-
- def hold_the_lock():
- with stream._close_lock:
- holding.set()
- release.wait(30)
-
- holder = threading.Thread(target=hold_the_lock, daemon=True)
- holder.start()
- self.assertTrue(holding.wait(5), "lock was never taken")
-
- # The child inherits _close_lock held by a thread that does not exist
- # there, so close() has to take the foreign-process path without
- # acquiring it.
- pid = os.fork()
- if pid == 0:
- try:
- stream.close()
- # Exit 3 rather than 0 if close() returned without marking the
- # stream closed, so a silent no-op cannot pass as success.
- marked = stream._closed and not stream._initialized
- os._exit(0 if marked else 3)
- except BaseException:
- os._exit(2)
-
- deadline = time.time() + 15
- status = None
- while time.time() < deadline:
- done, wait_status = os.waitpid(pid, os.WNOHANG)
- if done:
- status = wait_status
- break
- time.sleep(0.05)
-
- if status is None:
- os.kill(pid, signal.SIGKILL)
- os.waitpid(pid, 0)
- release.set()
- holder.join(5)
- self.fail("close() in the forked child blocked on the inherited "
- "lock instead of taking the foreign-process path")
-
- release.set()
- holder.join(5)
- self.assertEqual(
- os.WEXITSTATUS(status), 0,
- "close() in the forked child raised (2) or returned without "
- "closing the stream (3)")
-
-
class TestConsumeReservationWindow(unittest.TestCase):
"""The consume reservation must outlast ownership classification.
@@ -3680,7 +3668,7 @@ def gated_read():
def observer():
if not reading.wait(10):
return
- # The consuming call is mid-classification right now.
+ # The consuming call is mid-classification at this point.
seen_valid.append(resource.is_valid)
may_finish.set()
@@ -3825,6 +3813,251 @@ def make_and_drop(index):
self.assertEqual(set(counts.values()), {1},
"a dropped resource was freed more than once")
+ def test_cross_closing_inside_lock_regions_does_not_deadlock(self):
+ """Tests cocnurrent closes do not deadlock.
+ """
+ first = _ConcreteResource()
+ first._activate(0x40001)
+ second = _ConcreteResource()
+ second._activate(0x40002)
+
+ holding = threading.Barrier(2, timeout=5)
+ queued = threading.Barrier(2, timeout=5)
+ failures = []
+
+ def worker(mine, theirs):
+ try:
+ with mine._guarded_op():
+ # Both locks required,
+ holding.wait()
+ theirs.close()
+ # Teardowns queue.
+ queued.wait()
+ except BaseException as error:
+ failures.append(error)
+
+ threads = [
+ threading.Thread(target=worker, args=(first, second), daemon=True),
+ threading.Thread(target=worker, args=(second, first), daemon=True),
+ ]
+ for thread in threads:
+ thread.start()
+ self._join_all(threads, "cross-closing workers")
+
+ self.assertEqual(failures, [], "workers raised: {}".format(failures))
+
+ counts = {handle: value
+ for handle, value in self._free_counts().items()
+ if handle in (0x40001, 0x40002)}
+ self.assertEqual(counts, {0x40001: 1, 0x40002: 1},
+ "cross-closed handles were not each freed once")
+
+ def test_failed_locked_region_still_flushes_a_queued_teardown(self):
+ resource = _ConcreteResource()
+ resource._activate(0x50001)
+
+ holding = threading.Event()
+ queued = threading.Event()
+
+ def holder():
+ try:
+ with resource._guarded_op():
+ holding.set()
+ queued.wait(self.JOIN_TIMEOUT)
+ raise RuntimeError("locked region failed")
+ except RuntimeError:
+ pass
+
+ def closer():
+ holding.wait(self.JOIN_TIMEOUT)
+ resource.close()
+ queued.set()
+
+ threads = [
+ threading.Thread(target=holder, daemon=True),
+ threading.Thread(target=closer, daemon=True),
+ ]
+ for thread in threads:
+ thread.start()
+ self._join_all(threads, "failing locked region")
+
+ self.assertEqual(self._free_counts().get(0x50001), 1,
+ "a teardown queued during the region was orphaned")
+
+ def test_close_racing_a_consumed_handle_does_not_free_it(self):
+ resource = _ConcreteResource()
+ resource._activate(0x50002)
+
+ resource._inflight = 1
+ resource._teardown(free_handle=False)
+ self.assertIs(resource._pending_teardown, False,
+ "the consume was not recorded")
+
+ resource._inflight = 0
+ resource.close()
+ self.assertIsNone(self._free_counts().get(0x50002),
+ "a consumed handle was freed by a racing close")
+
+ resource._maybe_flush_pending()
+ self.assertIsNone(self._free_counts().get(0x50002),
+ "a later flush freed a consumed handle")
+
+ def test_close_against_a_bare_lock_holder_is_not_orphaned(self):
+ resource = _ConcreteResource()
+ resource._activate(0x50003)
+
+ holding = threading.Event()
+ release = threading.Event()
+
+ def holder():
+ with resource._live_op_lock():
+ holding.set()
+ release.wait(self.JOIN_TIMEOUT)
+ resource._release_handle()
+
+ def closer():
+ holding.wait(self.JOIN_TIMEOUT)
+ resource.close()
+ release.set()
+
+ threads = [
+ threading.Thread(target=holder, daemon=True),
+ threading.Thread(target=closer, daemon=True),
+ ]
+ for thread in threads:
+ thread.start()
+ self._join_all(threads, "bare lock holder")
+
+ self.assertEqual(self._free_counts().get(0x50003), 1,
+ "a teardown queued against the lock was orphaned")
+
+ def test_close_queued_inside_a_flush_hold_is_not_orphaned(self):
+ resource = _ConcreteResource()
+ resource._activate(0x60001)
+
+ real_lock = resource._op_lock
+ closed = threading.Event()
+ join_timeout = self.JOIN_TIMEOUT
+
+ class GatedLock:
+ def acquire(self, blocking=True, timeout=-1):
+ if timeout == -1:
+ return real_lock.acquire(blocking)
+ return real_lock.acquire(blocking, timeout)
+
+ def release(self):
+ return real_lock.release()
+
+ def __enter__(self):
+ real_lock.acquire()
+ return self
+
+ def __exit__(self, exc_type, exc_val, exc_tb):
+ if not closed.is_set():
+ worker = threading.Thread(
+ target=lambda: (resource.close(), closed.set()),
+ daemon=True)
+ worker.start()
+ worker.join(join_timeout)
+ real_lock.release()
+ return False
+
+ resource._op_lock = GatedLock()
+ try:
+ resource._maybe_flush_pending()
+ finally:
+ resource._op_lock = real_lock
+
+ self.assertEqual(self._free_counts().get(0x60001), 1,
+ "a teardown queued during a flush was orphaned")
+
+ def test_close_recording_after_a_flush_is_not_orphaned(self):
+ resource = _ConcreteResource()
+ resource._activate(0x70001)
+
+ real_lock = resource._op_lock
+ real_record = ManagedResource._record_pending_intent
+ reached_record = threading.Event()
+ flusher_done = threading.Event()
+ closer_done = threading.Event()
+ join_timeout = self.JOIN_TIMEOUT
+
+ def gated_record(target, free_handle):
+ if (target is resource
+ and threading.current_thread().name == "delayed-closer"):
+ reached_record.set()
+ flusher_done.wait(join_timeout)
+ return real_record(target, free_handle)
+
+ class GatedLock:
+ def acquire(self, blocking=True, timeout=-1):
+ if timeout == -1:
+ return real_lock.acquire(blocking)
+ return real_lock.acquire(blocking, timeout)
+
+ def release(self):
+ return real_lock.release()
+
+ def __enter__(self):
+ real_lock.acquire()
+ return self
+
+ def __exit__(self, exc_type, exc_val, exc_tb):
+ if not closer_done.is_set() and not reached_record.is_set():
+ worker = threading.Thread(
+ target=lambda: (resource.close(), closer_done.set()),
+ name="delayed-closer",
+ daemon=True)
+ worker.start()
+ reached_record.wait(join_timeout)
+ real_lock.release()
+ return False
+
+ ManagedResource._record_pending_intent = gated_record
+ resource._op_lock = GatedLock()
+ try:
+ resource._maybe_flush_pending()
+ finally:
+ resource._op_lock = real_lock
+ flusher_done.set()
+ closer_done.wait(join_timeout)
+ ManagedResource._record_pending_intent = real_record
+
+ self.assertEqual(self._free_counts().get(0x70001), 1,
+ "a teardown recorded after a flush was orphaned")
+
+ def test_stream_finalizer_does_not_block_on_a_held_close_lock(self):
+ stream = Stream(io.BytesIO(self.image_bytes))
+ self.addCleanup(stream.close)
+
+ holding = threading.Event()
+ release = threading.Event()
+ returned = threading.Event()
+
+ def holder():
+ with stream._close_lock:
+ holding.set()
+ release.wait(self.JOIN_TIMEOUT)
+
+ def finalizer():
+ stream.__del__()
+ returned.set()
+
+ holder_thread = threading.Thread(target=holder, daemon=True)
+ holder_thread.start()
+ self.assertTrue(holding.wait(self.JOIN_TIMEOUT),
+ "holder never took the close lock")
+
+ finalizer_thread = threading.Thread(target=finalizer, daemon=True)
+ finalizer_thread.start()
+ finalizer_thread.join(5)
+ blocked = not returned.is_set()
+
+ release.set()
+ self._join_all([holder_thread, finalizer_thread], "stream finalizer")
+ self.assertFalse(blocked,
+ "__del__ waited for a close lock held elsewhere")
+
def test_settings_relayed_across_threads_stays_usable(self):
ManagedResource._free_native_ptr = self._real_free
@@ -3932,12 +4165,13 @@ def test_finalizer_inside_locked_operation(self):
class Dropped:
def __del__(self):
- # Runs on this thread, inside the locked region below.
- with resource._lock():
+ # Runs on this thread, inside the locked region body()
+ # holds.
+ with resource._guarded_op():
observed.append(True)
def body():
- with resource._lock():
+ with resource._guarded_op():
dropped = Dropped()
del dropped
gc.collect()
@@ -4156,32 +4390,36 @@ def test_no_nested_op_locks(self):
data = self.image_bytes
held = threading.local()
violations = []
- real_lock = ManagedResource._lock
-
- def tracking_lock(resource):
- lock = real_lock(resource)
- depth = getattr(held, 'stack', None)
- if depth is None:
- depth = held.stack = []
-
- class Tracked:
- def __enter__(self):
- others = [r for r in depth if r is not resource]
- if others:
- violations.append(
- "{} while holding {}".format(
- type(resource).__name__,
- [type(o).__name__ for o in others]))
- depth.append(resource)
- return lock.__enter__()
-
- def __exit__(self, *exc):
- depth.pop()
- return lock.__exit__(*exc)
-
- return Tracked()
-
- ManagedResource._lock = tracking_lock
+ real_lock = ManagedResource._guarded_op
+ real_live_op_lock = ManagedResource._live_op_lock
+
+ def make_tracking(real):
+ def tracking(resource, **kw):
+ lock = real(resource, **kw)
+ depth = getattr(held, 'stack', None)
+ if depth is None:
+ depth = held.stack = []
+
+ class Tracked:
+ def __enter__(self):
+ others = [r for r in depth if r is not resource]
+ if others:
+ violations.append(
+ "{} while holding {}".format(
+ type(resource).__name__,
+ [type(o).__name__ for o in others]))
+ depth.append(resource)
+ return lock.__enter__()
+
+ def __exit__(self, *exc):
+ depth.pop()
+ return lock.__exit__(*exc)
+
+ return Tracked()
+ return tracking
+
+ ManagedResource._guarded_op = make_tracking(real_lock)
+ ManagedResource._live_op_lock = make_tracking(real_live_op_lock)
try:
reader = Reader("image/jpeg", io.BytesIO(data))
reader.json()
@@ -4190,7 +4428,8 @@ def __exit__(self, *exc):
reader.get_remote_url()
reader.close()
finally:
- ManagedResource._lock = real_lock
+ ManagedResource._guarded_op = real_lock
+ ManagedResource._live_op_lock = real_live_op_lock
self.assertEqual(violations, [],
"a thread held two operation locks at once")
@@ -4232,6 +4471,49 @@ def closer_worker():
self._join_all(threads, "concurrent storm")
self.assertEqual(errors, [])
+ def test_native_section_deferred_free_is_thread_local(self):
+ """Two threads each with their own open native-error section: one
+ thread's section closing must not flush a free deferred inside
+ the other thread's still-open section.
+ """
+ freed = self._counted_free()
+ resource = _ConcreteResource()
+ resource._activate(0x1001)
+
+ thread_ready = threading.Event()
+ release_thread = threading.Event()
+
+ def worker():
+ with _native_section():
+ resource.close()
+ thread_ready.set()
+ release_thread.wait(self.JOIN_TIMEOUT)
+ # Flush happens here, on the worker thread, once its own
+ # section closes.
+
+ thread = threading.Thread(target=worker)
+ thread.start()
+ try:
+ self.assertTrue(
+ thread_ready.wait(self.JOIN_TIMEOUT),
+ "worker thread did not reach its open section in time")
+
+ # A section opened and closed entirely on this (main) thread,
+ # while the worker's section is still open on its own thread.
+ with _native_section():
+ pass
+
+ self.assertEqual(
+ freed, [],
+ "a different thread's section flushed this thread's "
+ "pending resource")
+ finally:
+ release_thread.set()
+ self._join_all([thread], "native-section worker")
+
+ self.assertEqual(freed, [0x1001],
+ "worker thread's own section never flushed")
+
def _counted_free(self):
"""Patch _free_native_ptr to count frees; returns the list."""
freed = []
@@ -4279,6 +4561,31 @@ def write(self, buffer):
self.assertIsNone(reader._pending_teardown)
self.assertEqual(reader._lifecycle_state, LifecycleState.CLOSED)
+ def test_with_fragment_closes_main_stream_when_second_stream_fails(self):
+ """Streams in with_fragment on failure must not get into a broken state"""
+ opened = []
+ real_init = Stream.__init__
+
+ def tracking_init(wrapper, source):
+ if opened:
+ raise ValueError("fragment stream could not be built")
+ real_init(wrapper, source)
+ opened.append(wrapper)
+
+ reader = Reader("image/jpeg", io.BytesIO(self.image_bytes))
+ self.addCleanup(reader.close)
+
+ with patch.object(Stream, '__init__', tracking_init):
+ with self.assertRaises(ValueError):
+ reader.with_fragment(
+ "video/mp4",
+ io.BytesIO(self.image_bytes),
+ io.BytesIO(self.image_bytes))
+
+ self.assertEqual(len(opened), 1, "main stream was never built")
+ self.assertTrue(opened[0].closed,
+ "main stream was left open for the collector")
+
def test_cross_thread_close_during_callback_defers_free(self):
"""A close() from inside a stream callback must not free the handle
the native call is still using."""
@@ -4312,8 +4619,8 @@ def closer():
self.assertEqual(reader._inflight, 0)
def test_deferred_teardown_still_closes(self):
- """After a deferred free the resource is closed and a later close()
- is a no-op rather than a second free."""
+ """After a deferred free the resource is closed and a later
+ close() frees nothing."""
freed = self._counted_free()
reader = Reader("image/jpeg", io.BytesIO(self.image_bytes))
uri = self._thumbnail_uri(reader)
@@ -4621,8 +4928,8 @@ def test_concurrent_close_runs_release_once(self):
The native free is already single (the handle is nulled after the
first teardown), so a free-counting test cannot see this: it is
- _release() -- the Python-side stream/cache cleanup a subclass
- overrides -- that must not run twice. _teardown() has to be
+ What must not run twice is _release(), the Python-side
+ stream/cache cleanup a subclass overrides. _teardown() has to be
idempotent under its own lock.
Gate _teardown so the first close() pauses on entry, before taking
@@ -4776,14 +5083,13 @@ def test_every_callback_running_method_is_guarded(self):
checked += 1
if key in class_a:
continue
- if "_native_call()" not in body:
+ if ("_native_call()" not in body
+ and "_exclusive_native_call()" not in body
+ and "_consume_and_swap(" not in body):
unguarded.append("{}.{}".format(*key))
self.assertGreater(checked, 0, "coverage scan found no methods")
- self.assertEqual(
- unguarded, [],
- "these hand a Stream to native without _native_call(): {}".format(
- unguarded))
+ self.assertEqual(unguarded, [])
def test_every_borrowed_handle_is_guarded(self):
"""When a method hands a second object's handle to the native library,
@@ -4893,7 +5199,6 @@ def visit(node, active):
"borrowed handles used without their own guard:\n "
+ "\n ".join(unguarded))
-
def test_consume_during_concurrent_sign_does_not_crash(self):
"""Consuming a shared Signer must not free it under a live sign.
@@ -4927,7 +5232,7 @@ def sign():
io.BytesIO(img), io.BytesIO())
builder.close()
except Exception:
- # A consumed signer may legitimately be rejected;
+ # A consumed signer may be rejected;
# only a crash is a failure here.
pass
@@ -5014,7 +5319,7 @@ def worker():
io.BytesIO())
builder.close()
except Exception:
- # A closed context may legitimately be rejected;
+ # A closed context may be rejected;
# only a crash is a failure here.
entered.set()
@@ -5065,6 +5370,101 @@ def test_context_close_during_sign_defers_teardown(self):
"the deferred teardown never ran")
self.assertIsNone(context._pending_teardown)
+ def test_deferred_teardown_survives_a_flush_inside_a_section(self):
+ """A flush blocked by a section must re-register, not drop the free.
+
+ The teardown defers on _inflight, so it is queued for the in-flight
+ call rather than for a section. When that call finishes inside a
+ section opened later on this thread, the flush cannot free yet, and
+ without re-registering nothing would ever free this handle.
+ """
+ context = Context()
+ freed = []
+ real_free = ManagedResource._free_native_ptr
+ ManagedResource._free_native_ptr = staticmethod(
+ lambda ptr: (freed.append(ptr), real_free(ptr))[1])
+ try:
+ with context._native_call():
+ closer = threading.Thread(target=context.close)
+ closer.start()
+ closer.join()
+ self.assertIsNotNone(
+ context._pending_teardown,
+ "close() during a native call should defer")
+ section = _native_section()
+ section.__enter__()
+
+ self.assertEqual(
+ freed, [],
+ "the flush freed while a native section was still open")
+ self.assertIsNotNone(
+ context._pending_teardown,
+ "the deferral was dropped")
+
+ section.__exit__(None, None, None)
+ self.assertEqual(
+ len(freed), 1,
+ "the deferred teardown was stranded and never freed")
+ self.assertIsNone(context._pending_teardown)
+ finally:
+ ManagedResource._free_native_ptr = real_free
+
+ def test_abort_consume_leaves_a_queued_teardown_closed(self):
+ """A resource whose free is already queued must not become usable.
+
+ The deferred free still runs when the section drains, so restoring
+ ACTIVE would hand the caller a resource that closes underneath it.
+ """
+ context = Context()
+ with _native_section():
+ context.close()
+ self.assertIsNotNone(context._pending_teardown)
+
+ context._abort_consume(LifecycleState.ACTIVE)
+ self.assertEqual(
+ context._lifecycle_state, LifecycleState.CLOSED,
+ "a resource with a queued teardown was revived")
+ self.assertFalse(
+ context.is_valid,
+ "a resource with a queued teardown reported itself usable")
+
+ def test_section_drain_error_does_not_mask_the_body_error(self):
+ """The body's exception is what the caller asked for, so it wins."""
+
+ class FlushRaises:
+ _pending_teardown = True
+
+ def _maybe_flush_pending(self):
+ raise RuntimeError("flush failed")
+
+ class BodyError(Exception):
+ pass
+
+ with self.assertLogs('c2pa', level='ERROR') as logs:
+ with self.assertRaises(BodyError):
+ with _native_section():
+ c2pa_module._register_for_section_flush(FlushRaises())
+ raise BodyError("the error the caller cares about")
+
+ self.assertTrue(
+ any("flush failed" in line for line in logs.output),
+ "the flush failure was not logged")
+
+ def test_drain_errors_log(self):
+ """Log flushing failures."""
+
+ class FlushRaises:
+ _pending_teardown = True
+
+ def _maybe_flush_pending(self):
+ raise RuntimeError("flush failed")
+
+ with self.assertLogs("c2pa", level="ERROR") as captured:
+ with _native_section():
+ c2pa_module._register_for_section_flush(FlushRaises())
+ self.assertTrue(
+ any("flush failed" in message for message in captured.output))
+
def test_context_sign_after_close_raises_rather_than_skipping_signer(self):
"""Signing through a closed Context must raise, not silently succeed.
@@ -5158,7 +5558,7 @@ def sign():
io.BytesIO(img), io.BytesIO())
b.close()
except Exception:
- # A closed signer may legitimately be rejected;
+ # A closed signer may be rejected;
# only a crash is a failure here.
pass
@@ -5188,5 +5588,315 @@ def sign():
self.assertIn("OK", result.stdout)
+class TestSwapConsumeExclusion(unittest.TestCase):
+ """with_archive, with_fragment must be rejected during other in-flight calls.
+ """
+
+ _MANIFEST = {
+ "claim_generator": "c2pa_python_test",
+ "claim_generator_info": [{
+ "name": "c2pa_python_test",
+ "version": "0.1.0",
+ }],
+ "format": "image/jpeg",
+ "title": "Python Test",
+ "ingredients": [],
+ "assertions": [],
+ }
+
+ def _archive_bytes(self):
+ builder = Builder(self._MANIFEST)
+ try:
+ archive = io.BytesIO()
+ builder.to_archive(archive)
+ archive.seek(0)
+ return archive
+ finally:
+ builder.close()
+
+ def test_with_archive_rejected_when_to_archive_in_progress(self):
+ archive = self._archive_bytes()
+ builder = Builder(self._MANIFEST)
+
+ inside = threading.Event()
+ release = threading.Event()
+
+ class BlockingSink(io.BytesIO):
+ def write(self, data):
+ inside.set()
+ release.wait(10)
+ return super().write(data)
+
+ def seek(self, *args):
+ inside.set()
+ release.wait(10)
+ return super().seek(*args)
+
+ borrow_errors = []
+
+ def borrow():
+ try:
+ builder.to_archive(BlockingSink())
+ except Exception as e: # noqa: BLE001 - asserted below
+ borrow_errors.append(e)
+
+ worker = threading.Thread(target=borrow, daemon=True)
+ worker.start()
+ try:
+ self.assertTrue(
+ inside.wait(10), "to_archive never reached its callback")
+
+ with self.assertRaises(Error) as raised:
+ builder.with_archive(archive)
+ self.assertIn("in use", str(raised.exception))
+ finally:
+ release.set()
+ worker.join(10)
+
+ self.assertFalse(worker.is_alive(), "to_archive hung")
+ self.assertEqual(borrow_errors, [])
+
+ # The refusal must leave the builder untouched and usable.
+ self.assertEqual(builder._lifecycle_state, LifecycleState.ACTIVE)
+ builder.add_action('{"action": "c2pa.color_adjustments"}')
+ builder.close()
+
+ def test_with_fragment_rejected_when_native_in_progress(self):
+ init_path = os.path.join(FIXTURES_FOLDER, "dashinit.mp4")
+ fragment_path = os.path.join(FIXTURES_FOLDER, "dash1.m4s")
+ with open(init_path, "rb") as f:
+ init_bytes = f.read()
+ with open(fragment_path, "rb") as f:
+ fragment_bytes = f.read()
+
+ reader = Reader("video/mp4", io.BytesIO(init_bytes))
+ try:
+ with reader._native_call():
+ with self.assertRaises(Error) as raised:
+ reader.with_fragment(
+ "video/mp4",
+ io.BytesIO(init_bytes),
+ io.BytesIO(fragment_bytes))
+ self.assertIn("in use", str(raised.exception))
+
+ # The refusal must leave the reader untouched: the swap still
+ # works once the borrow is gone.
+ self.assertEqual(reader._lifecycle_state, LifecycleState.ACTIVE)
+ reader.with_fragment(
+ "video/mp4",
+ io.BytesIO(init_bytes),
+ io.BytesIO(fragment_bytes))
+ reader.json()
+ finally:
+ reader.close()
+
+ def test_close_during_with_archive_defers_and_frees(self):
+ archive_bytes = self._archive_bytes().getvalue()
+ builder = Builder(self._MANIFEST)
+
+ inside = threading.Event()
+ release = threading.Event()
+
+ class BlockingArchive(io.BytesIO):
+ def read(self, *args):
+ inside.set()
+ release.wait(10)
+ return super().read(*args)
+
+ def seek(self, *args):
+ inside.set()
+ release.wait(10)
+ return super().seek(*args)
+
+ outcome = {}
+
+ def consume():
+ try:
+ builder.with_archive(BlockingArchive(archive_bytes))
+ outcome["result"] = "ok"
+ except Exception as e: # noqa: BLE001 - asserted below
+ outcome["result"] = e
+
+ worker = threading.Thread(target=consume, daemon=True)
+ worker.start()
+ try:
+ self.assertTrue(
+ inside.wait(10), "with_archive never reached its callback")
+ # Defers: the swap is counted in flight.
+ builder.close()
+ finally:
+ release.set()
+ worker.join(10)
+
+ self.assertFalse(worker.is_alive(), "with_archive hung")
+ # The deferred teardown freed the replacement handle: closed for
+ # good, nothing left to free, exactly one release.
+ self.assertEqual(builder._lifecycle_state, LifecycleState.CLOSED)
+ self.assertIsNone(builder._handle)
+ self.assertTrue(builder._released)
+ self.assertIsNone(builder._pending_teardown)
+
+ def test_calling_close_should_not_corrupt_other_objects(self):
+ """Other threads asking for close() should not corrupt objects.
+ """
+ real_free = ManagedResource._free_native_ptr
+
+ k = 1
+ while True:
+ archive = self._archive_bytes()
+ builder = Builder(self._MANIFEST)
+
+ freed = []
+ ManagedResource._free_native_ptr = staticmethod(
+ lambda p, _real=real_free: (freed.append(int(
+ ctypes.cast(p, ctypes.c_void_p).value or 0)),
+ _real(p))[1])
+
+ real_live_op_lock = builder._live_op_lock
+ enters = [0]
+ injected = []
+
+ class LockProxy:
+ def __init__(self, inner):
+ self._inner = inner
+
+ def __enter__(self):
+ enters[0] += 1
+ self._n = enters[0]
+ self._inner.__enter__()
+ return self
+
+ def __exit__(self, *exc):
+ result = self._inner.__exit__(*exc)
+ if self._n == k and not injected:
+ injected.append(True)
+ builder._live_op_lock = real_live_op_lock
+ closer = threading.Thread(target=builder.close)
+ closer.start()
+ closer.join(10)
+ builder._live_op_lock = gated
+ return result
+
+ def gated(_lock=real_live_op_lock):
+ return LockProxy(_lock())
+
+ builder._live_op_lock = gated
+ try:
+ try:
+ builder.with_archive(archive)
+ except Error:
+ pass
+ finally:
+ builder._live_op_lock = real_live_op_lock
+ ManagedResource._free_native_ptr = real_free
+
+ with self.subTest(injection_point=k):
+ self.assertFalse(
+ builder._released
+ and builder._lifecycle_state == LifecycleState.ACTIVE,
+ "resource resurrected to ACTIVE after its close()")
+ self.assertEqual(
+ len(freed), len(set(freed)),
+ f"a pointer was freed twice: {freed}")
+ builder.close()
+ self.assertIsNone(
+ builder._handle,
+ "a handle survived every close(): it leaks")
+
+ if not injected:
+ # k exceeded the number of lock releases in the
+ # operation: the sweep is complete.
+ self.assertGreater(k, 2, "sweep never covered the "
+ "historical bug's window")
+ break
+ k += 1
+
+ def test_second_mutating_call_is_rejected(self):
+ builder = Builder(self._MANIFEST)
+
+ inside = threading.Event()
+ release = threading.Event()
+
+ class BlockingSink(io.BytesIO):
+ def write(self, data):
+ inside.set()
+ release.wait(10)
+ return super().write(data)
+
+ def seek(self, *args):
+ inside.set()
+ release.wait(10)
+ return super().seek(*args)
+
+ worker = threading.Thread(
+ target=lambda: builder.to_archive(BlockingSink()), daemon=True)
+ worker.start()
+ try:
+ self.assertTrue(
+ inside.wait(10), "to_archive never reached its callback")
+
+ with self.assertRaises(Error) as second_mut:
+ builder.to_archive(io.BytesIO())
+ self.assertIn("mutating operation", str(second_mut.exception))
+
+ # A _lock-path native call is refused too: the in-flight
+ # mutating call holds `&mut` on the same native object.
+ with self.assertRaises(Error) as read_call:
+ builder.add_action('{"action": "c2pa.color_adjustments"}')
+ self.assertIn("mutating operation", str(read_call.exception))
+ finally:
+ release.set()
+ worker.join(10)
+
+ self.assertFalse(worker.is_alive(), "first to_archive hung")
+ # Both refused calls work once the mutating call has returned.
+ builder.to_archive(io.BytesIO())
+ builder.add_action('{"action": "c2pa.color_adjustments"}')
+ builder.close()
+
+ def test_read_during_mutation_is_rejected(self):
+ with open(os.path.join(FIXTURES_FOLDER, "C.jpg"), "rb") as f:
+ image = f.read()
+ reader = Reader("image/jpeg", io.BytesIO(image))
+ manifest = reader.get_active_manifest()
+ uri = (manifest or {}).get("thumbnail", {}).get("identifier")
+ self.assertTrue(uri, "fixture must carry a thumbnail resource")
+
+ inside = threading.Event()
+ release = threading.Event()
+
+ class BlockingSink(io.BytesIO):
+ def write(self, data):
+ inside.set()
+ release.wait(10)
+ return super().write(data)
+
+ def seek(self, *args):
+ inside.set()
+ release.wait(10)
+ return super().seek(*args)
+
+ worker = threading.Thread(
+ target=lambda: reader.resource_to_stream(uri, BlockingSink()),
+ daemon=True)
+ worker.start()
+ try:
+ self.assertTrue(
+ inside.wait(10),
+ "resource_to_stream never reached its callback")
+
+ with self.assertRaises(Error) as raised:
+ reader.detailed_json()
+ self.assertIn("mutating operation", str(raised.exception))
+ finally:
+ release.set()
+ worker.join(10)
+
+ self.assertFalse(worker.is_alive(), "resource_to_stream hung")
+ # Works again once the mutating call has returned.
+ self.assertTrue(reader.detailed_json())
+ reader.close()
+
+
if __name__ == '__main__':
unittest.main()