Crash when terminating process if Py_Finalize isn't called

[tonyroberts]

What happens?

If Python is embedded in another application and the process terminates without a chance for Py_Finalize to be called, there's a DuckDBPyConnection object that gets destroyed by the C runtime when the process ends, and that causes the process to crash.

In ~DuckDBPyConnection it's assumed the GIL is already held, but this isn't true when the process is being terminated (gil_scoped_release calls PyEval_SaveThread, and PyEval_SaveThread crashes because the GIL is not held). This object is cleaned up if Py_Finalize is called via a capsule, but that shouldn't be relied on.

A quick fix would be to add Py_IsInitialized and PyGILState_Check checks to ~DuckDBPyConnection, but a better fix would be to not do this cleanup in the destructor of a static and instead to move this to module state so Python can manage its lifecycle. I see there's a comment about PEP 489 already in the code suggesting the same.

To Reproduce

# Importing duckdb is enough to cause this script to crash.
# Comment this line out and it will work fine.
import duckdb
                                                               
import ctypes                                                                                                               
                                                                                                                                                                                                                                                           
# ctypes releases the GIL around this call (standard behavior for foreign DLL calls,
# unlike ctypes.pythonapi calls). ExitProcess never returns, so the GIL stays released
# from here on - (no Python code running, GIL not held).
# Py_Finalize is never called. ExitProcess still triggers DLL_PROCESS_DETACH for every                                       
# loaded DLL, so _duckdb.pyd's static destructor runs with no GIL held and no valid
# thread state for this thread.
ctypes.windll.kernel32.ExitProcess(0)

OS:

Windows

DuckDB Package Version:

1.5.4

Python Version:

3.14.5

Full Name:

Tony Roberts

Affiliation:

PyXLL Ltd

What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have tested with a source build

Did you include all relevant data sets for reproducing the issue?

Not applicable - the reproduction does not require a data set

Did you include all code required to reproduce the issue?

• Yes, I have

Did you include all relevant configuration to reproduce the issue?

• Yes, I have

[abas-coder]

Possibly-related data point from a non-embedded process, offered because it may affect how this is triaged. I want to be explicit up front about what is verified and what is not: I have the same faulting frame, but no stack, so I cannot confirm the caller is ~DuckDBPyConnection.

Environment

• Windows 11, CPython 3.14.4 (3.14.4150.1013, MSC v.1944, x64), standard GIL-enabled build (Py_GIL_DISABLED=0, sys._is_gil_enabled() == True)
• duckdb 1.5.4 (_duckdb.cp314-win_amd64.pyd)
• Not embedded. A plain scheduled python -m <package> run — no host application, no PyXLL, no ExitProcess call anywhere in the codebase.

The frame matches your description exactly

Two crashes, identical WER bucket and fault offset:

Faulting module : python314.dll  3.14.4150.1013
Exception       : 0xC0000005 (STATUS_ACCESS_VIOLATION)
Fault offset    : 0x10298

python314.dll+0x10298 resolves to PyEval_SaveThread+0x34. Resolved without symbols (python.org ships no PDB for python314.dll), by two independent routes:

• The DLL's own .pdata / IMAGE_DIRECTORY_ENTRY_EXCEPTION table: the RUNTIME_FUNCTION covering 0x10298 is BeginAddress=0x00010264, EndAddress=0x00010326. Since the x64 unwind table is a complete function-extent map emitted by the linker, that is a real function boundary rather than a nearest-export guess.
• The export table maps RVA 0x00010264 to PyEval_SaveThread (ordinal 279). Delta +0x34.

Corroboration that this really is the GIL-release path: every indirect call inside 0x10264–0x10326 resolves through the IAT to AcquireSRWLockExclusive, WakeConditionVariable, ReleaseSRWLockExclusive — i.e. drop_gil() on Windows.

The faulting instruction, and why it is a NULL thread state

Bytes at RVA 0x10298 are 83 67 20 ef:

and dword ptr [rdi+0x20], 0FFFFFFEFh    ; clear bit 4, 32-bit read-modify-write

The preceding instructions load rdi straight from thread-local storage and then clear the slot:

0x10273  mov  rax, gs:[0x58]        ; TEB->ThreadLocalStoragePointer
0x1027C  mov  ecx, [rip+0x5DD986]   ; _tls_index
0x10282  mov  r8d, 8
0x10288  mov  rdx, [rax+rcx*8]      ; this thread's TLS block
0x1028C  mov  rdi, [rdx+r8]         ; <- current PyThreadState
0x10290  mov  qword [rdx+r8], 0     ; clear the slot
0x10298  and  dword ptr [rdi+0x20], 0FFFFFFEFh   ; <- FAULTS

There is no test rdi, rdi between the load and the dereference. So if the thread has no attached thread state, rdi == 0 and the write targets address 0x20. That is consistent with your analysis — PyEval_SaveThread called with the GIL not held / no valid thread state — and it is a write, not a read, since AND with a memory destination is read-modify-write.

Worth noting for anyone tempted to treat this as an interpreter bug: the fault site is source-identical from 3.14.4 through 3.14.7, so a CPython patch bump does not move it.

What I have NOT verified

• No call stack. Windows had LocalDumps at the default DumpType=1, so the minidumps contain no thread stack memory (the faulting thread's RSP falls outside every captured range). I therefore cannot show that the caller is ~DuckDBPyConnection, or that this is your bug rather than a different route to the same frame.
• My association is only that _duckdb.cp314-win_amd64.pyd was loaded in both same-bucket crashes, alongside pydantic_core, numpy, yaml/_yaml and markupsafe. That is circumstantial.
• Your repro is Python 3.14.5; mine is 3.14.4.

Why it might still be relevant to triage

Your reproducer needs an embedded interpreter terminating without Py_Finalize. Mine is an ordinary long-running python -m process that had been working for 2h44m and died mid-task, which — if it is the same root cause — would mean the destructor path is reachable outside the embedded/ExitProcess scenario, and the needs triage severity would be higher than the report implies.

One hypothesis I cannot yet test: the crash landed ~21 s after the process logged a lost network connection, and none of its normal shutdown log lines were ever written. If the run had begun tearing down, a static DuckDBPyConnection destructor firing during process detach would fit — but that is speculation until I have a stack.

Offer

I am arming LocalDumps with DumpType=2 (full memory) scoped to python.exe. This crash has recurred twice in six days on real workloads, so I expect another. If a full dump would help, I am happy to post the resolved stack above PyEval_SaveThread — which would either confirm ~DuckDBPyConnection or rule this out as unrelated noise on your issue. Say the word and I will follow up either way.