cuda.core: define the error handling policy and report failures that cannot be raised

cuda_core/AGENTS.md

@@ -101,6 +101,66 @@ and agents should flag violations.
   (kernel arguments, memcpy/memset operands, `dst_owner`/`src_owner`, and
   host-callback closures) inherit this contract.
 
+## Failure handling
+
+The user-facing contract lives in `docs/source/error_handling.rst`; the rules
+below are for contributors. Reviewers and agents should flag violations.
+
+- **Raise by default**: any failure on a path where an exception can propagate
+  raises. Driver statuses go through `HANDLE_RETURN` (Cython) or are returned as
+  `CUresult` from the C++ handle layer and then `HANDLE_RETURN`ed; never
+  replace a `CUresult` with a generic `RuntimeError`, and drain
+  `get_last_error()` immediately after a handle constructor returns empty so a
+  stale status cannot be misattributed later.
+- **Guarantees**: a call that creates a resource must create nothing when it
+  raises (undo the creation if a later step fails). Every call except
+  `Device.set_current` must leave the calling thread's current context as it
+  found it. Do not hand-roll `cuCtxPush/Pop/SetCurrent` sequences in Cython; use
+  the handle layer's scoped-context helpers (`invoke_in_context`,
+  `invoke_in_context_or_undo`, `cleanup_in_context`, `context_get_device`,
+  `graph_node_set_params`) so the failure handling exists in one place.
+- **Publish before you raise**: when a driver mutation has succeeded and a later
+  step can still fail, commit whatever keeps that mutation memory-safe (for
+  example the graph attachment that retains a node's new owners) before raising
+  the later error. Rolling back the retention of a live mutation creates a
+  dangling reference. When ownership cannot be established, retain the
+  resources anyway (leak) rather than release them; a leak is always preferred
+  to a use-after-free.
+- **Non-propagating paths never raise and never discard a status**: shared_ptr
+  deleters, `__dealloc__` and CUDA callbacks report through one channel, `report_cuda_error()` / `report_message()` in C++ (the
+  `pw_*` wrappers) or `warnings.warn(..., CUDAWarning)` in Cython and Python,
+  which emits `cuda.core.CUDAWarning`. No `print(file=sys.stderr)` and no
+  `fprintf` outside that helper. `CUDA_ERROR_DEINITIALIZED` is filtered by the
+  helper because it means the driver is shutting down.
+- **Rollback failure**: the original exception propagates; the failed rollback
+  is attached to it with `note_or_report_cuda_error()` (a PEP 678 note on
+  Python 3.11+, reported out-of-band on 3.10), or chained with
+  `raise ... from` when a second exception must be raised. Catching everything
+  (bare `except:` or `except BaseException:`) is acceptable only for
+  rollback-then-`raise` blocks, where the rollback must also run for
+  `KeyboardInterrupt`.
+- **Finalization**: once `py_is_finalizing()` is true, do no Python work from
+  destructors or callbacks and accept the leak (see
+  `_cpp/resource_handles.hpp` and `_cpp/GRAPH_ATTACHMENTS.md`).
+- **Aborting**: `std::abort` (or any process termination) is reserved for an
+  internal invariant violation where continuing could corrupt memory or produce
+  silently wrong results *and* no leak-based fallback exists. A failed CUDA
+  call, including a failed context restoration, never qualifies: raise or
+  report instead. There is currently no such path; if one is ever needed it
+  must go through a single helper that writes a diagnostic (call, CUDA error,
+  invariant, "please report") and a Python traceback of all threads to stderr
+  before aborting (as `faulthandler` does, via the GIL-free
+  `_Py_DumpTracebackThreads`; no Python-object work), must never trigger
+  during interpreter finalization or for driver-shutdown errors, and must be
+  called out in the docs and release notes. An *implicit* abort (an exception
+  escaping a `noexcept` function or a deleter, including `std::bad_alloc` from
+  an allocation inside `noexcept` code) is a bug (#1489, #2417), not a policy
+  choice: `noexcept` helpers must not allocate, or must catch what they call.
+- **Testing**: inject restoration failures with
+  `cuda.core._resource_handles._set_context_restore_fault_for_testing`; assert
+  reports with `pytest.warns(CUDAWarning)` or `warnings.catch_warnings`, never
+  by matching stderr text.
+
 ## API design guidelines
 
 These are some API design guidelines we try to follow when adding new APIs to

cuda_core/cuda/core/__init__.py

@@ -102,8 +102,10 @@ class _PatchedProperty(metaclass=_PatchedPropMeta):
 from cuda.core._stream import __all__ as _stream_all
 from cuda.core._tensor_map import *
 from cuda.core._tensor_map import __all__ as _tensor_map_all
+from cuda.core._utils.cuda_utils import CUDAWarning
 
 __all__ = [
+    "CUDAWarning",
     *_context_all,
     *_device_all,
     *_device_resources_all,

cuda_core/cuda/core/_cpp/DESIGN.md

@@ -275,6 +275,52 @@ Related functions:
 - `peek_last_error()`: Returns the error without clearing it
 - `clear_last_error()`: Clears the error state
 
+The C++ layer never raises Python exceptions: it runs `nogil` and `noexcept`,
+and is called from deleters, CUDA callbacks and GIL-released code where raising
+is impossible. Status is turned into `CUDAError` in one place, `HANDLE_RETURN`
+in the Cython layer. Which status convention a function uses is decided by its
+return value. Factories return the handle, so their status goes to thread-local
+`err` and is read with `get_last_error()`. Functions that do not produce a
+handle (`context_synchronize`, `context_get_device`, `graph_node_set_params`,
+the `graph_*_attachment` family, `deviceptr_alloc_raw`) return the `CUresult`
+directly and deliver results through out-parameters, mirroring the driver API;
+their callers `HANDLE_RETURN` the value. The two conventions never mix.
+
+### Context-scoped operations
+
+Operations that must run in a specific context use `invoke_in_context` /
+`invoke_in_context_or_undo` (propagating paths) and `cleanup_in_context`
+(deleters). They switch the current context, run the operation, and restore the
+caller's context. When restoration fails after the operation succeeded, the
+creation is undone and the restoration status is returned. When both fail, the
+operation status is returned. Either way the helper records a thread-local
+detail keyed to the returned status (`take_last_error_detail(status)`) that
+`_check_driver_error` attaches to the raised `CUDAError` as a PEP 678 note
+(appended to the message on Python 3.10), so the user learns that the caller's
+context was not restored, which context is current and, for a double failure,
+why restoration failed. Keying the detail to its status keeps it from attaching
+to an unrelated error if the caller never raises that status; `enter_context`
+clears any stale detail. Tests inject restoration failures with
+`set_context_restore_fault_for_testing()`.
+
+### Reporting from non-propagating paths
+
+Deleters and CUDA callbacks cannot raise. They report through
+`report_cuda_error()` / `report_message()` (the `pw_*` wrappers decorate
+destroy calls with it), which emit a `cuda.core.CUDAWarning` through
+the Python warnings machinery when the interpreter is usable, deliver an
+escalated warning as an unraisable exception, and fall back to stderr when the
+GIL cannot be taken (for example during finalization). `CUDA_ERROR_DEINITIALIZED`
+is never reported because it means the driver is shutting down. No status is
+discarded silently anywhere in this layer, and nothing in this layer terminates
+the process; see `docs/source/error_handling.rst` and the "Failure handling"
+section of `AGENTS.md` for the policy.
+
+A rollback that fails inside a Cython `except` block is not a non-propagating
+path: `note_or_report_cuda_error()` attaches it as a note to the exception being
+handled (`PyErr_GetHandledException`, Python 3.11+) and falls back to a report
+only when there is no such exception or notes are unavailable.
+
 ## Usage from Cython
 
 ```cython