Extension module work - #200
Merged
Merged
Conversation
three independent type-inference defects the native backend work ran into while checking real projects. a parameter-shape base spelled as a parenthesised tuple was inferred as a runtime tuple value and reported `invalid-base`, so ty rejected code the transpiler accepts and emits as `class C(tuple[A, B])`. a `src/` holding files named after stdlib modules made `by check` panic and take every other file in the run with it. a panic that cancels the whole run is worse than any diagnostic being wrong, and both of these reproduced on the baseline. the first is `known_class_to_instance`. a first-party module that shadows a stdlib one makes a known class resolve into code whose own inference asks for that same known class again; specializing what was found reopens the recursion a level lower, through `generic_context` and `explicit_bases`. the sibling query that performs the lookup half already recovers, so this one gets the same treatment rather than a new policy. the recovery answers `Unknown` for one fixpoint iteration inside the cycle and nothing outside it. the second is `inferred_return_type`'s recovery running a salsa query of its own, which salsa forbids. the query was reached through the union builder asking a literal for its fallback *instance* — which means resolving the class's module and reading the symbol out of it. it now asks the element it already has which class it is instead, keeping "which class is this literal's fallback" apart from "what type does that class make". where the lookup used to fail, which is exactly the shadowed case, the old comparison matched nothing and kept a redundant literal beside its own instance, so the new form is strictly more precise. `object & ~int`, which is what `if not isinstance(x, int)` narrows to, carries no positive element of its own. both the subscript store and the delete path iterated those elements alone, so they visited nothing at all there. for the store that was a crash: `infer_loud` was never reached, so a `MultiInferenceGuard` was dropped unfinalised and its `debug_assert` aborted the run. it was also losing real work — the key and the assigned value were never inferred with diagnostics, so `x[missing_key] = missing_value` silently dropped two `unresolved-reference` errors as well. for the delete it was quieter: nothing was reported at all where `object` has no `__delitem__`. an intersection with only negatives still has a positive bound, because everything is an `object`. the read side already fell back to it through `positive_elements_or_object`, which exists for this — so this is a known pattern applied in the two places that had missed it, not a new rule. Co-Authored-By: Claude Opus 5 <[email protected]>
seven defects, each one a case where an emitted module gave a different answer from the interpreted twin and said nothing about it. a silent wrong answer is the worst thing this compiler can do — a decline costs speed, a crash is at least visible, and these were neither. five of them shared one shape: a hand-maintained list that had to agree with another list, and did not. `__dict__` on an emitted instance is now a real `dict` over the whole state that every write updates, rather than a mapping that named only half of it. a decline protects the function it is raised in, not the class that function is handed — `render` declined correctly, the interpreted twin ran, and it still failed because the emitted class it was given had no `__dict__` to reach. `walk` now descends into `match` case bodies, and `written_names` and `local_representations` learned pattern captures alongside it; widening `walk` alone traded an honest decline for a wrong answer, so all three moved together. `dead-registers` no longer carries its own copy of every operand's shape: it goes through `Op::dest_mut`, `Op::loop_cursor_mut` and `Op::operands_mut`, so there is one list rather than two that had to agree, and `ArraySet` was the one that had drifted. the `warnings.warn` class-body gate is lifted — the two defects its first re-costing turned up are both fixed — and the weakref gate now asks whether an instance of ours could stand where the referent is read, so `ref(self.attr)` is left alone where `ref(self)` still refuses. the sixth is binding. a `PyCFunction` is not a descriptor, so `Cls.method = mod.fn` — which is what `functools.total_ordering` does — installed something that never received the receiver. with a default on that receiver it answered quietly rather than raising: `Thing().label()` gave `'bound'` interpreted and `'UNBOUND'` compiled, from ordinary code with nothing reported. cpython offers no way to make a `PyCFunction` bind, and a custom binding type loses `inspect.isroutine` and `copy`'s atomic handling, so neither of those was the answer. each exported module-level definition now gets a real `function` written into the interpreted twin, which forwards to the native and takes its name, docstring, defaults, annotations and `__wrapped__` from the definition standing under the same name. it takes `*args, **kwargs` deliberately: written with the exact parameter list it fails, because the transpiler rewrites a mutable default into a sentinel plus a body test, so the twin's `__defaults__` holds a sentinel the native has never heard of. a transparent forwarder makes no arity or default decision at all, so none of them can be got wrong. the cost is one forwarder frame on the single entry through a module's own name: measured at ~40ns, constant in the work done, because an intra-module call is `Op::CallNative` by symbol and never touches the module namespace. two more were found on the way — a module defining a function called `globals` made the installer fail on its own first line, and the forwarder's frame broke `warnings.warn(stacklevel=…)`, which now steps over it. the seventh is comparison. one `tp_richcompare` backs all six, so a class writing `__lt__` took the slot over from `object` for the other five as well, and two things followed. `object`'s richcompare is not empty — it is where `!=` gets its meaning, negate `__eq__` — and answering `NotImplemented` in its place threw that away: `Money(5) != Money(5)` was `True` compiled against `False` interpreted, from ordinary code with no decorator anywhere. an unwritten comparison now goes to the base's slot, where python would have answered it. and `PyType_Ready` publishes a wrapper under every name a filled slot backs, so the type advertised a `__le__`, `__gt__` and `__ge__` this body never wrote — which is why `functools.total_ordering` saw all four roots present and set none. those names come back off after the type is built; the slot is untouched, so what a comparison *does* is unchanged and only what the class *says* moves. the same goes for the pairs a binary number slot backs and for `__setitem__` alongside `__delitem__`. with that gone, the class-statement decline for a decorated class is no longer needed and `@total_ordering` on a class compiles where it used to fall back entirely — a decline recovered rather than replaced. removing it exposed a cascade that was always there: a class whose decline is settled at the end of its lowering has a layout by then, and every class holding one declines too. `tracemalloc` went 50 compiled to 26 that way. the question is asked in `class_fields` now, where the layout is decided, so a declining class is an ordinary object to the rest of the module. the semantic-delta table is updated throughout for what moves. `type(f)` and the binding row go, since both now match python; `__code__`, `__wrapped__` and the forwarder's traceback frame arrive; and applying `@total_ordering` from another module raises at the `setattr`, since an emitted class is sealed — loud where it used to be quiet. Co-Authored-By: Claude Opus 5 <[email protected]>
…oup of one reading `cell.v` on an emitted class cost the whole `PyObject_GenericGetAttr` trip per iteration — an interned string, a type lookup, the real `property` object, `property_descr_get`, a `METH_FASTCALL` PyCFunction, the wrapper with argument binding and instance unboxing — then the body, then a box and an immediate unbox, and the same again for the write. the body at the end of it is one struct field load already marked infallible. both halves are now called directly. soundness is the method path's argument plus one it does not get: a `property` is a *data* descriptor, so the class binding wins over the instance dict and no shadowing test is needed. halves stay on the protocol where the protocol is the answer — no setter, or a body taking more than python hands it. a group of *one* is now lowered too, which recovers a decline rather than only speed: the name answered with a real `property` carried over from the twin, but the object it carried held the interpreted function, so the getter's body never ran compiled at all. two attempts were needed. the first moved the census — classes that had compiled now declined and cascaded — so a group of one is optional, since a lone getter is a single ordinary `def` and can be left where it stands. the second was census-clean and still wrong: `ABCMeta` computes `__abstractmethods__` from the namespace it is handed, and a published property is not in it, so compiled `Integral` reported its abstract methods unimplemented. it is taken only for a class `type` itself builds, which computes nothing from what the body bound. and the arity gate comes off a defaulted half. it was written for `tp_getset`, whose function pointers really do pass exactly one and two arguments, but a half is published over the function's wrapper and binds its arguments as a call through the name would. a half stacked under a second decorator is still declined — it now says so as the property it is, rather than falling through to the generic message about a name written twice. Co-Authored-By: Claude Opus 5 <[email protected]>
KotlinIsland
force-pushed
the
extension-module-work
branch
from
September 4, 2026 17:28
e916d7e to
dfe18d0
Compare
three speedups, none of which changes an answer. the census is byte-identical across all three — 550 modules, 7381 compiled and 1111 interpreted, with no decline reason moving anywhere in the corpus. two error checks asked a second question when the value in hand already carried the answer. a tagged register's sentinel is the tag bit alone, so `By_LongOf` already turns it into NULL and the X-forms drop it; testing `x != BY_INT_ERROR` as well put a second compare and branch on the straight line of every release, which a loop pays every trip. a static assert now pins the sentinel's shape, because this is the first thing that depends on it rather than on its value. and a call returning a fixed-length tuple was checked by asking the thread whether an exception was set. a struct reserves no bit pattern of its own, but a member usually does, so a tuple now borrows a sentinel from its first member that has one and the check is a compare against that member; each member also takes its own undefined value rather than a zero, so the pattern is really there to read. only a tuple holding nothing but doubles and fixed-width integers still falls back to the thread. the compare is not merely the cheaper of the two: `PyErr_Occurred` is an opaque call, so the C compiler has to assume it clobbers everything and stops inlining the callee into the loop around it. `split`, `startswith`, `join` and `upper` are the `str` methods with a C-API entry point of their own. a method site already skips the attribute lookup, but it still ends in an indirect call into the method's python-facing wrapper, which unpacks the argument array again before doing the work; for these four the work itself is reachable directly, so the wrapper and the array both go. each keeps the site it had, and takes the direct path only where the receiver is an *exact* `str` and the arguments are the shape the entry point serves — a `str` subclass with a method of its own, a separator that is not a string, a `startswith` given a tuple of prefixes or a start and end range all fall through to `str`'s own method. so the answer stays python's own rather than an approximation of it, and a receiver the fast path refuses is no worse off than before. and `tp_free` now parks a dead instance's block on a small per-class array instead of handing it to the allocator, with `tp_alloc` taking it from there. it sits entirely below `tp_dealloc`, so the object is really destroyed, a written `__del__` really runs and the next instance really is a new one — only the block is reused. cpython does the same for floats, tuples and frames. this was hand-written first and timed before any of it was designed: on alloc, an object built and dropped every iteration, 1.99x against a 1.003x floor; on objects, 1.12x against 1.012x, both in one process with the legs interleaved ABBA. the gate is `mutable_type`, false only for a sealed leaf — no base, nothing derived, no decorator, no written `__new__` — which is exactly the shape whose `tp_alloc` and `tp_free` nothing can inherit, so every block that reaches the array is this class's own size. a free-threaded build gets none of it, because the array is shared mutable state and the module says it keeps none. a class whose instances have a finalizer is turned away as well. the collector keeps 'this object has been finalized' in its own header, in front of the block rather than in it, and untracking masks `_gc_prev` down to exactly that bit. a recycled block would therefore arrive already-finalized, and python has no call that clears it, so the second object to live there is one `__del__` never runs for. the bit is only ever set after a non-null `tp_finalize` has been called, and the two things that fill that slot are a written `__del__` and a generator's own, so turning both away leaves a parked block's header all zeroes, which is what the allocator hands back. the rest of the enumeration is on `recycles_instances`: the object header's trace links, which a tracing build refuses over; a managed dict or inline values, which an emitted class never asks for; a weakref list, likewise; and the block's size, which the sealed-leaf gate settles. the module drains its arrays on the way out, so it is memory held rather than memory leaked. differential tests sit either side of that gate and over the three other routes a recycled block could show the last instance's state. an optional field written on only every other construction is the one part of an instance a constructor does not overwrite, so it is where the reset is load-bearing: with the reset cut back to the object header the compiled leg reads TrueFalseTrueTrueTrueTrue against python's TrueFalseTrueFalseTrueFalse. the other two are the allocator pairing for a class with no dict, which is allocated through the plain allocator rather than the collector's, and a run of cycles collected before the blocks are handed out again — the collector's other header flag, which the deallocator's own untrack is what clears. the suite rows moved 1.01x to 2.42x on alloc, 1.43x to 3.91x on alloc_slots, 0.77x to 1.23x on tuples — from behind to ahead — and 1.12x to 1.51x on calls. the tagged-register half on its own is worth 1.25-1.27x on methods and 1.08-1.11x on inherit, measured in one process with the legs interleaved ABBA and confirmed by swapping which build each name got; the byte-identical null leg read 1.00-1.03x either way. Co-Authored-By: Claude Opus 5 <[email protected]>
four gaps, each of which had cost real time. `bg.sh` had no way to end a job, so it was done by hand with `pkill`, which kills the script the wrapper is running and leaves the wrapper to spawn the next one. a sweep chain killed that way ran orphaned for 98 minutes while benchmarks were being timed against the load it made. `start` now runs the job under job control so it leads a process group, and `stop` signals the group, escalates, and re-checks rather than trusting the kill. signalling the group is guarded on the job actually leading one: `kill -TERM -$pid` names the group *numbered* `$pid`, which for a job started before this change is somebody else's. `cdiff` compares emitted C, and the runtime header is included rather than inlined — so a change to a helper left all 550 modules byte-identical and the rung named nothing to walk. each leg's `by.h` is captured now, and a module whose C is unchanged but which calls a changed helper reads as `header`. the attribution is conservative on purpose: every changed line must name a `By_` symbol of its own or nothing is scoped out, because `diff -p` blames a change after the last function in a file on that function, which would drop real modules from the walk. `fntwin` asks what a module *body* captured, which no rung did — every other one reads through the module's own names, where the compiled function answers. it is one import per module with no second leg and no alarm, so it stays trustworthy on a busy machine: contention can drop a module, but it cannot invent a twin. and the benchmark harness ran a fixed repeat count, which spends milliseconds on a slow row and seconds on a fast one, so the fast rows carried all the noise. each build now runs to a duration instead, and the row reports the floor it measured rather than borrowing a global one. the new rows are the three property shapes — a plain pair, an extension property and a lone getter — and pairs, a tuple whose slot holds a reference rather than an int. pairs lands at 0.40x, the worst row in the tuple group, and the error-check work did not move it: a tuple parameter is still passed as an object and indexed through the object protocol, where mypyc passes the struct by value. Co-Authored-By: Claude Opus 5 <[email protected]>
KotlinIsland
force-pushed
the
extension-module-work
branch
from
September 4, 2026 18:19
dfe18d0 to
c7c13bb
Compare
Contributor
ecosystem checkLinter (stable)✅ ecosystem check detected no linter changes. Linter (preview)✅ ecosystem check detected no linter changes. Formatter (stable)✅ ecosystem check detected no format changes. Formatter (preview)✅ ecosystem check detected no format changes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.