Compiler work - #205
Merged
Merged
Compiler work#205
Conversation
python 3.12 split ob_refcnt so an immortal object is recognised from the sign of its low half, and Py_INCREF has written only that low half ever since while Py_DECREF tests and decrements the whole word. a 32-bit store followed by a 64-bit load of one address cannot be served from the store buffer, so every retain an emitted module makes stalls the release after it. one small module carried 86 of these. By_IncRef/By_XIncRef keep the immortality check — asking _Py_IsImmortal, the predicate Py_DECREF itself uses, so both halves of the pair compile to the same test — and do the increment at 64 bits. it is cpython's own 32-bit-host arm of Py_INCREF, taken on a 64-bit host because the split it avoids is what costs us. it skips the increment on a superset of what the split form skips, and every object in the difference is one Py_DECREF also leaves alone, so no reference is dropped early and no immortal sentinel is written. the fast path is opt-in behind a positive test of every name it uses: 3.12 and 3.13 only, GIL-enabled, 64-bit, no REF_DEBUG/TRACE_REFS/STATS/LIMITED_API, and _Py_IsImmortal actually defined. checked against 3.11 through 3.15, free-threaded builds included. inc_ref's Primitive and Instance arms are where every emitted retain of a PyObject * comes from, so the codegen takes the same store. on one small module the whole-artefact count of 32-bit refcount stores goes 86 -> 13 and the 64-bit ones 3 -> 78; what is left is by.h's operational helpers.
four families of silent wrong answer, all of them a compiled module disagreeing
with the interpreted twin it was built from: about a class written in a class
body, about the names its own namespace holds, about what a decorator, a calling
frame or a nested scope leaves behind, and about which of its classes install.
## carry a class written in a class body off its interpreted definition
three things a class statement nested in another class body got wrong.
the outer class now compiles while the inner one keeps its interpreted
definition, copied into the type dict as a class-level constant.
a base such a class stands on gives up its emission. the nested class is never
emitted, so it stands on whatever its base name held while the interpreted
definition's body ran; an emitted type under that name left the copy on a
second, orphaned copy of it, and isinstance answered False where python answers
True. that is what a module-level class this module does not emit already does.
and a module whose class bodies are captured ran its interpreted twin against a
*copy* of the builtins mapping, so that `__build_class__` reached that body and
nothing else. python gives a function the builtins its module dict held when the
function was made, so every function and method the body defined kept that copy
for life — and those are exactly the definitions a declined function runs from.
the copy was a snapshot frozen at import: a builtin rebound afterwards still
answered with the old one, a deleted one still answered, an added one was never
found, and a body writing through `__builtins__` wrote where nobody would read,
while the compiled half of the same module saw all four. the hook now displaces
`__build_class__` in the real mapping, calls the entry it displaced, records only
where the `class` statement was written in this module's own globals, and puts
the entry back only if it is still the one standing. the globals filter closes a
second silent wrong answer: a `class` exec'd into another namespace during a
capturing body was recorded as if it were this module's, and the emitted class
then carried the foreign body's constants.
## answer for what a module's own namespace holds
a module-level definition is bound under many names — `ALIAS = fn`, a class
body's `direct = fn`, a container the body built, a default argument. every one
was bound to the *interpreted* definition while the name the definition came from
answered the forwarder the module publishes. the two compute the same thing and
are not the same object, so only an `is` sees it: a dispatch table keyed on a
function, an unregister, a set of callbacks all silently took the wrong branch.
five shapes now agree with python. this reverses a decision the tests recorded —
that comment called the divergence tolerable because it "never reaches an answer",
which is exactly the shape of a silent wrong answer, so the test is replaced. a
decorated definition is left out: the twin's source has its decorators taken out
of it, so pairing it with the undecorated forwarder swaps one wrong answer for
another.
the guard on a class the module body writes on an emitted base gave a reason that
is not the one — that a type of ours refuses to be a base. it does not. the reason
is the order module init runs in: the whole interpreted source runs first and the
emitted types are installed over the names afterwards, so such a class is already
built on the interpreted definition of its base and is left on an orphaned copy.
relaxing the guard on the wrong reason takes webbrowser from 1 compiled unit to 21
and makes isinstance(MacOSXOSAScript("default"), BaseBrowser) False.
the rule that a module rebinding a name must not have a compiled definition
installed over it was blanket for a removal through `globals()`: `del
globals()[k]`, `pop`, `popitem` and `clear` set one flag with no name attached,
and every definition the module wrote was then treated as the one that went. `ast`
is the only module in the 3.13 stdlib that does this, and it does it with five
string literals in plain sight, so the key is read where it can be — a string
literal, or a loop target walking a display of them, for a `for` statement and the
four comprehensions alike. a key that does not resolve keeps the blanket rule, and
so does a body that assigns the target or iterates anything but a display. ast.py
goes from 9 compiled units and 59 interpreted to 167 and 13, and the census over
the 550 stdlib modules from 7435/1103 to 7593/1057.
two constructs the source writes as punctuation were lowered as reads of a name:
`a[i:j]` as a call to `slice`, and `...` as a read of `Ellipsis`. python emits
BUILD_SLICE for one and loads a constant for the other and never touches a
namespace, so a module binding either name for itself was obeyed where python
ignores it — `ast` binds both, and `self._source[i:]` raised `TypeError: slice()
takes no arguments`. `Op::MakeSlice` and `Op::LoadEllipsis` read no module dict.
and a call handed back to the interpreted definition kept only its positional
arguments, so `ast.literal_eval` parsed in `exec` mode and refused every literal
with `malformed node or string`; such a call goes through the unpacked form now.
also: a `def` counts among the definitions a computed removal reaches, and the two
namespace-removal enums are `Copy`, which is what clippy's needless_pass_by_value
asks about.
## stop assuming what a decorator, a calling frame or a nested scope leaves behind
four reads a compiled function was not entitled to make.
the window a decorator moved to module init opens was tested for a body that
reads the decorated name and for an effect the body sees under another name. the
third shape had no test and is the one a real module fails on: an attribute that
exists only on what the decorator handed back. `pkgutil` writes it as
`@simplegeneric` with `@iter_importer_modules.register` below, and with the gate
lifted its import stops with `AttributeError: 'function' object has no attribute
'register'`. the gate is coarse and looks expensive, so its price is written down
where the next reader will be: sixty definitions over the 42 modules that carry a
decorator decline, against three modules that then stop importing, with the two
obvious weakenings named alongside the module each one fails on.
a removal key read from the sequence a loop walks is disqualified by every
binding of the target inside the loop, compared by position — but a `global`
write cannot be compared by position, since it is in a frame of its own and the
module body only has to call the function that makes it. reading the tuple would
then decline the name the tuple says and install a compiled definition over the
name the module actually took out of its namespace, which is a silent wrong
answer where the unreadable key was an honest decline. so a target some nested
scope declares `global` is not read, and the walk that collects those declarations
descends into a `def` and a `class`, which is the point.
a compiled function pushes no python frame, so a call that walks the stack from
its caller starts one frame further out than the source meant and answers about
another module, raising nothing. only `sys._getframe` and `warnings.warn` were
guarded, and only when the call spelled them that way — `from sys import _getframe
as grab` compiled and answered about the caller. the callee decides now, by
identity, against the eight functions in `FRAME_WALKERS`. over the 550-module
corpus this costs 4 compiled functions: 7367 -> 7363 compiled. the differential
tests reach each walk through a caller, because a walk called straight from the
driver's module body has no frame between the two legs to differ over and passed
with the defect fully present.
and a compiled function read a module global and, where the checker typed it as an
instance of one of this module's own classes, narrowed it to that class's emitted
representation. the interpreted definitions run first and the whole module body
binds its globals against them, so such a name holds an instance of the
interpreted definition until module init moves it — and where the class takes its
layout from a base outside the module there is no move at all. `Annotated` is one
of those values, so `annotation_origin is Annotated` raised `TypeError: expected
_TypedCacheSpecialForm, got _TypedCacheSpecialForm` and every `TypedDict('T',
{...})` call against a compiled standard library failed. the read is left an object
now.
## install a family of classes together, and publish what the class body built
a `@property` group left the method table for one object written onto the finished
type, and where module init built that type by *calling the metaclass* the name
was never in the namespace the metaclass read. `abc.ABCMeta` decides what is still
abstract from exactly that namespace, so a class overriding an abstract property of
its base came out with the property in `__abstractmethods__` and refused to be
instantiated — compiled cleanly, declined nothing, diverged silently from its own
twin. the namespace now carries the `property` the interpreted body built, and
`By_PublishProperty` leaves that object standing rather than replacing it. that is
what lets the gate on a group of one come off a base. a property half's
`PyMethodDef` also carried no `ml_doc`, so `C.value.__doc__` was `None` where the
interpreted class answered with the getter's docstring. measured over the 76 stdlib
modules holding a lone getter with a new `propcensus.sh` rung: 461 lone getters, of
which 74 ran compiled before and 150 after, none moved back.
a class whose fields sit past a base's instance is built from a type spec, and the
spec can refuse. where compiled code that still runs would have read one of those
instances the answer has to be the whole module, and that was worked out one class
at a time — so a class any other class named, and any class named as a base, took
the module down with it. module init now works out the largest family of classes
the module could leave interpreted together and installs each member behind a test
of every class it reaches. the wait is transitive, because one compiled method
calls another's emitted body directly with no type object in between, and it
follows the base relation both ways. `asyncio.unix_events` goes from 0 of its 4
classes and 0 of its 3 functions to 1 class and all 3 functions, and nothing else
in the 130 stdlib modules that write a class on an outside base installs less.
an implicit dunder call — `x[k]`, `x + 1` — is resolved on the type rather than the
value, and a structural protocol keeps its members on the instance with no class
object for such a lookup to find them on. a bound signature recovery writes is an
intersection, and only its other elements have class objects, so a method none of
them states was not found at all. `_Unframer.readinto` is that shape: `len(buf)`
bounds `buf` by `Sized` while the assignment asks for a `__setitem__` only the
recovered protocol states, so the assignment was reported invalid, its right-hand
side never inferred, and `_Unframer`, `_Unpickler`, `_load` and `_loads` declined
with 83 compiled definitions behind them. the structural half answers only what the
rest of the bound did not, so a forwarded type that states the method keeps its
precision.
`X.register(Y)` records `Y` inside `X`'s own `_abc_impl`, and the fallback source
runs the whole module body before module init builds anything, so every
registration landed on the interpreted definition while `ABCMeta.__new__` handed
the replacement a fresh, empty one — `issubclass(dict, MutableMapping)` answered
False off the compiled `_collections_abc` and True off the interpreted one.
`By_AdoptTwinAttributes` hands the twin's `_abc_impl` over. and a class keyword no
longer turns down a base this module emits: only a class placing storage of its own
needs the type spec a keyword has nowhere to sit in, and that question is asked
where the fields are known. `_collections_abc.ByteString` is the shape, and it took
`Sequence`, `Reversible`, `Collection`, `Iterable`, `Container` and two generators
down with it; that module goes from 100 compiled / 16 declined to 115 / 8.
a module-level instance is moved onto the type that replaced its class by reading
each of the layout's fields off the twin. a name the twin carried that the layout
has none of refused the whole move, so `obj = A(1); obj.extra = 7` left the instance
standing on the interpreted definition and `isinstance(obj, Cls)` answered False. an
emitted class keeps a dict beside its layout wherever the source did not declare
`__slots__` throughout, so such a name is carried across with the fields and the
refusal applies only where the emitted type keeps no dict at all. a move also
registers itself before its fields are filled so a cyclic graph resolves to one
object, which left a window where an instance dropped after a later field failed
was still standing in somebody else's field with the rest of its layout zeroed —
`r.peer is a` False, and reading the unwritten field raised `SystemError: error
return without exception set`. a failed move now refuses every move begun inside it.
`_colorize` is the one value over the 550-module corpus the carried extras rescue.
`bench()` is the whole of what the native benchmark suite times, and eight programs built their inputs inside it — `dot` two fifty-thousand-element lists, `prefix` one of a hundred thousand, `dicthist` twenty thousand keys, `dictget` the keys its own docstring said were "handed in already built". each of those rows reported two costs as one, and it distorts a ratio more than the share of the timed region suggests, because preparation does not speed up by what the work does. those inputs now live in module state a `setup()` fills, called once per process after the import and before any clock. the hoist is silently reversible, so a declared setup has to be shown to matter: the probe calls `bench()` once with the setup withheld and refuses a program whose two answers agree. the recorded milliseconds those eight rows had measured the build as well as the kernel, so they are withdrawn rather than carried forward — a stale figure a reader cannot tell is stale is worse than no figure. every existing row measured a runtime primitive, so nothing in the set reached a language construct or a syntactic shape. ten new rows do. five for basedpython and python syntax alike — `with_` (1.08x cpython, 0.02x mypyc: the protocol was re-resolved by name every iteration), `match_` (1.11x, 0.23x: captures came out as `object`), `comp`, `dunder` (2.86x, 0.31x: both operands known and the call still went out through `By_ObjAdd`) and `globals_`. five for python shapes read against the row that does the same work by hand — `dataclass` 1.12x against `alloc`'s 12.78x, `kwargs` 2.98x against `calls`'s 63.56x, `slices` 1.16x, `closure` 0.88x and `sortkey` 0.72x. a program under `programs/` may now be written as `.by`. `by compile` takes that source as written and `by transpile` lowers it once, outside every clock, into the python the interpreted and mypyc builds run, so the row says both what the native backend makes of our own syntax and what that syntax costs against the python it replaces. nothing in `programs.toml` declares the language — the extension decides, and a name on disk both ways is refused. both commands are given `--soundness none`, because `by compile` emits no soundness checks into native code while the transpiled python gets an `isinstance` per checked call; that is worth 0.90x against 1.16x on `payloads`, the difference between reporting that compiling the program makes it slower and reporting that it makes it faster. the first three basedpython rows are `payloads`, `destructure` and `accessors`, and their decline counts are the point of them rather than a defect in them. also written down: handing `by compile` a `.by` and handing it that `.by`'s transpiled python are not the same compilation, so a `.by` row's decline count is a statement about the `.by` path rather than about the program. and `--by` now defaults to the debug build. the suite times the extension module `by` emitted, not `by` itself; the emitted C is byte-identical across the two profiles over eight diverse programs, and a release `by` stages eight of them 0.12s faster on a phase that is not timed. release is also the wrong direction on correctness, since the one profile-dependent line in the whole `by_*` path guards `verify_module`. the same change repairs `benchmarks.md`, which was committed with conflict markers still in it that mdformat had rewritten into a heading and a blockquote, so the file looked clean.
five paths resolved at runtime what the compiler already knew: the error label's release set, an operator between two operands of one class, a property pair on an extended class, a `with` block's protocol, and a class pattern's captures. ## release only what a jump to the error label could find held, and call a class's own operator body directly the shared `by_error:` label released every register the frame owns, because no one block's release set describes a label every block jumps to. that is correct and it is not free: the error path is dead code the c compiler still has to *cost*, and that cost is what decides whether a caller inlines the function. `split` in the `pairs` benchmark came to 240 against clang's inline threshold of 225, and 35 of that was releasing two registers that cannot be holding anything at the one point that jumps there — so `run` called it 300,000 times instead of inlining a magic-multiply modulo and a field read. the label now releases the union, over every fragment that jumps to it, of what may have been written by then; anything less than plainly-not-yet-written counts as written, which releases more rather than less. an operator between two operands of one emitted class went through the object protocol: `a + b` widened both sides, called `PyNumber_Add`, reached the type's `nb_add` adapter, went through the method wrapper and came back as an object that then had to be unboxed. a same-typed pair leaves the protocol nothing to do — cpython hands it to the left type's slot alone — so the whole of `a + b` is `C.__add__(a, b)`. the licence is the static-type one the direct call by name already runs on, and unlike a call by name an operator is looked up on the type, so no value stored on the instance can shadow it and there is no test to write. this also settles a wrong answer: the protocol call was `By_ObjCompare`, which is `PyObject_RichCompareBool` and takes identity as equality before it asks the type, so `a == a` inside a compiled function answered `True` however the class's `__eq__` voted. `dunder` goes 0.32x to 0.67x of mypyc, measured back to back against ±3.0% and ±2.3% floors, with the census unchanged at 7472 compiled / 1049 interpreted. ## lower a basedpython accessor block as the property it is, and test a property pair before calling it a class another class in the module extends is emitted as a mutable heap type, because an interpreted subclass may override a half and an attribute may be rebound on the class after import. a read or a write of one of its `@property` pairs had no middle option between the direct call and the full descriptor protocol, so it took the protocol: `By_GetAttr`/`By_SetAttr` by name, with the receiver increfed and the name interned on every access. it now takes the tested call that method dispatch on such a class already takes — `By_ArmAccessor` works out once at import whether the type still answers the name with the pair this module compiled and records the version tag that held, and `By_AccessorStands` is two loads and two comparisons at the site. an override in an interpreted subclass, a subclass in the module and a half rebound on the class all fail the test. the instance-dict probe a method call needs is absent on purpose: a `property` is a data descriptor, so a value stored on the instance is never what a lookup answers with. on `props_ext`, 13.56m -> 0.49m, 1.27x over cpython to 35.05x and 0.18x over mypyc to 4.85x, with the 550-module census byte-identical. the `fset is None` branch of the arming had no test over it and now does. `by compile` reads the parsed `.by` ast, and a `var v: int` with a `get`/`set` suite under it arrives there as the members a hand-written pair has. the markers the parser puts on the two halves are decorators no source spelled, and the lowering recognised only written ones, so it saw two `def v`s in one class body and turned the whole class down for a name defined twice — every class holding an accessor block was declined, and with it everything that read one. the getter's `__property__` marker is read as the `@property` the transpiler writes it out as, and the setter's as the `@value.setter` it stands for. a block declaring storage with an initialiser declines instead, saying so, and the decline moved into `class_fields` where the class loses its layout rather than keeping one everything else was already lowered against — so `accessors` goes from three declined functions to one. ## resolve a with block's protocol once, then call it directly when the class is one we emitted `By_Enter`, `By_ExitContext` and their two `async with` counterparts reached `__enter__` and `__exit__` through `PyObject_GetAttrString`, which builds a fresh `str` on every call — hashed from scratch, compared byte for byte in each dict along the mro, freed again, and with a new address every time, which is what the interpreter's type attribute cache is keyed on, so none of these lookups had ever hit it. a `sample` profile of the `with_` loop put 74% of the whole function in those two lookups against 22% in the two calls they resolve. interning each name once per module takes `with_` from 20.28m to 7.22m, 1.02x to 2.81x of cpython. that left the lookup at a quarter of the loop, and one `with` block enters and leaves the same manager every pass. `ByProtocolSite` is a memo with a validity test rather than an assumption, and the test is the one the interpreter's own specialiser makes: the manager's type, and that type's version tag. rebinding `__enter__` on the class or any base runs `PyType_Modified`, so a site holding the replaced method stops matching at the next call; a manager whose `__class__` was reassigned misses on the pointer; only a class whose metaclass is plain `type` is served, and only a value the type's own dict holds, because the memo holds it borrowed. a free-threaded build has no memo at all. 7.22m to 5.34m, 2.81x to 3.96x. after the memo the remaining cost was the calls themselves — 58% of the loop, over half of that in `By_BindArgs` laying arguments back out for a body that already had them. an emitted class licenses the direct call for the two reasons `o.m()` already relies on, and the licence is simpler here because python reaches `__enter__` and `__exit__` on the type and never on the instance. only the exits that are not unwinding an exception are direct; the unwinding exit keeps `By_ExitContext`, where the exception is taken apart into the triple and a truthy answer decides whether it goes on. 5.34m to 0.96m, 3.96x to 21.53x of cpython — over the three changes together 20.28m to 0.96m, 1.02x to 21.53x, and 0.01x to 0.29x of mypyc. `a_with_block_sees_its_manager_rebound_after_it_first_ran` arms a site then rebinds through every route the validity test covers, and `a_with_block_over_a_class_the_module_declares_reaches_both_halves_of_it` walks every exit a block has. ## read a class pattern's captures out of an emitted layout's fields `case Point(a, b)` asked `__match_args__` for the attribute name at runtime and then looked the attribute up, once per position per arm — `PyObject_GetAttrString` on the class, a tuple walk, and `PyObject_GetAttr` through the field's descriptor. what it bound was an `object`, so `a > b` and `a - b` went out through the object protocol even though `Point.x` and `Point.y` are declared `int`. where the pattern names a class this module emits and that class is sealed — no decorator, no base, nothing derives from it — a value that passes the pattern's `isinstance` is exactly that layout, and python can neither derive from the class nor rebind its `__match_args__`. so the positions the checker resolves become loads at compile-time offsets, and each capture is held in the field's own representation; the field's setter unboxes, so an `int` field can only ever hold an `int` and the narrowing needs no trust in the annotation. a position that resolves to anything else keeps the lookup it had, which covers a `@property`, an unsettled `__match_args__`, python's match-self builtins, and an attribute `__init__` writes on only some paths — python answers an absent one by moving to the next case rather than by raising, which a field read at a fixed offset has no way to say. `reaches_the_end` now reads a `match` too, so `classify` returns an `int` rather than an `object`. two of `By_Enter`'s neighbours resolved a fixed name the same wasteful way, and both are on a per-operation path: `By_MatchPositional` asks for `__match_args__` once per positional sub-pattern, and `By_Extend` asks a mapping for `keys` on every `**x` merge. `By_ProtocolName` becomes `By_FixedName` beside `By_InternedStr`, no longer being about the context-manager protocol. no benchmark reaches a `**` merge, so that half is unmeasured. `match_` goes 1.15x to 1.92x of cpython on the interning, and to 5.07x/4.67x on the field reads against 1.13x before — measured over 31 rounds on a machine carrying other work, so each figure is the row's own reading with its floor. against mypyc, 0.24x before and 1.01x after. declines unchanged at 0, census unchanged at 7472/1049.
…function
a licence, a lowering and an install are all claims nothing was checking. each is
now asked out loud — at compile time where it can be, at import where it has to
be, and per call behind a flag where that is the only place to ask.
## decline the basedpython surface forms the backend read as plain python, and check four more ir invariants
`by_irbuild` lowers the basedpython ast, markers and all, and twelve
basedpython-only forms reached a body and were read by their plain-python meaning.
every one compiled with no decline recorded and then answered differently from its
interpreted twin: `item?.value` and `item?.shout()` dropped the `None` guard and
raised `AttributeError`; `(name="ada", age=36)` became the plain tuple
`("ada", 36)`; `v cast! int` and `v cast? int` were lowered as a call of `v`;
`def area(Rect(w, h): Rect)`, `if let P := x:`, `for P in xs:` and `with e as P:`
read their captures as module globals; `p.1` became a dynamic attribute read of the
name "1"; an `extension` member and the prelude's grapheme string surface became an
attribute read the receiver does not answer; and a reified type parameter, declared
or inferred, read its name as a module global while `f[int]` found an
unsubscriptable function.
`by_irbuild::surface` is now the one place every marker the parser can set is
decided: a form the lowering understands is named and allowed, and every other one
declines, which puts the function back on the twin the transpiler lowered. an
annotation is not scanned, since a type expression is resolved by ty rather than
lowered. an extension member is asked about through a new
`SemanticModel::resolves_through_extension`, because it carries no marker. the ten
differential tests assert both legs agree on the *answer*, and all ten fail with the
gate removed.
the verifier already typed every operation, checked every call against its callee,
proved definite assignment per edge and demanded a release set that covers every
reference a block may hold. four things it did not say: every register an operation
mentions must be one its accessors report (`dest`, `operands`, `loop_cursor` and
`unbinds` are four hand-written matches over ninety-odd variants, and `ArraySet` once
did not report the value it writes — the check is the derived `Debug`, which cannot
forget a field); a borrowed register may only be filled by an operation that lends,
the narrowings being the sharp edge; a consuming concatenation may not take over a
parameter's register or a borrowed one, which once moved a refcount 4 -> 4389120980
when the guard was removed from `str_append`; and an error edge must lead to a block
that exists, the one edge `Terminator::successors` does not name. none fires, and the
corpus census is unchanged byte for byte. the accessor check says nothing about the
module in front of it, so it is asked only of a compiler built with debug assertions.
## take the argument-passing steps out of reaching a compiled function, and offer to re-ask every licensed lookup
a call from python into compiled code crossed two generic argument passes that a
call between two compiled functions never touches, which is why `calls` runs 60x
while `closure` and `sortkey` were both slower than the interpreter.
the first is the forwarder. a compiled module publishes each function as a python
forwarder onto the native one, because a `PyCFunction` is not a descriptor, and the
forwarder took `*args, **kwargs` and passed them on — a tuple and a dict per call
plus the `CALL_FUNCTION_EX` that takes them apart again. `sorted(values, key=fn)`
reaches `fn` once per element, so that is on the hot path rather than at the edge of
one. a function with no defaults, no `*args`, no `**kwargs` and no positional-only or
keyword-only parameters has nothing for the forwarder to decide, so those forwarders
name their parameters; every other shape passes the call straight on, so no default
is filled in and no refusal moves. the second is the wrapper's own binding: a
boundary whose parameters are all required and all ordinary positional-or-keyword
takes the call as it stands, and every other call still goes through `By_BindArgs`
so the refusals and their wording stay in one place. `sortkey` 0.72x to 0.93x on the
forwarder and to 0.99x on the binding, which three runs agree on.
two smaller costs on the same path. a call handing a keyword to a callee's
`**kwargs` filled a register with the module's interned string first and put the
register in the dict; the name is fixed where the call is written, so the literal
goes in directly and a call in a loop stops retaining and releasing one object per
trip. and the general indexed read keeps one arm inline that answers a `list`, so
`rest[0]` on a `*args` tuple missed the head on every trip and paid a call into the
cold tail — the site knows which container the calling convention packed, so it takes
an arm that suits it, with the exact type still tested and everything it turns down
falling through to python's own answer and wording. worth 7-8% of `kwargs`.
`by compile --licence-recheck` puts a check in front of every call the compiler
licensed to go straight to a compiled body: it does the lookup the call skipped — on
the instance for a method, on the type for a property half — and aborts naming the
class, the member and what changed when the answer is not the body about to run. all
four licensed shapes are covered. a dunder is answered through a slot whose wrapper
carries no `PyMethodDef`, so an operator's check is the receiver's class alone, which
is the whole of that licence anyway. the mode rides in the IR as `Op::LicenceHolds`
and the generated C `#define`s `BY_LICENCE_RECHECK` when it holds one, so a build with
the flag off is byte for byte what it was. the differential suite runs with the mode
on by default, so all 1309 cases exercise every licence they reach.
## check at import that every class a module reported compiling actually installed
a class that quietly leaves its interpreted definition standing answers exactly as
that definition does, so it agrees with every differential rung at once while
`--annotate` goes on counting it compiled. that is how the compiled-class figures
here became upper bounds rather than counts.
module init now ends by asking the finished namespace what is in it. for every class
the module publishes: the name holds the emitted type, `__bases__` holds what this
module's own bases mean by identity, `_abc_impl` is the twin's object where the twin
had one, and every name in the method table answers as a descriptor rather than as a
`function`. a violation raises `ImportError` naming the class. a class the layout
guard or the install gate stood down is not one — the interpreted definition keeping
its name is what those are for — and that is recorded rather than raised.
`BY_INSTALL_CENSUS` names a file the import writes one row per published class into,
`installed`/`interpreted`/`twin`, including for the exits that leave the whole module
interpreted; `scripts/native-sweeps/installcensus.sh` compares those rows against the
report's class headings over the 550-module corpus, which is what turns "the census
counts what ran" into a checked fact. on by default; `--no-verify-install` leaves it
out. it runs once per module at import, not per call: on a synthetic 200-class module
with 1200 lowered methods it costs 0.050ms of a 9.23ms import, inside the ±0.37ms
run-to-run spread.
it finds three live wrong answers. `logging/config.py` is one of the 550:
`ConvertingDict(dict, ConvertingMixin)` cannot be laid out over the emitted
`ConvertingMixin`, so it is left as the definition the body built on the interpreted
one, and `isinstance(ConvertingDict(), ConvertingMixin)` answers False where python
answers True. two differential fixtures had the same shape and asked only for
`__mro__` names, which agree; their broken halves are now their own tests, asserting
the refusal.
the install check's hand-written `Default` and a verifier test's `ModuleIr` literal
were each written against a tree without the other lane's new field, so both are
named here.
Contributor
by ecosystem round-tripbase: regressions: 0, changed: 22, improvements: 1, error changes: 0 (across 24955 files in 148 projects) ℹ️ changed round-trip outputmongo-python-driver — _by_sourcemap.py--- base/_by_sourcemap.py
+++ head/_by_sourcemap.py
@@ -574,5 +574,5 @@
"/tmp/tmpow2_us0b/mongo-python-driver/build/pymongo/write_concern.py": {"by": "sha256:2a0b3e89612bde0fa03067ac786c58b7fa4f8ee919f0c2900d295f5ba19a572e", "py": "sha256:15ac3c2168a60d64634ddc86f6e723cf9dd8eab13a4cfbf7bee366a3ee94c0d7"},
"/tmp/tmpow2_us0b/mongo-python-driver/build/setup.py": {"by": "sha256:967fe6128d71cebe9cce479f64121c80b87ec13fd6221f82aa744783917f50f1", "py": "sha256:967fe6128d71cebe9cce479f64121c80b87ec13fd6221f82aa744783917f50f1"},
- "/tmp/tmpow2_us0b/mongo-python-driver/build/test/__init__.py": {"by": "sha256:56689880719dbf422fd7ca1f1ada6711a10ce5c7b25fce32a15f7ae27c581a6c", "py": "sha256:749aa1829e7dffa1d4c1f83379e2d964ff519cc0e8324d7c9938464e982936b2"},
+ "/tmp/tmpow2_us0b/mongo-python-driver/build/test/__init__.py": {"by": "sha256:56689880719dbf422fd7ca1f1ada6711a10ce5c7b25fce32a15f7ae27c581a6c", "py": "sha256:fbaa63c2df5bfe3a0893ef209a302a218ab619e17f298a3425cef5f34708f2bd"},
"/tmp/tmpow2_us0b/mongo-python-driver/build/test/asynchronous/conftest.py": {"by": "sha256:196e3ef090d349ba21da7e2d53308a673ee5345335aa1a2847fc5b7e6039d10c", "py": "sha256:a3ad5c507bcf2101062efc83b0c7ef4f93a8ddce7af46b8c063fbb7520ef5736"},
"/tmp/tmpow2_us0b/mongo-python-driver/build/test/asynchronous/helpers.py": {"by": "sha256:42de755b0ab1a637e70fc6ec63d3fbc53b944ba59c951c130bd96f132e1dfb5b", "py": "sha256:034118866e12cf41e7d3f3cc1927cafb657bab291dc9073379f4fda383812805"},mongo-python-driver — test/__init__.py--- base/test/__init__.py
+++ head/test/__init__.py
@@ -257,5 +257,5 @@
# May not have this if OperationFailure was raised earlier.
- self.cmd_line = _soundness_check(self.client.admin.command("getCmdLineOpts"), dict)
+ self.cmd_line = self.client.admin.command("getCmdLineOpts")
self.server_status = _soundness_check(self.client.admin.command("serverStatus"), dict)pytest-autoprofile — lib/pytest_autoprofile/__init__.py(only produced on base)pytest-autoprofile — lib/pytest_autoprofile/_doctest.py(only produced on base)pytest-autoprofile — lib/pytest_autoprofile/_json.py(only produced on base)pytest-autoprofile — lib/pytest_autoprofile/_multiprocessing.py(only produced on base)pytest-autoprofile — lib/pytest_autoprofile/_patches.py(only produced on base)pytest-autoprofile — lib/pytest_autoprofile/_test_util_capture_warnings.py(only produced on base)pytest-autoprofile — lib/pytest_autoprofile/_test_utils.py(only produced on base)pytest-autoprofile — lib/pytest_autoprofile/_typing.py(only produced on base)pytest-autoprofile — lib/pytest_autoprofile/_warnings.py(only produced on base)pytest-autoprofile — lib/pytest_autoprofile/_xdoctest.py(only produced on base)pytest-autoprofile — lib/pytest_autoprofile/importers.py(only produced on base)pytest-autoprofile — lib/pytest_autoprofile/option_hooks.py(only produced on base)pytest-autoprofile — lib/pytest_autoprofile/plugin.py(only produced on base)pytest-autoprofile — lib/pytest_autoprofile/profiler.py(only produced on base)pytest-autoprofile — lib/pytest_autoprofile/rewriting.py(only produced on base)pytest-autoprofile — lib/pytest_autoprofile/setup.py(only produced on base)pytest-autoprofile — lib/pytest_autoprofile/startup_hook.py(only produced on base)pytest-autoprofile — lib/pytest_autoprofile/utils.py(only produced on base)stone — _by_sourcemap.py--- base/_by_sourcemap.py
+++ head/_by_sourcemap.py
@@ -132,5 +132,5 @@
"/tmp/tmpwxdhevur/stone/build/stone/frontend/ir_generator.py": {"by": "sha256:c1efb1175da48629a8358ff943f6430c0acf82d7466b586f7030593a65ab375a", "py": "sha256:08d30fc19fc7d6efd8fbd93587901fad442d0e051dfde9e38ee382d32f8e0dfe"},
"/tmp/tmpwxdhevur/stone/build/stone/frontend/lexer.py": {"by": "sha256:d398e5e8f9ce8776c4803079541cd5125f489ffeb22b8cf94fe50d646c35f7fd", "py": "sha256:f5aca1e8c878e055c3de096ca884663693684a1ba8a91ad362ed0b0d9606a5ee"},
- "/tmp/tmpwxdhevur/stone/build/stone/frontend/parser.py": {"by": "sha256:98e06aa0d429cb94db882f86ebe20d6261912f90ed7ca1a7700e225eca8c28a7", "py": "sha256:4e33d6361a7a8493884311f3734ebba7934e9fd04bb03d4118c4657c583df833"},
+ "/tmp/tmpwxdhevur/stone/build/stone/frontend/parser.py": {"by": "sha256:98e06aa0d429cb94db882f86ebe20d6261912f90ed7ca1a7700e225eca8c28a7", "py": "sha256:cfedd6eeca831cad653b5b6f314aa135b5f1d4d963549ebebaed9f6d49f22a7c"},
"/tmp/tmpwxdhevur/stone/build/stone/ir/__init__.py": {"by": "sha256:481b036e26ed7f3a77f32037fe42127f64c69de98e6ae44b8754663930ca8bcb", "py": "sha256:481b036e26ed7f3a77f32037fe42127f64c69de98e6ae44b8754663930ca8bcb"},
"/tmp/tmpwxdhevur/stone/build/stone/ir/api.py": {"by": "sha256:5587a3e044d40a2460d9ae52495799e09fe1e280c80c391f4200d6d479bcb8ae", "py": "sha256:92f822975f93c1dd0c1565e61a9664f437681b5ad47c6ae1aef11feb496f0cbb"},stone — stone/frontend/parser.py--- base/stone/frontend/parser.py
+++ head/stone/frontend/parser.py
@@ -575,8 +575,8 @@
p[0] = AstVoidField(self.path, p.lineno(1), p.lexpos(1), p[1])
if len(p) > 3:
- if p[4] is not None:
+ if (p[4], True)[1]:
p[0].set_annotations(p[4])
- if p[5] is not None:
+ if (p[5], True)[1]:
p[0].set_doc(p[5])💥 fails to round-trip (unchanged from base)
✅ improvements (failed on base, now builds)
⏭️ skipped
|
Contributor
ecosystem checkLinter (stable)✅ ecosystem check detected no linter changes. Linter (preview)✅ ecosystem check detected no linter changes. Formatter (stable)✅ ecosystem check detected no format changes. Formatter (preview)✅ ecosystem check detected no format changes. |
a destructuring binder, an accessor block's storage, a class statement written with a keyword, and a data class's generated members: each was declined or, worse, compiled to something the twin does not answer. ## lower a destructuring `let` and a `for` pattern, and name every statement once `let (a, b) := pair` and `for Rect(w, h) in rects:` both bind a pattern's captures from a value, which is a single-case `match` — the shape the transpiler writes them as. both lower through `pattern_branch` now, so a pattern that does not match binds nothing and falls through, and a `let`'s `else` block runs on that edge and carries on to the statement after. three things the lowering needed beside the branch. a name bound only by a pattern has to be recorded as a local of the frame, or a nested function reads it out of the module namespace and the call raises `NameError` where the twin answers. a `for` pattern's captures are the loop's per-iteration binding — the target beside them is the synthetic binder no closure can name — so closures made in different trips must not share one cell; they did, and every one answered with the last trip's value. and `walk` descends into a `let`'s `else` block, which is a suite like any other. the two statement-naming tables are one exhaustive table. the one the body lowering read had no entry for `let`, so a `let` it could not lower declined as "`this statement`"; with no `_` arm a statement kind added to the ast cannot reach a decline without being named first. `class`, a type alias and an ipython escape gain names by the same move. `for` comes off the surface decline table, since it is lowered now; `if let` stays, being a condition as well as a binding. the `destructure` bench row measures a fully compiled build rather than an interpreted one. ## emit the class an accessor block's initialised storage stands in `var v: int = 0` with a `get`/`set` block under it declined the whole class. the transpiler moves that initialiser into an `__init__` it injects, so the twin's class body no longer binds the name and there was nothing for the emitted class to take the value from — the class ran interpreted, and every read and write in the module went round the interpreted descriptor with it. the storage carries the value as the field's *constructor* default now, which is the same thing the injected `self.__v = 0` says. it must not be a class-level value, which is one object every instance reads through to and is published on the type where the twin publishes nothing, so the storage is kept out of the class-level defaults and out of the constants module init copies across. only an immediate: a `[]` there is a fresh list per instance in the twin, and one value written into every instance would be the one list they all share. that declines, as do the two shapes with nowhere to put the write — a `data class`, whose annotations are its constructor's parameters, and a class with a written `__init__`. `ClassIr` gains `fields_are_parameters` because `inherited_init` was answering two questions at once. it now says only whether the class has no constructor at all, which decides the message a call with arguments is refused with; whether the fields are the constructor's parameters is a `data class`'s question and is asked separately. the emitted C for python is unchanged. measured on the `accessors` row, two runs each at load 6.2-8.1: `by` 14.43m and 14.46m before, 0.26m and 0.24m after, against a row floor of ±0.2% to ±7.5%. that is 1.16-1.25x to 63.6-71.7x against cpython and 0.03x to 1.5-1.6x against mypyc, which puts the row level with `props` — what it exists to say is that the surface syntax costs nothing. the row's decline count goes 1 to 0, no other row's count moves, and the stdlib census is unchanged at 7472 compiled / 1049 interpreted. ## name the construct a class statement stands for when it declines, and pin the enum decline a class written with a keyword — `enum class`, `case`, `extension`, `build`, `protocol` — reaches the native build as an ordinary class statement carrying a synthetic marker decorator, in the same place a modifier keyword sits. every one fell through `class_modifier` to the arm meant for `private`, which told the reader their code had a modifier that "changes what the class is". `payloads.by` in the benchmark set is the case that says so out loud: its decline names no construct the program contains. the decline itself is right and stays. lifting it was measured on both enum lowerings: with an emitted `Shape` taking the twin's namespace entry, the variant types stay hung on the twin's class and `Shape.Circle(...)` raises `AttributeError` at the first construction — and the all-unit form fails silently instead, answering `Color.Red` with a bare object that has no `name`, no `value`, and a `Color` that will not iterate, while a `match` over it still returns the right string. so the four tests here pin it, and the two differential ones ask the enum surface rather than the match, which is the half only the silent failure moves. mypyc raises an internal `AssertionError: RefExpr not resolved` on a class pattern that reaches its class through an attribute, which is what a payload enum matches on, so that column was never going to fill. the row stops asking for it, and both of its notes now say why they stand. ## generate the members `@dataclass` puts on a data class a compiled `data class` answered none of them. `repr(Pair(1, 2))` printed `<m.Pair object at 0x…>`, two instances built from the same fields were never equal, `is_dataclass` said False and `fields`/`astuple`/`replace` all raised — and worst, two equal frozen instances hashed differently, which quietly corrupts any set or dict holding them. `data class C` is `@dataclass(slots=True)` on the twin and `frozen data class C` is `@dataclass(frozen=True, slots=True)`, and a written `@dataclass(...)` declines for its own reasons, so those two are the whole option matrix. what the decorator grows on the twin's class splits in two. the members that fill a type slot are emitted: `__repr__`, `__eq__` through `tp_richcompare`, `__hash__` — the tuple hash for a frozen class and python's own `PyObject_HashNotImplemented` for a mutable one, which is what `__hash__ = None` means — and a frozen class's `__setattr__`/`__delattr__`, which now raise `FrozenInstanceError` where they used to raise a bare `AttributeError`. these cannot be carried off the twin: a name written into `tp_dict` does not fill a slot, so a carried `__eq__` would answer `a.__eq__(b)` while `a == b` still went to the slot. the rest has no slot to disagree with and is carried — `__dataclass_fields__` is a dict of `dataclasses.Field` objects rather than code we could emit, and `__dataclass_params__`, `__match_args__`, `__replace__` and `__doc__` go with it, which is what makes `fields`, `asdict`, `astuple`, `replace` and a `case Pair(a, b)` written outside the module work. one slot backs all six comparisons, so the five `@dataclass` does not generate are unpublished, leaving `Pair.__lt__` `object`'s as it is on the twin. the generated `__eq__` is `self.a==other.a and self.b==other.b`, not a comparison of two tuples, and the difference shows in three places: `and` hands back the term rather than a bool, so a field whose `__eq__` answers with something that is neither True nor False is what the whole comparison answers with; the chain stops at the first falsy term; and a plain `==` has none of the identity shortcut a tuple comparison applies to each pair, so two instances sharing one NaN object are not equal. the body tests `self is other` before it looks at a field, which is why an instance holding a NaN is equal to itself — an unboxed field has no object to share, so reading it twice builds two floats and NaN is not equal to NaN. `ClassVar`, `InitVar` and the `KW_ONLY` marker are annotations python's decorator does not turn into storage, and laying each out as a field gave the emitted constructor a parameter the generated `__init__` does not take — `C(1)` interpreted against `C(kind, x)` compiled — while an `InitVar` left the instance answering an attribute the interpreted one raises for. a data class standing on a base that is not itself one had the same shape from the other direction, since `@dataclass` takes a base's fields from its `__dataclass_fields__` while the inherited layout carries everything the base's `__init__` assigns. each of the four declines, read off the annotation's inferred type so an import under another name is caught too.
KotlinIsland
force-pushed
the
compiler-work
branch
from
September 8, 2026 04:07
c5f1359 to
f6f00f6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.