GML part M5: one functor per layer family, the Bool keywords as type parameters, and the MHA head keys built once by michakraus · Pull Request #349 · JuliaGNI/GeometricMachineLearning.jl

michakraus · 2026-10-19T20:56:58Z

Part M5 — layers and architectures

Part M5 of the plan Simplify GeometricOptimizers and GeometricMachineLearning for 0.9: the layer
and architecture consolidations of GeometricMachineLearning, and three defects.

What changes

One functor per layer family. GradientLayer, ActivationLayer and LinearLayer each had a
Q functor and a P functor; each has one now, through _shift(Val(C), z, f).
SymplecticAttention and LinearSymplecticAttention lose their per-direction functors the same
way. The outputs are unchanged bit for bit.
The Bool keywords are Bool type parameters: skew_sym in VolumePreservingAttention,
use_bias in the volume-preserving feedforward layers, symmetric in SymplecticAttention, and
the two init keywords of SymplecticAutoencoder. A caller that names these types by their
parameters writes true or false where it wrote a symbol.
VolumePreservingLowerLayer and VolumePreservingUpperLayer are aliases of one concrete
VolumePreservingFeedForwardLayer{M, N, UseBias, AT, Triangle}. Both names stay exported, and
the bias initialiser has one spelling, init_bias.
MultiHeadAttention gets one initialparameters method for both values of Stiefel, and its
head keys are built once rather than in every forward pass. The forward pass reduces over the
parameter NamedTuples and names no key.
The architectures build their gradient layers through one _gradient_layers; LASympNet's
four Chain methods become one; the DataLoader constructors delegate to the dimension ones;
GSympNet.act is activation; and encoder/decoder dispatch on the architecture family rather
than repeat themselves for SymplecticCompression.
Smaller type work: LinearLayerQ/LinearLayerP lose an unused third parameter,
GradientLayer.second_dim is Int, and five layers annotate their element type as ::Type{T}
with no new bound on T. C19 leaves KNOWN_ISSUES.md.
Fixed: LinearSymplecticTransformer's inner constructor built new{typeof(tanh)} and
converted its activation to it, so activation = sin raised a MethodError;
StandardTransformerIntegrator's field upsacling_activation is spelled upscaling_activation;
and PositionalEncoding allocates its encoding on the input's backend by a kernel, so the layer
runs on a device array under allowscalar(false). B11 and K5 leave KNOWN_ISSUES.md.
Tests: test/layers/sympnets_shift.jl, attention_shift.jl, bool_type_parameters.jl,
multi_head_attention_allocation.jl, test/architectures/transformer_constructor_bugs.jl,
gradient_layer_builders.jl and forward_pass_inference.jl are new. The shift tests pin the
layers' outputs bit for bit against an independent formula; forward_pass_inference.jl runs
@inferred on the forward pass of every exported architecture in Float32 and Float64.

The MHA draw order, and the seeded tests

MultiHeadAttention now draws every query projection, head by head, then every key, then every
value; it drew a query, a key and a value per head before.

No seeded pinned value in the suite changed, and no test needed re-pinning. No test pins a drawn
MultiHeadAttention weight: the layer's tests assert types and shapes
(test/layers/transformer_setup.jl:12, test/layers/multi_head_attention_stiefel_setup.jl:22),
orthonormality to a tolerance (@test check(A) < tol), the optimizer-cache types, and the type of
the pullback; test/layers/transformer_optimizer.jl:34-35 compares one network's parameters with
themselves before and after a step; and the new allocation test compares the layer's own output with
the same expression written out with literal head keys, which no draw order changes. The two full
suite runs — 1.13.2 and 1.12.7, both green — are what confirms it: a seeded pin on these weights
would have failed.

The CHANGELOG entry says the same, under Changed (breaking): "A seeded MultiHeadAttention
therefore draws different initial weights from the ones it drew before, so a caller that pins seeded
initial weights must re-pin them." That is a statement about callers outside this repository, not
about this suite.

Known issues added

K20 — the VolumePreservingFeedForwardLayer docstring's formula omits the input term
(src/layers/volume_preserving_feedforward.jl:7). Docs, not code: the functors and the manual
page both add the input.
K21 — the MultiHeadAttention allocation comparison is sensitive to what ran before it in the
same process (test/layers/multi_head_attention_allocation.jl:67-83). It passes in the suite and
as a named run; a supplemental reporter that replays the forward-inference coverage first reports
332240 B of difference for Float32 with one head. The tolerance is not widened.
K22 — the SymplecticAutoencoder docstring says decoder_init_q makes the first decoder
layer of p type, and it makes it a q layer
(src/architectures/symplectic_autoencoder.jl:36). The sentence is wrong, not the dispatch; the
line predates the branch.

K20, K21 and K22 are the entries this branch adds. It also removes B11, C19 and K5, whose fixes are
in it.

Critic

Verdict: PASS. Branch simplify/part-m5 at 5b48e09. Judged 0d00b98f..HEAD.

clause	domain	my command	on origin/main	on the branch	verdict
Independent scalar-state restoration	JLArray, both precisions; independently allowed/disallowed task and process states	`scalar_restoration.jl`	Test-only correction	154/154	PASS
Restoration beyond the reported combination	Absent/Allowed/Disallowed × Absent/Allowed/Disallowed	`scalar_states.jl`, warm and fresh process	Not a source regression	1395/1395 each run	PASS
Static-array precision and element type	Float32 and Float64 static matrices	`static_precision_coverage.jl`; unit	Pins existing behaviour	probe 1/1; unit 152/152	PASS
MHA output element and container types	Dense and Stiefel weights; matrix/tensor; both precisions	`mha_type_coverage.jl`; unit	Pins existing values	probe 1/1; unit 106/106	PASS
Correct key-cost comment and unchanged tolerance	One/two heads, 1000 passes, both precisions	`key_cost_comment.jl`; key mutants	Per-pass key construction differs	probe 1/1; 192000/384000 B excess; tolerance 1024 B with positive control	PASS
Accurate host-storage explanation	JLArray broadcast and encoding seam	Read header; D3	Host allocation present	D3 CAUGHT; header correct	PASS
Fixed extraction pins	Four distinct, exactly orthonormal Stiefel leaves; SAE/PSD; both precisions and Julia versions	`extraction_pins.jl` on parent, 1.13.2 and 1.12.7	28/28 each version; all eight vectors identical	builders unit 81/81	PASS
Earlier guards, dimensions and coverage	Named earlier reproducers	`reproducers.jl`	Guard pinned	guard 6/6, Int32 8/8, second guard 12/12, inference 69/69, classification 3/3	PASS
Earlier numerical and precision coverage	SympNet, attention, VPFF, Bool	`pins.jl … origin/main …`	Numerical pins pass	145, 66, 70, 66 pass	PASS
Earlier encoder-family decision	SAE, PSDArch, PlainAE	builders unit	Corrected family contract	in 81/81	PASS
Earlier 36 plus S1, S2, P1	Named behavioural mutations	`mutate.jl --check`; bounded `--warm`	Mutation guards	all 43 CAUGHT	PASS
Inference, allocation barrier and quality	Reached hot path and quality units	cold `hot_path.jl`; selected run	Unchanged pin	shift 12/12; Aqua 17/17; selected 853 pass, 1 known Broken	PASS
Scope and helper ownership	Three authorised test files	`git diff`; method sweep	—	only authorised files; no `src/` change	PASS

Per-file counts: positional encoding 152; MHA 106; architecture builders 81; forward inference 68;
SympNet 145; attention 66; Bool parameters 70; VPFF 66; constructor bugs 14; Aqua 17; JET 52 plus one
Broken; exports 8; reachability 2; docs-reference checks 6. Scoped 1.12.7 run: 407/407.

Full suite, 1.13.2: PASS, exit 0, ran to the end (Testing GeometricMachineLearning tests passed),
6821 pass + 1 broken = 6822, no failing testset; Metal 40/40 on an Apple M4 Max. Full suite, 1.12.7:
PASS, exit 0, ran to the end, 6821 pass + 1 broken = 6822, the same testsets row for row. (1.12 runs
--check-bounds=yes, so the guarded @allocated assertions are skipped there; that run is not
evidence for an allocation bound.)

Pre-PR verification

julia-branch-verifier ran twice, once at ddb908d8 and once at 75e1ba85.

Pass 1 (3 unresolved, all text, all settled):

Comments and docstrings: the parent commit and still phrasings of four test files were
rewritten, and the positional_encoding(backend, T, dim, seq_length) method the branch added on
an exported function is now documented. The cited path
test/layers/symplectic_attention_network_parameters_gradient.jl exists, and the family claim
holds (SymplecticAutoencoder <: SymplecticCompression <: AutoEncoder).
CHANGELOG: every claim checked against the source and test diff. The MultiHeadAttention
draw-order entry moved to Changed (breaking), and the upscaling_activation rename keeps one
entry.
KNOWN_ISSUES: form and file:line citations checked; K21 stays recorded (see below).
Exports, aliases and renames: the @docs block and test/integration/documented_defaults.jl
follow them; no act, upsacling, init_bias!, :bias, :skew_sym or EncoderInit… remains
in src/, docs/src/ or the tests.
Tests: all seven new files are registered in test/runtests.jl and JLArrays is in
test/Project.toml; architectures/gradient_layer_builders.jl ran 81 pass and
layers/positional_encoding.jl 152 pass on 1.13.2.
Docs page: the six @assert typeof(Chain(arch).layers[i]) <: …{3, 3, <Bool>, …} lines evaluate
to true in a session. No docs build was run locally, so the added docstring lines are checked by
the Documentation workflow.
Scope: no @show, println, @info or TODO, and no scratch file in the diff.

Pass 2: Unresolved: 0.

The Documentation entry now names the Transformer docstring as well, because the diff removes
the "makes the network CPU-only" sentence from both.
K20 sits in the Documentation group, where K5 sat.
The draw-order entry reads correctly under Changed (breaking), and the deleted rename entry
leaves nothing dangling.
K20's two citations were read line by line, and K21's location is exactly its testset.

quality/doctests.jl — doctest(GeometricMachineLearning) — PASS, 88 s, at 75e1ba85.

Pass 2's two edits, and two of mine — K21's per-pass figure, which the pass called a nit, and the
K22 entry it reported as a pre-existing defect — are in df6f2416.

Unresolved: 1. K21 records the allocation comparison's order sensitivity instead of making the
measurement independent of earlier allocations. That is a decision taken above the part, and the
tolerance is unchanged.

Advisor decisions

Decided (advisor, 2026-10-19, high): check 4's type sentence: SAE keeps
NonLinearSymplecticEncoder/NonLinearSymplecticDecoder, PSDArch keeps UnknownSymplectic…,
PlainAE gives Unknown….
Decided (advisor, 2026-10-19, medium): check 3 measures k = 1000 passes with
abs(measured - literal) ≤ 1024 B, control literal > 0.
Decided (arbitrator, 2026-10-19, high; M5, second FAIL): the redesign of the scalar-indexing
restoration and the extraction pins, the corrected 192 B/pass figure, and the plan-wide pin rule.

… VPFF struct, and the MHA keys built once Each SympNet layer family and each symplectic attention weighting had one functor per half of the input; each is one functor on _shift(Val(C), z, f) now, and the outputs are pinned bit for bit. The Bool keywords of VolumePreservingAttention, the volume-preserving feedforward layers, SymplecticAttention and SymplecticAutoencoder are their type parameters, so no if/else ladder over new{...} remains. VolumePreservingLowerLayer and VolumePreservingUpperLayer are const aliases of one struct with a triangle parameter, and init_bias is the one bias-initialiser keyword. MultiHeadAttention has one initialparameters for both Stiefel flags, and the head keys are built there rather than per forward pass; a forward pass allocates no more than 16 bytes above the same products with literal keys, in both element types, on a matrix and a 3-tensor, with one and two heads. The architectures build their gradient layers through one _gradient_layers, LASympNet has one Chain, the DataLoader constructors delegate, GSympNet's field is activation, encoder and decoder dispatch on the architecture family, and the two _make_block_for_initialization methods are gone. LinearSymplecticTransformer keeps the activation it is given, and StandardTransformerIntegrator's upscaling Dense takes its keyword. PositionalEncoding allocates its matrix on the input's backend, so it runs on a JLArray under allowscalar(false). Every exported architecture's forward pass is checked with @inferred in Float32 and Float64. Closes known issues B11, C19 and K5, which leave KNOWN_ISSUES.md.

…he pins The consolidated SympNet functors had dropped the `size(z.q, 1) == M ÷ 2` check the six origin/main functors opened with, so a half of the wrong length reached the broadcast and, for an ActivationLayer, returned a value. The check is back at the top of all three functors, and the tests cover the NamedTuple, array and uneven-half paths in both precisions. The six layer constructors convert the dimension to an `Int`, so `GradientLayerQ(Int32(4), 8, tanh)` is the same type as `GradientLayerQ(4, 8, tanh)`. `LinearLayerP` loses the unused third alias parameter that the changelog already said was gone. `PositionalEncoding` builds its encoding on `KernelAbstractions.get_backend` rather than `networkbackend`, so a static array is accepted again, and the test asserts the backend of the array the layer adds. The test also restores the `allowscalar` setting it found. The forward-pass test lists `ClassificationTransformer` and covers a Stiefel and a `SkewSymMatrix` leaf; the allocation check measures 1000 passes against a 1024-byte tolerance; and the VPFF, LASympNet, extraction and shift pins carry the parent's numbers with element-type assertions in both precisions. The extraction test states all three autoencoder families over `REAL_ELTYPES`. Three false or stale CHANGELOG lines, four false or historical comments and the dead type assertion in the MHA tensor method are corrected. K20 records the VPFF docstring's formula, which omits the input term.

…nd pin the extraction on fixed leaves The JLArray testset of `positional_encoding` set scalar indexing with `allowscalar(false)`, which writes the process-wide default as well as the task-local key, and restored both from one captured value. It now runs under an outer `@allowscalar` and an inner `task_local_storage(:ScalarIndexing, ScalarDisallowed) do`, each of which puts back what it found; the test reads `requested_scalar_indexing[]` and never writes it, asserts that a scalar read raises inside the block, and asserts both states after it. The extraction pins of `gradient_layer_builders` overwrite every `StiefelManifold` leaf with a fixed, exactly orthonormal signed selection before the pinned pass, because a leaf is the CholeskyQR2 of a seeded draw and OpenBLAS 0.3.29 and 0.3.30 factorise the same draw to different last bits. The four vectors are recorded from the parent commit and are identical on Julia 1.12.7 and 1.13.2. The MHA value pins assert the element type and the shape of both outputs, the static-array testset runs in both precisions with `eltype(y) === T`, and the key-cost comment states the measured 192 bytes a pass.

…-order finding The family comment in `autoencoder.jl` said a `SymplecticCompression` always gets the symplectic `Unknown…` architecture, which a `SymplecticAutoencoder` does not. The header of `forward_pass_inference.jl` counted the volume-preserving transformer twice, `attention_shift.jl` referred to the plan, and the `Stiefel` note in the allocation test said the method branches on the flag. `KNOWN_ISSUES.md` gains K21, the order sensitivity of the allocation comparison in a process that ran the forward-inference coverage first, and its K20 entry names the two manual lines in the order they state their formulas.

`positional_encoding` documents its backend method, which the branch added on an exported function. Four test comments lose a word that recounted the change. The `MultiHeadAttention` draw-order entry moves to *Changed (breaking)*, because a seeded caller must re-pin, and the `upscaling_activation` rename keeps one entry, the breaking one.

The documentation entry names the `Transformer` docstring as well as `PositionalEncoding`, since the diff removes the CPU-only sentence from both. K20 moves to the Documentation group, and K22 records the pre-existing `decoder_init_q` sentence of the `SymplecticAutoencoder` docstring, which says `p` where the dispatch makes a `q` layer. K21 states the per-pass key figure as a figure and not as a comparison.

codecov · 2026-10-20T06:39:32Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.76%. Comparing base (4c4a832) to head (f5001d1).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #349      +/-   ##
==========================================
+ Coverage   86.72%   91.76%   +5.04%     
==========================================
  Files          58       58              
  Lines        1341     1239     -102     
==========================================
- Hits         1163     1137      -26     
+ Misses        178      102      -76

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

michakraus · 2026-10-20T07:28:02Z

@coderabbitai full review

coderabbitai · 2026-10-20T07:28:11Z

✅ Action performed

Full review finished.

coderabbitai · 2026-10-20T07:28:23Z

📝 Summary

Summary by CodeRabbit

New Features
- Positional encoding now preserves the input device/backend, including GPU and static-array inputs.
- Transformer architectures support custom activation functions throughout their layers.
- Symplectic and volume-preserving layers offer more consistent Boolean configuration options.
- Architecture constructors provide more flexible DataLoader-based configuration.
Bug Fixes
- Corrected the transformer upscaling activation setting and behavior.
- Improved attention execution efficiency and consistency across supported input formats.
Documentation
- Updated architecture documentation and clarified known issues.
Tests
- Expanded coverage for architecture inference, attention, device compatibility, configuration options, and numerical accuracy.

The pull request updates layer type parameters and shared dispatch, changes architecture construction and extraction, adds backend-aware positional encoding, and revises attention initialization and forward paths. New tests cover these changes across input types and precisions.

Changes: Layer and architecture updates

Layer / File(s)	Summary
Boolean layer type parameters `src/layers/volume_preserving_attention.jl`, `src/layers/volume_preserving_feedforward.jl`, `src/architectures/symplectic_autoencoder.jl`, `test/layers/bool_type_parameters.jl`, `docs/src/architectures/volume_preserving_feedforward.md`	Boolean options now appear as Boolean type parameters. Lower and upper volume-preserving feedforward layers use aliases for one concrete layer type. Related tests and architecture assertions use the Boolean forms.
SympNet layers and architecture builders `src/layers/sympnets.jl`, `src/architectures/sympnet.jl`, `src/architectures/linear_symplectic_transformer.jl`, `src/architectures/symplectic_transformer.jl`, `test/layers/sympnets_shift.jl`, `test/architectures/gradient_layer_builders.jl`	SympNet layer calls share shift dispatch, and architecture builders share gradient-layer construction. Constructors forward keywords, and tests cover layer ordering, dimensions, and outputs.
Attention initialization and forward paths `src/layers/linear_symplectic_attention.jl`, `src/layers/symplectic_attention.jl`, `src/layers/multi_head_attention.jl`, `test/layers/attention_shift.jl`, `test/layers/multi_head_attention_allocation.jl`	Q/P attention methods share implementations. Multi-head attention initializes query, key, and value projections in groups and computes per-head outputs for matrix and tensor inputs. Tests cover output values, types, shapes, and allocations.
Backend-aware positional encoding `src/layers/positional_encoding.jl`, `src/layers/transformer.jl`, `test/layers/positional_encoding.jl`	Positional encoding now accepts a backend and fills its matrix with a kernel. The layer uses the input backend. Tests cover backend preservation, element types, and static arrays.
Architecture extraction and transformer construction `src/architectures/autoencoder.jl`, `src/architectures/standard_transformer_integrator.jl`, `test/architectures/gradient_layer_builders.jl`, `test/architectures/transformer_constructor_bugs.jl`, `test/architectures/forward_pass_inference.jl`	Encoder and decoder extraction selects architecture types through dispatch. Transformer constructors propagate activations and use the corrected `upscaling_activation` field. Tests cover extraction and inferred forward-pass output types.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~60 minutes

Change: Refactor

Merge Risk: 🔵 Low · up to f5001

The layer and architecture consolidation is broadly safe to merge. One small issue remains. An extracted decoder can report the wrong block count when the encoder and decoder depths differ, although the computation itself is unaffected. It is a one-line fix and worth making before or shortly after merge.

Pre-merge checks: 5

Check name	Status	Explanation
Title check		The title clearly summarizes the main changes: unified layer functors, Boolean type parameters, and one-time MHA head-key construction.
Description check		The description is directly related to the changeset and provides detailed coverage of the implementation, fixes, tests, and known issues.
Docstring Coverage		No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files. (33 skipped: 33 unsupported.)

✨ Finishing Touches: 💡 1

🛠️ Fix failing CI checks: 💡

Commit to this branch
Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 1

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI
Review profile: CHILL
Plan: Advanced
Run ID: f72e7edd-ed3c-40f2-b6e6-c7daa173f5fd

📥 Commits

Reviewing files that changed from the base of the PR and between 4c4a832 and f5001d1.

📒 Files selected for processing: 33

CHANGELOG.md
KNOWN_ISSUES.md
docs/src/architectures/volume_preserving_feedforward.md
src/architectures/autoencoder.jl
src/architectures/linear_symplectic_transformer.jl
src/architectures/standard_transformer_integrator.jl
src/architectures/symplectic_autoencoder.jl
src/architectures/symplectic_transformer.jl
src/architectures/sympnet.jl
src/layers/linear_symplectic_attention.jl
src/layers/multi_head_attention.jl
src/layers/positional_encoding.jl
src/layers/psd_like_layer.jl
src/layers/symplectic_attention.jl
src/layers/sympnets.jl
src/layers/transformer.jl
src/layers/volume_preserving_attention.jl
src/layers/volume_preserving_feedforward.jl
test/architectures/forward_pass_inference.jl
test/architectures/gradient_layer_builders.jl
test/architectures/transformer_constructor_bugs.jl
test/helpers/cayley.jl
test/helpers/ulps.jl
test/integration/documented_defaults.jl
test/kernels/cayley.jl
test/layers/attention_shift.jl
test/layers/bool_type_parameters.jl
test/layers/multi_head_attention_allocation.jl
test/layers/positional_encoding.jl
test/layers/sympnets_shift.jl
test/layers/volume_preserving_attention.jl
test/layers/volume_preserving_feedforward.jl
test/runtests.jl

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

coderabbitai · 2026-10-20T07:33:20Z

    NeuralNetwork(
-        UnknownDecoder(nn.architecture.full_dim, nn.architecture.reduced_dim, nn.architecture.n_encoder_blocks),
-        decoder_model(nn.architecture),
+        _unknown_decoder(arch)(arch.full_dim, arch.reduced_dim, arch.n_encoder_blocks),


🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Use arch.n_decoder_blocks for the extracted decoder.

UnknownDecoder and UnknownSymplecticDecoder declare the third field as n_decoder_blocks. Line 239 fills that field from arch.n_encoder_blocks. The code then builds the decoder model from compute_decoder_iterations(arch), which uses n_decoder_blocks. If an AutoEncoder has n_encoder_blocks != n_decoder_blocks, the extracted decoder reports the wrong block count. The chain it wraps still has the correct layers. This line changed in this PR, so fix the field source now.

🐛 Proposed fix

--- "a/src/architectures/autoencoder.jl" +++ "b/src/architectures/autoencoder.jl" @@ -236,7 +236,7 @@ function decoder(nn::NeuralNetwork{<:AutoEncoder}) arch = nn.architecture NeuralNetwork( - _unknown_decoder(arch)(arch.full_dim, arch.reduced_dim, arch.n_encoder_blocks), + _unknown_decoder(arch)(arch.full_dim, arch.reduced_dim, arch.n_decoder_blocks), decoder_model(arch), decoder_parameters(nn), networkbackend(nn))

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

_unknown_decoder(arch)(arch.full_dim, arch.reduced_dim, arch.n_encoder_blocks),

_unknown_decoder(arch)(arch.full_dim, arch.reduced_dim, arch.n_decoder_blocks),

- Pins compare at 16 ULPs of the pinned vector's largest magnitude, and a failure prints the measured deviation (pin_ulps), as decided for M5. - ClassificationTransformer(dl) calls the dimension constructor, as in #350. - ActivationLayerQ/P and LinearLayerQ/P convert the dimension to Int, as in #349, so an Int32 dimension names the same layer. - Tests: the Stiefel branch of the MHA initialisation and its draw order; inference at 32 heads (layer and StandardTransformerIntegrator(32)); the alias test reads the source, because Julia drops an unused alias variable; the stale header of the MHA inference test; a narrower name for the Bool-ladder testset. Co-Authored-By: Claude Opus 5.5 <[email protected]>

michakraus · 2026-10-24T08:40:42Z

Closed in favour of #351, which is now the one build of part M5.

#351 takes this PR's rule that every SympNet layer constructor converts its dimension to Int (ActivationLayerQ/P and LinearLayerQ/P beside the gradient layers). It keeps its own attention, which reads the heads by name and infers at every head count: the forward pass here infers Any at 32 heads or more, and PositionalEncoding here raises on an SMatrix.

The branch simplify/part-m5 stays on the remote.

michakraus added 6 commits October 19, 2026 18:00

github-project-automation Bot added this to Geometric Machine Learning Development Oct 19, 2026

GML part M5 fix 3: compare the recorded vectors at 8 ULPs

f5001d1

coderabbitai Bot reviewed Oct 20, 2026

View reviewed changes

michakraus closed this Oct 24, 2026

github-project-automation Bot moved this to Done in Geometric Machine Learning Development Oct 24, 2026

michakraus mentioned this pull request Oct 24, 2026

GML part M5: layers and architectures share one method per Q/P pair, Bool choice and triangle #351

Merged

Navigation Menu

Repository navigation

GML part M5: one functor per layer family, the Bool keywords as type parameters, and the MHA head keys built once - #349

GML part M5: one functor per layer family, the Bool keywords as type parameters, and the MHA head keys built once#349
michakraus wants to merge 7 commits into
mainfrom
simplify/part-m5

michakraus commented Oct 19, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Oct 20, 2026

Uh oh!

michakraus commented Oct 20, 2026

Uh oh!

coderabbitai Bot commented Oct 20, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Oct 20, 2026 •

edited

Loading

Summary by CodeRabbit

Uh oh!

coderabbitai Bot left a comment

Uh oh!

coderabbitai Bot Oct 20, 2026

Uh oh!

michakraus commented Oct 24, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

	_unknown_decoder(arch)(arch.full_dim, arch.reduced_dim, arch.n_encoder_blocks),
	_unknown_decoder(arch)(arch.full_dim, arch.reduced_dim, arch.n_decoder_blocks),

Conversation

michakraus commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Part M5 — layers and architectures

What changes

The MHA draw order, and the seeded tests

Known issues added

Critic

Pre-PR verification

Advisor decisions

Uh oh!

codecov Bot commented Oct 20, 2026

Codecov Report

Uh oh!

michakraus commented Oct 20, 2026

Uh oh!

coderabbitai Bot commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Oct 20, 2026

Choose a reason for hiding this comment

Uh oh!

michakraus commented Oct 24, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

michakraus commented Oct 19, 2026 •

edited

Loading

coderabbitai Bot commented Oct 20, 2026 •

edited

Loading

coderabbitai Bot commented Oct 20, 2026 •

edited

Loading