Royer Research Labs
Popular repositories Loading
-
-
TriGLU
TriGLU PublicToken-local triple-product gating for selectively replacing attention in decoder-only Transformers, with reproducible quality and efficiency ablations.
Python
-
Dilution-Attention
Dilution-Attention PublicDilution attention: a parameter-free causal attention operator that divides each query's bid by the demand its key has already absorbed. Fused Triton kernels, and a controlled study where 209M mode…
Python
Repositories
- Dilution-Attention Public
Dilution attention: a parameter-free causal attention operator that divides each query's bid by the demand its key has already absorbed. Fused Triton kernels, and a controlled study where 209M models with no position encoding retrieve at 8x their training context.
- Niah Public
- TriGLU Public
Token-local triple-product gating for selectively replacing attention in decoder-only Transformers, with reproducible quality and efficiency ablations.
People
This organization has no public members. You must be a member to see who’s a part of this organization.
Top languages
Loading…
Most used topics
Loading…