Repository navigation
11.0.0: Build feature pipelines at runtime. - #13
Merged
Merged
Conversation
Every combinator takes its input as `V: View<T>`, so a chain whose shape is only known at runtime (parsed from a spec) needs boxed views to be views.
Building blocks for return-magnitude and log-price features (e.g. the mean absolute return as a volatility estimate). Ln yields None for non-positive input instead of a non-finite value that would poison downstream windows.
A small expression language over the crate's views: `x`, + - * /, unary minus and calls `name(expr, params...)` for 35 functions, e.g. `zscore(ema(x, 10) - ema(x, 50), 200)`. Features become data that can be stored, deduplicated (the Display form is canonical) and written by a search or an LLM without compiling code. Built chains are safe for untrusted specs: Limits bound windows, nodes and depth; every node passes only finite values on; division by zero and logs of non-positive values are "not ready"; per-function minimum windows keep the library's index arithmetic in bounds (cyber_cycle needs 3, pfe 3). An adversarial test runs every function on trends, constants, zeros, alternating signs and huge values with debug assertions on.
The flat pipeline keeps every stateful view in one enum and derives Clone on it, so a warmed-up pipeline can be copied. MadScaler was the only sliding window without Clone.
First piece of the flat pipeline that replaces the spec module. One table declares each function's name, parameters, argument requirement, output range and description, so they cannot drift apart (esr-core repeats a match per property across 900 lines). Unlike esr-core, which only knows a window length, parameters are typed: windows, reals and fractions. Domain is an interval with open, closed or infinite ends, as in esr-core's domain analysis, but derived per node from the inputs instead of fixed per operator. It lets the compiler report calls that are never ready and constant outputs before any data is seen. A randomized test checks that every operation contains all results of its members. The revision is larger than 500 lines because the interval operations are only used by the function table; split apart, either half would be dead code.
The first update set the sum to the value and then added it again, and eviction removed it only once, so every later output carried the first value: a window of 1 over a constant 5 gave 10. Found by the pipeline's check that outputs stay in their declared domain.
Eviction compared the queue length with `>` before pushing, so a full window held window_len + 1 values and sma(x, 10) averaged 11 values. Found by comparing a pipeline against hand-computed values.
A Builder assembles a Pipeline from inputs, constants, binary operators and function calls, and names its outputs. It hash-conses nodes, so features that share ema(x, 10) compute it once, which a Box<dyn View> tree cannot do, and folds stateless calls on constants. Limits bound the distinct node count, the depth and the windows, now for whole multi-input, multi-output pipelines instead of one expression. The pipeline lowers the nodes its outputs read into a flat list of small operations over a value buffer, with the sliding views in one State enum fed by Echo (as in esr-core) and kept apart from the operations, so a large view does not widen every slot. NaN marks a value that is not ready, so stateless operations need no branch; a view only sees values that meet its requirement and holds its last output otherwise, which matches the semantics of chained views. Pipelines are Clone and can be reset, and a single-input single-output pipeline is a View (Feature). Unlike esr-core, evaluation is a loop instead of recursion over a prefix order, stateless functions are not-ready outside their domain instead of silently clamped, and every value is f64. The revision is large because builder, graph, lowering and runtime only make sense together; about 600 of its lines are tests, including adversarial input for every function and a check that outputs stay in their declared domains, which found the Cumulative and Sma bugs and that PFE is not bounded to [-1, 1] as documented.
Analysis derives the domain and warm-up of every node in one pass from the declared input domains, the functions' requirements and output ranges, the way esr-core's domain checks guide tree generation, but per node instead of per operator, so a pipeline gets feedback instead of being generated around it. build() now rejects calls that are never ready (ln of -abs(x)), divisions by zero and constant outputs (x - x is recognised because equal subexpressions are one node), listing every error at once. It warns about calls that are not always ready, with a hint to declare the input's domain, entropy of an argument that never changes sign, duplicate outputs and unused helpers. Messages quote the expressions, so an LLM can act on them. Each output reports its domain and warm-up estimate; a test checks the estimate suffices for every function.
Pipeline::compile parses statements into a Builder: `name = expr` for outputs, `let name = expr` for helpers and bare expressions as outputs named by their text, over named inputs with declared domains. The expression grammar is the spec module's, refactored into a lexer and a parser that drive the builder, so text and code build the same graphs and share all checks. Line breaks end statements only after a complete expression, so long expressions can span lines, and `#` starts a comment. Every diagnostic now carries a line and column: syntax errors, builder errors and the analysis findings, which point at the expression they are about. Unknown names and functions suggest the closest match (smaa -> sma), a value called like a function and a function used like a value say so, and nesting is bounded so a malicious text cannot overflow the stack (the spec parser had no such bound). Display prints the canonical text: helpers by name, parentheses only around nested operations. A randomized test checks that random pipelines print to text that compiles to the same text and computes the same values. reference() renders the language guide and function table for a prompt. About 400 of the lines are tests.
A test compiles every spec function (on three arguments) and composite
specs as pipelines and checks they compute bit-identical values to the
spec module's boxed trees on adversarial input, so moving to pipelines
changes no feature.
The bench measures per-update cost on 100k values (this machine):
one feature, zscore(ema - sma): static chain 10.4 ns, spec 26.4 ns,
pipeline 18.5 ns
mostly stateless nodes: spec 57.0 ns, pipeline 39.6 ns
8 features sharing nodes: spec 256 ns, pipeline 217 ns
The flat program removes about a third of the boxed tree's cost. The
gain from sharing is smaller than hoped because the windows themselves
(mad over 100 values, zscore over 200) dominate the feature set.
Every spec compiles as a pipeline to bit-identical values (tested), and pipelines add several inputs and outputs, shared subexpressions, domain checks with located diagnostics, and run about a third faster. The public items are deprecated rather than the module, because a deprecated module also flags the test harness's references to its own tests. Inside the module the lint is allowed until it is removed. The README now shows the pipeline text form instead of specs.
The normalizer scales the current value by the previous window's range, so it does not see itself, but a new high or low then mapped beyond the documented [-1, 1] (5/3 in the rising-sequence test). Clamping keeps the documented range and stays causal: a new extreme reads as -1 or 1. The pipeline now declares [-1, 1] for min_max, and its domain test checks it.
Three deviations from Hannula's PFE let the output leave its documented [-1, 1]: the path length summed window_len - 2 segments while the straight line spanned window_len steps, instead of both spanning the window_len - 1 steps between its values, and the sign came from the last step instead of the move over the window, so the output flipped on every one-step reversal. Both distances now span the same steps, so the ratio is at most 1 by the triangle inequality (rounding is clamped), and the sign is that of last - first. A window of 1 underflowed window_len - 2; the constructor now asserts at least two values. The pipeline declares [-1, 1] for pfe again.
There is now one way to assemble features at runtime: the pipeline module, whose text form accepts every spec and computed bit-identical values in the test that compared them. The View impl for Box<V> goes too: it only existed so spec could nest boxed views, method calls on a Box<dyn View> work without it, and pipeline::Feature is the runtime view. The spec equivalence test goes with the module. The bench now compares a feature set as one pipeline against one pipeline per feature, which isolates the gain from shared subexpressions (24.6 ms -> 19.1 ms per 100k updates of 8 features here). The README explains when to use chained views and when pipelines. This is a breaking change for users of spec and of View for Box.
Removing spec and the View impl for Box breaks their users, and the Sma, Cumulative, PolarizedFractalEfficiency and MinMaxNormalizer fixes change outputs, so this is a major release.
MathisWellmann
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
New Features:
Domainstruct for each sliding view, helps validate combinations of chained views for validity. e.g. it can flag ln of a negative number.BinaryOpenum holds all available binary operatorsRequirementis used to restrict the possible operators to use with a type to ensure domain validity.Functionand macrofunctionsare used to declare all operators behind a unified APILimitsbound the distinct node count, depth and the window lengths, for whole multi-input, multi-output pipelines.(ln of -abs(x))is rejected outright as its never ready.Bug fixes: