Repository navigation
mad_scaler: O(log w) MAD selection and a ring-buffer sorted window, 4… - #11
Merged
Merged
Conversation
….6x-150x faster Every update merged all w absolute deviations into a buffer to read the middle one or two, and kept the sorted window in a Vec whose insert/remove shifted ~w values. Both made an update O(w): at w = 8192 about 21 us, which dominated the feature extraction of the agent-symbiont runners (92 us per trade for 9 MadScalers). - The MAD is now selected, not merged: the deviations form two non-decreasing sequences left and right of the median, and the (n/2)-th smallest of their union is found by a binary search over the split, computing only ~2 log2(w) deviations. - The sorted window is a VecDeque: insert/remove move only the shorter side, so monotone input (timestamps) slides without moving anything and other input moves at most w/2 values. The output is bit-identical: every deviation uses the same expression and an order statistic does not depend on tie order. New tests check that against an oracle that sorts the window and its deviations from scratch, comparing integer_decode() exactly for f64 and f32 over windows 1..=10, 16, 31, 32, 63, 64, 255, 256 and streams with ties, a zero MAD, monotone input and outliers. The oracle passed on the old implementation before the change, and catches mutated selections (max for min, inverted search predicate, wrong rank). benches/mad_scaler (100k realistic prices, vs the previous commit): w=128 47 -> 10 ms (4.6x), w=1024 285 -> 13 ms (21x), w=8192 2103 -> 28 ms (75x), w=32768 5468 -> 77 ms (71x) for f64; f32 up to 150x. End to end in nexus (feature_day bench, one real BTCUSDT day, 18 production extractors): pipeline 121 -> 2.0 CPU-s per day, sequential 22.8 -> 0.51 CPU-s per day, same feature digest.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
….6x-150x faster
Every update merged all w absolute deviations into a buffer to read the middle one or two, and kept the sorted window in a Vec whose insert/remove shifted ~w values. Both made an update O(w): at w = 8192 about 21 us, which dominated the feature extraction of the agent-symbiont runners (92 us per trade for 9 MadScalers).
The output is bit-identical: every deviation uses the same expression and an order statistic does not depend on tie order. New tests check that against an oracle that sorts the window and its deviations from scratch, comparing integer_decode() exactly for f64 and f32 over windows 1..=10, 16, 31, 32, 63, 64, 255, 256 and streams with ties, a zero MAD, monotone input and outliers. The oracle passed on the old implementation before the change, and catches mutated selections (max for min, inverted search predicate, wrong rank).
benches/mad_scaler (100k realistic prices, vs the previous commit): w=128 47 -> 10 ms (4.6x), w=1024 285 -> 13 ms (21x), w=8192 2103 -> 28 ms (75x), w=32768 5468 -> 77 ms (71x) for f64; f32 up to 150x.