Add time-series simplification to di.analytics - #133
Open
jonnypress wants to merge 1 commit into
Open
jonnypress wants to merge 1 commit into
jonnypress wants to merge 1 commit into
Conversation
Adds shrink, rdprecur and rdpiter: the Ramer-Douglas-Peucker line simplification described in "Dynamically shrinking big data using timeseries database kdb+" by Sean Keevey and Kevin Smyth (https://code.kx.com/q/wp/ts-shrink/), credited in the code header and in a References section in the module docs. shrink is the dictionary entry point over a table (table/xcol/ycol/ tolerance, optional by and method) and returns the source table restricted to the rows worth keeping. rdprecur and rdpiter are the recursive and iterative kernels, exposed for callers working with plain vectors; both return the indices of the retained points rather than (x;y) pairs, which is what lets shrink carry every column through and simplify each by group independently. Three deliberate differences from the paper's listings: - Segment endpoints have their distance to their own chord pinned to zero. The paper computes it as slope*x1+(y1-slope*x1)-y1, which is not exactly zero in floating point; when that noise is the maximum, the breakpoint lands on an endpoint, the split drops nothing and the recursion never terminates. It is reachable - rdpRecur[0f;...] over ten ordinary points exhausts the stack. - The x axis is rebased on its first value before the cast to float, so a nanosecond timestamp keeps its resolution instead of rounding to ~100ns. - The iterative kernel splits every pending segment per pass rather than one. The paper's one-per-pass queue costs about 3x against recursion; level-wise brings it within ~6% (106ms vs 112ms over a 20k point walk), so shrink defaults to the iterative method - the speed argument for recursion no longer outweighs its stack risk. Tests cover both kernels against a worked example, the short-series and no-x-extent edge cases, the table entry point over plain, keyed, empty, grouped, timestamp and date input, and every validation path. The property tests check the algorithm's actual guarantee - that each dropped point lies within tolerance of the chord joining its surviving neighbours - against an independent restatement of the distance formula. Also fixes the module load line in the test csv, which read use`analytics rather than use`di.analytics and meant no analytics test could run at all. Co-Authored-By: Claude Opus 5 <[email protected]>
jonnypress
requested review from
Papamaci444,
jonathonmcmurray and
submartingle
September 18, 2026 15:24
DIReview Summary0 critical | 0 warning(s) | 1 suggestion(s)
Suggestions
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds shrink, rdprecur and rdpiter: the Ramer-Douglas-Peucker line simplification described in "Dynamically shrinking big data using timeseries database kdb+" by Sean Keevey and Kevin Smyth (https://code.kx.com/q/wp/ts-shrink/), credited in the code header and in a References section in the module docs.
shrink is the dictionary entry point over a table (table/xcol/ycol/ tolerance, optional by and method) and returns the source table restricted to the rows worth keeping. rdprecur and rdpiter are the recursive and iterative kernels, exposed for callers working with plain vectors; both return the indices of the retained points rather than (x;y) pairs, which is what lets shrink carry every column through and simplify each by group independently.
Three deliberate differences from the paper's listings:
Tests cover both kernels against a worked example, the short-series and no-x-extent edge cases, the table entry point over plain, keyed, empty, grouped, timestamp and date input, and every validation path. The property tests check the algorithm's actual guarantee - that each dropped point lies within tolerance of the chord joining its surviving neighbours - against an independent restatement of the distance formula.
Also fixes the module load line in the test csv, which read use
analytics rather than usedi.analytics and meant no analytics test could run at all.