Skip to content

Add time-series simplification to di.analytics - #133

Open
jonnypress wants to merge 1 commit into
mainfrom
feature-analytics-shrink
Open

jonnypress wants to merge 1 commit into
mainfrom
feature-analytics-shrink

Conversation

@jonnypress

Copy link
Copy Markdown
Contributor

Adds shrink, rdprecur and rdpiter: the Ramer-Douglas-Peucker line simplification described in "Dynamically shrinking big data using timeseries database kdb+" by Sean Keevey and Kevin Smyth (https://code.kx.com/q/wp/ts-shrink/), credited in the code header and in a References section in the module docs.

shrink is the dictionary entry point over a table (table/xcol/ycol/ tolerance, optional by and method) and returns the source table restricted to the rows worth keeping. rdprecur and rdpiter are the recursive and iterative kernels, exposed for callers working with plain vectors; both return the indices of the retained points rather than (x;y) pairs, which is what lets shrink carry every column through and simplify each by group independently.

Three deliberate differences from the paper's listings:

  • Segment endpoints have their distance to their own chord pinned to zero. The paper computes it as slopex1+(y1-slopex1)-y1, which is not exactly zero in floating point; when that noise is the maximum, the breakpoint lands on an endpoint, the split drops nothing and the recursion never terminates. It is reachable - rdpRecur[0f;...] over ten ordinary points exhausts the stack.
  • The x axis is rebased on its first value before the cast to float, so a nanosecond timestamp keeps its resolution instead of rounding to ~100ns.
  • The iterative kernel splits every pending segment per pass rather than one. The paper's one-per-pass queue costs about 3x against recursion; level-wise brings it within ~6% (106ms vs 112ms over a 20k point walk), so shrink defaults to the iterative method - the speed argument for recursion no longer outweighs its stack risk.

Tests cover both kernels against a worked example, the short-series and no-x-extent edge cases, the table entry point over plain, keyed, empty, grouped, timestamp and date input, and every validation path. The property tests check the algorithm's actual guarantee - that each dropped point lies within tolerance of the chord joining its surviving neighbours - against an independent restatement of the distance formula.

Also fixes the module load line in the test csv, which read useanalytics rather than usedi.analytics and meant no analytics test could run at all.

Adds shrink, rdprecur and rdpiter: the Ramer-Douglas-Peucker line
simplification described in "Dynamically shrinking big data using
timeseries database kdb+" by Sean Keevey and Kevin Smyth
(https://code.kx.com/q/wp/ts-shrink/), credited in the code header and in
a References section in the module docs.

shrink is the dictionary entry point over a table (table/xcol/ycol/
tolerance, optional by and method) and returns the source table
restricted to the rows worth keeping. rdprecur and rdpiter are the
recursive and iterative kernels, exposed for callers working with plain
vectors; both return the indices of the retained points rather than
(x;y) pairs, which is what lets shrink carry every column through and
simplify each by group independently.

Three deliberate differences from the paper's listings:

- Segment endpoints have their distance to their own chord pinned to
  zero. The paper computes it as slope*x1+(y1-slope*x1)-y1, which is not
  exactly zero in floating point; when that noise is the maximum, the
  breakpoint lands on an endpoint, the split drops nothing and the
  recursion never terminates. It is reachable - rdpRecur[0f;...] over ten
  ordinary points exhausts the stack.
- The x axis is rebased on its first value before the cast to float, so a
  nanosecond timestamp keeps its resolution instead of rounding to ~100ns.
- The iterative kernel splits every pending segment per pass rather than
  one. The paper's one-per-pass queue costs about 3x against recursion;
  level-wise brings it within ~6% (106ms vs 112ms over a 20k point walk),
  so shrink defaults to the iterative method - the speed argument for
  recursion no longer outweighs its stack risk.

Tests cover both kernels against a worked example, the short-series and
no-x-extent edge cases, the table entry point over plain, keyed, empty,
grouped, timestamp and date input, and every validation path. The
property tests check the algorithm's actual guarantee - that each dropped
point lies within tolerance of the chord joining its surviving
neighbours - against an independent restatement of the distance formula.

Also fixes the module load line in the test csv, which read
use`analytics rather than use`di.analytics and meant no analytics test
could run at all.

Co-Authored-By: Claude Opus 5 <[email protected]>
@DI-Software-Engineering

Copy link
Copy Markdown

DIReview Summary

0 critical | 0 warning(s) | 1 suggestion(s)

⚠️ Spec check skipped — tracker lookup failed (NO_REF_FOUND). Standards axis only.

Suggestions

  • di/analytics/analytics.q:164 — The iterative kernel's convergence check (iterate[...]/[state]) can loop forever when tolerance is exactly 0 and the series contains collinear points. segdist pins the two endpoints of every segment to 0f, so for a collinear interior segment the max distance is 0f, which is NOT strictly less than tolerance=0, meaning split is 0b for that segment and it is correctly retired — but a segment whose interior points all have distance exactly 0 except for floating-point noise could produce a furthest point that is actually an interior point with distance > 0 and > tolerance=0. The segdist zero-pinning of endpoints prevents the endpoints being chosen as breakpoints, but floating-point arithmetic on the interior points can still yield a non-zero max distance causing an infinite bisection on a three-point segment where the interior point measures at machine epsilon. The analogous recursion guard is documented in analytics.md for rdprecur ("Without this a segment can pick one of its own endpoints as the breakpoint and fail to shrink, which recurses forever"), but for rdpiter the converge operator (/) will also loop forever if any segment never reaches the 'retire' branch. Verify that pinning endpoints to 0f is sufficient to guarantee convergence for rdpiter at tolerance=0 on all degenerate cases, or add an explicit guard for zero-length segments (where s=e after a bisect).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants