From 3aa664a1886d50b55eeec359ee7f9a62435569c0 Mon Sep 17 00:00:00 2001 From: Jonny Press Date: Fri, 18 Sep 2026 16:17:16 +0100 Subject: [PATCH] Add time-series simplification to di.analytics Adds shrink, rdprecur and rdpiter: the Ramer-Douglas-Peucker line simplification described in "Dynamically shrinking big data using timeseries database kdb+" by Sean Keevey and Kevin Smyth (https://code.kx.com/q/wp/ts-shrink/), credited in the code header and in a References section in the module docs. shrink is the dictionary entry point over a table (table/xcol/ycol/ tolerance, optional by and method) and returns the source table restricted to the rows worth keeping. rdprecur and rdpiter are the recursive and iterative kernels, exposed for callers working with plain vectors; both return the indices of the retained points rather than (x;y) pairs, which is what lets shrink carry every column through and simplify each by group independently. Three deliberate differences from the paper's listings: - Segment endpoints have their distance to their own chord pinned to zero. The paper computes it as slope*x1+(y1-slope*x1)-y1, which is not exactly zero in floating point; when that noise is the maximum, the breakpoint lands on an endpoint, the split drops nothing and the recursion never terminates. It is reachable - rdpRecur[0f;...] over ten ordinary points exhausts the stack. - The x axis is rebased on its first value before the cast to float, so a nanosecond timestamp keeps its resolution instead of rounding to ~100ns. - The iterative kernel splits every pending segment per pass rather than one. The paper's one-per-pass queue costs about 3x against recursion; level-wise brings it within ~6% (106ms vs 112ms over a 20k point walk), so shrink defaults to the iterative method - the speed argument for recursion no longer outweighs its stack risk. Tests cover both kernels against a worked example, the short-series and no-x-extent edge cases, the table entry point over plain, keyed, empty, grouped, timestamp and date input, and every validation path. The property tests check the algorithm's actual guarantee - that each dropped point lies within tolerance of the chord joining its surviving neighbours - against an independent restatement of the distance formula. Also fixes the module load line in the test csv, which read use`analytics rather than use`di.analytics and meant no analytics test could run at all. Co-Authored-By: Claude Opus 5 --- di/analytics/analytics.md | 161 ++++++++++++++++++++++++++++++++++++- di/analytics/analytics.q | 162 +++++++++++++++++++++++++++++++++++++- di/analytics/init.q | 2 +- di/analytics/test.csv | 74 ++++++++++++++++- 4 files changed, 392 insertions(+), 7 deletions(-) diff --git a/di/analytics/analytics.md b/di/analytics/analytics.md index c241e5ee..d5b7b5b8 100644 --- a/di/analytics/analytics.md +++ b/di/analytics/analytics.md @@ -2,7 +2,7 @@ A set of analytical utilities designed to streamline and make common data manipulation operations more efficient in kdb+/q. -The library provides specialized functions for handling typical analytical workflows, including forward filling missing values, creating custom time intervals, pivoting tables, and generating cross-product expansions. Each function accepts dictionary parameters or a table for flexible configuration and includes robust error handling with informative messages. +The library provides specialized functions for handling typical analytical workflows, including forward filling missing values, creating custom time intervals, pivoting tables, generating cross-product expansions, and simplifying time series down to the points that carry their shape. Each function accepts dictionary parameters or a table for flexible configuration and includes robust error handling with informative messages. --- @@ -13,6 +13,8 @@ The library provides specialized functions for handling typical analytical workf - **`intervals`** – Generate custom time/value intervals with configurable step and rounding. - **`pivot`** – Transform tables into cross-tab (wide) format using a pivot column. - **`rack`** – Build cross products of key columns, optionally with time intervals and base tables. +- **`shrink`** – Reduce a time series to the points that carry its shape (Ramer-Douglas-Peucker). +- **`rdprecur`** / **`rdpiter`** – The recursive and iterative simplification kernels behind `shrink`. --- @@ -195,6 +197,138 @@ rack[`table`keycols!(quotes; `sym`exchange)] ``` --- +
+ +### ⚙️`shrink` + +**Description** + +Reduces a time series to the handful of points that carry its shape, discarding those that add +nothing. Spikes, turning points and trend changes survive; flat or near-linear runs collapse to +their endpoints. Unlike bucketing, nothing is averaged or moved — every returned row is a real row +from the source table, so neither the time nor the value domain is distorted. + +This is the Ramer-Douglas-Peucker algorithm described in +[Dynamically shrinking big data using timeseries database kdb+](https://code.kx.com/q/wp/ts-shrink/) +— see [References](#references). + +**Parameters** +- Dictionary containing: + - `table`: Source table (**required**) + - `xcol`: Name of the x-axis column — one numeric or temporal column (**required**) + - `ycol`: Name of the y-axis column — one numeric column (**required**) + - `tolerance`: Non-negative numeric atom; how far a point may sit from the line through its + neighbours before it is worth keeping (**required**) + - `by`: Grouping column(s) — each series is simplified independently (optional) + - `method`: `` `recursive`` or `` `iterative`` (optional, default: `` `iterative``) + +**Behaviour** +1. Draw a chord between the first and last points of the series. +2. Find the point furthest from that chord. +3. If it is further away than `tolerance`, keep it as a breakpoint and repeat on the two halves + either side of it; otherwise discard every point between the endpoints. + +- The first and last points of each series are always retained. +- The guarantee this gives: every discarded point lies within `tolerance` of the straight line + joining the two retained points that bracket it. +- Rows must already be ordered by `xcol` — within each `by` group where `by` is supplied. Out of + order input is rejected rather than silently simplified against meaningless chords. +- `xcol` and `ycol` must not contain nulls; filter or forward fill (see `ffill`) beforehand. +- All columns are carried through untouched — `shrink` selects rows, it does not project columns. +- With `by`, groups are simplified independently and the retained rows are returned in the order + they appear in the source table, not grouped. +- Keyed tables are simplified on their unkeyed form and returned unkeyed. + +**Examples** +```q +// Thin a day of prices down to its shape, to a tolerance of half a tick +shrink[`table`xcol`ycol`tolerance!(trades; `time; `price; 0.005)] + +// Simplify each symbol separately +shrink[`table`xcol`ycol`tolerance`by!(trades; `time; `price; 0.005; `sym)] + +// Use the recursive kernel instead of the default iterative one +shrink[`table`xcol`ycol`tolerance`method!(trades; `time; `price; 0.005; `recursive)] + +// Works on any ordered numeric axis, not just time +shrink[`table`xcol`ycol`tolerance!(curve; `strike; `vol; 0.001)] +``` + +--- +
+ +### ⚙️`rdprecur` / `rdpiter` + +**Description** + +The two simplification kernels that `shrink` dispatches to, exposed for callers working with plain +vectors rather than a table. Both take the same arguments and return exactly the same answer — they +differ only in how the work is sequenced. + +**Parameters** + +Called positionally as `[tolerance; x; y]`: +- `tolerance`: Non-negative numeric atom (**required**) +- `x`: x-axis vector — numeric or temporal, non-decreasing (**required**) +- `y`: y-axis vector — numeric, same length as `x` (**required**) + +**Behaviour** +- Returns the **indices** of the retained points, in ascending order — not the points themselves. + Indices compose better than values: they can be used to select from any parallel vector, or from + the whole table, which is what `shrink` does. +- A series of fewer than three points is returned whole. +- Neither kernel checks that `x` is ordered; `shrink` does that before it calls them. + +**Examples** +```q +// Indices of the points worth keeping +keep: rdpiter[0.005; trades`time; trades`price] + +// Use them to select from the source table +trades keep + +// The two kernels agree, by construction +rdprecur[0.005; x; y] ~ rdpiter[0.005; x; y] / 1b +``` + +--- +
+ +### Choosing a tolerance + +The distance being measured is *perpendicular* to the chord, so it mixes the two axes and its +meaning depends on their relative scale: + +- When `x` is a timestamp, its magnitude dwarfs any realistic `y`. The chord is effectively flat in + the rescaled space, so the perpendicular distance is the vertical gap between the point and the + chord — and `tolerance` reads directly in `y` units (price, volume, basis points). +- When the axes are comparable — a row number against a price, say — the distance is a genuine + perpendicular one and `tolerance` is a distance in that plane. + +A tolerance of `0` removes only points that are exactly redundant (collinear runs, repeated values). +A tolerance wider than the whole series collapses it to its two endpoints. In between, start from a +small fraction of the `y` range — a tick, a basis point — and adjust against the reduction achieved. + +### Recursive or iterative? + +| | `rdprecur` | `rdpiter` | +|---|---|---| +| Work queue | q's call stack | an explicit list of pending segments, walked with converge (`/`) | +| Depth risk | recursion depth is driven by the data; the paper reports stack exhaustion on volatile series at a low tolerance | none | +| Speed | marginally faster | within a few percent | + +On a 20,000-point random walk at `tolerance` 0.05 (18% of points discarded), the two kernels ran in +roughly 106 ms and 112 ms respectively. The paper's own iterative implementation walks its queue one +segment per pass, which costs it about 3x against recursion; splitting *every* pending segment in a +single pass instead reduces the number of passes from one per retained point to the depth of the +split tree, which is what closes the gap here. + +Because the difference is small and the failure mode of deep recursion is a hard `'stack` error, +`shrink` defaults to `` `iterative``. Reach for `` `recursive`` when the series is known to be +well behaved and the last few percent matter. + +--- +
## Error Handling @@ -210,4 +344,29 @@ The functions implement comprehensive validation with descriptive error messages 'Input parameter must be a dictionary with at least three keys (an optional key round):-start-end-interval 'some columns provided do not exist in the table 'interval start and end data type mismatch +'tolerance must be a non-negative number +'xcol must be non-decreasing within each series - sort the table on xcol first +'xcol and ycol must not contain nulls - remove them before shrinking ``` + +--- + +## References + +The time-series simplification functions (`shrink`, `rdprecur`, `rdpiter`) implement the +Ramer-Douglas-Peucker approach set out in: + +> Sean Keevey and Kevin Smyth, *Dynamically shrinking big data using timeseries database kdb+*, +> KX whitepaper — + +The paper supplies the algorithm, the perpendicular-distance formulation and the recursive / +iterative split. This implementation differs from the listings in it in three respects, all noted +in the code: + +- the kernels return **indices** rather than `(x;y)` pairs, so every column of the source table can + be carried through; +- the x axis is **rebased on its first value** before the distance arithmetic, which keeps full + resolution for nanosecond timestamps; +- the endpoints of each segment have their distance **pinned to zero** rather than left to + floating-point noise. Without this a segment can pick one of its own endpoints as the breakpoint + and fail to shrink, which recurses forever at a tolerance of `0`. diff --git a/di/analytics/analytics.q b/di/analytics/analytics.q index 565b849b..6621dbac 100644 --- a/di/analytics/analytics.q +++ b/di/analytics/analytics.q @@ -95,7 +95,161 @@ rack:{[d] :$[`base in fkey; (cross/)(d`base;rackkeycol;timeinterval); (cross/)(rackkeycol;timeinterval)]]; :$[`base in fkey; (cross/)(d`base;rackkeycol); rackkeycol]; } - - - - + +/ ============================================================ +/ time-series simplification +/ ============================================================ + +/ ramer-douglas-peucker line simplification, following "Dynamically shrinking big data using +/ timeseries database kdb+" by Sean Keevey and Kevin Smyth (https://code.kx.com/q/wp/ts-shrink/). +/ a chord is drawn between the first and last points of a series and the point furthest from +/ that chord is measured: if it sits further away than the caller's tolerance it is kept as a +/ breakpoint and the two halves either side of it are simplified the same way, otherwise every +/ point between the endpoints is discarded. what survives is the small set of points that carry +/ the shape of the series - spikes and turning points are preserved while flat runs collapse. + +/ the x axis may be numeric or temporal - short, int, long, real, float, timestamp, month, +/ date, timespan, minute, second, time. the y axis must be numeric +xaxistypes:5 6 7 8 9 12 13 14 16 17 18 19h; +yaxistypes:5 6 7 8 9h; + +checktolerance:{[tolerance] + / a null or negative tolerance would keep nothing sensible and, worse, would let a segment + / split on a point it has already kept - reject it before either kernel runs + if[not (type tolerance) in neg yaxistypes;'`$"tolerance must be a numeric atom"]; + if[(null tolerance) or 0>tolerance;'`$"tolerance must be a non-negative number"]; + }; + +pdist:{[x1;y1;x2;y2;px;py] + / perpendicular distance from each point (px;py) to the line through (x1;y1) and (x2;y2) + / a chord with no x extent has no gradient, so fall back to distance from the line x=x1 + if[x1=x2;:abs px-x1]; + slope:(y2-y1)%x2-x1; + intercept:y1-slope*x1; + :abs((slope*px)+intercept-py)%sqrt 1f+slope*slope; + }; + +/ every index in the segment running from point s to point e inclusive +segpoints:{[s;e] s+til 1+e-s}; + +segdist:{[px;py;idx] + / distance from every point of the segment spanned by idx to that segment's own chord. + / both endpoints lie on the chord by construction, so pin them to zero rather than leave + / them to floating-point noise: that guarantees the furthest point is an interior one and + / so every split strictly shrinks the segment + d:pdist[px first idx;py first idx;px last idx;py last idx;px idx;py idx]; + :@[d;0,-1+count d;:;0f]; + }; + +furthest:{[px;py;s;e] + / index of the point furthest from the chord joining points s and e, and its distance + d:segdist[px;py;segpoints[s;e]]; + :(s+first where d=max d;max d); + }; + +preppoints:{[px;py] + / casts both axes to float for the distance arithmetic. x is rebased on its first value + / before the cast: perpendicular distance is unchanged by that shift, and it preserves + / nanosecond resolution on timestamps, which a direct cast to float would round away + :("f"$px-first px;"f"$py); + }; + +recurse:{[tolerance;px;py;s;e] + / recursive kernel - the indices kept from the segment between points s and e + brk:furthest[px;py;s;e]; + :$[tolerancecount px;:til count px]; + pts:preppoints[px;py]; + :recurse[tolerance;pts 0;pts 1;0;-1+count px]; + }; + +/ the two segments a parent segment splits into at breakpoint b +bisect:{[seg;b] (seg[0],b;b,seg 1)}; + +/ every point of the given segments bar their endpoints - the points a retired segment drops +interiors:{[segs] `long$raze {1_-1_segpoints . x} each segs}; + +iterate:{[tolerance;px;py;state] + / one pass of the iterative kernel. every pending segment is measured against its chord and + / is either split at its furthest point or, if that point is within tolerance, retired - + / dropping all of its interior points. state is (pending segments;keep flags) + pending:state 0; + if[not count pending;:state]; + brk:flip furthest[px;py] ./: pending; + split:tolerancecount px;:til count px]; + pts:preppoints[px;py]; + :where last iterate[tolerance;pts 0;pts 1]/[(enlist 0,-1+count px;count[px]#1b)]; + }; + +/ simplification kernels selectable through shrink's method argument +kernels:`recursive`iterative!(rdprecur;rdpiter); + +checkorder:{[px] + / each point is measured against the chord joining its segment endpoints, which only + / describes the series if the points arrive in x order + if[any 0>1_deltas "f"$px-first px; + '`$"xcol must be non-decreasing within each series - sort the table on xcol first"]; + }; + +shrinkseries:{[kernel;tolerance;px;py] + / retained row indices for a single ordered series + checkorder[px]; + :kernel[tolerance;px;py]; + }; + +shrinkgroups:{[kernel;tolerance;px;py;grp] + / simplify each group of row indices independently, returning the retained rows in the + / order they appear in the source table + :`long$asc raze grp@'shrinkseries[kernel;tolerance]'[px@grp;py@grp]; + }; + +shrink:{[d] + / discards the points of a series that lie within tolerance of the line joining the points + / bracketing them, returning the input table restricted to the rows worth keeping. + / rows must already be ordered by xcol, within each by group where by is supplied + $[99h<>type d;'`$"input should be a dictionary"; + not all `table`xcol`ycol`tolerance in fkey:key[d];'`$"Input parameter must be a dictionary with at least four keys (with optional keys by and method):\n\t-",sv["\n\t-";string `table`xcol`ycol`tolerance]; + not .Q.qt d`table;'`$"table must be a table"; + any not -11h=type each d`xcol`ycol;'`$"xcol and ycol must each name a single column"; + not (method:$[`method in fkey;d`method;`iterative]) in key kernels;'`$"method must be one of:\n\t-",sv["\n\t-";string key kernels]]; + + checktolerance[d`tolerance]; + / keyed tables are simplified on their unkeyed form and returned unkeyed + t:0!d`table; + bycols:$[`by in fkey;(),d`by;`symbol$()]; + if[count missing:(d[`xcol],d[`ycol],bycols) except cols t; + '`$"some columns provided do not exist in the table: ",sv[", ";string missing]]; + + px:t d`xcol; + py:t d`ycol; + if[not (abs type px) in xaxistypes;'`$"xcol must be a numeric or temporal column"]; + if[not (abs type py) in yaxistypes;'`$"ycol must be a numeric column"]; + if[any raze null (px;py);'`$"xcol and ycol must not contain nulls - remove them before shrinking"]; + + :t $[count bycols; + shrinkgroups[kernels method;d`tolerance;px;py;value group flip bycols!t bycols]; + shrinkseries[kernels method;d`tolerance;px;py]]; + }; diff --git a/di/analytics/init.q b/di/analytics/init.q index 801d8f65..320e5a74 100644 --- a/di/analytics/init.q +++ b/di/analytics/init.q @@ -1,3 +1,3 @@ \l ::analytics.q -export:([ffill;ffillzero;intervals;pivot;rack]) +export:([ffill;ffillzero;intervals;pivot;rack;shrink;rdprecur;rdpiter]) diff --git a/di/analytics/test.csv b/di/analytics/test.csv index f9e483ed..a79856fb 100644 --- a/di/analytics/test.csv +++ b/di/analytics/test.csv @@ -1,5 +1,5 @@ action,ms,bytes,lang,code,repeat,minver,comment -before,0,0,q,analytics:use`analytics,1,1,load module into session +before,0,0,q,analytics:use`di.analytics,1,1,load module into session run,0,0,q,N:20;prob:0.2,1,,initialize testing parameters run,0,0,q,zerotable:table:`time xasc ([]time:N?.z.P;sym:N?`AMD`AAPL`MSFT`IBM;ask:N?100f;bid:N?100f;asize:N?500i;bsize:N?500i;ex:N?`NYSE`CME`LSE),1,,define testing table run,0,0,q,update ask:?[prob>N?1f;0n;ask] from `table,1,,add null values in testing table @@ -99,3 +99,75 @@ run,0,0,q,args:(`table`by`piv`var)!(quote;`date`sym;`side`level;`price`size),1,, run,0,0,q,res:analytics.pivot args,1,,call pivot function on input true,0,0,q,"(99h=type res)&(((),args`by)~cols key res)&(((count distinct (,/') flip string quote args`piv)*count args`by)~count cols value res)",1,,verify output in the correct format true,0,0,q,(asc (raze value flip value res) except 0n)~(asc raze quote args`var),1,,verify pivoted values conform to original data + +run,0,0,q,shx:0 1 2 3 4 5 6 7 8 9f,1,,x axis of a worked example - a flat series carrying one spike at x=3 +run,0,0,q,shy:0 0.1 0 5 0 0.2 0 0 0.1 0f,1,,y axis of the worked example +true,0,0,q,(0 2 3 4 9)~analytics.rdprecur[1f;shx;shy],1,,recursive kernel keeps the endpoints plus the spike and its shoulders +true,0,0,q,analytics.rdprecur[1f;shx;shy]~analytics.rdpiter[1f;shx;shy],1,,both kernels return identical indices +true,0,0,q,(til 10)~analytics.rdpiter[0f;shx;shy],1,,a zero tolerance discards nothing from a series with no collinear runs +true,0,0,q,(0 9)~analytics.rdpiter[100f;shx;shy],1,,a tolerance wider than the series collapses it to its endpoints +true,0,0,q,(0 3)~analytics.rdprecur[0f;0 1 2 3f;0 0 0 0f],1,,collinear points are redundant and go even at zero tolerance +true,0,0,q,(0 2 3 5)~analytics.rdpiter[1f;til 6;0 0 9 0 0 0],1,,integer x and y axes are accepted + +comment,,,,,,,kernel edge cases - series too short to simplify or with no x extent +true,0,0,q,(`long$())~analytics.rdprecur[1f;`float$();`float$()],1,,an empty series simplifies to nothing +true,0,0,q,(`long$())~analytics.rdpiter[1f;`float$();`float$()],1,,an empty series simplifies to nothing +true,0,0,q,(enlist 0)~analytics.rdprecur[1f;enlist 1f;enlist 2f],1,,a single point is always kept +true,0,0,q,(0 1)~analytics.rdpiter[1f;1 2f;3 4f],1,,a two point series is always kept in full +true,0,0,q,(0 3)~analytics.rdpiter[1f;5 5 5 5f;1 2 3 4f],1,,a series with no x extent collapses to its endpoints + +comment,,,,,,,properties that must hold for any series - checked against a random walk +run,0,0,q,M:5000,1,,length of the random walk used for the property tests +run,0,0,q,rwx:`float$til M,1,,evenly spaced x axis +run,0,0,q,rwy:sums -0.5+M?1f,1,,random walk on the y axis +run,0,0,q,"chorddist:{[px;py;a;b] s:(py[b]-py a)%px[b]-px a; c:(py a)-s*px a; m:1+a+til (b-a)-1; max 0f,abs((s*px m)+c-py m)%sqrt 1f+s*s}",1,,independent restatement of the distance the algorithm is supposed to bound - how far the points dropped between retained points a and b sit from their chord +run,0,0,q,maxresid:{[px;py;idx] max chorddist[px;py]'[-1_idx;1_idx]},1,,largest distance from any dropped point to the chord of the two retained points bracketing it +run,0,0,q,rwkeep:analytics.rdprecur[0.05;rwx;rwy],1,,simplify the random walk with the recursive kernel +true,0,0,q,rwkeep~analytics.rdpiter[0.05;rwx;rwy],1,,both kernels agree on a 5000 point random walk +true,0,0,q,rwkeep~asc distinct rwkeep,1,,retained indices come back ascending with no duplicates +true,0,0,q,"(0,M-1)~(first rwkeep),last rwkeep",1,,the first and last points of a series are always retained +true,0,0,q,count[rwkeep]=maxresid[rwx;rwy;rwkeep],1,,every dropped point lies within tolerance of the chord joining its surviving neighbours +true,0,0,q,all {[tol] tol>=maxresid[rwx;rwy;analytics.rdpiter[tol;rwx;rwy]]} each 0.01 0.25 1 5f,1,,the tolerance guarantee holds across a range of tolerances +true,0,0,q,all 0>=1_deltas count each analytics.rdpiter[;rwx;rwy] each 0.01 0.05 0.25 1 5f,1,,raising the tolerance never increases the number of retained points + +comment,,,,,,,shrink - the table entry point +run,0,0,q,"shtab:([]tm:shx;px:shy;sym:10#`A)",1,,table form of the worked example +run,0,0,q,shres:analytics.shrink `table`xcol`ycol`tolerance!(shtab;`tm;`px;1f),1,,simplify through the table entry point +true,0,0,q,shres~select from shtab where i in 0 2 3 4 9,1,,shrink keeps the rows the kernel selects and carries every column through +true,0,0,q,shres~analytics.shrink `table`xcol`ycol`tolerance`method!(shtab;`tm;`px;1f;`iterative),1,,shrink defaults to the iterative method +true,0,0,q,shres~analytics.shrink `table`xcol`ycol`tolerance`method!(shtab;`tm;`px;1f;`recursive),1,,the recursive method returns the same table +true,0,0,q,(0#shtab)~analytics.shrink `table`xcol`ycol`tolerance!(0#shtab;`tm;`px;1f),1,,an empty table simplifies to an empty table + +run,0,0,q,shkeyed:1!([]tm:shx;px:shy),1,,keyed form of the worked example +true,0,0,q,(delete sym from shres)~analytics.shrink `table`xcol`ycol`tolerance!(shkeyed;`tm;`px;1f),1,,a keyed table is simplified on its unkeyed form and returned unkeyed + +run,0,0,q,"shts:([]tm:2024.01.01D09:00:00+1000000000*til 10;px:shy;sym:10#`A)",1,,worked example on a timestamp x axis +true,0,0,q,(select from shts where i in 0 2 3 4 9)~analytics.shrink `table`xcol`ycol`tolerance!(shts;`tm;`px;1f),1,,a timestamp x axis is rebased before the distance arithmetic rather than losing resolution to a float cast +run,0,0,q,"shdt:([]tm:2024.01.01+til 10;px:shy)",1,,worked example on a date x axis +true,0,0,q,(select from shdt where i in 0 2 3 4 9)~analytics.shrink `table`xcol`ycol`tolerance!(shdt;`tm;`px;1f),1,,a date x axis is accepted + +comment,,,,,,,shrink - grouped series +run,0,0,q,"shby:([]sym:(10#`A),10#`B;tm:shx,shx;px:shy,reverse shy)",1,,two independent series stacked in one table +run,0,0,q,shbyres:analytics.shrink `table`xcol`ycol`tolerance`by!(shby;`tm;`px;1f;`sym),1,,simplify each sym on its own +true,0,0,q,shbyres~select from shby where i in 0 2 3 4 9 10 15 16 17 19,1,,each group is simplified independently and rows come back in source order +run,0,0,q,shby2:update ex:`N from shby,1,,add a second grouping column +true,0,0,q,shbyres~delete ex from analytics.shrink `table`xcol`ycol`tolerance`by!(shby2;`tm;`px;1f;`sym`ex),1,,multiple by columns are supported + +comment,,,,,,,shrink - input validation +fail,0,0,q,analytics.shrink 5,1,,a non dictionary argument is rejected +fail,0,0,q,analytics.shrink `table`xcol!(shtab;`tm),1,,missing required keys are rejected +fail,0,0,q,analytics.shrink `table`xcol`ycol`tolerance!(5;`tm;`px;1f),1,,a non table is rejected +fail,0,0,q,analytics.shrink `table`xcol`ycol`tolerance!(shtab;`tm`px;`px;1f),1,,xcol must name exactly one column +fail,0,0,q,analytics.shrink `table`xcol`ycol`tolerance`method!(shtab;`tm;`px;1f;`quick),1,,an unknown method is rejected +fail,0,0,q,analytics.shrink `table`xcol`ycol`tolerance!(shtab;`nope;`px;1f),1,,a missing column is rejected +fail,0,0,q,analytics.shrink `table`xcol`ycol`tolerance`by!(shtab;`tm;`px;1f;`nope),1,,a missing by column is rejected +fail,0,0,q,analytics.shrink `table`xcol`ycol`tolerance!(shtab;`sym;`px;1f),1,,a non numeric x axis is rejected +fail,0,0,q,analytics.shrink `table`xcol`ycol`tolerance!(shtab;`tm;`sym;1f),1,,a non numeric y axis is rejected +fail,0,0,q,analytics.shrink `table`xcol`ycol`tolerance!(shtab;`tm;`px;-1f),1,,a negative tolerance is rejected +fail,0,0,q,analytics.shrink `table`xcol`ycol`tolerance!(shtab;`tm;`px;0n),1,,a null tolerance is rejected +fail,0,0,q,analytics.shrink `table`xcol`ycol`tolerance!(shtab;`tm;`px;1 2f),1,,a non atomic tolerance is rejected +fail,0,0,q,analytics.shrink `table`xcol`ycol`tolerance!(update px:0n from shtab where i=3;`tm;`px;1f),1,,nulls in the measured columns are rejected +fail,0,0,q,analytics.shrink `table`xcol`ycol`tolerance!(`px xdesc shtab;`tm;`px;1f),1,,an x axis that is not in order is rejected +fail,0,0,q,analytics.rdprecur[-1f;shx;shy],1,,the kernels validate the tolerance too +fail,0,0,q,analytics.rdpiter[0n;shx;shy],1,,the kernels validate the tolerance too