diff --git a/di/analytics/analytics.md b/di/analytics/analytics.md index c241e5ee..d5b7b5b8 100644 --- a/di/analytics/analytics.md +++ b/di/analytics/analytics.md @@ -2,7 +2,7 @@ A set of analytical utilities designed to streamline and make common data manipulation operations more efficient in kdb+/q. -The library provides specialized functions for handling typical analytical workflows, including forward filling missing values, creating custom time intervals, pivoting tables, and generating cross-product expansions. Each function accepts dictionary parameters or a table for flexible configuration and includes robust error handling with informative messages. +The library provides specialized functions for handling typical analytical workflows, including forward filling missing values, creating custom time intervals, pivoting tables, generating cross-product expansions, and simplifying time series down to the points that carry their shape. Each function accepts dictionary parameters or a table for flexible configuration and includes robust error handling with informative messages. --- @@ -13,6 +13,8 @@ The library provides specialized functions for handling typical analytical workf - **`intervals`** – Generate custom time/value intervals with configurable step and rounding. - **`pivot`** – Transform tables into cross-tab (wide) format using a pivot column. - **`rack`** – Build cross products of key columns, optionally with time intervals and base tables. +- **`shrink`** – Reduce a time series to the points that carry its shape (Ramer-Douglas-Peucker). +- **`rdprecur`** / **`rdpiter`** – The recursive and iterative simplification kernels behind `shrink`. --- @@ -195,6 +197,138 @@ rack[`table`keycols!(quotes; `sym`exchange)] ``` --- +
+ +### ⚙️`shrink` + +**Description** + +Reduces a time series to the handful of points that carry its shape, discarding those that add +nothing. Spikes, turning points and trend changes survive; flat or near-linear runs collapse to +their endpoints. Unlike bucketing, nothing is averaged or moved — every returned row is a real row +from the source table, so neither the time nor the value domain is distorted. + +This is the Ramer-Douglas-Peucker algorithm described in +[Dynamically shrinking big data using timeseries database kdb+](https://code.kx.com/q/wp/ts-shrink/) +— see [References](#references). + +**Parameters** +- Dictionary containing: + - `table`: Source table (**required**) + - `xcol`: Name of the x-axis column — one numeric or temporal column (**required**) + - `ycol`: Name of the y-axis column — one numeric column (**required**) + - `tolerance`: Non-negative numeric atom; how far a point may sit from the line through its + neighbours before it is worth keeping (**required**) + - `by`: Grouping column(s) — each series is simplified independently (optional) + - `method`: `` `recursive`` or `` `iterative`` (optional, default: `` `iterative``) + +**Behaviour** +1. Draw a chord between the first and last points of the series. +2. Find the point furthest from that chord. +3. If it is further away than `tolerance`, keep it as a breakpoint and repeat on the two halves + either side of it; otherwise discard every point between the endpoints. + +- The first and last points of each series are always retained. +- The guarantee this gives: every discarded point lies within `tolerance` of the straight line + joining the two retained points that bracket it. +- Rows must already be ordered by `xcol` — within each `by` group where `by` is supplied. Out of + order input is rejected rather than silently simplified against meaningless chords. +- `xcol` and `ycol` must not contain nulls; filter or forward fill (see `ffill`) beforehand. +- All columns are carried through untouched — `shrink` selects rows, it does not project columns. +- With `by`, groups are simplified independently and the retained rows are returned in the order + they appear in the source table, not grouped. +- Keyed tables are simplified on their unkeyed form and returned unkeyed. + +**Examples** +```q +// Thin a day of prices down to its shape, to a tolerance of half a tick +shrink[`table`xcol`ycol`tolerance!(trades; `time; `price; 0.005)] + +// Simplify each symbol separately +shrink[`table`xcol`ycol`tolerance`by!(trades; `time; `price; 0.005; `sym)] + +// Use the recursive kernel instead of the default iterative one +shrink[`table`xcol`ycol`tolerance`method!(trades; `time; `price; 0.005; `recursive)] + +// Works on any ordered numeric axis, not just time +shrink[`table`xcol`ycol`tolerance!(curve; `strike; `vol; 0.001)] +``` + +--- +
+ +### ⚙️`rdprecur` / `rdpiter` + +**Description** + +The two simplification kernels that `shrink` dispatches to, exposed for callers working with plain +vectors rather than a table. Both take the same arguments and return exactly the same answer — they +differ only in how the work is sequenced. + +**Parameters** + +Called positionally as `[tolerance; x; y]`: +- `tolerance`: Non-negative numeric atom (**required**) +- `x`: x-axis vector — numeric or temporal, non-decreasing (**required**) +- `y`: y-axis vector — numeric, same length as `x` (**required**) + +**Behaviour** +- Returns the **indices** of the retained points, in ascending order — not the points themselves. + Indices compose better than values: they can be used to select from any parallel vector, or from + the whole table, which is what `shrink` does. +- A series of fewer than three points is returned whole. +- Neither kernel checks that `x` is ordered; `shrink` does that before it calls them. + +**Examples** +```q +// Indices of the points worth keeping +keep: rdpiter[0.005; trades`time; trades`price] + +// Use them to select from the source table +trades keep + +// The two kernels agree, by construction +rdprecur[0.005; x; y] ~ rdpiter[0.005; x; y] / 1b +``` + +--- +
+ +### Choosing a tolerance + +The distance being measured is *perpendicular* to the chord, so it mixes the two axes and its +meaning depends on their relative scale: + +- When `x` is a timestamp, its magnitude dwarfs any realistic `y`. The chord is effectively flat in + the rescaled space, so the perpendicular distance is the vertical gap between the point and the + chord — and `tolerance` reads directly in `y` units (price, volume, basis points). +- When the axes are comparable — a row number against a price, say — the distance is a genuine + perpendicular one and `tolerance` is a distance in that plane. + +A tolerance of `0` removes only points that are exactly redundant (collinear runs, repeated values). +A tolerance wider than the whole series collapses it to its two endpoints. In between, start from a +small fraction of the `y` range — a tick, a basis point — and adjust against the reduction achieved. + +### Recursive or iterative? + +| | `rdprecur` | `rdpiter` | +|---|---|---| +| Work queue | q's call stack | an explicit list of pending segments, walked with converge (`/`) | +| Depth risk | recursion depth is driven by the data; the paper reports stack exhaustion on volatile series at a low tolerance | none | +| Speed | marginally faster | within a few percent | + +On a 20,000-point random walk at `tolerance` 0.05 (18% of points discarded), the two kernels ran in +roughly 106 ms and 112 ms respectively. The paper's own iterative implementation walks its queue one +segment per pass, which costs it about 3x against recursion; splitting *every* pending segment in a +single pass instead reduces the number of passes from one per retained point to the depth of the +split tree, which is what closes the gap here. + +Because the difference is small and the failure mode of deep recursion is a hard `'stack` error, +`shrink` defaults to `` `iterative``. Reach for `` `recursive`` when the series is known to be +well behaved and the last few percent matter. + +--- +
## Error Handling @@ -210,4 +344,29 @@ The functions implement comprehensive validation with descriptive error messages 'Input parameter must be a dictionary with at least three keys (an optional key round):-start-end-interval 'some columns provided do not exist in the table 'interval start and end data type mismatch +'tolerance must be a non-negative number +'xcol must be non-decreasing within each series - sort the table on xcol first +'xcol and ycol must not contain nulls - remove them before shrinking ``` + +--- + +## References + +The time-series simplification functions (`shrink`, `rdprecur`, `rdpiter`) implement the +Ramer-Douglas-Peucker approach set out in: + +> Sean Keevey and Kevin Smyth, *Dynamically shrinking big data using timeseries database kdb+*, +> KX whitepaper — + +The paper supplies the algorithm, the perpendicular-distance formulation and the recursive / +iterative split. This implementation differs from the listings in it in three respects, all noted +in the code: + +- the kernels return **indices** rather than `(x;y)` pairs, so every column of the source table can + be carried through; +- the x axis is **rebased on its first value** before the distance arithmetic, which keeps full + resolution for nanosecond timestamps; +- the endpoints of each segment have their distance **pinned to zero** rather than left to + floating-point noise. Without this a segment can pick one of its own endpoints as the breakpoint + and fail to shrink, which recurses forever at a tolerance of `0`. diff --git a/di/analytics/analytics.q b/di/analytics/analytics.q index 565b849b..6621dbac 100644 --- a/di/analytics/analytics.q +++ b/di/analytics/analytics.q @@ -95,7 +95,161 @@ rack:{[d] :$[`base in fkey; (cross/)(d`base;rackkeycol;timeinterval); (cross/)(rackkeycol;timeinterval)]]; :$[`base in fkey; (cross/)(d`base;rackkeycol); rackkeycol]; } - - - - + +/ ============================================================ +/ time-series simplification +/ ============================================================ + +/ ramer-douglas-peucker line simplification, following "Dynamically shrinking big data using +/ timeseries database kdb+" by Sean Keevey and Kevin Smyth (https://code.kx.com/q/wp/ts-shrink/). +/ a chord is drawn between the first and last points of a series and the point furthest from +/ that chord is measured: if it sits further away than the caller's tolerance it is kept as a +/ breakpoint and the two halves either side of it are simplified the same way, otherwise every +/ point between the endpoints is discarded. what survives is the small set of points that carry +/ the shape of the series - spikes and turning points are preserved while flat runs collapse. + +/ the x axis may be numeric or temporal - short, int, long, real, float, timestamp, month, +/ date, timespan, minute, second, time. the y axis must be numeric +xaxistypes:5 6 7 8 9 12 13 14 16 17 18 19h; +yaxistypes:5 6 7 8 9h; + +checktolerance:{[tolerance] + / a null or negative tolerance would keep nothing sensible and, worse, would let a segment + / split on a point it has already kept - reject it before either kernel runs + if[not (type tolerance) in neg yaxistypes;'`$"tolerance must be a numeric atom"]; + if[(null tolerance) or 0>tolerance;'`$"tolerance must be a non-negative number"]; + }; + +pdist:{[x1;y1;x2;y2;px;py] + / perpendicular distance from each point (px;py) to the line through (x1;y1) and (x2;y2) + / a chord with no x extent has no gradient, so fall back to distance from the line x=x1 + if[x1=x2;:abs px-x1]; + slope:(y2-y1)%x2-x1; + intercept:y1-slope*x1; + :abs((slope*px)+intercept-py)%sqrt 1f+slope*slope; + }; + +/ every index in the segment running from point s to point e inclusive +segpoints:{[s;e] s+til 1+e-s}; + +segdist:{[px;py;idx] + / distance from every point of the segment spanned by idx to that segment's own chord. + / both endpoints lie on the chord by construction, so pin them to zero rather than leave + / them to floating-point noise: that guarantees the furthest point is an interior one and + / so every split strictly shrinks the segment + d:pdist[px first idx;py first idx;px last idx;py last idx;px idx;py idx]; + :@[d;0,-1+count d;:;0f]; + }; + +furthest:{[px;py;s;e] + / index of the point furthest from the chord joining points s and e, and its distance + d:segdist[px;py;segpoints[s;e]]; + :(s+first where d=max d;max d); + }; + +preppoints:{[px;py] + / casts both axes to float for the distance arithmetic. x is rebased on its first value + / before the cast: perpendicular distance is unchanged by that shift, and it preserves + / nanosecond resolution on timestamps, which a direct cast to float would round away + :("f"$px-first px;"f"$py); + }; + +recurse:{[tolerance;px;py;s;e] + / recursive kernel - the indices kept from the segment between points s and e + brk:furthest[px;py;s;e]; + :$[tolerancecount px;:til count px]; + pts:preppoints[px;py]; + :recurse[tolerance;pts 0;pts 1;0;-1+count px]; + }; + +/ the two segments a parent segment splits into at breakpoint b +bisect:{[seg;b] (seg[0],b;b,seg 1)}; + +/ every point of the given segments bar their endpoints - the points a retired segment drops +interiors:{[segs] `long$raze {1_-1_segpoints . x} each segs}; + +iterate:{[tolerance;px;py;state] + / one pass of the iterative kernel. every pending segment is measured against its chord and + / is either split at its furthest point or, if that point is within tolerance, retired - + / dropping all of its interior points. state is (pending segments;keep flags) + pending:state 0; + if[not count pending;:state]; + brk:flip furthest[px;py] ./: pending; + split:tolerancecount px;:til count px]; + pts:preppoints[px;py]; + :where last iterate[tolerance;pts 0;pts 1]/[(enlist 0,-1+count px;count[px]#1b)]; + }; + +/ simplification kernels selectable through shrink's method argument +kernels:`recursive`iterative!(rdprecur;rdpiter); + +checkorder:{[px] + / each point is measured against the chord joining its segment endpoints, which only + / describes the series if the points arrive in x order + if[any 0>1_deltas "f"$px-first px; + '`$"xcol must be non-decreasing within each series - sort the table on xcol first"]; + }; + +shrinkseries:{[kernel;tolerance;px;py] + / retained row indices for a single ordered series + checkorder[px]; + :kernel[tolerance;px;py]; + }; + +shrinkgroups:{[kernel;tolerance;px;py;grp] + / simplify each group of row indices independently, returning the retained rows in the + / order they appear in the source table + :`long$asc raze grp@'shrinkseries[kernel;tolerance]'[px@grp;py@grp]; + }; + +shrink:{[d] + / discards the points of a series that lie within tolerance of the line joining the points + / bracketing them, returning the input table restricted to the rows worth keeping. + / rows must already be ordered by xcol, within each by group where by is supplied + $[99h<>type d;'`$"input should be a dictionary"; + not all `table`xcol`ycol`tolerance in fkey:key[d];'`$"Input parameter must be a dictionary with at least four keys (with optional keys by and method):\n\t-",sv["\n\t-";string `table`xcol`ycol`tolerance]; + not .Q.qt d`table;'`$"table must be a table"; + any not -11h=type each d`xcol`ycol;'`$"xcol and ycol must each name a single column"; + not (method:$[`method in fkey;d`method;`iterative]) in key kernels;'`$"method must be one of:\n\t-",sv["\n\t-";string key kernels]]; + + checktolerance[d`tolerance]; + / keyed tables are simplified on their unkeyed form and returned unkeyed + t:0!d`table; + bycols:$[`by in fkey;(),d`by;`symbol$()]; + if[count missing:(d[`xcol],d[`ycol],bycols) except cols t; + '`$"some columns provided do not exist in the table: ",sv[", ";string missing]]; + + px:t d`xcol; + py:t d`ycol; + if[not (abs type px) in xaxistypes;'`$"xcol must be a numeric or temporal column"]; + if[not (abs type py) in yaxistypes;'`$"ycol must be a numeric column"]; + if[any raze null (px;py);'`$"xcol and ycol must not contain nulls - remove them before shrinking"]; + + :t $[count bycols; + shrinkgroups[kernels method;d`tolerance;px;py;value group flip bycols!t bycols]; + shrinkseries[kernels method;d`tolerance;px;py]]; + }; diff --git a/di/analytics/init.q b/di/analytics/init.q index 801d8f65..320e5a74 100644 --- a/di/analytics/init.q +++ b/di/analytics/init.q @@ -1,3 +1,3 @@ \l ::analytics.q -export:([ffill;ffillzero;intervals;pivot;rack]) +export:([ffill;ffillzero;intervals;pivot;rack;shrink;rdprecur;rdpiter]) diff --git a/di/analytics/test.csv b/di/analytics/test.csv index f9e483ed..a79856fb 100644 --- a/di/analytics/test.csv +++ b/di/analytics/test.csv @@ -1,5 +1,5 @@ action,ms,bytes,lang,code,repeat,minver,comment -before,0,0,q,analytics:use`analytics,1,1,load module into session +before,0,0,q,analytics:use`di.analytics,1,1,load module into session run,0,0,q,N:20;prob:0.2,1,,initialize testing parameters run,0,0,q,zerotable:table:`time xasc ([]time:N?.z.P;sym:N?`AMD`AAPL`MSFT`IBM;ask:N?100f;bid:N?100f;asize:N?500i;bsize:N?500i;ex:N?`NYSE`CME`LSE),1,,define testing table run,0,0,q,update ask:?[prob>N?1f;0n;ask] from `table,1,,add null values in testing table @@ -99,3 +99,75 @@ run,0,0,q,args:(`table`by`piv`var)!(quote;`date`sym;`side`level;`price`size),1,, run,0,0,q,res:analytics.pivot args,1,,call pivot function on input true,0,0,q,"(99h=type res)&(((),args`by)~cols key res)&(((count distinct (,/') flip string quote args`piv)*count args`by)~count cols value res)",1,,verify output in the correct format true,0,0,q,(asc (raze value flip value res) except 0n)~(asc raze quote args`var),1,,verify pivoted values conform to original data + +run,0,0,q,shx:0 1 2 3 4 5 6 7 8 9f,1,,x axis of a worked example - a flat series carrying one spike at x=3 +run,0,0,q,shy:0 0.1 0 5 0 0.2 0 0 0.1 0f,1,,y axis of the worked example +true,0,0,q,(0 2 3 4 9)~analytics.rdprecur[1f;shx;shy],1,,recursive kernel keeps the endpoints plus the spike and its shoulders +true,0,0,q,analytics.rdprecur[1f;shx;shy]~analytics.rdpiter[1f;shx;shy],1,,both kernels return identical indices +true,0,0,q,(til 10)~analytics.rdpiter[0f;shx;shy],1,,a zero tolerance discards nothing from a series with no collinear runs +true,0,0,q,(0 9)~analytics.rdpiter[100f;shx;shy],1,,a tolerance wider than the series collapses it to its endpoints +true,0,0,q,(0 3)~analytics.rdprecur[0f;0 1 2 3f;0 0 0 0f],1,,collinear points are redundant and go even at zero tolerance +true,0,0,q,(0 2 3 5)~analytics.rdpiter[1f;til 6;0 0 9 0 0 0],1,,integer x and y axes are accepted + +comment,,,,,,,kernel edge cases - series too short to simplify or with no x extent +true,0,0,q,(`long$())~analytics.rdprecur[1f;`float$();`float$()],1,,an empty series simplifies to nothing +true,0,0,q,(`long$())~analytics.rdpiter[1f;`float$();`float$()],1,,an empty series simplifies to nothing +true,0,0,q,(enlist 0)~analytics.rdprecur[1f;enlist 1f;enlist 2f],1,,a single point is always kept +true,0,0,q,(0 1)~analytics.rdpiter[1f;1 2f;3 4f],1,,a two point series is always kept in full +true,0,0,q,(0 3)~analytics.rdpiter[1f;5 5 5 5f;1 2 3 4f],1,,a series with no x extent collapses to its endpoints + +comment,,,,,,,properties that must hold for any series - checked against a random walk +run,0,0,q,M:5000,1,,length of the random walk used for the property tests +run,0,0,q,rwx:`float$til M,1,,evenly spaced x axis +run,0,0,q,rwy:sums -0.5+M?1f,1,,random walk on the y axis +run,0,0,q,"chorddist:{[px;py;a;b] s:(py[b]-py a)%px[b]-px a; c:(py a)-s*px a; m:1+a+til (b-a)-1; max 0f,abs((s*px m)+c-py m)%sqrt 1f+s*s}",1,,independent restatement of the distance the algorithm is supposed to bound - how far the points dropped between retained points a and b sit from their chord +run,0,0,q,maxresid:{[px;py;idx] max chorddist[px;py]'[-1_idx;1_idx]},1,,largest distance from any dropped point to the chord of the two retained points bracketing it +run,0,0,q,rwkeep:analytics.rdprecur[0.05;rwx;rwy],1,,simplify the random walk with the recursive kernel +true,0,0,q,rwkeep~analytics.rdpiter[0.05;rwx;rwy],1,,both kernels agree on a 5000 point random walk +true,0,0,q,rwkeep~asc distinct rwkeep,1,,retained indices come back ascending with no duplicates +true,0,0,q,"(0,M-1)~(first rwkeep),last rwkeep",1,,the first and last points of a series are always retained +true,0,0,q,count[rwkeep]=maxresid[rwx;rwy;rwkeep],1,,every dropped point lies within tolerance of the chord joining its surviving neighbours +true,0,0,q,all {[tol] tol>=maxresid[rwx;rwy;analytics.rdpiter[tol;rwx;rwy]]} each 0.01 0.25 1 5f,1,,the tolerance guarantee holds across a range of tolerances +true,0,0,q,all 0>=1_deltas count each analytics.rdpiter[;rwx;rwy] each 0.01 0.05 0.25 1 5f,1,,raising the tolerance never increases the number of retained points + +comment,,,,,,,shrink - the table entry point +run,0,0,q,"shtab:([]tm:shx;px:shy;sym:10#`A)",1,,table form of the worked example +run,0,0,q,shres:analytics.shrink `table`xcol`ycol`tolerance!(shtab;`tm;`px;1f),1,,simplify through the table entry point +true,0,0,q,shres~select from shtab where i in 0 2 3 4 9,1,,shrink keeps the rows the kernel selects and carries every column through +true,0,0,q,shres~analytics.shrink `table`xcol`ycol`tolerance`method!(shtab;`tm;`px;1f;`iterative),1,,shrink defaults to the iterative method +true,0,0,q,shres~analytics.shrink `table`xcol`ycol`tolerance`method!(shtab;`tm;`px;1f;`recursive),1,,the recursive method returns the same table +true,0,0,q,(0#shtab)~analytics.shrink `table`xcol`ycol`tolerance!(0#shtab;`tm;`px;1f),1,,an empty table simplifies to an empty table + +run,0,0,q,shkeyed:1!([]tm:shx;px:shy),1,,keyed form of the worked example +true,0,0,q,(delete sym from shres)~analytics.shrink `table`xcol`ycol`tolerance!(shkeyed;`tm;`px;1f),1,,a keyed table is simplified on its unkeyed form and returned unkeyed + +run,0,0,q,"shts:([]tm:2024.01.01D09:00:00+1000000000*til 10;px:shy;sym:10#`A)",1,,worked example on a timestamp x axis +true,0,0,q,(select from shts where i in 0 2 3 4 9)~analytics.shrink `table`xcol`ycol`tolerance!(shts;`tm;`px;1f),1,,a timestamp x axis is rebased before the distance arithmetic rather than losing resolution to a float cast +run,0,0,q,"shdt:([]tm:2024.01.01+til 10;px:shy)",1,,worked example on a date x axis +true,0,0,q,(select from shdt where i in 0 2 3 4 9)~analytics.shrink `table`xcol`ycol`tolerance!(shdt;`tm;`px;1f),1,,a date x axis is accepted + +comment,,,,,,,shrink - grouped series +run,0,0,q,"shby:([]sym:(10#`A),10#`B;tm:shx,shx;px:shy,reverse shy)",1,,two independent series stacked in one table +run,0,0,q,shbyres:analytics.shrink `table`xcol`ycol`tolerance`by!(shby;`tm;`px;1f;`sym),1,,simplify each sym on its own +true,0,0,q,shbyres~select from shby where i in 0 2 3 4 9 10 15 16 17 19,1,,each group is simplified independently and rows come back in source order +run,0,0,q,shby2:update ex:`N from shby,1,,add a second grouping column +true,0,0,q,shbyres~delete ex from analytics.shrink `table`xcol`ycol`tolerance`by!(shby2;`tm;`px;1f;`sym`ex),1,,multiple by columns are supported + +comment,,,,,,,shrink - input validation +fail,0,0,q,analytics.shrink 5,1,,a non dictionary argument is rejected +fail,0,0,q,analytics.shrink `table`xcol!(shtab;`tm),1,,missing required keys are rejected +fail,0,0,q,analytics.shrink `table`xcol`ycol`tolerance!(5;`tm;`px;1f),1,,a non table is rejected +fail,0,0,q,analytics.shrink `table`xcol`ycol`tolerance!(shtab;`tm`px;`px;1f),1,,xcol must name exactly one column +fail,0,0,q,analytics.shrink `table`xcol`ycol`tolerance`method!(shtab;`tm;`px;1f;`quick),1,,an unknown method is rejected +fail,0,0,q,analytics.shrink `table`xcol`ycol`tolerance!(shtab;`nope;`px;1f),1,,a missing column is rejected +fail,0,0,q,analytics.shrink `table`xcol`ycol`tolerance`by!(shtab;`tm;`px;1f;`nope),1,,a missing by column is rejected +fail,0,0,q,analytics.shrink `table`xcol`ycol`tolerance!(shtab;`sym;`px;1f),1,,a non numeric x axis is rejected +fail,0,0,q,analytics.shrink `table`xcol`ycol`tolerance!(shtab;`tm;`sym;1f),1,,a non numeric y axis is rejected +fail,0,0,q,analytics.shrink `table`xcol`ycol`tolerance!(shtab;`tm;`px;-1f),1,,a negative tolerance is rejected +fail,0,0,q,analytics.shrink `table`xcol`ycol`tolerance!(shtab;`tm;`px;0n),1,,a null tolerance is rejected +fail,0,0,q,analytics.shrink `table`xcol`ycol`tolerance!(shtab;`tm;`px;1 2f),1,,a non atomic tolerance is rejected +fail,0,0,q,analytics.shrink `table`xcol`ycol`tolerance!(update px:0n from shtab where i=3;`tm;`px;1f),1,,nulls in the measured columns are rejected +fail,0,0,q,analytics.shrink `table`xcol`ycol`tolerance!(`px xdesc shtab;`tm;`px;1f),1,,an x axis that is not in order is rejected +fail,0,0,q,analytics.rdprecur[-1f;shx;shy],1,,the kernels validate the tolerance too +fail,0,0,q,analytics.rdpiter[0n;shx;shy],1,,the kernels validate the tolerance too