Repository navigation
feat: read a market's volume and traders from the node - #1444
Conversation
Time Submission Status
Submit or update total time with: Add time on top of previous submission with: See available commands to help comply with our Guidelines. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configuration
📒 Files selected for processing (3)
Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review. 📝 WalkthroughWalkthroughAdds an index for order-event time-window queries and a public ChangesMarket activity query
Priority: ➖ Normal Estimated code review effort: 3 (Moderate) | ~25 minutes Change: Feature · Severity of issue fixed: Medium Sequence Diagram(s)sequenceDiagram
participant Client
participant get_market_activity
participant ob_order_events
Client->>get_market_activity: query_id and time window
get_market_activity->>get_market_activity: validate required inputs and time window
get_market_activity->>ob_order_events: aggregate matching market events
ob_order_events-->>get_market_activity: event aggregates and retained-event coverage
get_market_activity-->>Client: activity row or no row for an unknown market
Merge Risk: ⚪ Minimal · up to This adds a read-only market activity action and a supporting index. No concrete merge-blocking risk was found. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@holdex pr submit-time 4h |
## Description A Python user can now ask how much traded on one order book over a time window. `client.get_market_activity(query_id, from_ts, to_ts)` returns a `MarketActivity` dict with the filled volume and its direct and mint and burn parts, the unique traders, the fill and share counts, the first and last fill inside the window, and how far back the node's order events reach. The call goes through sdk-go's `OrderBook.GetMarketActivity`, which reads the node action `get_market_activity`. The node action holds the definition of volume, so Python reads the same figures as Go and JavaScript, and a change to the definition needs no SDK release. The action and the other two SDKs' methods come from: - trufnetwork/node#1444 - trufnetwork/sdk-go#206 - trufnetwork/sdk-js#181 What this adds: - `go.mod` pins sdk-go v0.7.8, the release that carries `GetMarketActivity`. Only the sdk-go lines change, and kwil-db stays where it was. - `GetMarketActivity` in `bindings/bindings.go` calls sdk-go and returns one JSON object whose keys are the action's column names. The three cents fields stay strings because `NUMERIC(78,0)` exceeds what a JSON number holds exactly, and the fill times are `null` when nothing traded. sdk-go checks the arguments and reports a missing market, and the binding passes both errors on, so Python raises `RuntimeError` with sdk-go's message in it. - `TNClient.get_market_activity` and the `MarketActivity` TypedDict in `client.py`, next to `get_full_market_depth`. - A Market Activity section in `docs/api-reference.md` with the key table, an example, and the notes on bridges and coverage. The linked issue describes ad hoc SQL running in sdk-go. sdk-go calls the node action instead, as its pull request above explains. For Python nothing changes, since the binding, the dict and its keys are the ones the issue specifies. The issue also asks for two notes in the docs, both written for the ad hoc SQL. Neither applies to the action, so I left both out, as the Go and JavaScript docs do: - The note "Requires a node that allows ad hoc queries (not Kwil private mode)". A node in private mode refuses ad hoc SQL but still serves action calls that carry the caller's signature. In private mode the client adds that signature on its own, with the key every `TNClient` is created with. - The note on the sequential scan. The node now indexes order events on `(query_id, block_timestamp)`, the fix the issue points to. ### Where it works Mainnet has carried `get_market_activity` since 8 October, and testnet since 13 October. ## Related Problem - resolves: #139 Part of: - trufnetwork/node#1429 ## How Has This Been Tested? - `tests/test_market_activity.py` monkeypatches the binding, as `tests/test_full_market_depth.py` does, so it needs no node. It checks that the three arguments reach the binding in order, that the dict decodes to every field, that cents above 2**63 stay exact strings, that `null` fill times and a false coverage survive, that an error from the binding propagates, and that an empty answer raises instead of reading as no activity. - `bindings/market_activity_test.go` checks the JSON the binding builds: the action's column names, cents above `math.MaxInt64` as strings, and `null` fill times with block 0 and false coverage when nothing traded. - The existence checks in `tests/test_order_book.py` include the new method and type. I broke the change in 15 ways, one at a time: 6 in the Python wrapper (argument order, the market id, an empty answer read as no activity, a swallowed error, cents turned into numbers, fill times defaulted to 0), 8 in the JSON the binding builds (a key name, cents as a JSON number, each fill time, the coverage fields, a count read from the wrong field), and the window arguments swapped inside the binding. A test failed every time. Only a live run can catch the last one, and the live suite did. `tests/test_market_activity_live.py` runs when `TN_LIVE_NODE_URL` is set, like the other live suites, so CI skips it. It finds the newest market with a fill in the last 30 days and checks that `volume_cents` equals `direct_cents + mint_burn_cents`, that `direct_fill_count <= fill_count`, that the fills fall inside the window, that both ends of the window are inclusive, and that a market that does not exist raises. - Mainnet: all 3 tests passed in 35 seconds, through Python, the binding, sdk-go and the node. The suite picked market 1957, with 1000 cents of volume from one mint or burn fill of 10 shares, and complete coverage. The JavaScript suite picked the same market and read the same figures. - Testnet: none of the newest 100 markets had a fill in the last 30 days, so the two checks that need fills skipped, and the missing market test passed. Called directly, mainnet market 1835 over [1788891555, 1791483555] returns 1145 cents (545 direct, 600 mint and burn), 2 traders, 6 fills (4 direct) and 20 shares with complete coverage, the figures the Go and JavaScript methods read. An inverted window and `query_id` 0 raise `RuntimeError` with sdk-go's messages. `make gopy_build`, `go build`, `go vet` and `gofmt` are clean. The 157 tests in the 11 test files that need no node pass. pyright reports the same 2 errors on `client.py` as on main, and none new. The tests that need a Docker node run in CI. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added market activity reporting for an inclusive time range, including trading volume, trader and fill counts, traded shares, and first and last fill times. * Results include per-bridge volume details and coverage information to help interpret activity, including when no trades are reported. Missing markets and invalid time ranges return errors. * **Documentation** * Added API reference details and an example for retrieving market activity, including volume units and guidance for interpreting coverage. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Description
A caller can now ask the node how much traded on one order book over a time window.
get_market_activity($query_id, $from_ts, $to_ts)returns the filled volume with its direct and mint/burn parts, the unique traders, the fill and share counts, the first and last fill inside the window, and how far back the node's order events reach. Every SDK will call this action instead of carrying its own copy of the SQL, so the definition of volume lives in one place.058-order-book-market-activity.sqlcreates the action. It is a new object, so it gets a new file, and it references no bridge, so there is no.prod.sqltwin.direct_buy_fillcountsprice * amount, and a mint or burn counts100 * amounton its YES row. The sell row and the NO row add traders but no volume. Splits, placements, cancels, amends and settlement add nothing. The fill types are an allowlist, and the window includes both of its ends.044-order-book-events.sqlgains an index onob_order_events (query_id, block_timestamp), so a read costs what its window holds rather than a scan of the whole retained table.get_market_depthdoes.One change from the Goal's statement
On a node that holds no order events, the statement returned NULL for
coverage_from_blockandcoverage_complete. The action returns 0 and false, so every SDK can type both as non-null, and false still says a zero from that market cannot be trusted.Worth knowing before the release
Coverage errs on the safe side.
coverage_completecompares the market's creation height with the oldest event the node holds. For a market older than every held event it reads false even when nothing was trimmed, but it never reads true when fills were trimmed. The test covers both cases.The index is built when the migration runs.
CREATE INDEX IF NOT EXISTSlocksob_order_eventsagainst writes while it builds. The table held about 174k rows on mainnet on 6 Oct, and on 175k synthetic rows the build took 178 ms on my workstation.The SDK calls need this on the network first. The sdk-go pull request below moves to this action once it ships, and the JS and Python calls will use it from the start:
Related Problem
resolves: #1443
Part of:
How Has This Been Tested?
go test -tags kwiltest ./tests/streams/order_book/ -run TestMarketActivitytrades on a real order book, setting each call's block height and time. All five function tests pass.trim_order_eventsruns as the block leader. When the market's oldest events go,coverage_completeturns false and the volume holds. When the direct fill goes, the volume drops to 2000.block_heightindex.kwil-cli utils parseaccepts both migration files, andgo vet -tags kwiltest ./tests/streams/order_book/is clean.Summary by CodeRabbit