Skip to content

feat: read a market's volume and traders from the node - #1444

Merged
MicBun merged 1 commit into
mainfrom
get-market-activity-action
Oct 8, 2026
Merged

MicBun merged 1 commit into
mainfrom
get-market-activity-action

Conversation

@MicBun

@MicBun MicBun commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Description

A caller can now ask the node how much traded on one order book over a time window. get_market_activity($query_id, $from_ts, $to_ts) returns the filled volume with its direct and mint/burn parts, the unique traders, the fill and share counts, the first and last fill inside the window, and how far back the node's order events reach. Every SDK will call this action instead of carrying its own copy of the SQL, so the definition of volume lives in one place.

  • 058-order-book-market-activity.sql creates the action. It is a new object, so it gets a new file, and it references no bridge, so there is no .prod.sql twin.
  • The definition is the Goal's, unchanged. A direct_buy_fill counts price * amount, and a mint or burn counts 100 * amount on its YES row. The sell row and the NO row add traders but no volume. Splits, placements, cancels, amends and settlement add nothing. The fill types are an allowlist, and the window includes both of its ends.
  • 044-order-book-events.sql gains an index on ob_order_events (query_id, block_timestamp), so a read costs what its window holds rather than a scan of the whole retained table.
  • The action refuses a NULL argument and a window that ends before it starts. A market that does not exist returns no row, as get_market_depth does.

One change from the Goal's statement

On a node that holds no order events, the statement returned NULL for coverage_from_block and coverage_complete. The action returns 0 and false, so every SDK can type both as non-null, and false still says a zero from that market cannot be trusted.

Worth knowing before the release

Coverage errs on the safe side. coverage_complete compares the market's creation height with the oldest event the node holds. For a market older than every held event it reads false even when nothing was trimmed, but it never reads true when fills were trimmed. The test covers both cases.

The index is built when the migration runs. CREATE INDEX IF NOT EXISTS locks ob_order_events against writes while it builds. The table held about 174k rows on mainnet on 6 Oct, and on 175k synthetic rows the build took 178 ms on my workstation.

The SDK calls need this on the network first. The sdk-go pull request below moves to this action once it ships, and the JS and Python calls will use it from the start:

Related Problem

resolves: #1443

Part of:

How Has This Been Tested?

go test -tags kwiltest ./tests/streams/order_book/ -run TestMarketActivity trades on a real order book, setting each call's block height and time. All five function tests pass.

  • Every column. One market gets a split and resting orders, then a direct fill of 30 shares at 55, a mint of 10 and a burn of 10. Over the whole window the action returns 3650 cents: 1650 direct and 2000 from the mint and burn, with 4 traders, 3 fills, 50 shares, and the times of the first and last fill. The test first checks that the split rows, the direct sell row and the NO rows of the mint and burn exist, so every exclusion is exercised.
  • Both ends of the window. A one-second window on each fill returns that fill alone. The seconds before the first fill and after the last return zeros with no fill times.
  • Coverage follows the trim. trim_order_events runs as the block leader. When the market's oldest events go, coverage_complete turns false and the volume holds. When the direct fill goes, the volume drops to 2000.
  • Nothing traded. A market on a node with no order events returns one row of zeros with coverage 0 and false. A market that does not exist returns no row.
  • Bad input. A NULL argument and an inverted window are refused.
  • Index use. On a throwaway Postgres 16.8 with 175k synthetic events across 1,000 markets, a one-week read of one market went from a parallel sequential scan (21 ms) to a bitmap index scan on the new index (0.3 ms). The coverage minimum already reads the block_height index.
  • Mutation check. I changed the SQL ten ways and a test caught each one: counting the sell row, the NO row of a mint or burn, or a split as volume; making either end of the window exclusive; leaving sellers out of the traders; taking the first fill time from any event; leaving empty coverage NULL; allowing an inverted window; and comparing coverage with the window instead of the market's creation.

kwil-cli utils parse accepts both migration files, and go vet -tags kwiltest ./tests/streams/order_book/ is clean.

Summary by CodeRabbit

  • New Features
    • Added market activity reporting for a selected time window, including trading volume, direct and mint/burn activity, trader and fill counts, shares traded, and first and last event times.
    • Reports indicate whether event history is complete and the block where retained history begins. Existing markets with no matching trades return zero-valued totals; unknown markets return no result.
    • Reversed time windows and missing required inputs are rejected.

@MicBun MicBun self-assigned this Oct 8, 2026
@holdex

holdex Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Time Submission Status

Member # Time Running Total Status Last Update
MicBun 4h ✅ Submitted Oct 8, 2026, 2:49 PM

Submit or update total time with:

@holdex pr submit-time 2h

Add time on top of previous submission with:

@holdex pr add-time 1h30m

See available commands to help comply with our Guidelines.

@coderabbitai

coderabbitai Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 769c6663-14bc-4046-9920-a217a63564c5
📥 Commits

Reviewing files that changed from the base of the PR and between b64fe03 and ca7bd3b.

📒 Files selected for processing (3)
  • internal/migrations/044-order-book-events.sql
  • internal/migrations/058-order-book-market-activity.sql
  • tests/streams/order_book/market_activity_test.go

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.


📝 Walkthrough

Walkthrough

Adds an index for order-event time-window queries and a public get_market_activity action. The action returns market activity aggregates and retained-event coverage. Integration tests exercise its results, time filtering, coverage, and input handling.

Changes

Market activity query

Layer / File(s) Summary
Indexed activity query
internal/migrations/044-order-book-events.sql, internal/migrations/058-order-book-market-activity.sql
Adds an index on (query_id, block_timestamp) and the public get_market_activity action. The action validates inputs, aggregates matching events, and reports retained-event coverage.
Activity action integration tests
tests/streams/order_book/market_activity_test.go
Tests activity totals, time-window filtering, coverage before and after event trimming, empty and unknown markets, and invalid inputs.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Feature · Severity of issue fixed: Medium

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant get_market_activity
  participant ob_order_events
  Client->>get_market_activity: query_id and time window
  get_market_activity->>get_market_activity: validate required inputs and time window
  get_market_activity->>ob_order_events: aggregate matching market events
  ob_order_events-->>get_market_activity: event aggregates and retained-event coverage
  get_market_activity-->>Client: activity row or no row for an unknown market
Loading

Merge Risk: ⚪ Minimal · up to ca7bd

This adds a read-only market activity action and a supporting index. No concrete merge-blocking risk was found.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the main change: adding a node action to read a market's volume and trader activity. It is concise and specific enough for pull request history.
Linked Issues check ✅ Passed The PR satisfies the coding requirements in directly linked issue #1443, with the Goal context from #1429. Migration 058 adds the public get_market_activity($query_id, $from_ts, $to_ts) action and r…
Out of Scope Changes check ✅ Passed The changed files stay within issue #1443. The index supports the new action, and the integration test verifies its required behavior. No unrelated product behavior or unrelated files appear in the re…
Docstring Coverage ✅ Passed Docstring coverage is 91.67% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 12 functions across 1 files. (2 skipped: 2 …
✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Commit to this branch
  • Create a new PR
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@MicBun

MicBun commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

@holdex pr submit-time 4h

@MicBun
MicBun merged commit e848d39 into main Oct 8, 2026
8 checks passed
@MicBun
MicBun deleted the get-market-activity-action branch October 8, 2026 18:04
MicBun added a commit to trufnetwork/sdk-py that referenced this pull request Oct 15, 2026
## Description

A Python user can now ask how much traded on one order book over a time
window. `client.get_market_activity(query_id, from_ts, to_ts)` returns a
`MarketActivity` dict with the filled volume and its direct and mint and
burn parts, the unique traders, the fill and share counts, the first and
last fill inside the window, and how far back the node's order events
reach.

The call goes through sdk-go's `OrderBook.GetMarketActivity`, which
reads the node action `get_market_activity`. The node action holds the
definition of volume, so Python reads the same figures as Go and
JavaScript, and a change to the definition needs no SDK release. The
action and the other two SDKs' methods come from:

- trufnetwork/node#1444
- trufnetwork/sdk-go#206
- trufnetwork/sdk-js#181

What this adds:

- `go.mod` pins sdk-go v0.7.8, the release that carries
`GetMarketActivity`. Only the sdk-go lines change, and kwil-db stays
where it was.
- `GetMarketActivity` in `bindings/bindings.go` calls sdk-go and returns
one JSON object whose keys are the action's column names. The three
cents fields stay strings because `NUMERIC(78,0)` exceeds what a JSON
number holds exactly, and the fill times are `null` when nothing traded.
sdk-go checks the arguments and reports a missing market, and the
binding passes both errors on, so Python raises `RuntimeError` with
sdk-go's message in it.
- `TNClient.get_market_activity` and the `MarketActivity` TypedDict in
`client.py`, next to `get_full_market_depth`.
- A Market Activity section in `docs/api-reference.md` with the key
table, an example, and the notes on bridges and coverage.

The linked issue describes ad hoc SQL running in sdk-go. sdk-go calls
the node action instead, as its pull request above explains. For Python
nothing changes, since the binding, the dict and its keys are the ones
the issue specifies.

The issue also asks for two notes in the docs, both written for the ad
hoc SQL. Neither applies to the action, so I left both out, as the Go
and JavaScript docs do:

- The note "Requires a node that allows ad hoc queries (not Kwil private
mode)". A node in private mode refuses ad hoc SQL but still serves
action calls that carry the caller's signature. In private mode the
client adds that signature on its own, with the key every `TNClient` is
created with.
- The note on the sequential scan. The node now indexes order events on
`(query_id, block_timestamp)`, the fix the issue points to.

### Where it works

Mainnet has carried `get_market_activity` since 8 October, and testnet
since 13 October.

## Related Problem

- resolves: #139

Part of:

- trufnetwork/node#1429

## How Has This Been Tested?

- `tests/test_market_activity.py` monkeypatches the binding, as
`tests/test_full_market_depth.py` does, so it needs no node. It checks
that the three arguments reach the binding in order, that the dict
decodes to every field, that cents above 2**63 stay exact strings, that
`null` fill times and a false coverage survive, that an error from the
binding propagates, and that an empty answer raises instead of reading
as no activity.
- `bindings/market_activity_test.go` checks the JSON the binding builds:
the action's column names, cents above `math.MaxInt64` as strings, and
`null` fill times with block 0 and false coverage when nothing traded.
- The existence checks in `tests/test_order_book.py` include the new
method and type.

I broke the change in 15 ways, one at a time: 6 in the Python wrapper
(argument order, the market id, an empty answer read as no activity, a
swallowed error, cents turned into numbers, fill times defaulted to 0),
8 in the JSON the binding builds (a key name, cents as a JSON number,
each fill time, the coverage fields, a count read from the wrong field),
and the window arguments swapped inside the binding. A test failed every
time. Only a live run can catch the last one, and the live suite did.

`tests/test_market_activity_live.py` runs when `TN_LIVE_NODE_URL` is
set, like the other live suites, so CI skips it. It finds the newest
market with a fill in the last 30 days and checks that `volume_cents`
equals `direct_cents + mint_burn_cents`, that `direct_fill_count <=
fill_count`, that the fills fall inside the window, that both ends of
the window are inclusive, and that a market that does not exist raises.

- Mainnet: all 3 tests passed in 35 seconds, through Python, the
binding, sdk-go and the node. The suite picked market 1957, with 1000
cents of volume from one mint or burn fill of 10 shares, and complete
coverage. The JavaScript suite picked the same market and read the same
figures.
- Testnet: none of the newest 100 markets had a fill in the last 30
days, so the two checks that need fills skipped, and the missing market
test passed.

Called directly, mainnet market 1835 over [1788891555, 1791483555]
returns 1145 cents (545 direct, 600 mint and burn), 2 traders, 6 fills
(4 direct) and 20 shares with complete coverage, the figures the Go and
JavaScript methods read. An inverted window and `query_id` 0 raise
`RuntimeError` with sdk-go's messages.

`make gopy_build`, `go build`, `go vet` and `gofmt` are clean. The 157
tests in the 11 test files that need no node pass. pyright reports the
same 2 errors on `client.py` as on main, and none new. The tests that
need a Docker node run in CI.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added market activity reporting for an inclusive time range, including
trading volume, trader and fill counts, traded shares, and first and
last fill times.
* Results include per-bridge volume details and coverage information to
help interpret activity, including when no trades are reported. Missing
markets and invalid time ranges return errors.
* **Documentation**
* Added API reference details and an example for retrieving market
activity, including volume units and guidance for interpreting coverage.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Problem: SDKs can't read market activity from the node

1 participant