Repository navigation
Conversation
…eviation when read (#767) * perf(statistic)!: Compute a dataset summary's variance and standard deviation when read Every closed position updated its instrument's tear sheet, whose PnL-return summaries recomputed a `Decimal` square root for the standard deviation on each update, twice for a losing exit. Only `TearSheetGenerator::generate` reads it. That was about 30% of a simulated step that fills an order. `Dispersion` now holds the range and the Welford recurrence relation M only. The variance and standard deviation are derived on read by `DataSetSummary::variance`/`std_dev`, or `Dispersion::variance`/`std_dev` given the count. `Dispersion::update` no longer takes the count. A summary serialised with the removed fields still loads. Adds `benches/sim_stepper`, which drives one simulated venue bar by bar: a bar only, a bar plus one filled market order, and building a session. Round trip per step: 4.14 us -> 2.93 us. * refactor(statistic): Keep Dispersion's count-taking accessors private, and pin the derived values - `Dispersion::variance` and `Dispersion::std_dev` are `pub(super)`: the count they take must match the one the dispersion was updated with, and `DataSetSummary` holds it. Read them through `DataSetSummary`. - Test that a variance rounded below zero gives the root of its magnitude, and that a loaded summary ignores stored values. - The bench checks that every round trip closes a position into the tear sheet. * docs(statistic): Say the dispersion module holds the range and M, from which variance and std dev are derived
* test(engine): Pin HistoricalClock's range, monotonicity and sharing, and bench it Tests that hold for the current implementation and any replacement: every instant round-trips exactly (MIN_UTC, MAX_UTC, a leap second, a sub-microsecond instant), advance_to and process never move the clock backwards, an event at the current instant is accepted, clones share one instant, and clones advanced from several threads end at the latest instant while a concurrent reader never sees time move backwards or a mixed read. The clock bench times a read, an advance plus three reads, a process at the current instant, and the advance plus reads on several threads at once, each on its own clock. * perf(engine): Read HistoricalClock's instant without a lock The instant is held in a sequence lock over its seconds (AtomicI64) and nanoseconds (AtomicU32), versioned by an AtomicU64, instead of a parking_lot RwLock. A read loads the version, both fields, then the version again after an acquire fence, and retries if a write overlapped it. A write takes an odd version by compare-and-swap, compares and stores, then publishes the next even version, so writers on clones serialise and the clock stays monotonic whatever order they arrive in. An advance to an instant no later than the one held stores nothing. Every DateTime<Utc> is still held exactly, MIN_UTC to MAX_UTC with leap seconds. No unsafe and no new runtime dependency. The shared state is aligned to 128 bytes so clocks on different threads do not share a cache line. process now logs after storing, not while holding the clock, and Debug shows time_exchange_last directly. A loom model test (new dev-dependency) compiles the seqlock's source against loom's atomics and checks a read overlapping an advance, and two racing advances with a reader: no mixed read, no regression, the later instant wins. * docs(changelog): Note HistoricalClock's lock-free reads, and bench its clock on 16 threads The clock bench's per-thread cases now include 16 threads, where clocks sharing a cache line slow each other down. * build: Limit the lockfile change to the loom dev-dependency Adding loom had also re-pointed existing crates to newer versions already in the lockfile (socket2 0.5 to 0.6 for hyper-util, quinn and quinn-udp; older windows-sys to 0.61). The lockfile now differs from develop's by loom, generator and scoped-tls, and rustrade's new edge, alone. * perf(engine): Compare HistoricalClock instants as seconds and nanoseconds An advance compared DateTimes, so it rebuilt the held instant from its fields on the lock-free check and again while holding the version. It now compares the stored seconds and nanoseconds, which order exactly as the DateTime does, leap seconds included, and process rebuilds the held instant only to log it. Nothing that can panic runs while the version is odd any more. A test pins the order against DateTime's across a leap second and at both ends of the range. The rustdoc now says that a read finding a write in progress spins until it ends, and that the seqlock takes every atomic from its parent's sync module. * docs: Qualify the clock's benchmark figures The CHANGELOG rounds the stepper gains, and says the multi-thread gain is mostly the alignment: the previous clocks could share a cache line, so their cost depended on where they were allocated. The clock bench's doc says advance_and_read includes computing the next instant, and that the per-thread figure also rises with scheduling noise and shared cores. * test(engine): Pin HistoricalClock's Debug output * test(engine): Pin the level HistoricalClock logs an event at An event at the clock's instant or later is logged at DEBUG, and an older one by how much older it is: DEBUG within a second, WARN within 30 seconds, ERROR beyond. The rustdoc now says an advance that stores nothing waits only as a read does.
#776) * fix(execution): Place missed-fill recovery's read on the venue's clock (#770) When the ExecutionManager reopened an account stream, the read of the fills it missed started from the host time the last event arrived (less up to max_clock_skew) and ended at the host's present. A host clock ahead of the venue's by more than the allowance, or one that stepped back before the reconnect, lost fills with no FillRecoveryGaveUp. - Add VenueClock (rustrade-execution): the earliest a venue's clock can read, from the times it stamped on events, counted on from the arrival on tokio's monotonic clock with a 100 ppm drift allowance. A step of the host's wall clock does not move it. - The watermark samples the venue's clock from delivered fills. The read starts at the venue's estimated time at the last event (or the local time if earlier) less overlap, and ends max_clock_skew past the later of the local and venue times. With no fill delivered it starts max_clock_skew before the local time. - Warn when the measured clock difference exceeds max_clock_skew. - Binance margin fetch_trades no longer cuts the span's end to the host clock, only to one window past it. The fetch_trades rustdoc states that end can lie past the present. * test(execution): Pin the venue clock's sample replacement and the replay cut-off at their boundaries * fix(execution): End a recovery read past the host time counted on from the last event - End the read past the later of the local time, the venue's estimated time, and the local time of the last event counted on on tokio's clock, so a host clock stepped back before the reopening does not end it early on a stream that delivered no fill. - Saturate the read's bounds instead of panicking on an allowance beyond the time range. - Read the clocks before taking the watermark's lock on each event. - Say that a replayed fill younger than max_span widens the reads until a fill delivered on time replaces it, and word the skew warning to allow for one. - Correct VenueClock's rustdoc: the monotonic clock stops while the host is suspended and can be slewed by NTP, and why bounding a read's start by the wall clock too stays covered. - Test the manager's skew warning through a captured log. * test(execution): Give the reopened read's end a margin for real time, and say suspend stops the monotonic clock only on some platforms
… describe (#779) * fix(execution): Stamp REST balance reads no later than the state they describe Every client stamped the balances from `fetch_balances` and `account_snapshot` with this host's time taken after the response arrived. The engine orders balance and debt state by that time, so a snapshot stamped later than its state shut out the stream's balance and debt events from between the two. That happened when the venue's REST view trailed its stream (a Binance cross-margin read straight after a fill has returned the pre-trade balances), and when this host's clock ran ahead of the venue's. Each balance is now stamped with the host time taken before the request that read it was sent: the retry, for a request retried after a rate limit. Binance spot and margin, cross and isolated, stamp 5 s earlier still, to cover the REST view trailing the stream. Alpaca, IBKR and Hyperliquid stream no balances and subtract nothing. The rule is stated on `ExecutionClient::fetch_balances` and `AssetBalance::time_exchange`, with a host clock ahead of the venue's left as the caller's to bound. Closes #777 * fix(execution): Time a Binance REST attempt with one clock reading, and pin the documented lag The weight tracker's send time and a balance read's time now come from the same reading, so the stamp is never a moment after the request's recorded send. The 5 s REST balance lag allowance is pinned by a test, as the clients' rustdoc state it as a number. The CHANGELOG now says only Binance stamps the retry of a rate-limited request. * docs(changelog): Rewrap the REST balance stamp entry * test(execution): Check the IBKR live balance read returns balances stamped within the call The paper-account test only checked that the read succeeded, so a read returning no balances, or one stamped after the call, passed.
…reconnects by itself (#780) * fix(execution): Restate a Binance account stream's balances after it reconnects by itself The spot and margin account streams recovered the fills and order endings they missed while disconnected, but no balances. A balance update or margin debt change sent in the gap was lost, and the engine's balances and debt stayed wrong until the next REST snapshot, which a long-running process may never take. After each reconnect, once the gap's fills have been read, the stream now reads the account's balances over REST and delivers each of the assets it was opened for as a `BalanceSnapshot`, debt included on margin. An isolated margin stream delivers each pair's base and quote as the new `AccountEventKind::InstrumentBalanceSnapshot`. Each balance is stamped with the earlier of this host's time and the venue's time at the read's start, less the 5 s REST lag allowance. The venue's time comes from a `VenueClock` fed by the stream's own events. The read waits until the venue's clock is that allowance past the latest balance or debt event the stream delivered, and is never stamped earlier than that event, which the engine would otherwise keep. A failed read is logged and retried while connected, backing off from 5 s to 2 minutes, and is never given up. Closes #778 * refactor(execution): Run a Binance balance restatement through one shared driver Spot and margin each carried the same restatement driver: the due check, the wait for the read to be allowed to start, the 30 s bound, the send and the backoff. They now share one, and each account supplies only its read. A read that does not finish in time is now logged as unfinished, rate-limit waits included, rather than as unanswered. The stream locks its venue clock only for an event the clock learns from. Docs: the restatement gets its own section of the Binance module docs, out of the reconnect-budget list. It now states that a gap whose fill read failed does not hold the restatement back, that the read can delay a fill or order check retry by up to its 30 s, and the order of an isolated stream's pairs. Tests: the shared driver's deferral, failure, timeout, drop part-way and dropped-consumer paths; each event kind the venue clock samples; indexing an `InstrumentBalanceSnapshot` and its engine time. The test WS server's pushing connection now pushes every 100 ms rather than once after a fixed delay, and the spot floor test allows for the event reaching the test after the stream. * refactor(execution): Keep a stream's latest balance time as a plain maximum * test(execution): Allow a second for a balance event to reach the spot floor test
…781) * test: Capture a test's logs reliably while other tests run beside it tracing caches whether a call site is of interest the first time any thread reaches it. While only one subscriber is registered, it asks just the subscriber of the thread that got there first. A parallel test whose thread has no subscriber could therefore cache a call site as of no interest, and a test capturing it with a thread-local subscriber missed the event. the_manager_warns_of_a_clock_difference_beyond_the_allowance_only failed about two runs in three this way when run with the other clock tests, and passed alone. rustrade-instrument's test-utils feature gains test_utils::logs: capture, Capture::start for async tests, and with_layer for a test that acts when an event is logged. Before capturing, each installs once per process a global default subscriber that is interested "sometimes" in every call site and enables none. Interests from different subscribers then combine to "sometimes", so no call site is cached as of no interest. All eight log-capturing tests move onto it, which also replaces their copies of a recording layer. A new integration test, alone in its binary so no other capture hides the race, reaches a call site first from another thread and checks the capture still sees it; it fails without the global subscriber. * test: Check a thread outside a capture enables nothing, and derive Capture's Debug * docs: Document Capture::start and Logged::level, and say the log capture tests need test-utils
* fix(engine): Measure in-flight deadlines on a monotonic clock The in-flight watch measured each order's deadline from its wall-clock `time_sent`. Under `LiveClock` a step of the host's clock back delayed every overdue flag by the step, and lost it for orders sent after a step larger than the deadline; a jump forward flagged every order at once. - `EngineClock::instant` gives a monotonic `EngineInstant`. `LiveClock` reads `std::time::Instant`, `TokioClock` tokio's clock; `HistoricalClock` and the default read `time()`, so backtests and existing clocks are unchanged. - The watch logs each send's instant under the `time_sent` it stamped and finds each order's by its own stamp, so a resent cancel keeps the first one's deadline. An order it was not told of is placed by its wall age when first checked. - `LiveClock`'s rustdoc states that its time can step back or jump forward. Closes #771 * perf(engine): Read the clocks the in-flight watch needs only when it needs them - A scan reads the engine's instant to decide whether anything is due, and its wall time only to place an order the send log lacks, at most once per scan. - A send reads the instant only when some exchange has a deadline. - The watch's rustdoc states that a stale log entry can share its instant with a later send stamped the same, and names coarse custom clocks. - `LiveClock`'s rustdoc drops advice on configuring the host's clock. - CHANGELOG lists `EngineClock::instant` and `EngineInstant` under Added.
…t it reached (#783) * fix(execution): Hold the simulated venue's clock at the latest instant it reached `SimulatedVenue::advance_time` took whatever instant it was given. An earlier one moved the venue's clock back: it restamped every balance and stamped later requests' fills before what the venue had already reported, and the engine drops a balance older than the one it holds. A request-priced `MockExchange` stamping from a wall clock that stepped back, or a request arriving behind a later `apply_market` tick, did this. `advance_time` now holds an earlier instant at the latest one either input has reached (`accrued_through`, which both inputs raise and nothing lowers). It charges nothing again, and what follows is stamped at that instant, whichever driver calls it. The clock reads the epoch before it first moves, so clamping against it would have held a run that starts before 1970; the clamp applies only once the clock has moved. The forward-jump policy is now documented: a move, however large, charges every funding and interest boundary it crosses once each, at its own stamp, on what was held across it. Closes #772 * docs(execution): State when the simulated venue's request clock catches up with a market tick The request clock is set only by advance_time, so an apply_market tick can raise the venue's high-water mark past it until the next advance. Say so on the field and on time_exchange(), and describe the apply_market path in the CHANGELOG as one a consumer driving the venue directly can reach.
…BKR times in a known zone (#784) * fix(execution): Mark fills stamped with this host's time, and read IBKR times in a known zone A fill that arrived without a readable venue time was stamped with this host's clock, unmarked, and IBKR read an execution time whose zone was missing or unknown as UTC. The fill-recovery watermark and the Binance balance restatement estimate the venue's clock from delivered fills, so such a fill skewed the estimate by however far this host's clock, or the guessed zone, was off. - Add `Trade::time_source` (`TimeSource::{Venue, Host}`, serde default `Venue`) and `Trade::with_time_source`. Both venue-clock samplers skip `Host` fills. - Fall back to another venue-stamped time first: Binance `executionReport` `T` then `E`; Alpaca `trade_updates` `timestamp` then the order's `updated_at`. Only then this host's, marked `Host`. - Drop a Binance `outboundAccountPosition` with neither `u` nor `E`, rather than stamp it with this host's time, which could hold back genuine balance updates. - `parse_ib_timestamp` takes the gateway's session zone: a time naming no zone is read in it, an unknown zone is not read, a repeated daylight-saving hour is its earlier instant, and `YYYYMMDD-HH:MM:SS` is UTC. Add `IbkrClient::session_zone`. - Document the host-time stamps that remain: REST order listings and request answers as a last resort, IBKR order statuses and corrections, which carry no time. - Add a read-only IBKR probe of the raw time formats. Closes #775. * fix(execution): Mark Binance REST fills without a time, and keep unreadable IBKR executions - Binance Spot and Margin `myTrades` fills without a readable `time` were stamped with this host's clock but marked as the venue's. They are now marked `TimeSource::Host`. - `ExecutionBuffer::clear_stale` cleared every execution whose time did not read, however fresh, before its commission report could arrive. It now ages such an execution by when it was buffered. - An IB time zone that is not a tz database name is cached as unknown and warned of once per thread, not on every time. Each execution's time is parsed once, and passed to `resolve_execution`. - A good-till-date in a zone this crate cannot read is read as UTC again, with a `warn!`, so the order can still be reported from a listing. - `parse_ib_timestamp`'s rustdoc says when `session_zone` is ignored, and that the `-` form takes no zone. Test that a time-selected read skips an execution whose time does not read. * fix(execution): Say a good-till-date could not be read in its zone, not that the zone is unknown The fallback also reads a time in a skipped daylight-saving hour of a known zone as UTC, so the warning and the rustdoc now say the date could not be read in its zone. * test(execution): Clear an IBKR execution buffered for exactly the age limit Pins the boundary of ExecutionBuffer::clear_stale: an execution aged exactly max_age is cleared, not kept. The rustdoc of the test now says "for max_age or longer".
…ue's clock (#790) * fix(execution): Place the clients' in-stream fill recovery on the venue's clock Alpaca, Binance Spot and Margin, and IBKR recover the fills their account stream missed after it reconnects by itself, and bounded that span by this host's wall clock. A host clock ahead of the venue's by more than the client's lookback started the read after fills the venue stamped during the outage, and one stepped back before the reconnect ended a Binance gap early, losing those fills with no FillRecoveryGaveUp. Each stream now keeps a VenueClock, sampled from the times the venue stamped on the events it delivered, an order's as well as a fill's: every Binance event's E, every Alpaca trade_updates event's timestamp or updated_at, and IBKR's executions. The span starts at the earlier of the venue's and the host's time at the disconnect, less the client's lookback, and ends past the latest of the host's time, the venue's estimate, and the disconnect counted on on tokio's monotonic clock. FillRecoveryGaveUp reports the span on the same clock. Before an Alpaca stream has seen an event, its read starts 30 s further back, since nothing else bounds the difference between the clocks there. VenueClock gains span_start and span_end, over a new HostMoment, and the ExecutionManager's watermark uses them too. Closes #774 * test(execution): Pin the frame-time sample, spot's disconnect lookback and IBKR's notice routing The local mutation run found the stream clock's frame-time sample, its venue accessor, spot's heartbeat-timeout lookback and IBKR's notice routing untested. Spot's lookback moves into a helper so it can be tested, as margin's already is. * fix(execution): Test the reconnect wiring end to end, and read each frame's venue time once - An Alpaca and a Binance margin test drive a reconnect with this host's clock two minutes ahead of the venue's, and check the recovery read starts at the venue's time. They fail if either stream anchors on host time. - A Binance gap's end saturates rather than panicking on a far-future venue stamp, and a malformed `E` costs only the clock sample, not the event. - `HostMoment` is `#[non_exhaustive]`; `start` and `end` are named arguments again rather than a same-typed tuple. - IBKR's give-up span no longer falls back to this host's clock. - A Binance frame takes the stream clock's lock once, for its `E` and its events together. Alpaca reads a trade update's time once, for the clock sample and the events. IBKR's watcher reads the shared clock only when a recovery is due. * docs(execution): Say an Alpaca frame without a venue time warns even when it is dropped
…sets, read market notional limits (#791) * feat(execution)!: Round onto an order rule's grid, and check trailing offsets and read market notional limits Rounding (#786): - `ValueRule::round_down` / `round_up`: the nearest multiple of the rule's step at or below / at or above a value, counted from zero as the pre-send check counts it. No clamping to the range. `None` only when a `Decimal` cannot hold the result. - `OrderRules::round_quantity_down` / `round_quantity_up` take the order's kind: a market order's quantity rounds to the grid that `quantity` and `market_quantity`'s steps share (their least common multiple). Trailing offsets and market notional (#787): - `OrderRules::trailing_offset_above` / `trailing_offset_below`, in basis points, read from Binance's `TRAILING_DELTA`. The pre-send check refuses a trailing order's offset outside its side's range (buy: above, sell: below) with `OutOfBounds` on `TrailingOffset`, in the offset's own unit; an absolute offset is not checked. - `OrderRules::market_notional`: the notional limits that also hold for a market order, from `NOTIONAL`'s `applyMinToMarket` / `applyMaxToMarket` or `MIN_NOTIONAL`'s `applyToMarket`. Not applied by the check, which has no price for a market order. Breaking: `OrderRules::check` takes the order's side, and `OrderRules` gains public fields. `OrderRules::new` keeps its four arguments; `with_market_notional` and `with_trailing_offsets` set the new parts. Rules serialised without the new fields still load. * fix(execution): Compare a percent trailing offset without a silent fallback, and tighten the rounding docs - The trailing offset check divides the basis-point limits by 100 directly for a percent offset; the helper whose failed division would have turned a limit off is gone. - Document that a rounded value keeps the finer scale of the value and the step. - The market notional rustdoc says to compare price times quantity with min and max, as no public check takes it. - #[must_use] on the rounding methods and the OrderRules builders. - CHANGELOG: OrderRules is non-exhaustive, so its new fields break no downstream literal. * docs(execution): Say that a value already on the grid keeps its own scale when rounded
…xt hour's margin borrow rates (#792) * feat(execution): Read a Binance account's commission rates and the next hour's margin borrow rates `BinanceSpot::fetch_commission_rates` and `BinanceMargin::fetch_commission_rates` read `GET /api/v3/account/commission` into `CommissionRates`. The standard, special and tax parts stay separate, because Binance charges them differently when the fee is paid in BNB: only the standard part is discounted. Each part keeps its maker, taker, buyer and seller rates. The discount block is `CommissionDiscount`, whose `multiplier` is Binance's `discount`, the share of the standard part that is charged. `CommissionRates::rate(side, liquidity)` adds up the three parts' rates for a trade not paid in BNB. A missing or unreadable rate is an error, never zero, and an absent discount block or discount asset is `None`. Margin has no commission endpoint, so the margin client keeps its own spot REST client and spot-pool tracker, which its rules loader now shares. `BinanceMargin::fetch_next_hourly_interest_rates(assets)` reads the next hourly rate on the client's cross or isolated margin. It removes duplicate assets and reads them 20 per request, Binance's limit. One failed request fails the whole read, so a partial map is never returned. The private single-asset read behind `fetch_shortability` now goes through it. Closes #788, closes #789. * docs(execution): Say that Spot Testnet reports zero commission rates only for an account not charged commissions there
…793) * feat(execution): Stamp Binance's signed requests on its server time Binance refuses a signed request stamped 1 s or more ahead of its clock, or more than recvWindow behind it (-1021), so a host clock that stays that far off failed every signed request. Both clients now read GET /api/v3/time and give the SDK a timestamp provider (binance-sdk 77.2.0) that stamps each signed request with the earliest Binance's clock can read, counted on from the read on tokio's monotonic clock (a VenueClock). The stamp is never ahead of Binance's clock, and a wall-clock step between reads does not move it. - The first request waits for one read; a failed read leaves this host's clock until one succeeds. A request that finds the last read 5 minutes old starts one in the background. - A -1021 drops the reading and reads again before the error returns, so the caller's retry is stamped on the new reading. - host_clock_lead on each client reports the measured lead; a read warns when the clocks are 1 s or more apart. - The account streams take the reads as a bound on Binance's clock too, so fill recovery and balance restatement stay on Binance's time before a stream's first event, now that a skewed host no longer fails those reads. - server_time_sync on each config (default true) turns it off. - VenueClock::merge takes another clock's sample of the same venue. Bumps binance-sdk 77.0.0 -> 77.2.0 (reviewed; Cargo.lock delta is its version and checksum). Closes #773 * test(execution): Pin the server-time warning, the signing clock's sharing and host_clock_lead Also drops ServerTime's hand-written Debug, which nothing used. * fix(execution): Keep Binance's signing estimate until a new answer replaces it, and re-read after a suspend - A -1021 dropped the estimate before its new read: a read dropped or failing left signed requests on this host's clock, and requests sent meanwhile were stamped on it too. The answer now replaces the estimate only when it arrives. - The estimate counts on on the monotonic clock, which stops while the host is suspended. A request that finds the wall clock gone more than 1 s further than the monotonic one since the last read now waits for a new read, instead of being stamped behind Binance by the suspend. - No read after a -1021 during a rate-limit cooldown; the reads' weight is documented as uncounted. - VenueClock::merge is crate-private; a test pins the spot WS order and cancel -1021 hooks; two misplaced comments moved back. * fix(execution): Test the cooldown skip, state the suspend read's limits, and pin the warning threshold without log capture A per-thread log-capture test raced other tests hitting the same warn! callsite, so the 1 s threshold is now a function with a boundary test.
* fix(execution): Deduct accrued interest in `Balance::net_asset` `net_asset` now returns `total - borrowed - interest` for a margin balance, as Binance defines its margin accounts' `netAsset`. Interest is owed as the principal is, so the figure falls as it accrues and a repayment out of the asset's holdings leaves it unchanged. Before, it left interest out until the repayment, so net-asset drawdown on the simulated venue stepped down by all of it at once. `BalanceBasis::NetAsset` follows. `EngineState::free_base` and the simulated venue's lending equity, which each deducted interest on their own, now read `net_asset`. * docs(execution): State `net_asset` on account-level margin venues and its interest's freshness On Alpaca and IBKR `margin` is `None` and `total` is already the account's cash, negative while it borrows, so `net_asset` returns that figure rather than a no-debt holding. Interest is held at its last reported total, so the figure falls in steps as snapshots or debt events report it, not continuously.
…e's time when the host clock leads (#798) * fix(execution): Stamp Binance REST balance reads no later than Binance's time when the host clock leads `fetch_balances` and `account_snapshot`, on spot and on cross and isolated margin, stamped each balance with this host's time before the request, less the 5 s REST lag allowance. The engine orders balance and debt state by that stamp, so a stamp ahead of Binance's time dropped the stream's balance and debt events until Binance's clock caught up. Since the clients sign on Binance's server time, a host clock any distance ahead reads successfully, so nothing bounded it. - `rest_read_with_retry` returns the send as a `HostMoment` (wall and monotonic clocks). - `balance_read_time` takes the earlier of the host's time and an estimate of Binance's clock at the send, less the lag. The consumer-facing reads pass the signing clock's estimate (`RateLimitTracker::venue_clock`); `restatement_time` passes the stream's, and now times its read at the send, after any rate-limit wait, rather than before. - Rustdoc: `ExecutionClient::fetch_balances` says when the caller must bound a leading host clock; the Binance module docs say the clients bound it themselves unless `server_time_sync` is off. Closes #796 * refactor(execution): Stamp Binance balance reads through the rate-limit tracker - `RateLimitTracker::balance_read_time(sent)` pairs a read's send with the client's own estimate of Binance's clock, so a consumer read cannot stamp against another clock. - `venue_clock` says why a copy taken after the read still bounds Binance's time at the send. - The Binance module docs say a balance read falls back to this host's time only while no server-time read has succeeded, as after a failed first read: the first request waits for one. - The test constructor is `ServerTime::trailing_host`, apart from the server-time tests' own `behind_host` helper.
… session's zone (#799) * fix(execution): Read an IBKR good-till-date that names no zone in the session's zone Listing open orders, or reading how an order ended, reads back the good-till-date of an order this client did not place or no longer tracks. IB's `yyyyMMdd HH:mm:ss` with no zone was read as UTC, but IB reads such a time in the TWS/Gateway session's zone, so the expiry was off by that zone's offset. - `parse_gtd_datetime` takes the session zone, as `parse_ib_timestamp` does: a zone-less good-till-date is read in it, the dash form this client sends stays UTC, a named zone stays that zone. - It still falls back to the wall-clock reading as UTC, with a `warn!`, when the zone is unknown, when no session zone is known, or when the time falls in the hour skipped as clocks go forward, so the order can still be listed. The rustdoc names each cause. - `order_shape_from_ib` and the open-order listing take the session zone; a completed order keeps the zone its times were read in. Closes #797 Closes #785 * fix(execution): Read IBKR ended orders in the session zone read at connect - The ended-order reader takes the gateway's zone from the client, read once at connect, as the open-order listing does, rather than reading it again for each lookup, which repeated its warning for a zone name `chrono-tz` does not know. The recovery watcher carries it for the order check. - An adopted order's test keeps its own doc comment.
…r owing a loan (#801) * feat(sim)!: Let `SimStart` open a spot margin account already short or owing a loan On a simulated venue whose `spot_margin` is set, a `SimStart` can now hold a spot short and list loans: - Spot positions net per base asset, as a cross margin account nets them: a net long is held, a net short opens as a loan of the base, holding none of it. The Engine is seeded with each position at its entry price, as before. - `StartBalance` carries a loan (borrowed principal and interest owed), set with `StartBalance::with_loan`, so a start can hold a long bought on margin, interest already owed, or a base loan larger than the net short. A listed base asset must hold its positions' net quantity net of its borrowed principal. - A short's proceeds are listed in `balances`, as a long's leftover cash is. Interest on what the start owes runs from the interest model's first boundary after the venue's clock first moves, with no charge for the borrow. - The start's debt may not exceed `(max_leverage - 1) x equity`, valued at the start's own prices since no market exists yet: the valuation asset at one, and an asset a spot position pairs with it at that position's entry price. An asset with no such price adds nothing to equity, and a debt in one is refused. BREAKING CHANGE: `StartBalance` gains a `loan` field and is `#[non_exhaustive]` (build it with `StartBalance::new`); `SimStartError::BaseBalanceConflict` reports `listed` (total less borrowed principal) in place of `balance`. New `SimStartError` variants: `NegativeLoan`, `LoanWithoutMargin`, `UnpricedDebt`, `OverLeveraged`. Closes #794 * test(sim): Cover a listed base loan in a conflict and a pair sharing only the quote * test(sim): Pin a start at the lending limit against the venue, and leverage one - A start whose debt is exactly the venue's lending limit is lent nothing more by the venue, while the same start one leverage higher borrows, so the start's check and the venue's limit cannot drift apart unnoticed. - A max leverage of one refuses any debt and accepts a start owing nothing. - Several assets held without a price are named in order. - `SimStart`'s rustdoc says an entry price must be positive to price an asset.
…to a position (#802) * feat(engine)!: Let the strategy attribute an account-level cash flow to a position A venue can report a cash flow on the account as a whole, such as cross-margin interest, with `instrument: None`. The engine applies it to no position, and its docs told a consumer to set the instrument before the event reached the engine. `SimStepper`, `backtest` and the live run loops all hand events to `Engine::process` with no such point, so the flow could never reach a position's carry there. Add `CashFlowAttributionStrategy`, a strategy trait with one method, `attribute_cash_flow`. `Engine::process` calls it for each unattributed `CashFlow`, before the flow updates the state or is recorded in the audit, so a state replica applies the attribution too. `Some(instrument)` sets the flow's instrument; `None` leaves it on the account. An answer naming no instrument the state tracks on the flow's exchange is ignored with a warning. `route_to_base` ships as an opt-in helper: the only instrument on the exchange whose base asset is the flow's asset and that holds an open position. `DefaultStrategy` returns `None`. BREAKING CHANGE: every strategy used with the `Engine` must implement `CashFlowAttributionStrategy`. Returning `None` keeps the previous behaviour. Closes #800 * refactor(strategy): Filter route_to_base by exchange without allocating, and pin an answer holding no position route_to_base compares each instrument's exchange directly instead of building a one-exchange InstrumentFilter, which allocated on every call. The trait's rustdoc says what happens when the answer holds no open position: the flow keeps the instrument and changes no carry, which a new test pins. The State rustdoc no longer names the upstream project.
…nd gate feature-only helpers (#804) IBKR fill recovery read the gateway's zone from the client on every recovery, while the order stream reads executions in the zone taken at connect. Recovery now takes the zone the watcher already holds, so a recovered fill and a streamed one are read in the same zone, and a zone `chrono-tz` does not know is warned about once rather than on every recovery. Three crate-private helpers were dead code in builds without the features of their callers, each raising a warning: - `VenueClock::merge`: Binance only. - `OrderKind::is_conditional_unresolved`: any client feature. - `SharedKnownLiveOrders::holders`: Binance tests only. Each is now compiled only with those features.
…ests can script (#805) * test(execution): Read IBKR fill recovery's executions from a source tests can script `recover_fills` read its executions straight from `ibapi`'s blocking client, so only a live gateway exercised it. It now reads them through two private traits, one that requests them and one that reads the replies, implemented for `ibapi`'s client and subscription. It takes a `RecoveredFills` and the read's timeout instead of building the one and fixing the other. New tests drive it from a script: replies read to the end marker (a notice skipped, times that name no zone read in the session's), a refused request, an error reply, a silent gateway timing out, and a consumer that goes. * fix(execution): Report the timeout an IBKR fill recovery read actually ran out `RecoveryError::TimedOut` stated `RECOVERY_TIMEOUT` although `recover_fills` now takes its timeout as a parameter. The variant now carries the timeout the read ran out, and `FillRecoveryGaveUp` reports that one. * test(execution): Pin the timeout a timed-out IBKR fill recovery states to the one it ran out
…rategy's State without the old name (#806) * chore: Clear the Binance spot test clippy warnings, and describe a strategy's State without the old name - Split `test_convert_order_kind_tif` into plain, conditional and trailing tests, which brings each under clippy's cognitive-complexity limit. - Keep the struct update in `test_convert_account_position_is_at_u_else_at_e`, with an allow saying why: it sets no field today, but keeps the test compiling when the SDK's generated model gains one. - `AlgoStrategy::State` and `ClosePositionsStrategy::State` described the full `EngineState` as what "Barter ecosystem strategies" use. They now say it is what a strategy run by the `Engine` uses. * chore: Drop the redundant struct update in a Binance spot test, and give every engine-run State the same rustdoc - `test_convert_account_position_is_at_u_else_at_e` sets every field of the SDK's `OutboundAccountPosition`, so the `..Default::default()` and its allow go. An SDK bump that adds a field is a deliberate change, and a compile error there is the clearer signal. - `CashFlowAttributionStrategy::State` said "typically" the full `EngineState`, which the `Engine` requires. It now matches `AlgoStrategy` and `ClosePositionsStrategy`, and `RiskManager::State` gains the same rustdoc.
…nce spot fills after a heartbeat timeout (#807) * fix(execution): Reach back two heartbeat windows when recovering Binance spot fills after a heartbeat timeout The spot account stream reads its activity flag once every 30 s window and declares a heartbeat timeout at the end of the first window without a frame. A connection that died just after the frame that set the flag early in the window before is therefore declared dead almost 60 s later. Recovery started the gap only one window (plus the signal margin) before the timeout, so fills from the first part of that span could be missed. `disconnect_lookback` now reaches back two windows; fills read twice are dropped by the dedup cache. The overflow guard on `HEARTBEAT_TIMEOUT_SECS` now covers the doubled value. * refactor(execution): Share the Binance disconnect lookback between the spot and margin streams The spot and margin account streams each encoded how far back a disconnect's gap starts, and the spot copy had drifted to one heartbeat window while margin's used two. `disconnect_lookback` now lives in `shared.rs` beside `disconnect_error`, and margin's `SessionEnd::gap_lookback` calls it. Its rustdoc says the signal margin covers timer lateness as well as scheduling jitter.
… at the clock (#808) `SimulatedVenue::apply_market` took its tick's instant as given. A tick earlier than an instant the venue had already reached stamped its fills, the balances they restate, its expiries and the market it stores behind what the venue had already reported, and the engine drops a balance older than the one it holds. It now takes such a tick as that latest instant, as `advance_time` does, so the clock never moves back whichever driver calls the venue. Both inputs share one private helper for the clamp. Drivers that feed their ticks in order see no change.
…art, and once the order ends (#809) * test(execution): Read an IBKR good-till-date expiry back after a restart, and once the order ends A paper test places a good-till-date limit order and reads its expiry back three ways: from the client that placed it, from a new client under the same API client id that knows only what IB lists, and from `fetch_ended_orders` after cancelling it. Each must equal the instant placed. The expiry is five days out, because IB rejects one that falls before the next market open, and so can fall across a daylight saving change in the session's zone. * test(execution): Read the IBKR good-till-date only through clients that never saw the order The client that places or adopts an order answers from what it tracks, so reading the expiry back through it never parsed IB's good-till-date. The placing client's read is dropped, and the completed orders are now read by another new client after a second restart, so both reads parse what IB lists. The reconnect loop is shared with the restart test. * test(execution): Say which read of an IBKR order parses its good-till-date, and that the test does not choose a daylight saving window
…and what it must return (#810) * docs(core): Document RiskManager::check: when the engine calls it, and what it must return * docs(engine): Name the events after which the engine skips RiskManager::check, and that it ignores a refusal's Unrecoverable
…at a configured margin level (#812) * feat(execution): Liquidate the simulated venue's spot margin account at a configured margin level `SpotMarginConfig::with_liquidation(SpotMarginLiquidation)` turns it on: a liquidation level, a fee rate on the debt repaid, and an optional margin-call level. After every market tick and clock move, an account at or below the level is sent a `Liquidation` notice, has its open orders cancelled, and has every debt closed by forced taker trades reported as `TradeOrigin::Liquidation` fills, selling other assets for the valuation asset as needed. The fee is reported as the new `CashFlowKind::LiquidationFee`. A debt it cannot close at once stays owed, and the liquidation continues on later inputs. The margin-call level sends `MarginCall` and `MarginRestored` once per crossing. The leverage limit and the liquidation check share one valuation of the account. * refactor(execution): Drop two guards the liquidation check and its config already imply * test(execution): Pin each liquidation step a mutant could change, and fix the rounding nudge A forced trade the account's balance limits, whose quantity rounded up to cost more than the balance, was nudged down a digit and then back up, until its four tries ran out, so it traded nothing. A quantity that costs too much now lowers the cap, and the largest affordable quantity tried is traded. New tests cover selling collateral before a buy-back, a fee the free balance partly covers, the nudge in both directions, the quantity the fill model is asked to price, a pair without a market, a leg whose fee takes all it delivers, distinct fee ids, and the warnings that are given once. `raise` already does nothing for a need that is not positive, so its callers no longer check, and the lending limit takes the headroom's maximum with zero rather than matching on its sign. * fix(execution): Skip the liquidation check for a debt that values at zero, and pin the cancel order - A dust debt at a tiny price can value at zero, and the margin level divides by the debt, so the check now treats a zero valuation as nothing to liquidate rather than panicking. - A test cancels a held market order and two resting orders placed out of order, and pins that each releases what it held, by client order id. - `SpotMarginLiquidation` is `Copy`, and `SpotMarginConfig::liquidation` returns it by value. - The docs say a forced trade draws from the slippage stream once priced, even if it trades nothing, and that `apply_market`'s margin check runs once, at the end of the tick. * feat(execution): Lend nothing while a simulated liquidation is under way, and document the fee's priority - While a liquidation that could not close every debt is still under way, the simulated venue lends nothing, so an order that would need a loan is rejected as unaffordable, its reason showing nothing borrowable. An order the account can pay for from its own balance is still accepted, and cancelled by the next input if it is still open. Once nothing is owed, the venue lends again. - The docs and CHANGELOG say that each pass charges its fee before any debt it leaves owed, selling for it as for the debt, as a venue takes its fee out of the proceeds of the liquidation's trades. - The dust-debt check's comment says the liquidation state is left as it was. * test(execution): Pin the liquidation lending gate for market and held orders, and clarify the docs - The lending-gate test also has a market sale refused for a loan the leverage limit would grant, and asserts the self-funded order is reported Cancelled, not just named. - A new test holds a self-funded market order for the next bar while a liquidation is under way; the bar's open makes its fill need a loan, so the order is cancelled and the liquidation then closes the remaining debt. - The docs and CHANGELOG say each pass's fee comes out of what the account holds after that pass's repayments, that a held order's fill needing a loan is cancelled, and that a debt the venue cannot trade out of keeps lending off until the account repays it.
…nance balance-stamp tests (#814) * test(execution): Allow for the server-time read's own delay in the Binance balance-stamp tests The tests that check a balance read is stamped on Binance's clock bounded the stamp from below by one millisecond, for the server time's truncation. The estimate also dates that server time to the moment its answer is received, so any delay before then puts the stamp further behind; under load it exceeded the millisecond. A shared test constant now allows 1 ms plus 50 ms, explained where it is defined. The upper bound, which shows the stamp is not on this host's leading clock, is unchanged. Closes #811 * test(execution): Keep the host-clock balance-stamp bound at a millisecond, and correct the allowance's rustdoc - The margin balance-stamp test widens its lower bound only for a request signed on a server time read; on this host's clock it keeps the 1 ms bound it had. - TRAILING_HOST_BEHIND's rustdoc names the delay as task scheduling and polling (not the read's timeout, which adds none), and mentions the clock's drift allowance. * test(execution): Word the host-clock bound's comment without reference to the earlier code
… of spot pairs (#817) * perf(execution): Add a benchmark of the simulated venue's per-tick spot margin check against the number of instruments and assets held Times one SimulatedVenue::apply_market tick on an account that owes a loan, with liquidation configured and without, across 1 to 1000 spot pairs and 1 to 100 assets held. The difference is what valuing the account costs on every tick. Refs #813 * perf(execution): Value a simulated spot margin account from an index of spot pairs The venue valued each balance by scanning every instrument it trades, so each market input's margin check cost instruments x balances while the account owed anything. It now indexes its spot instruments by the two assets each pairs, once at construction, and both the valuation and a liquidation's choice of trade read that index. Which instrument values an asset is unchanged: the first by name that has a positive price. BREAKING CHANGE: `SimulatedVenue::instruments` is no longer a public field, since a map changed after construction would leave the index stale and the account valued wrongly. Read it through the new `instruments()` and `instrument(&name)` accessors; to trade a different set of instruments, construct a new venue. Refs #813 * test(execution): Pin a simulated venue's instrument accessors, and that a request's snapshot values only the pair it names Refs #813 * docs(changelog): Add the simulated venue's spot margin valuation benchmark result Refs #813 * refactor(execution): Drop the spot pair index's unused Clone derives Refs #813
…on parameter (#818) `RiskManager::check` returns refusals with a `String` reason, which is never unrecoverable, and the engine never asked whether a refusal was. The impl could not affect anything, and the `Reason` parameter could not be used through `check`. A refusal concerns only the request it refuses; stopping the strategy's orders is `TradingState::Disabled`, which the rustdoc now says. BREAKING CHANGE: `RiskRefused<T, Reason = String>` is now `RiskRefused<T>` with `reason: String`, and no longer implements `Unrecoverable`. The serialized form is unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Promotes
develop→mainfor the v0.18.0 release. This is step 4 of the flow inCONTRIBUTING.md. The release-prep PR, #819, is already merged.Merge this as a merge commit, not a squash. Earlier releases were merged the same way.
Closes #786
Closes #787
Closes #789
Closes #768
Closes #770
Closes #803
Closes #813
Each of these was fixed by a PR merged into
develop: #786 and #787 (#791), #789 (#792), #768 (#769), #770 (#776), #803 (#812), #813 (#817). A closing reference only fires when it reaches the default branch, so these lines close the issues when the release reachesmain. The other issues fixed in this release (#771–#775, #777, #778, #785, #788, #794, #796, #797, #800, #811) are named with a closing keyword in their own commits, so they close when those commits reachmain.Scale
30 commits since v0.17.0, about 16,500 lines added and 1,600 removed.
rustrade-executioncarries most of it (about 11,300 lines added).rustradegains account-level cash flow attribution, the engine and venue clocks, monotonic in-flight deadlines, and theRiskRefusedand dataset summary changes.rustrade-instrumentgains a test-only log capture helper.rustrade-data,rustrade-integrationandrustrade-macrochange only their version (and, forrustrade-data, one test).Version
The version is 0.18.0, a minor bump: the release has breaking changes, each flagged in the CHANGELOG. On
develop, the six crate versions, the workspace dependency pins andCargo.lockare already at 0.18.0, and the root README requires0.18.What is in it
See
[0.18.0]in the CHANGELOG for the full list. In short:CashFlowAttributionStrategy, feat(engine)!: Let the strategy attribute an account-level cash flow to a position #800).SimStartcan open a simulated spot margin account already short or owing a loan (feat(sim): LetSimStartseed a spot short on a venue that lends spot margin #794).Balance::net_assetdeducts accrued interest.OrderRules::checktakes the order's side, and checks a trailing order's offset (binance: Read TRAILING_DELTA and NOTIONAL's market flags into OrderRules #787).SimulatedVenue's instruments are fixed at construction: the field is private, read throughSimulatedVenue::instrumentsandSimulatedVenue::instrument(perf(execution): The simulated venue's margin valuation scans every instrument for each balance #813).RiskRefused<T>has aStringreason and no longer implementsUnrecoverable.VenueClock,HostMoment,Trade::time_source,EngineClock::instantandAccountEventKind::InstrumentBalanceSnapshot.HistoricalClockreads its instant without a lock (HistoricalClock: replace the RwLock with a lock-free seqlock that keeps the full DateTime range #768), and the simulated venue values a margin account from an index of spot pairs (perf(execution): The simulated venue's margin valuation scans every instrument for each balance #813).binance-sdk77.2.0 (binance: estimate the server-time offset and resync on -1021, instead of relying on the host clock #773).Checks
2951c4cd, includingcargo audit,cargo deny, the mutants shards and the Alpaca integration tests.RELEASE_TAG=v0.18.0 DRY_RUN=1 .github/scripts/publish-crates.shpassed on the release-prep branch (6 crates). The dry run is repeated onmainbefore tagging.