Skip to content

Fast-forward main onto the fork - #409

Open
userFRM wants to merge 1653 commits into
deepentropy:mainfrom
userFRM:main
Open

userFRM wants to merge 1653 commits into
deepentropy:mainfrom
userFRM:main

Conversation

@userFRM

@userFRM userFRM commented Aug 26, 2026

Copy link
Copy Markdown

This is userFRM:main, offered as a fast-forward.

TL;DR

Commits 1,465 — 797 fixes, 162 features, 131 test, 201 docs, 73 refactors
Files 452
Merge Clean fast-forward. main..upstream/main is 0, nothing of yours rewritten or dropped
Undo git reset --hard 9367845
CI Green on the head — actions
Tested 154 phases against a real session, on paper and live — last run green in full 9 September 2026. 3,421 offline tests + 197 broker-session tests
Soak 14 hours held open on one session, through the venue's maintenance window — 0 losses, 0 reconnects
Order fields dropped 0 of 154
Call arguments dropped 0 of 572
API parity 77 of 78 calls served on both languages
Order types 24
Requests on the wire 72 of 80

Merge safety

  • git merge-base upstream/main main = 9367845 = the tip of main here.
  • git rev-list --count main..upstream/main = 0 — nothing to resolve, no conflicts.
  • Nothing of yours is rewritten, reordered or dropped.
  • One command undoes it: git reset --hard 9367845.
  • Partial take: cut at 7cfa6e56 (see Taking it).

How it was tested

  • Reproduced first. Fixes were reproduced before they were made, and landed with tests that fail without them — 127 test commits.
  • Against a real session. 153 phases end to end: order placement, replacement, cancellation, brackets and legs, executions, commission reports, account and position downloads, market data, depth, historical series, contract lookups, reconnection.
  • Live as well as paper. Paper and live speak the same wire — identical messages, fields and sequencing, differing only in which prints arrive. Order paths have additionally been run against a live account, real fills, a real book.
  • No silent passes. A phase cannot skip on silence — each names what refused it. A phase cannot pass on an order that never reached the wire.
  • Held open. One session run for 14 unbroken hours across the venue's nightly maintenance window: no losses, no reconnects, quotes and orders still answering at the end.
  • Counted by script, re-derived in CI. A documentation claim fails the build when the source stops supporting it.
python scripts/gate.py          # runs what CI runs; --list prints the suites first
cargo test --lib
git diff --exit-code docs/      # the generated reference matches its source

IB_USERNAME=... IB_PASSWORD=... cargo test --test ib_paper_compat -- --test-threads=1 --nocapture

Orders — 110 fixes

Was Now
Algo params, conditions and unmodelled params dropped between caller and wire Travel as stated; condition flags sit on the tags that mean them
Flags the caller never set were sent anyway An algo flag, time or aversion not stated is not sent
Pegged orders lost offset, cap, snap-to offset, benchmark price All carried, including an offset of zero
Prices truncated; sign lost on prices with no whole part Rounded; sign kept
13 order types the venue replaces were refused Accepted, plus volatility, both relative combinations, passively relative, peg-to-best, peg-by-minute, delayed activation
Replace kept terms the venue had refused The venue's word settles it; refused terms do not stand
A correction the venue sent twice reopened a finished order and republished its quantity as working Read once; the second copy changes nothing
A second refusal for the same cancel read the echoed name as digits and retired an unrelated live order Resolved through the cancel's own name
The async client's two withdrawal calls waited out a reconnect's replay on the reactor Off it, as every other order call there
An order's quantity was bounded where a double stops multiplying exactly, so a cash order stated in currency units was refused Bounded where the field itself ends, four orders of magnitude further; the whole part scales in integer arithmetic
A rejection answering a replace was read as how the order finished It is not the order's state
A replacement could not itself be replaced It can
Type change on a linked order corrupted links Refused; links kept both ways
Trigger survived a replace to a type without one Dropped
A working order this client did not place was placed again Replaced
Bracket legs lost group and parent on replace Kept
A held parent withdrawn orphaned its children Children go with it; a parent releases them
A set that could not be placed whole left partial legs Withdrawn
Filled orders reported as working Not reported as working
Orders done but unfilled tracked forever Stop being tracked
Order ids counted from a file Counted from what the account is working, past every id the venue has named
A cancel the venue refused left the order reading as working Reads correctly
A failed write reported as a rejection Reported as unknown
A refusal with no words did not end the order Ends it

Python surface — 74 fixes

Was Now
Programs written against the reference client would not import Its module layout, constants, enumerations and run-together names all answered
A model calculation on a described contract claimed the watch the request already held, and cancelling it withdrew that subscription Only a watch this call opened is its own
A handler that closed the session from inside connectAck still received the accounts and the next order id The startup stops with the session
Constructor spellings rejected The reference API's own spelling accepted
Enum-typed fields handed back raw values Hand back the member
KeyboardInterrupt / SystemExit swallowed in callbacks Propagate
A raising callback lost its answer Keeps it
A raising position handler ended the dispatch loop Does not
A handler raising during connect failed the session Does not
GIL held across blocking engine calls and blocking waits Released
Two connects could both win Cannot
A client holding itself was never collected Collected
Request ids truncated; callbacks handed partial contracts Whole ids, whole contract
Ticks named differently from the reference Every tick the venue numbers named as the reference names it
5 places Python answered differently from Rust Answer the same
4 things the model classes changed or dropped in transit Carried

Engine — 65 fixes

Was Now
Slots reused while something still pointed at them Not reused; resident while holding market data
Withdrawing a quote took the decoder mapping for a news subscription asked for under its own request News is withdrawn only by its own withdrawal
A historical or calendar farm that came up long past its deadline was installed anyway Each farm's clock is read before its result is taken
A slot that changed hands carried the wrong contract's order An order states the contract it means
A slot did not return when its last order ended Returns to the table
5 answers belonged to another question Go to the request that asked
5 requests answered from something other than the venue Answer from the venue
4 answers taken after the moment they belonged to Discarded
6 answers said something had happened when it had not Do not
Transports shared one recovery clock Each has its own
Hot loop spun with every transport down Parks
Hot loop could wait on something that never comes Cannot
A quote feed the venue would not take back killed the session Ends the feed only
Fills lost for orders the session does not track Booked
A duplicate execution suppressed the whole report Suppresses the fill only
Missing position quantity read as flat Read as no quantity
Orders unaccounted for after recovery were silent Reported

Historical data — 40 fixes

Was Now
7 series the venue carries were refused, incl. ADJUSTED_LAST Served
Midpoint, rate, volatility asked under names the venue does not answer Asked under the venue's names
TRADES meant something other than the vendor's TRADES Same meaning
Corporate actions discarded Read
Bars dated on the wrong clock, written in the wrong shape Dated on the venue's clock, written as the request asked
A bar the fold could not build was folded anyway Refused
A refusal ended queries it did not name Reaches only the query it names
A withdrawal took more than it withdrew Takes what it withdrew, only
A sloppy answer came back as a plausible wrong number Refused
The historical farm gave up after six attempts Keeps retrying

Market data — 41 fixes

Was Now
A contract named by id alone was refused Looked up at the venue
The slot getter answered from the cache after the engine had given the slot to another contract Answers through the same lookup every other reader uses
Dropping a live-bar stream never sent its withdrawal, so the venue went on closing bars at a reader that had gone Withdrawn under the mark its subscribe was numbered with
An unknown contract was assumed a US stock The venue is asked what it is
A request joining a live subscription got no quote Owed the quote as it stands, and its tickReqParams
A snapshot was handed to a second caller as a subscription It is not; the chargeable snapshot is asked for
A finished snapshot withdrew what replaced it Withdraws itself
A snapshot ended on a pause Ends where the venue ends it
A stream was withdrawn by the wrong number Withdrawn by the number the venue gave it
A feed given up on kept taking subscriptions Takes none; refused where the caller asks
L1 did not survive a disconnect Re-subscribes from the surviving record
A drop mid-replay lost what the replay had not reached Kept
A refused book kept being asked for Stops

Session, second factor, connection — 41 fixes

Was Now
The logon was judged on the last message type in the answer Judged on what it answered with
An acknowledgement sharing its type with account traffic was missed Found wherever in the answer it sits
A fill riding in with the reconnect acknowledgement was dropped Kept
4 ways a login ended that the venue had not ended Closed
5 ways a connection could be held open or given up wrongly Closed
Second factor ran on first login only Runs on reconnect too
The code provider ran on the socket thread Runs off it
A code was spent on a login already lost Not spent
The code was submitted while the socket was busy Submitted only while quiet, as soon as it resolves
The session token reached the log Kept out
An unstamped reconnect left the logon time behind, so the next reconnect read this session's own logon as a competitor and gave the account up The logon time moves on
A takeover could not be told from this session's stale logon Told apart
A signal interrupting a read counted as silence, so the venue's routing answer went unread and the connection failed A signal is retried, not counted

Protocol — 17 fixes

Was Now
Fields searched for rather than read where the format puts them Read at stated positions
Length and signature read from the wrong offsets Read where the header puts them
A message borrowed the next one's bits Ends where its own length says
A compressed header split across reads failed to frame Frames
A read taking two messages kept one Keeps both
A pause inside a frame read as a failed frame Does not
9 ways a frame could say something it did not carry Closed

Account, contracts, options — 42 fixes

Was Now
The account could be read before it was stated whole Reads wait for the download; every download ends
A holding the new statement never names stayed open Closed
P&L did not survive a trading reconnect Renewed
Contract hours stated on the wrong clock Stated on the clock they are named with
Market-rule ids not surfaced Surfaced on contract details
A lookup bounded on answer length Bounded on silence
con_id=0 reported as a contract Reported as "not found"
An option's model read from the wrong contract Read for the contract asked about
An option on a future priced as one on a share Priced as a future option
A non-number price yielded a volatility; an overflowed tree yielded a price Both refused
Time to expiry taken from the local clock Taken from the venue

What the gateway does that this did not — 15

The sweep above asks where this client is more than the gateway. This one asks the reverse,
against the decompiled build: what does the gateway do that a caller here cannot reach? Every
row was verified by three independent readers before it was written.

Was Now
A change to an order reached the venue only for the terms the replace names; a new display size, discretionary amount, all-or-none or algorithm parameter was dropped, while this client's own answer already read back the new value Every replace goes behind the caller's own statement, and the latest statement stands
A site that was down was reported as credentials being refused, and stopped the retry ladder The venue numbers its refusals; the two that mean "not now" are waited out
A host that stopped answering ended the session Its two standbys are named after it and tried first, as the venue names them
An order naming a model portfolio was refused outright The model rides the placement, the replacement and the withdrawal
Day-till-cancelled went out as good-till-date, carrying an expiry nobody named Good-till-cancelled with the flag that stands it down at the day's end
A combination's legs stated open or close on the field that carries a leg's number On the field that carries the intent; a short leg states its borrow in full
An account summary asked for by currency answered nothing The per-currency set is named by a prefix, not a figure name
Delayed and frozen quotes were collapsed to one entry, so the trade half never arrived Both halves, as every live subscription already asked for
A withdrawal omitted three companions its subscription had stated States what it withdraws
Six bar and tick series the venue serves were refused before they were sent Sent under the venue's own names
A news article delivered as a document, or as a refusal, reached the caller as article text Delivered as what it is
What the venue says to the account holder was counted as wire nobody reads Reaches the caller
A symbol search was sent exactly as typed Trimmed and collapsed, refused only where the venue refuses it

Thirty-nine further differences were found and left alone, each recorded with what closing it
would take. Most are one shape: a family of order types, or a request whose answer has nowhere
to go — a new type, a new channel, or a refactor across files rather than a smaller diff.


Matching the gateway rather than improving on it — 13

This client stands in for the gateway, so a caller who switches must get the same
answers. In each of these it was being more careful than the thing it replaces.

Was Now
Calendar requests expired locally after 30s; a later answer was discarded The venue's answer ends them; the gateway sets no timer
An execution reported the reconciled difference as its own quantity Reports the quantity the venue stated; the reconciliation still uses the difference
An unmodelled risk level or algo flag was refused, then silently dropped Travels as the caller wrote it; the venue answers for its own vocabulary
Adjusted history was refused on a contract stated by description Sent; the venue resolves it
A historical-news request naming a time window was refused Carries the window; the element names and their format are the ones this client's other queries already send to the same service
A caller asking the time got the stamp on the last message the venue happened to send — frozen on a quiet link, refused before any arrived The venue states how far its clock sits from this one; the answer is this machine's clock moved by that, and a connected session is always told the time
An order stating an advisor group, method or percentage had them dropped, under a note claiming the protocol carries no field All three ride the order and its replacement
A profit request naming another account was warned about, then answered with this account's figures under the caller's own number Refused, and refused before it takes the one subscription slot
A market rule this session has not seen was refused under this client's own number and code Refused under the ones the venue uses, on both surfaces
A withdrawal arriving while its subscription was still opening was refused, and the subscription went on Recorded and answered; the registration reads it before publishing and takes back down what it opened
A session held as many quote subscriptions as this client's own table, ignoring the number the venue states on the logon Reads the allowance and refuses past it under the venue's own code
Margin figures the venue had not stated carried a stand-in number Stay empty, as the reference client leaves them
One account summary at a time, ended at its own first answer Two at once, each lasting until it is cancelled
An ordinary exception from a Python callback was logged and dispatch continued Leaves dispatch and closes the session, as the reference client's loop does
Values the reference decoder reads as absent arrived as sentinel numbers Arrive as
Repeated order-status reports were suppressed when the status was unchanged All delivered; the venue sends them and the reference client hands them on

Kept deliberately, with reasons: the SRP group check, the server-proof requirement
and the DH range check, since no caller can see them and dropping them would only
widen what a hostile peer can do; and the permanent-id derivation, because the
gateway's own parse throws on this venue's identifiers and zeroes the field.


Known gaps

  • request_fa / replace_fa reach the server; their reply is not parsed. Pinning the shape needs an advisor account I do not have. These are the 1 of 78 not served on both languages.
  • Some paper phases skip rather than pass. Each names what refused it — venue or account, not the client: an order type an exchange will not take for that security type, a lot size rule, a subscription the account does not hold.
  • One process holds one session. A gateway process rents its session out over a local socket; this has no socket. That is issue #2, unaddressed here.

Taking it

  • All of it: fast-forward.
  • Part of it: cut at 7cfa6e56 ("tests move beside the code they test"), which moves twelve source files into directories. Commits before it are written against your current layout and cherry-pick with ordinary conflicts; commits after it assume the new paths.
git log --reverse --oneline upstream/main..main | head -742
  • None of it: close it. If a particular subsystem would be more useful on its own, say which and I will open it against your layout.

This was referenced Aug 26, 2026
userFRM and others added 30 commits September 15, 2026 09:04
The page is read from three clients: the two it compares against and this one.
Installing the two was half the fix — the job that regenerates every generated
file and compares it against what is committed does not build the extension, so
`import ibx` fails there and the whole table of what this client carries beyond
the documented surface drops out of the page it produces.

The comparison moves to the job that already builds the extension for the Python
suite, and that job installs the two reference clients beside it. Nothing else
needs a second Rust build.

Reproduced before committing: a regeneration against the pinned clients and a
freshly built extension leaves `docs/` and the readme unchanged, which is what
the runner does.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01C9qDHkbkXiSNP7gjQ7PXxc
The venue states a count of entries and then a currency and a balance for each.
This client read the count as a selector saying which figure followed, and took
the last balance on the frame as its value — so it published "an account figure
of kind 3" carrying a number that belonged to whichever currency came last,
under a label that named nothing at all.

Captured whole from a live session:

    6040=77 1=DU… 6566=3 15=BASE 9806=1492751.1917
                         15=GBP  9806=1105714.0700
                         15=USD  9806=0.0000

and the sterling figure is that account's cash balance in sterling to the penny,
which it already states under a name of its own. So the frame carries nothing a
caller does not have, the kind was never a kind, and the diagnostic that said
otherwise is gone rather than reworded.

Two statements go with it. The estimated-opening record carries up to three
prices and two of them reach no caller; the documented API has one number for
it and this client will not pick a second, so the ones not handed over are
written down under their position rather than passing unseen. And the insider
and institutional series was documented as stating a float. It does not: it
states what institutions and insiders hold, the day each was counted, and how
many shares are on issue, and a float is worked out from those.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01C9qDHkbkXiSNP7gjQ7PXxc
… venue states one

Tick 586 carries three prices and a set of flags. The flags say which of the
three the venue is actually stating: all three where the estimate stands, the
first alone once the figure is final, and none otherwise. The note this client
keeps of the prices it cannot hand to a caller was written without consulting
them, so it named a price under flags that state none, and named the third
position under flags that state only the first.

It now writes down a position only where the flags make it a price, and only
where that price is a number, which is the same bar the published prices clear.

Two explanations went with it. The dispatch comment above the company series
said an ownership reply states a float; it states what institutions and insiders
hold beside the shares on issue, and a float is worked out from those.
… read

The venue states what it holds about the company behind a contract, and about
the terms of dealing in it, as runs of KEY=VALUE text on a series of its own.
This client read three of those series and let the other thirteen go past: the
ratios it keeps a history of, how a company scores against the principles an
account can screen on and whether it passes a religious screen, what margin a
contract takes and what a fund will take part in, the technical readings, the
fund-family figures, the two scores worked out from what is written about the
company, the lens over its accounts, and the reference price.

They are read now, and reach a caller through the same two calls the three
already did — company_data and company_data_series — keyed by the venue's own
number for the series, with the venue's own field names unchanged.

What separates them is only the frame in front of the text. Two analyst ratings
are text from the first byte. Four series state four bytes of their own first.
The remaining ten state the text's length as a count and pad what follows to a
four-byte boundary, so reading to the end of the payload puts padding inside the
last value and trusting a length that overruns reads bytes that never arrived.
Both are now read as the venue frames them.

Five of the series state their text in an alphabet of one byte to the character.
Held to the alphabet the rest are read in, a single accented character made the
whole record unreadable and nothing at all was published for it.

The request already sent any series a caller named by number, so nothing changes
in what goes out.
…wn count

The venue states that series' text behind a count of its length, and this
client read it as text beginning at the first byte. The count's low byte is a
letter for any record between sixty-five and ninety characters long, or between
ninety-seven and a hundred and twenty-two, and that letter then joined the first
name in the record: three contracts subscribed at once stated the same field
under three different names, each one the length of its own record.

Read from behind the count, the three state it under the one name.

The first analyst rating is stated the same way and was read the same way; its
records are long enough that the byte which leaked was punctuation rather than a
letter, so the name it produced was the right one and the fault did not show.
… session

The venue leaves figures out of the option model it publishes, and this surface
hands those on as nothing at all, which is what a caller written against the
reference client is given. The wrapper's own method declared them as numbers, so
that method refused the call, and the refusal travelled out of the reading loop
and closed the session.

It reached any caller that watched an option and did not write
`tick_option_computation` itself: one model without a dividend in it and the
session was over. Every test here had written the method, so the default was
never the one called.
…read

The venue states figures about a contract on series the documented API has no
call for: a bond's analytics and its accrued interest, the margin a future
takes, what one contract delivers, the volatility its own model settles on, the
volume a contract usually opens and closes on and what an average minute trades,
the close it keeps through the session and the close stated in money, what an
option is worth in and out of the money and what of that is time, the yield it
works out from a price, the two halves of a sentiment reading, what a perpetual
contract funds at, and a dozen more.

They reach a caller through stated_figures and stated_figures_series, on both
surfaces, keyed by the venue's own number for the series. The figures arrive in
the venue's order, read at the venue's widths, with the number it uses to say it
holds nothing left standing. None of them is republished under a tick number of
this client's own choosing: a caller reading the venue's numbers would be
reading an invention.

Three of the series end with fields the venue writes only sometimes, so a record
that stops short states the fields before it and no more.

Read against a live session on a share, an option, a future and an index. The
deliverable per contract came back as the contract's own multiplier times its
modelled price on each of them, the twenty-day volatility as the daily figure
for three shares whose volatilities differ threefold, and the four-week volume
as a figure that states a count, a flag and a second figure rather than the two
counts a record of two long figures would have made of it.
Four series state their figures as two numbered tables — whole numbers first,
then fractional ones, each opening with how many entries it carries and each
entry naming what it is before stating it. One of the four was read for the two
numbers that have a documented call to arrive on and for nothing else; the other
three were not read at all.

All four are read by one reader now, and every figure is kept under the number
the venue gave it, reachable through numbered_figures and
numbered_figures_series on both surfaces. The two numbers that have a documented
call still go out as the ticks they always did, and the share count and the
year-ago open are still kept against the contract as before.

The figures that have no documented call are not republished under a tick number
of this client's own choosing: read for a fixed list and dropped otherwise, a
figure the venue started stating went unseen until someone here wrote its number
down.

The two tables are kept apart because the venue numbers them apart: the whole
table states four bytes to the figure as a count and the fractional one four
bytes as a fraction, so the same number in each is not the same figure.

Read against a live session on three shares. One of the three series states its
figures against a span of days and dates each one, giving nine points of a
contract's price history back to five years on a single tick; another states the
average close over thirty days; the third states one month and twelve months
back. Their five-year points matched what those contracts traded at.
…e are read

Three more of the venue's series reach a caller.

Two of them state a run of paired figures: a count, then that many pairs of two
eight-byte figures. One holds the volatility the venue's own model puts against
each point of a curve; the other holds the weight it puts on each price a
contract might reach, and states a version in front of its count and a figure of
its own behind the pairs. They reach a caller through paired_figures and
paired_figures_series on both surfaces, under the venue's own number.

The third is the same option model this client already reads, worked out as the
contract closed rather than as it stands — the same shape down to the flags that
say which fields it carries, so it is read by the reader that already existed.
It is kept apart from the standing model and reached through
closing_option_model, because a figure worked out at yesterday's close standing
where a caller reads what a contract is worth today is worse than not having it.
That model carries every greek the venue states, including the several the
documented API has no field for at all.

Asked for and acknowledged on a live session against two options, and none of
the three stated anything: the venue assigns each a subscription and then
answers with silence, which is how it answers a series an account cannot see.
The third is the model at the close and the session was hours before one. So
these are read but not yet seen, unlike the series already landed.
…ing model is

The venue serves its option model only to a subscription that names the model's
own venue rather than the one the contract trades on. The model worked out at
the close is the same kind of record — same flags, same fields, same reader,
differing only in which moment it was worked out for — and it is asked for
through the same path. It was going out on the contract's own venue.

This does not make that model arrive. Asked for either way it is acknowledged
and then never stated, on a paper and a live login for the same account minutes
apart, so what withholds it is not the venue named in the request.
…or the default that broke

The page on what the API does not forward now covers the contract series. A
quote subscription carries dozens of them — the venue's own named fields as
text, figures at the venue's widths, two numbered tables, runs of paired
figures, and the option model as a contract closed — and the page shows how to
ask for each and what came back on a real session. One of the numbered series
is a five-year price history with the date of each point, off a quote
subscription and with no historical-data request behind it.

It also states plainly that a series can be refused by silence: the venue
acknowledges the subscription, assigns it a tag and says nothing, which is how
it answers a series an account is not entitled to. Six behaved that way on the
session this was written from, identically over a paper and a live login, so an
empty result means either the venue has nothing for that contract or the account
cannot see that series.

And a check, because the suite structurally cannot see what it guards. The
dispatcher hands a caller nothing at all for a figure the venue left out. A
wrapper method declaring that figure as a number refuses the call, and the
refusal leaves the reading loop and closes the session — on every caller that
did not write the method itself. Every test here writes its own, so the broken
default was never the one called. The check reads each call against the
signature it lands on and fails when one could be handed nothing it cannot take.
Reading what the venue states about a contract's company meant going through the
shared state and the reference store by hand — three hops and a type a caller
should not have to name. The two calls the Python surface has always had are on
the Rust client now, beside the other market data reads.

Read by the venue's id for the contract rather than by request number, because
what these state is a fact about the contract and outlives the subscription that
fetched it.
The venue states a contract's calendar — what the company has coming and when,
in which time zone — as the same runs of KEY=VALUE the rest of the company
series state, behind eight bytes of its own and compressed. It was the one of
these series this client did not read.

It reaches a caller through the same two calls the other sixteen do, under the
venue's own number, with the venue's own field names unchanged.

What it inflates to is bounded the way every other inflate here is: what arrives
is bounded on the wire and what it becomes is not. Bytes that are not what the
venue compressed publish nothing and leave what it last stated standing, rather
than replacing it with whatever they happen to inflate to.
…ng them

Five places said sixteen and listed the series without the calendar, which was
read after they were written. The count and the lists now match what the
dispatch reads.
What one contract states is kept for the life of the session and is not dropped
when the subscription that fetched it ends, because it is a fact about the
contract rather than about the watch. That was three series and is now
seventeen, so the cost per contract grew with it: about twenty-three kilobytes
for a share that states every one of them, held against contracts a session has
watched rather than the ones it is watching.

A program watching a few hundred contracts will not notice. One sweeping a
scanner across thousands is holding a couple of hundred megabytes of this, and
should be able to find that out from the call that returns it rather than from a
memory graph.
…he ceiling

Every job now states how long it may take. None did, so a step that stops making
progress sat until the six-hour ceiling and was killed there — and a job killed
that way keeps no log, so there was nothing to read afterwards to say where it
stopped. One of them has been doing exactly that.

Bounded, the same hang fails as a failure, and a failed job keeps its output.
The test runner names each test as it finishes, so the last one it named is the
one that did not.

The limits are what each job takes on a runner with room to spare, not what it
takes when it is healthy: the point is to catch a stall, not to fail slow work.
Fifty-nine series now reach a reader on an ordinary subscription, and the venue
is not the only thing that can put bytes in front of one. Each is handed an
empty record, records cut off at one, three and seven bytes, counts of nothing,
of minus one and of the largest a signed integer carries, a count that overruns
what arrived, and bytes that open like something compressed and are not.

None may panic, and none may hand a caller a figure that is not a number — the
venue states a figure it does not hold as the largest its field carries, and
that is its way of saying so rather than a reading.
The job limit added before does not do what it was meant to. A job that reaches
its limit is recorded as cancelled, and a cancelled job keeps no log at all —
so a run that stalls still says nothing about where it stalled. Two of them have
now been lost that way.

The bound belongs inside the step. A step that times out fails, the job fails
with it, and a failed job keeps every line it printed. The test runner names
each test as it finishes, so the last name in that log is the one that did not
finish — which is the whole question.

The job limits stay as a backstop, above the step limits so the step is always
the one that fires.
Bounding the step was not enough on its own. The runner kills the command it
started, which is cargo, and the binary cargo spawned to run the tests is a
generation further down — it survives, and it holds the pipe the step's output
is read through. So the step went on waiting for an end-of-output that the
orphan was never going to give it, and ran for an hour under a bound of
twenty-five minutes.

The output goes to a file instead, which severs that pipe: the step ends when
the bound fires, whatever the orphan does. What the file holds is printed
afterwards — all of it where the step failed, because that is the run worth
reading, and three lines where it passed.
Bytes the reader cannot find a header in are discarded, and what went was
written to the log as hex so a framing error could be told from a genuinely
malformed frame. The whole of it was written: this buffer holds whatever the
socket has delivered and not yet been read out of, so one discard of eight
megabytes built a string of sixteen million characters and wrote it as a single
line. Three of those in one run of the suite came to fifty megabytes of output.

Five hundred and twelve bytes of it now, with the size that went stated beside
it, which answers the same question. The longest line the suite produces falls
from sixteen million characters to one thousand two hundred, and its whole
output from about fifty megabytes to one.

It cost more than a large file. A reader that takes output a line at a time
stalls on a line that size, and the test step it was running under stalled with
it — for hours, against a suite that finishes in two minutes when its output
goes somewhere that does not care.
… read

Four more of the venue's series reach a caller, each read as its own handler
reads it.

The other news series states one story rather than a batch of them, and frames
every string of it with a count of its bytes and padding out to a multiple of
four. It reaches a caller on the callback the batch series already uses: it is
news about the contract, and which of the two carried it is not something a
caller asked about. Read against a live session, it carries stories the batch
series was not asked for.

Both news series now share one reading of a headline. The venue writes a marker
in braces in front of some of them and a caller is shown what follows it — which
the batch series did and the new one did not, so the same story reached a caller
in two spellings until they were made one.

The book the venue keeps for contracts dealt in size rather than on a screen is
stated in two forms, told apart by the record: an older one of two rows of a
count and a price, and a newer one of rows of a count and two prices behind a
version. A row of no quantity, or one withdrawn by stating its price as minus
one, is not a row the venue stands behind and is left out.

A scan states its strategies and the legs of each, and the legs are kept against
the strategy they belong to — the venue numbers neither, so the strategy's place
in the record is what names it. A leg sold reads as a negative size.

A scan's points are stated behind a version that says how much of the record is
there. Where that version is nought the six figures behind it are not in the
record at all, so a reader stepping over them anyway reads every figure after
them from the wrong place.
…e would not read

Five series were set aside as unreadable. They are not: the tool that produced
the source gave up on three classes, and the instructions those classes hold are
readable without it.

The moving averages the venue keeps over a contract's close state a number and a
figure in pairs from end to end — which is why the record's length is always a
multiple of eight, and which of two branches the build keeps is decided by that.
The first pair's number says a count follows and its figure is that count, read
whole; every pair behind it states its figure to four bytes. The reader's own
words for a record that does not start that way are what settled it.

Four more — a contract's quotes once dealing has closed, the same delayed, the
underlying's, and the request-for-quote stream — are written in the packed record
the quote stream itself uses, which this client has read all along. They are read
by that reader now, and kept as the record states them: the venue's own number
for each field, the figure, and how far its decimal point moves. No scale is put
on them, because which fields are prices and which are sizes is the one thing
only the unreadable class would say.

Two remain unread and will stay so for a reason rather than for want of trying.
One states no payload at all — its reader returns before touching the record.
The other hands the record to whichever component subscribed, and there are many
of those; there is no single layout to read, only the one belonging to a
subscription this client does not make.

And a test that failed about one run in eight, which took most of a day to
catch because it left nothing behind to identify it. Three tests spawn an engine
and then wait on a reply from it, and a test waits a millisecond by default:
under a suite running on every core that thread is not always scheduled inside
one, so the wait ended in a timeout whose code is not the one the engine refused
under. They wait long enough now. Verified across three runs with every core
busy, which is what provoked it.
…es are kept by

Both were read off a live session rather than left as the record's own numbering.

On the packed quotes the venue numbers its fields itself, and the numbers mean
the two sides of the quote and the size behind each. An option stated its
underlying's two sides on one of these series and the option's own on another,
and both matched what those contracts were quoted at to the penny. A side the
venue is not standing behind reads as minus one hundred. The figures are counted
in the contract's own increments, as every packed figure is, and no scale is put
on them here — which fields are prices and which are counts is the venue's to
change, and that is the one thing its own reader would say and this cannot read.

The moving averages are kept under the days they run over. A share around three
hundred and thirty stated twelve days at three hundred and twenty-three, fifty
at three hundred and fifteen and two hundred at two hundred and eighty-eight,
falling as the window lengthens the way a rising share's averages do. Two of
them are numbered below nought and are not days; nothing here says what they are.

The call a caller reads these from described one series and now carries all
three, on both surfaces.
…is read

The venue scans an underlying for combinations worth putting on. Nothing in the
documented API asks for that, and this client could not either.

The scan goes out as one run of named fields beside a subscription for the
series that answers it, which is how the venue takes it: the series carries the
answer and the scan says what to look for. A field the caller leaves unstated is
left out of the run entirely rather than sent empty.

What comes back is a version, an error where the venue refuses the scan rather
than answering it, how many it left out, and then the strategies — each with its
legs as a contract and a size apiece, a leg sold reading as a negative, which
shape of strategy it is and how pressing the venue takes it to be, thirteen
figures about it, where it comes out even at each price it can, and one figure
behind those. The thirteen are handed over in the order stated: what each one is
is the venue's, and it names them nowhere this client can read, so naming them
here would be naming them for it.

req_spread_scan and scanned_strategies, on both surfaces.

This is the one series that could not be read from the record alone. Its answer
has two shapes and which one arrives is decided by the kind of subscription it
answers, so reading it meant making that subscription rather than guessing which
shape to expect.
…n down

The venue declares a portfolio figure on a series of its own, and the plumbing
to subscribe it, and then nothing anywhere asks for it or says how to read what
it would answer with. Every class the terminal ships was searched: three mention
the reader that series would need — the reader itself, the handler that would
call it, and the thing that would subscribe it — and none of them is one.

So there is no layout to read here, not because the record is hard but because
nothing reads it. A message that does arrive is written into the record of what
this client cannot read, rather than passing in silence, so that it is visible
if the venue ever states one.

This was skipped before for a reason that was not true: that the record goes to
whichever component subscribed and there were many of those. There are none.
…lity is worth against its own past

The terminal keeps a register of the series it reads, and some are built there
rather than in a class of their own. Every inventory taken here was taken from
the classes, so those were never in it. Read from the register instead, fourteen
more are served and none of them needed a new reader.

Six say where a contract's volatility stands against its own history — its rank,
its percentile, and its high and low — for the volatility the market implies and
again for the one the contract has actually shown. They are stated as the same
two numbered tables the price extremes are, so the reader that already reads
those reads these. The number each figure is kept under is weeks, and the venue
keeps a quarter, a half year and a year; on the two that state a high and a low,
the window is written negative for the low and positive for the high. Read
against a live session, where a share stood at the same rank across all three
windows while a fund's moved with each.

Seven more state a single figure and one states a figure and a flag: a
volatility taken at the close, the close's own, a mark the venue keeps behind
the live one, and five it states under numbers of its own. All eight are layouts
the table of figures already had.

The documented API has no call for any of it.
…wn five

A second register builds five more series, all sharing one reader on the class
they descend from: a whole number and then a figure. Five volatilities taken at
or around the close.

And one more the venue compresses. It puts a single byte in front of what it
squeezed where the calendar puts eight, so the two differ in nothing but that
and are read together now.

What this does not read, and why, so nobody looks again:

- One states nothing at all: its reader logs that it cannot be served and
  returns before touching the record.
- One carries the dividends, line by line, each line tagged by a letter. This
  client already asks the venue for what a contract pays out and is answered
  with named fields; reading the same thing again as tagged letters would be
  worse than what a caller already has.
- Three are second numbers for series already read under their first.
…st not miss

Everything this client reads was already listed. What it does not read was in
the commit log and the guide and nowhere a reader would look, so somebody
pointing a program at a series that answers with nothing had no way to tell
whether it was the account, the contract, or a bug here.

It is stated now: the six series that are read and have never been seen, the
readers that have not met their instrument, the three that state no payload at
all, the dividends already served better elsewhere, the second numbers for
series read under their first, and the nine callbacks the venue states nothing
for. And the thing a reader most needs — an empty result means either the venue
holds nothing for that contract or the account cannot see the series, and this
client cannot tell them apart.

The page also uses the alert kinds the site renders, for the handful of things
that are worth stopping on: one session per login and the second factor, what a
migration does not have to change, the calls that will not run against a
gateway, and that this places orders against real money.

Written outside the block the matrix generator owns, and the generator was run
to confirm it leaves it alone.
Two pages described them as "the part of this client that is not a drop-in".
That reads as a hole in the thing, and it is the opposite of what is true: the
venue states all of it on an ordinary session, the documented API never gave it
a message, and so a program had no way to ask for a contract's float, where its
volatility stands against its own year, what margin it takes, or five years of
its price with the date of each point. Reaching those is why this exists.

What is actually true is narrower and worth saying once: the door opens one way.
A program moves to this client without changing a line; a program that then
calls one of these cannot move back to a gateway, because a gateway has no
message to carry it. That is the trade, and it is named where the calls are
named so it is visible before it is made.
It described what this is and listed what it carries, and stopped there. A
reader deciding whether to use it had no answer to why it exists, what it gets
them that a gateway does not, how it behaves when something drops, or what to
do when a series comes back empty.

Added, in the order someone reads them: why the arrangement it replaces costs
something; installation and a first program before three hundred lines of
tables rather than after; what the venue states that the documented API never
named, with the figures a session actually returned — a contract's float to the
share, where its volatility stands against its own year, five years of its
price with the date of each point, what one contract delivers; how the
connections are held and rebuilt; nine questions a reader arrives with, folded;
what the tests cover and how a reader of the venue's records is held to more
than passing; how to contribute something useful; and how not to leak an
account.

The alert kinds the site renders are used where stopping is worth it — one
session per login, the second factor, what a migration need not change, real
money — and the calls beyond the documented API are now framed as the reason
this exists rather than as a shortfall in it.

Everything stated is checked by what already checks this file: nothing here says
more than the client can see, every page and example named exists, and every
count matches what is there.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant