A resumable generator that turns HEY Research Lab's public API into research-friendly NDJSON snapshots of HEY's own builder records on Robinhood Chain: projects, ships, changes and evidence.
HEY's public API answers one project or one page at a time, which is right for an app and slow for
research. A notebook that asks "which projects shipped a release in September, and what did HEY
record about them?" wants the whole record in a file. hey-data walks the public API politely,
keeps only HEY's own builder facts with their provenance, and writes them as NDJSON with a
metadata.json that says what was read, when, and what was left out.
HEY researches Robinhood Chain (chain id 4663, CAIP-2 eip155:4663) and nothing else. Every
snapshot declares chainId: 4663; a record that names any other chain stops the run with
unsupported_chain rather than being written.
hey-data is a repository, not an npm package.
git clone https://github.com/hey-research-lab/hey-data.git
cd hey-data
corepack enable
pnpm install
pnpm buildNode 22 or newer. Runtime dependencies: @hey-research-lab/sdk (HEY's typed API client) and zod
(validates page envelopes and the resume cursor).
A quick snapshot without per-project detail reads (about 800 requests, roughly 15 minutes at the default pace):
pnpm generate --out snapshot --no-details --evidence-limit 0
head -n 1 snapshot/projects.ndjsonThe full snapshot reads each project's dossier and coverage too (about two requests per project; roughly three and a half hours for ~5,000 projects). Stop it at any time; the same command resumes.
pnpm generate --out snapshotThe test suite runs offline over recorded API answers: pnpm test.
| File | One line per | Read from |
|---|---|---|
projects.ndjson |
published project | GET /api/projects (partitioned by status), GET /api/projects/{slug}, GET /api/projects/{slug}/coverage |
ships.ndjson |
ship (release, code week, product ship) | GET /api/ships?sort=detected |
changes.ndjson |
change-ledger event, at its highest revision, retractions applied | GET /api/changes?after= from the start of the ledger |
evidence.ndjson |
evidence record referenced by a ship or change (bounded) | GET /api/evidence/{id} |
hooks.ndjson |
Uniswap v4 hook on Robinhood Chain that HEY's hooks page shows | hey-data hooks: HEY's daily /data/<day>/hooks.ndjson.gz, checked against its SHA256SUMS line |
metadata.json |
the snapshot | — |
builders.ndjson is not produced: HEY's only builder listing is the Builder Radar, a ranking
whose scores include an on-chain sub-score and a liquidity field. The builder facts outside that
ranking are already in projects.ndjson.
Every file has a field allowlist in code (src/allowlist.ts). Records are built by reading
those named fields from the API's JSON; nothing is passed through, so a field HEY adds later stays
out until someone adds it on purpose. Every line passes a final gate that fails the run if any key,
at any depth, is outside the allowlist or looks like a market, holder, usage, integrity or Radar
field. Field names and state words are the API's own.
- projects —
slug,name,url,symbol,shortDescription,projectKind,activityStatus,researchLevel,catalogStatus,lastShippedAt,primaryNarrative,narratives[],token {chainId, contractAddress},tokenVerification {status, reason},websiteUrl,officialX,logoUrl,hasBuilderSource,firstRecordedByHeyAt,latestShip {evidenceId, eventType, publishedAt, detectedAt, precision, verification}(the newest meaningful ship HEY lists),scoringVersion,scoredAt,coverage {<dimension>: {state, stateDetail, reason, cause, since, asOf}},coverageComputedAt, anddetailRead(read,dossier_only,not_foundwhen the project was unpublished between the listing and its detail read, orskippedwith--no-details). Coverage dimensions kept: identity, builderEvidence, repositories, releases, timeline, contractDeployment, contractSource, contractInterface, locks, officialDocs, apiDocs, sourceChanges, gitHost, package, securityContext. - ships —
evidenceId,eventType,title,publishedAt,detectedAt,precision,verification,sourceUrl,url,project {slug, name, url},codeWeek(a code week's fixed ISO week, repository and commit count) andcodeSubstance {verdict, classifierVersion, countsAsBuilding}. - changes —
id,revision,op,type,domain,origin,project,contract,token,occurredAt,occurredUntil,precision,detectedAt,recordedAt,summary,before,after,evidence[] {id, url, label},source,countsAsBuilding,facts(an allowlist of plain facts: ship type and verification, code-week counts, transition key and scoring version, ABI change counts, lock id and kind, site-change counts) andlinks. Types kept:build.*,contract.deployed,contract.followup_deployed,contract.implementation_changed,contract.source_verified,contract.source_unverified,contract.interface_changed,token.verification_changed,research.*,lock.*. Left out (and counted inmetadata.changes.excludedTypes): everymarket.*type,market_integrity.event,contract.usage_changed,contract.method_first_observed,contract.method_resumedandtoken.launch_stage_changed. - evidence —
id,withdrawn,withdrawalReason,contextReason,project,domain,claimType,summary,sourceType,sourceUrl,publishedAt,detectedAt,precision,verification,countsAsBuilding,recordedAt,metadata(release version, site-change counts) andsources[] {sourceType, sourceUrl, observedAt}. Only the familiesship,abi,impl,lock,source,claim,state,narrative,sourcechange,securityandv4hookare fetched, newest referenced first, up to--evidence-limit(default 500).
Price, market cap, FDV, valuation, liquidity, volume, trades, venue, token market status and launch stage; anything derived from them (Still Building, the Discovery Gap, Under the Radar); holder or distribution data; usage and caller counts; Market Integrity; the Builder Radar and Build Momentum; release notes and commit messages; anything keyed, partner-only or account-bound.
An absent field means HEY does not know it, or does not publish it. hey-data never fills a gap with
0, false, [], "" or "none". A null appears only where the API itself sends one with a
documented meaning (occurredAt: null with precision: "OBSERVED" for a change HEY observed but no
source dated). Coverage states (MEASURED, NO_SOURCE, NOT_ENOUGH_YET, STALE,
SOURCE_UNAVAILABLE, NOT_APPLICABLE, NOT_RESEARCHED, ERROR, WITHHELD) and activity states
(SHIPPING, ACTIVE, QUIET, DORMANT, RESUMED, UNKNOWN) are copied as the API words them.
A coverage entry also keeps HEY's additive stateDetail — MAPPING_BLOCKED under NO_SOURCE (HEY
holds a candidate source and nothing yet ties it to the project) or PARTIAL under MEASURED — and
cause on an unavailable or deferred read (own_source_unreachable, provider_unavailable,
budget_deferred) when HEY sends them; state keeps its meaning either way.
schemaVersion, generatedAt, startedAt, chainId (4663) and chain, source
(https://heyresearch.xyz), apiVersion, endpoints (every route read), recordCount per file,
files.*.readFrom/readUntil (a snapshot is read over a window, not at one instant), omitted,
checks (the totals checks below), changes (ledger cursor, excluded types, retractions applied),
evidence (referenced, eligible, limit, truncated), license (CC-BY-4.0), licenseUrl,
attribution, codeLicense, limitations[], disclaimer and nonAffiliation.
hey-data hooks --out snapshot copies HEY's newest daily hooks file into snapshot/hooks.ndjson and writes
snapshot/hooks.metadata.json (the source day and file, its checksum, the counts and the citation). It reads three
files: /data/latest/metadata.json for the day, then that day's SHA256SUMS and hooks.ndjson.gz, which never
change. A checksum that does not match, a key outside HOOK_RECORD, or a chain other than 4663 stops it. The public
API answers at most 100 hooks a call and counts factory instances, so the daily file is the complete record.
One line per hook: evidenceId (v4hook:4663:<address>), chainId, address, explorerUrl, receiptUrl,
callbacks (the PoolManager callbacks the address declares, comma-separated), tied; for a hook HEY ties to a
published project project {slug, name, url}, basis (chain, project, linked, factory, review),
basisLabel, tie, factory {address, explorerUrl} for a factory instance, deploymentEvidenceId and reviewedAt;
for any hook hooklist {name, url} when Uniswap's hooklist lists it, deployedAt, deployedBlock, firstPoolBlock,
firstPoolAt. The account that created a hook and pool counts are never in the file. How HEY ties a hook:
heyresearch.xyz/integrations/uniswap/hooks#method.
hey-data huggingface --out hf writes HEY's newest daily projects.csv, ships.csv, hooks.csv and metadata.json
(each checked against the day's SHA256SUMS, every CSV column against the allowlist) with a dataset card. The
Hugging Face copy workflow runs it every Monday at 03:47 UTC and uploads the result to
huggingface.co/datasets/heyresearch/robinhood-chain-builders.
It needs the repository secret HF_TOKEN (a fine-grained token with write access to that dataset only); without it
the run warns and copies nothing.
hey-data generate [options]
hey-data hooks [--out <dir>] [--quiet]
hey-data huggingface [--out <dir>] [--quiet]
--out <dir> output directory (default ./snapshot)
--delay-ms <n> pause between requests, at least 600 (default 1100)
--max-retries <n> retries of a 429, 5xx, network failure or timeout, 0-5 (default 1)
--evidence-limit <n> evidence records to fetch, 0 for none (default 500)
--no-details listing fields only: no per-project dossier or coverage reads
--no-coverage skip per-project coverage reads
--list-attempts <n> walks of a listing before a totals mismatch fails the run (default 3)
--max-resume-age <h> refuse to resume a partial run older than this (default 48)
--restart discard a partial run in the output directory
--keep-work keep the .work directory after a finished run
--quiet no progress lines on stderr
| Exit | Meaning |
|---|---|
| 0 | snapshot written |
| 1 | a totals check, the schema gate or the work state failed (nothing new is written) |
| 2 | usage error |
| 3 | HEY refused a request (4xx) |
| 4 | an unexpected not-found |
| 5 | rate limited past the retry budget — rerun the same command later to resume |
| 6 | network failure, timeout or HEY 5xx — rerun the same command to resume |
HEY's anonymous limit is 120 requests a minute per client. hey-data sends one request every
--delay-ms (1,100 ms by default, about 55 a minute; the CLI refuses less than 600 ms), honours
retry-after on a 429 or 503, and retries a retryable answer at most --max-retries times (a 429;
a 5xx, including the 520–524 errors the CDN in front of HEY answers when the origin errs; a network
failure or a timeout — every request is a GET). A 429 waits what the server asks; a server error or a
network failure backs off 30 s, 60 s, 120 s, 240 s, then 300 s, or longer when the server asks. It never follows a redirect and caps every response body at 8 MB. It sends no key.
Partial files and a cursor live in <out>/.work. Every page is appended before the cursor records
its new length, so a run stopped at any point resumes from its last whole page; bytes written past
the cursor are truncated first. A partial run older than --max-resume-age hours, or one started
with different options, is refused rather than stitched together. The finished files are written
atomically (temporary file, then rename) and .work is removed.
- Projects by status. The catalogue is walked once per activity status. Each partition must be
walked to the total the API reported for it, the total must not move during the walk, no project
may appear in two partitions, and the partitions must add up to the unfiltered total read before
and after the walk. A mismatch walks again, up to
--list-attempts; then the run exits 1 and names each discrepancy. A new status word from HEY, or a partition past the listing's offset cap, fails the same way instead of producing a short file. - Ships. Distinct ships collected must equal the feed's total, unchanged during the walk.
ci.yml— on push and pull request: leak scan, lint, typecheck, tests (offline), build.snapshot.yml— weekly, and by hand. Every Monday at 02:17 UTC it builds the full snapshot (dossiers, coverage, 500 evidence records), copies HEY's daily hooks file (hey-data hooks), and a separate job (the only one withcontents: write) attaches it to a GitHub Release taggeddata-<date>-<run id>, withLICENSE-DATA, a.tar.gzandSHA256SUMS— so a wholesale, citable dataset exists at a stable URL under Releases. A manual run (workflow_dispatch) takes the inputs above and attaches a release only withreleaseset. It runs with--max-retries 5; when HEY is still unreachable or rate-limiting after those, it waits 15 minutes and resumes from its own partial work, up to twice. A run that still stops uploads its.workashey-data-partial(pass that run's id asresume_from_runto continue it), and a scheduled one opens or comments on the issue "Weekly snapshot did not finish".
Snapshots are never committed to this repository.
HEY also publishes a daily snapshot of the same records, under the same licence and allowlist, at
heyresearch.xyz/data: projects.ndjson.gz, ships.ndjson.gz, that
UTC day's changes.ndjson.gz (one day of the public change ledger in seq order), hooks.ndjson.gz
(every Uniswap v4 hook its hooks page shows), uncapped projects.csv, ships.csv and hooks.csv, and a metadata.json naming the rule versions and the citation
("HEY Research Lab, , retrieved "). /data/latest/<file> is always the newest
day; thirty dailies and the first of every month are kept. The daily files carry no per-project
coverage and no per-project scoring fields; this generator reads both. One allowlist decides what
either may hold; a field added to one is added to the other on purpose.
examples/duckdb.sql— DuckDB SQL over the NDJSON files.examples/pandas_duckdb.py— Python with pandas and DuckDB.examples/node.mjs— Node.js, no dependencies.examples/curl-jq.sh— jq over the files, and curl for the same public routes.
hey-data reads only HEY's public, keyless read API at https://heyresearch.xyz (documented at
https://heyresearch.xyz/docs/public-api) through @hey-research-lab/sdk. It writes nothing to HEY,
calls no other host, and never calls a provider. A snapshot restates HEY's records; HEY's own
pages and API stay the authoritative, current source, and every record carries the HEY url or
evidence id it came from.
The datasets restate HEY's public records as of the snapshot time. An absent field means HEY does not know it. The data is not investment advice, carries no safety verdict, and market data is deliberately excluded.
HEY Research Lab is an independent research project and is not affiliated with, endorsed by or partnered with Robinhood Markets, Inc. or Robinhood Chain.
See SECURITY.md. hey-data makes HTTPS GET requests to https://heyresearch.xyz
only (HEY_BASE_URL exists for local tests: an https origin, or http on localhost), never follows
redirects, caps response bodies, treats every API text as data (bidirectional-override and
zero-width characters are removed; links are kept exactly as HEY recorded them, so check a
link's scheme before rendering it), reads only allowlisted keys (so __proto__ and friends are
never copied), and writes only fixed file names inside --out, refusing symbolic links.
See CONTRIBUTING.md. pnpm lint, pnpm typecheck, pnpm test, pnpm build
and pnpm scan must pass; tests never touch the network. Adding a field to an allowlist is a
product decision, not a refactor.
- Code: MIT — see LICENSE.
- Data (snapshot files produced by this generator): Creative Commons Attribution 4.0 International (CC BY 4.0) — see LICENSE-DATA. Attribution: HEY Research Lab, https://heyresearch.xyz.
HEY Research Lab, https://heyresearch.xyz. HEY data snapshot (generated <generatedAt from metadata.json>). Licensed under CC BY 4.0.
Keep the snapshot's generatedAt, link https://heyresearch.xyz, say whether you changed the
data, and keep each record's HEY url or evidence id with anything you republish from it.