From 4d9854669664433ff7796c188d103d9580d32009 Mon Sep 17 00:00:00 2001 From: Stefan Jansen Date: Wed, 23 Sep 2026 15:30:30 -0400 Subject: [PATCH 1/3] docs: add market data and alternative data source pages Market data: the vendor landscape for US equities and ETFs (daily panels and trade, quote and order-book data), futures, options, FX and crypto, with delisted coverage, adjustments, granularity, point-in-time index membership, access tier and ml4t-data support. Wrapped providers link to the existing provider comparison instead of repeating it. Alternative data: news and text, social, web and app, workforce, card transactions, foot traffic and satellite, ESG, supply chain, prediction markets and marketplaces, with history, what is documented about point-in-time behavior, access tier and ml4t-data support. Vendor facts verified September 2026; every external link checked. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01F6TR4nQDjiPtanA2n4F3Gw --- docs/providers/alternative_data.md | 210 +++++++++++++++++++++++++++++ docs/providers/index.md | 4 + docs/providers/market_data.md | 175 ++++++++++++++++++++++++ mkdocs.yml | 2 + 4 files changed, 391 insertions(+) create mode 100644 docs/providers/alternative_data.md create mode 100644 docs/providers/market_data.md diff --git a/docs/providers/alternative_data.md b/docs/providers/alternative_data.md new file mode 100644 index 0000000..c61696b --- /dev/null +++ b/docs/providers/alternative_data.md @@ -0,0 +1,210 @@ +# Alternative Data Sources + +This page maps sources of alternative data: news and text, social and web attention, app and +web usage, job postings, card transactions, foot traffic, satellite and shipping data, ESG +ratings, supply-chain relationships, and prediction markets, plus the marketplaces where such +datasets are found. For each source it lists what the data measures, how far back it goes, what +is known about its point-in-time behavior, the access tier, and whether `ml4t-data` wraps it. + +!!! note "Verified September 2026" + Alternative data vendors are acquired, merged and repackaged often. Every entry below was + checked against the vendor's own site or a published announcement in September 2026. Few + vendors publish their timestamp and revision policy; where it is not public, the table says + "not documented" and the question belongs in your due diligence. Confirm the current terms + before you build on any of them. + +--- + +## Choosing a Source + +### When was each record first observable? + +A backtest is only valid if every record is used after the time it could first have been seen. +Ask the vendor for the timestamp that marks first availability, not the event date, and whether +history was backfilled when the dataset launched or when coverage expanded. A backfilled history +looks continuous but was assembled later, often by a method that did not exist at the time. + +### Revisions and methodology changes + +Panels change composition (card and device panels gain and lose users), models are re-estimated, +and rating methodologies are revised. ESG ratings are the clearest case: providers disagree with +each other and revise their own methods over time. Ask whether old values are restated when the +method changes and whether earlier vintages are kept. + +### Coverage and representativeness + +Check which names, sectors and regions a dataset actually covers, and whether coverage is stable +across the sample. A signal that exists only for large US consumer companies is a narrower +signal than its marketing suggests. + +### Legal and privacy + +Provenance, permitted use and redistribution rights are hard constraints. Narrow panels can carry +material non-public information, and granular location or device data carries privacy risk even +without names. These questions are answered before any backtest, not after. + +### In the book + +- Chapter 4, "Understanding alternative data" (section 4.4): "The alternative data landscape", + "The evaluation framework", "Build versus buy for alternative data", and "Data sources and + implementation". +- Chapter 4, "Using text data for NLP features" (section 4.5): extracting and storing text from + filings point-in-time. Chapter 10 covers turning text into features. +- Chapter 2, "A modern taxonomy of financial data" (section 2.1), subsection "Alternative data - + High variety, high validation burden". + +--- + +## News, Text and Sentiment + +| Source | What it measures | History | Point-in-time and revisions | Access | ml4t-data | +|--------|------------------|---------|-----------------------------|--------|-----------| +| [GDELT Project](https://gdeltproject.org/data.html) | Global news events and tone (Event database); themes, entities and sentiment from full text (Global Knowledge Graph) | Events from 1979; Knowledge Graph from April 2013 | Published in 15-minute batches; the batch time is a usable first-availability time. Older history was built retrospectively. | Free and open | No | +| [RavenPack](https://www.ravenpack.com/) | Entity-level news analytics: relevance, sentiment and event classification | From 2000 (Dow Jones edition), 2007 (web edition) | Records carry the news timestamp; revision policy not documented publicly | Institutional; academic via WRDS | No | +| [LSEG MarketPsych Analytics](https://www.lseg.com/en/data-analytics/market-data/quantitative-economic-data-solutions/marketpsych-analytics-and-models) | Sentiment and emotion scores from news and social media across equities, macro, FX, crypto and commodities | From 1998 | Minute-level to daily series; revision policy not documented publicly | Institutional | No | +| [Dow Jones Factiva DNA](https://www.dowjones.com/business-intelligence/factiva/) | Licensed premium news archive with metadata, via API, snapshots and streams | Archive depth varies by publication | Publication timestamps per article | Institutional | No | +| [Benzinga News API](https://www.benzinga.com/apis/cloud-product/stock-news-api/) | Financial news headlines and articles, Wilshire 5000 and TSX coverage | Not documented | `created` and `updated` timestamps per article | Paid, licensed by news channel | No | +| [SEC EDGAR full-text search](https://www.sec.gov/edgar/search/efts-faq.html) | Full text of SEC filings | From May 2001 | Every filing carries its acceptance timestamp | Free | No | + +--- + +## Social Media and Web Attention + +Access to social-media data has narrowed since 2023, so check the current terms before planning +a project around any of these. + +| Source | What it measures | Status | Access | ml4t-data | +|--------|------------------|--------|--------|-----------| +| [Stocktwits API](https://api.stocktwits.com/developers) | Investor message stream with cashtags and self-reported sentiment | Closed to new developer registrations | Not open to new developers | No | +| [Reddit Data API](https://support.reddithelp.com/hc/en-us/articles/16160319875092-Reddit-Data-API-Wiki) | Posts and comments, including investing subreddits | Paid for commercial use since June 2023 | Commercial license required | No | +| [X API](https://docs.x.com/x-api/getting-started/pricing) | Posts and engagement | Pay-per-use credits since February 2026, replacing subscription tiers | Paid | No | + +--- + +## Web, App and Workforce Data + +| Source | What it measures | History | Point-in-time and revisions | Access | ml4t-data | +|--------|------------------|---------|-----------------------------|--------|-----------| +| [Similarweb](https://www.similarweb.com/corp/ourdata/) | Website and app traffic and engagement estimates | About 10 years | Modeled estimates; revision policy not documented publicly | Free tools, paid, enterprise | No | +| [Sensor Tower](https://sensortower.com/product/mobile-app/app-performance-insights) | App downloads, revenue, rankings and usage estimates; includes the former data.ai | Not documented | Modeled estimates; revision policy not documented publicly | Enterprise; free top charts | No | +| [Thinknum](https://www.thinknum.com/) | Web-collected company data: job listings, headcount, product prices, store locations | Not documented | Not documented | Institutional | No | +| [YipitData](https://www.yipitdata.com/) | Company KPI and revenue estimates from web, app and transaction data | Not documented | Not documented | Institutional | No | +| [Revelio Labs](https://www.reveliolabs.com/data) | Workforce composition and flows, job postings, employee sentiment, layoff notices | Workforce data from 2007 to 2008; postings from 2021 | Not documented | Paid; academic via WRDS | No | +| [LinkUp](https://www.linkup.com/data) | Job postings collected directly from employer websites | From 2007 | Each posting is timestamped when found, updated and removed | Institutional | No | + +--- + +## Card Transactions + +Card panels measure the spending of a sample of consumers. Panel composition changes over time, +so ask how the vendor weights the panel and whether history is reweighted when it changes. + +| Source | What it measures | History | Access | ml4t-data | +|--------|------------------|---------|--------|-----------| +| [Bloomberg Second Measure](https://secondmeasure.com/) | US consumer card spending by merchant and company | 8+ years; 2 to 7 day reporting lag | Bloomberg Terminal and enterprise | No | +| [Consumer Edge Transact](https://www.consumeredge.com/products/transact/) | Card transactions by merchant, company and demographic; now includes Earnest Analytics | About 9 years | Institutional | No | +| [Facteus](https://facteus.com/) | Debit and credit card transactions, synthesized for privacy | Not documented; about 1-day lag | Institutional feeds (API, AWS, Snowflake) | No | + +--- + +## Foot Traffic, Satellite and Shipping + +| Source | What it measures | History | Access | ml4t-data | +|--------|------------------|---------|--------|-----------| +| [Placer.ai](https://www.placer.ai/products/api) | Visits to stores and venues from a mobile-device panel | From January 2017 | Free tools, paid platform, enterprise API | No | +| [Advan Research Patterns+](https://advanresearch.com/products/patternsplus) | Foot traffic to points of interest; took over SafeGraph's Patterns product | Weekly data from January 2017 | Institutional | No | +| [Planet Labs](https://www.planet.com/industries/education-and-research/) | Satellite imagery: near-daily medium resolution and tasked high resolution | Archive from 2009 | Commercial; education and research program for university users | No | +| [RS Metrics](https://rsmetrics.com/) | Asset-level signals from satellite imagery: metal stockpiles, industrial activity | Operating since 2009 | Institutional; academic program | No | +| [Kpler](https://www.kpler.com/product/commodities) | Commodity flows from ship tracking (AIS), satellite, customs and port data; now owns MarineTraffic and Spire Maritime | MarineTraffic AIS archive from 2010 | Institutional | No | + +--- + +## ESG + +ESG ratings disagree across providers and change when methodologies change. Treat a rating +history as a sequence of vintages, and check which methodology produced each value. + +| Source | What it measures | History and methodology | Access | ml4t-data | +|--------|------------------|-------------------------|--------|-----------| +| [MSCI ESG Ratings](https://www.msci.com/data-and-analytics/sustainability-solutions/esg-ratings) | Industry-relative ESG ratings (AAA to CCC), about 17,000 issuers | Ratings since 2007; methodology revised several times since | Institutional; free public lookup for a subset | No | +| [Morningstar Sustainalytics ESG Risk Ratings](https://www.sustainalytics.com/esg-data) | Unmanaged ESG risk score (0 to 100), 16,000+ companies | A score change log exists; vintage policy not documented publicly | Institutional | No | +| [LSEG ESG Scores](https://www.lseg.com/en/media-centre/press-releases/2026/lseg-launches-new-suite-esg-scores-sustainability-analytics) | ESG scores from 220+ indicators, plus controversies | New suite launched March 2026; the earlier Refinitiv-era scores are being retired | Institutional | No | +| [S&P Global ESG Scores](https://www.marketplace.spglobal.com/en/datasets/s-p-global-esg-scores-(171)) | Scores from the annual Corporate Sustainability Assessment, about 13,000 companies | Annual assessments; history from 2013 | Free public scores for a subset; full data paid | No | +| [RepRisk](https://www.reprisk.com/insights/resources/methodology) | Reputational and conduct risk from news and stakeholder sources | Daily series with a consistent methodology since January 2007 | Institutional | No | + +--- + +## Supply Chain and Trade + +| Source | What it measures | History | Access | ml4t-data | +|--------|------------------|---------|--------|-----------| +| [FactSet Supply Chain Relationships](https://www.factset.com/marketplace/catalog/product/factset-supply-chain-relationships) (formerly Revere) | Customer, supplier, partner and competitor links between companies | Not documented | Institutional; academic via WRDS | No | +| [Bloomberg Supply Chain (SPLC)](https://professional.bloomberg.com/institutions/corporations/supply-chain/) | Supplier and customer relationships with revenue and cost exposure, 100,000+ companies | From 2006 | Bloomberg Terminal and enterprise feed | No | +| [S&P Global Panjiva](https://www.marketplace.spglobal.com/en/datasets/panjiva-supply-chain-intelligence-(22)) | Shipment-level customs records, 2 billion+ records | US data from 2007 | Limited free search; full data institutional | No | +| [ImportGenius](https://www.importgenius.com/pricing) | US and international bill-of-lading records | US imports from 2006, exports from 2014 on the full tier | Retail paid tiers | No | + +--- + +## Prediction Markets + +Prediction-market prices are market data: each trade and quote has an exchange timestamp and is +never revised. The caveats are thin liquidity outside headline contracts and markets that are +listed and resolved over time, so a set of markets picked today is a survivor sample. + +| Source | What it measures | History | Access | ml4t-data | +|--------|------------------|---------|--------|-----------| +| [Kalshi](https://docs.kalshi.com/welcome) | CFTC-regulated event contracts: economics, politics, weather, sports | From the 2021 launch | Public market-data API; trading needs a verified account | Yes: [`KalshiProvider`](kalshi.md) | +| [Polymarket](https://docs.polymarket.com/) | Event contracts on a crypto platform; a CFTC-regulated US exchange (QCX) relaunched in 2025 to 2026 | From 2020 | Public market data | Yes: [`PolymarketProvider`](polymarket.md) | + +--- + +## Marketplaces and Discovery + +Marketplaces simplify procurement; they do not validate point-in-time behavior or methodology. + +| Source | What it offers | Access | +|--------|----------------|--------| +| [Nasdaq Data Link](https://data.nasdaq.com/) (formerly Quandl) | Catalog of financial and alternative datasets from 400+ publishers, some free | Free account for free datasets; paid for premium | +| [Snowflake Marketplace](https://docs.snowflake.com/en/collaboration/collaboration-marketplace-about) | Datasets shared into a Snowflake account and queried in place | Free, trial and paid listings | +| [AWS Data Exchange](https://aws.amazon.com/data-exchange/) | Datasets delivered into AWS, billed through AWS | Per-product subscriptions | +| [Eagle Alpha](https://www.eaglealpha.com/) | Alternative data discovery and advisory, 2,500+ dataset profiles | Institutional | +| [Neudata](https://www.neudata.co/) | Alternative data discovery and research, about 2,800 datasets tracked | Institutional | +| [Datarade](https://datarade.ai/platforms) | Directory of data providers across industries | Free to browse; each provider sets its own terms | + +Of the free datasets that made Quandl popular, the WIKI prices table is frozen at April 2018 +(`ml4t-data` wraps it as [Wiki Prices](wiki_prices.md)), and the Zillow real-estate database is +no longer in the catalog. + +--- + +## Alternative Data in ml4t-data + +`ml4t-data` wraps two prediction-market sources, [Kalshi](kalshi.md) and +[Polymarket](polymarket.md). The FXMacroData provider also exposes news, risk-sentiment and CFTC +positioning endpoints alongside its release-timestamped macro data. Everything else on this page +needs its own client. + +--- + +## Name Changes and Closures + +| Then | Now | +|------|-----| +| data.ai (formerly App Annie) | Part of Sensor Tower since March 2024 | +| Earnest Analytics | Part of Consumer Edge since 2025 | +| SafeGraph Patterns (foot traffic) | Sold to Advan Research; SafeGraph now sells places, geometry and spend data | +| Orbital Insight | Acquired by Privateer, April 2024 | +| MarineTraffic | Acquired by Kpler, March 2023 | +| Spire Maritime | Acquired by Kpler, April 2025 | +| Refinitiv ESG scores | Replaced by the LSEG ESG Scores suite, March 2026 | +| Reddit and X APIs | Paid access (Reddit since June 2023; X pay-per-use since February 2026) | +| Stocktwits API | Closed to new developers | +| Polymarket (outside the US since 2022) | US exchange via the 2025 acquisition of CFTC-licensed QCEX | +| PredictIt | Operating under an amended CFTC no-action letter (2025) | +| Quandl | Nasdaq Data Link (September 2021) | + +## See Also + +- [Provider comparison](index.md) +- [Market data sources](market_data.md) +- [Fundamental data sources](fundamentals.md) diff --git a/docs/providers/index.md b/docs/providers/index.md index da77dc0..5a76718 100644 --- a/docs/providers/index.md +++ b/docs/providers/index.md @@ -2,6 +2,10 @@ ML4T Data supports 20+ live and specialized data providers, plus synthetic and testing providers. +For the wider vendor landscape, including sources the library does not wrap, see +[Market Data Sources](market_data.md), [Fundamental Data Sources](fundamentals.md) and +[Alternative Data Sources](alternative_data.md). + ## Provider Comparison | Provider | Asset Class | Free Tier | Async | API Key | diff --git a/docs/providers/market_data.md b/docs/providers/market_data.md new file mode 100644 index 0000000..547c181 --- /dev/null +++ b/docs/providers/market_data.md @@ -0,0 +1,175 @@ +# Market Data Sources + +This page maps the market data vendor landscape: US equities and ETFs (daily and intraday), +futures, options, FX and crypto, including the institutional and academic sources that +`ml4t-data` does not wrap. For the providers the library does wrap (free tiers, async support, +credentials), the [provider comparison](index.md) is the reference; this page links to it rather +than repeating it. + +!!! note "Verified September 2026" + Vendor coverage, history, access terms and ownership change often. Every entry below was + checked against the vendor's own documentation or a published announcement in September 2026. + Where a vendor does not document a property, the table says "not documented" instead of + guessing. Confirm the current terms before you build a dataset on any of them. + +--- + +## Choosing a Source + +### Survivorship + +A research universe must contain the securities that later delisted, merged or went bankrupt. +Free price sources usually carry only what trades today, so any cross-sectional backtest built +on them overstates returns. The "Delisted" columns below say whether a vendor keeps dead +securities. + +### Adjustments and corporate actions + +Splits, dividends and spin-offs break raw price series. Vendors differ in what they deliver: +unadjusted prices, adjusted prices, or both with the adjustment factors. Keep the unadjusted +series and the factors if you can: adjusted history is rewritten every time a new corporate +action occurs, so an adjusted series downloaded today differs from one downloaded last year. +Futures have the equivalent problem at each roll; a continuous contract is a construction choice, +not a market price. + +### Point-in-time universe membership + +An index strategy must trade the constituents as of each date, not today's members. Only a few +vendors sell historical index membership; without it, "S&P 500 stocks" silently means today's +survivors. + +### Granularity and timestamps + +Daily bars, minute bars, trades and quotes, and full order-book messages are different products. +For US equities, a consolidated (SIP) feed and a single-venue feed (for example IEX only) report +different volumes and prices. Check which timestamp a vendor records (exchange, SIP or receipt) +and how it defines the trading session. + +### Licensing + +Exchange data carries its own license terms. Redistribution, display and derived-data rights +vary by vendor and tier, and a dataset licensed for research may not be licensed for production. + +### In the book + +- Chapter 2, "The asset-class market data landscape" (section 2.2): what each asset class lets + you observe, its failure modes, and the engineering decisions it forces. +- Chapter 2, "A due diligence framework for data sourcing" (section 2.3): survivorship, + corporate actions, identifier integrity, and a vendor checklist. +- Chapter 3, "The anatomy of modern market data feeds" (section 3.2) and "Microstructure data + quality and sessionization" (section 3.6): feeds, timestamps and session handling for intraday + data. + +--- + +## Providers ml4t-data Wraps + +Yahoo Finance, Alpaca, EODHD, Tiingo, Twelve Data, Massive (formerly Polygon.io), Finnhub, +Databento, Oanda, Binance, Binance Public, OKX, CoinGecko, CryptoCompare, Wiki Prices and NASDAQ +ITCH samples all have `ml4t-data` providers. Their asset classes, free tiers and credentials are +in the [provider comparison](index.md) and on each provider's page. Three of them appear in the +tables below as well, because they are also the reference source for a data class: Databento +(order-book and CME data), Alpaca (US equities with a stated history start) and the NASDAQ ITCH +samples. + +--- + +## US Equities and ETFs: Daily Research Panels + +| Source | Coverage and history | Delisted | Adjustments | Finest granularity | Point-in-time index membership | Access | ml4t-data | +|--------|----------------------|----------|-------------|--------------------|--------------------------------|--------|-----------| +| [CRSP US Stock Databases (on WRDS)](https://wrds-www.wharton.upenn.edu/pages/about/data-vendors/center-for-research-in-security-prices-crsp/) | NYSE from December 1925, NASDAQ from December 1972 | Yes, keyed by PERMNO and PERMCO | Returns with and without dividends; full corporate-action history | Daily and monthly | Yes, index constituent files on WRDS | Academic and institutional, mainly via WRDS | No | +| [Norgate Data](https://norgatedata.com/data-content-tables.php) | US, Australian and Canadian stocks and ETFs; US stocks essentially complete from 1992 | Yes, on the higher tiers | Four modes: unadjusted, capital-reconstruction adjusted, plus special distributions, total return | Daily | Yes, on the higher tiers | Retail paid subscription | No | +| [Sharadar Equity Prices (SEP)](https://data.nasdaq.com/databases/SEP) | About 21,000 US equities and funds, from 1998 | Yes | Unadjusted, split-adjusted, and split, dividend and spin-off adjusted | Daily | Separate S&P 500 constituents table | Paid; direct at sharadar.com since July 2026, or Nasdaq Data Link | No | +| [FirstRate Data](https://firstratedata.com/about/FAQ) | 16,000+ US tickers from 2000, including 7,000+ delisted | Partly: included where the vendor could source them | Unadjusted, split-adjusted, and split and dividend adjusted | 1-minute bars | No | Retail paid, one-off purchase | No | +| [Kibot](https://www.kibot.com/faq.html) | US stocks and ETFs; minute bars from 1998 | Partly: active and delisted lists, stated as incomplete | Split and dividend adjusted series plus an adjustments API | Tick with bid and ask | No | Free guest access (daily only), paid tiers | No | +| [Alpaca](https://docs.alpaca.markets/us/docs/about-market-data-api) | US equities and ETFs, from 2016 | Not documented | `adjustment` parameter: raw, split, dividend or all | Trades and quotes | No | Free (IEX feed, delayed SIP); paid for real-time SIP | Yes: `AlpacaProvider` | + +The free wrapped sources (Yahoo Finance, Tiingo, EODHD free tier) carry currently listed +securities and vendor-adjusted prices; treat them as prototyping data, not as a survivorship-free +panel. Wiki Prices stopped updating in April 2018. + +--- + +## US Equities: Trades, Quotes and Order Books + +| Source | Coverage and history | Delisted | Adjustments | Finest granularity | Point-in-time index membership | Access | ml4t-data | +|--------|----------------------|----------|-------------|--------------------|--------------------------------|--------|-----------| +| [NYSE TAQ](https://wrds-www.wharton.upenn.edu/pages/about/data-vendors/nyse-trade-and-quote-taq/) (Daily TAQ) | All US exchanges from 1993; millisecond timestamps from 2003 | Not applicable: daily files of every security traded that day | Raw prices; adjust with CRSP factors | Trades and NBBO quotes | No | Academic and institutional via WRDS | No | +| [AlgoSeek](https://algoseek.com/us-equities-package/) | About 27,500 US securities from 1998 (SIP); adjustment factors from 2007 | Yes | Raw and adjusted prices plus adjustment-factor files | Trades and quotes with nanosecond timestamps; bars from 1 second | Yes, Index Components dataset | Institutional; free sandbox with up to one year of data | No | +| [Databento](https://databento.com/datasets/XNAS.ITCH) | US equities across exchanges and ATSs; Nasdaq TotalView-ITCH from 2018 | Not applicable: raw exchange feeds | Unadjusted; corporate actions sold as separate reference data | Full order book (market by order) | No | Usage-based; see the [Databento page](databento.md) | Yes: `DataBentoProvider` | +| [LSEG Tick History](https://www.lseg.com/en/data-analytics/market-data/data-feeds/tick-history) (formerly Refinitiv Tick History) | Cross-asset, 500+ venues, from 1996 | Not documented | Not documented | Trades, quotes and order-book depth | No | Institutional; some universities license it | No | +| [Nasdaq TotalView-ITCH samples](https://emi.nasdaq.com/ITCH/Nasdaq%20ITCH/) | Selected full-day files for Nasdaq-listed stocks | Not applicable | Raw messages | Every order add, cancel and execution | No | Free samples; the full archive is a paid Nasdaq product | Yes: [NASDAQ ITCH provider](nasdaq_itch.md) | +| [DTN IQFeed](https://www.iqfeed.net/dev/) | Equities, futures, options, FX; tick history rolling 180 days, minute bars from 2005 to 2007 depending on asset | Not documented | Not documented | Trades and quotes | No | Retail paid | No | +| [Kinetick](https://ninjatrader-live.ninjatrader.com/support/helpguides/nt8/data_by_provider.htm) | Market data service for NinjaTrader users; equities, futures, FX | Not documented | Not documented | Tick | No | Retail; free end-of-day tier | No | + +--- + +## Futures + +| Source | Coverage and history | Expired contracts | Continuous series | Finest granularity | Access | ml4t-data | +|--------|----------------------|-------------------|-------------------|--------------------|--------|-----------| +| [CME DataMine](https://www.cmegroup.com/datamine.html) | CME, CBOT, NYMEX and COMEX futures and options; history varies by dataset | Not documented | No, individual contracts | Market depth and time and sales | Paid per dataset; academic discount | No | +| [Databento CME Globex (GLBX.MDP3)](https://databento.com/datasets/GLBX.MDP3) | CME Group futures and options from June 2010 | Yes, per contract | Continuous symbology by roll rule | Full order book | Usage-based | Yes: `DataBentoProvider.fetch_continuous_futures()` | +| [Norgate Data](https://norgatedata.com/data-content-tables.php) | Global futures and cash commodities | Yes | Back-adjusted continuous contracts | Daily | Retail paid | No | +| [FirstRate Data](https://firstratedata.com/about/FAQ) | Futures, about 15 years intraday | Not documented | Unadjusted, absolute-adjusted and ratio-adjusted continuous series | 1-minute bars | Retail paid | No | + +--- + +## Options + +| Source | Coverage and history | Delisted underlyings | Analytics | Finest granularity | Access | ml4t-data | +|--------|----------------------|----------------------|-----------|--------------------|--------|-----------| +| [OptionMetrics IvyDB](https://optionmetrics.com/united-states/) | US equity and index options from 1996; Europe from 2002 | Yes | Implied volatility, Greeks, volatility surfaces; adjusts for dividends, splits and spin-offs | Daily; a separate intraday product exists | Academic via WRDS, institutional | No | +| [ORATS](https://orats.com/data-api) | US equity, ETF and index options (no futures options); end of day from 2007, 1-minute from August 2020 | Yes | Smoothed implied volatility and Greeks | 1-minute snapshots | Retail paid, enterprise | No | +| [Cboe DataShop](https://datashop.cboe.com/documentation) (LiveVol) | Options quotes, trades and end of day from 2010; VIX futures from 2004 | Not documented | Calculated implied volatility and Greeks in some datasets | Trades and quotes | Paid, with trial | No | +| [Databento OPRA](https://databento.com/datasets/OPRA.PILLAR) | All US equity options from March 2023 | Not applicable | None: raw feed | Trades and quotes | Usage-based | Yes: `DataBentoProvider.fetch_option_chain()`, `fetch_option_quotes()` | +| [AlgoSeek](https://algoseek.com/us-equities-package/) | US options as an add-on to equities | Not documented | Vendor-computed analytics on some products | Trades and quotes | Institutional | No | + +--- + +## Foreign Exchange + +Spot FX trades over the counter, so there is no consolidated tape: every source reports one +dealer's or one venue's prices, and volumes are not comparable across sources. + +| Source | Coverage and history | Finest granularity | Access | ml4t-data | +|--------|----------------------|--------------------|--------|-----------| +| [Dukascopy](https://www.dukascopy.com/swiss/english/marketwatch/historical/) | FX, CFDs, commodities, indices from one Swiss bank's platform | Tick | Free download | No | +| [TrueFX](https://www.truefx.com/truefx-historical-downloads-2/) | Major and cross pairs, top of book | Tick, millisecond timestamps | Free historical downloads with registration; real-time is institutional | No | +| [LSEG Tick History](https://www.lseg.com/en/data-analytics/market-data/data-feeds/tick-history) | Contributed and venue FX quotes from 1996 | Tick | Institutional | No | +| [Oanda](oanda.md) | One broker's quotes | Seconds-level candles | Account required | Yes: `OandaProvider` | + +--- + +## Crypto + +Crypto trades on many venues around the clock, and reported volume on some venues is unreliable. +A source that keeps delisted pairs and closed exchanges (FTX, for example) matters for the same +survivorship reason as delisted stocks. + +| Source | Coverage and history | Delisted pairs and venues | Finest granularity | Access | ml4t-data | +|--------|----------------------|---------------------------|--------------------|--------|-----------| +| [Kaiko](https://www.kaiko.com/products/l1-l2-data) | 100+ centralized exchanges plus DeFi; trades from 2010, order-book snapshots from 2015 | Not documented | Trades, best bid and offer, full order book | Institutional | No | +| [Tardis.dev](https://docs.tardis.dev/historical-data-details/overview) | 50+ exchanges, spot, derivatives and options; most from March 2019 | Yes: closed venues such as FTX and BitMEX history retained | Raw exchange messages, full order-book depth | Paid, including an academic tier | No | +| [CoinAPI](https://www.coinapi.io/products/market-data-api) | 400+ exchanges; spot, futures, perpetuals, options | Not documented | Trades, quotes, order books | Free credits, paid, enterprise | No | +| Binance, Binance Public, OKX, CoinGecko, CryptoCompare | Venue or aggregator data; see the [provider comparison](index.md) | Venue-specific | Bars; funding rates and premium index for perpetuals | Free or free tier | Yes | + +--- + +## Name Changes and Closures + +| Then | Now | +|------|-----| +| CRSP (University of Chicago) | Owned by Morningstar since February 2, 2026; CRSP indexes renamed Morningstar indexes | +| Thomson Reuters / Refinitiv Tick History | LSEG Tick History | +| Polygon.io | Massive (October 30, 2025); `api.polygon.io` still answers alongside `api.massive.com` | +| Sharadar on Nasdaq Data Link only | Also sold direct at sharadar.com since July 2026 | +| Quandl WIKI Prices | Frozen since April 2018; still available as a static dataset | +| IEX Cloud | Retired August 31, 2024 | + +## See Also + +- [Provider comparison](index.md) +- [Fundamental data sources](fundamentals.md) +- [Alternative data sources](alternative_data.md) diff --git a/mkdocs.yml b/mkdocs.yml index 83f771d..cc9b7fe 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -124,6 +124,8 @@ nav: - Providers: - providers/index.md - Fundamental Data Sources: providers/fundamentals.md + - Market Data Sources: providers/market_data.md + - Alternative Data Sources: providers/alternative_data.md - Yahoo Finance: providers/yahoo.md - Alpaca: providers/alpaca.md - CoinGecko: providers/coingecko.md From 53cc875c4f106cf015a53c0adbb2775d1f0c5bf7 Mon Sep 17 00:00:00 2001 From: Stefan Jansen Date: Wed, 23 Sep 2026 16:28:55 -0400 Subject: [PATCH 2/3] docs: correct market and alternative data source pages after a claim-by-claim audit Every factual claim on both pages was checked against vendor documentation, press releases or, for PredictIt and Polymarket's US access, published reporting. Corrections: - Wrong dates: Databento OPRA starts April 2013 (not March 2023), Cboe DataShop options data starts January 2012, Advan Patterns+ history starts 2019, Orbital Insight sale was May 2024, Reddit paid API began July 2023. - Wrong coverage: Sharadar SEP counts tickers, not funds, and only its close is dividend and spin-off adjusted; AlgoSeek covers every listed US equity from 2007; TAQ from 1993 is Monthly TAQ, Daily TAQ starts September 2003; CRSP daily NYSE data starts July 1962; Neudata lists 7,000+ datasets. - Outdated: PredictIt now has CFTC approval as a regulated exchange; Polymarket's US access runs through QCEX since December 2025; the Zillow database has no catalog page and was last refreshed July 2025. - Filled from vendor docs: Alpaca spin-off adjustment, FirstRate and Norgate individual futures contracts, FactSet supply-chain history by region, CME DataMine per-contract files, Tardis tier windows. - Removed claims no public source supports (ORATS futures-option exclusion, S&P ESG counts and history, Sustainalytics score range, Panjiva history, Nasdaq Data Link publisher count, and others) or marked them not documented. - Kinetick and Benzinga links now point to pages that describe the product. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01F6TR4nQDjiPtanA2n4F3Gw --- docs/providers/alternative_data.md | 64 +++++++++++++++--------------- docs/providers/market_data.md | 40 +++++++++---------- 2 files changed, 52 insertions(+), 52 deletions(-) diff --git a/docs/providers/alternative_data.md b/docs/providers/alternative_data.md index c61696b..7bda693 100644 --- a/docs/providers/alternative_data.md +++ b/docs/providers/alternative_data.md @@ -8,7 +8,7 @@ is known about its point-in-time behavior, the access tier, and whether `ml4t-da !!! note "Verified September 2026" Alternative data vendors are acquired, merged and repackaged often. Every entry below was - checked against the vendor's own site or a published announcement in September 2026. Few + checked in September 2026 against the vendor's own site, a press release, or published reporting. Few vendors publish their timestamp and revision policy; where it is not public, the table says "not documented" and the question belongs in your due diligence. Confirm the current terms before you build on any of them. @@ -63,8 +63,8 @@ without names. These questions are answered before any backtest, not after. | [RavenPack](https://www.ravenpack.com/) | Entity-level news analytics: relevance, sentiment and event classification | From 2000 (Dow Jones edition), 2007 (web edition) | Records carry the news timestamp; revision policy not documented publicly | Institutional; academic via WRDS | No | | [LSEG MarketPsych Analytics](https://www.lseg.com/en/data-analytics/market-data/quantitative-economic-data-solutions/marketpsych-analytics-and-models) | Sentiment and emotion scores from news and social media across equities, macro, FX, crypto and commodities | From 1998 | Minute-level to daily series; revision policy not documented publicly | Institutional | No | | [Dow Jones Factiva DNA](https://www.dowjones.com/business-intelligence/factiva/) | Licensed premium news archive with metadata, via API, snapshots and streams | Archive depth varies by publication | Publication timestamps per article | Institutional | No | -| [Benzinga News API](https://www.benzinga.com/apis/cloud-product/stock-news-api/) | Financial news headlines and articles, Wilshire 5000 and TSX coverage | Not documented | `created` and `updated` timestamps per article | Paid, licensed by news channel | No | -| [SEC EDGAR full-text search](https://www.sec.gov/edgar/search/efts-faq.html) | Full text of SEC filings | From May 2001 | Every filing carries its acceptance timestamp | Free | No | +| [Benzinga Newsfeed API](https://docs.benzinga.com/api-reference/news-api/overview) | Financial news headlines and articles, Wilshire 5000 and TSX coverage | Not documented | `created` and `updated` timestamps per article; `updatedSince` returns changes | Paid | No | +| [SEC EDGAR full-text search](https://www.sec.gov/edgar/search/efts-faq.html) | Full text of SEC filings | From 2001 | Every filing carries its acceptance timestamp | Free | No | --- @@ -76,7 +76,7 @@ a project around any of these. | Source | What it measures | Status | Access | ml4t-data | |--------|------------------|--------|--------|-----------| | [Stocktwits API](https://api.stocktwits.com/developers) | Investor message stream with cashtags and self-reported sentiment | Closed to new developer registrations | Not open to new developers | No | -| [Reddit Data API](https://support.reddithelp.com/hc/en-us/articles/16160319875092-Reddit-Data-API-Wiki) | Posts and comments, including investing subreddits | Paid for commercial use since June 2023 | Commercial license required | No | +| [Reddit Data API](https://support.reddithelp.com/hc/en-us/articles/16160319875092-Reddit-Data-API-Wiki) | Posts and comments, including investing subreddits | Paid for commercial use since July 2023 | Commercial license required | No | | [X API](https://docs.x.com/x-api/getting-started/pricing) | Posts and engagement | Pay-per-use credits since February 2026, replacing subscription tiers | Paid | No | --- @@ -89,8 +89,8 @@ a project around any of these. | [Sensor Tower](https://sensortower.com/product/mobile-app/app-performance-insights) | App downloads, revenue, rankings and usage estimates; includes the former data.ai | Not documented | Modeled estimates; revision policy not documented publicly | Enterprise; free top charts | No | | [Thinknum](https://www.thinknum.com/) | Web-collected company data: job listings, headcount, product prices, store locations | Not documented | Not documented | Institutional | No | | [YipitData](https://www.yipitdata.com/) | Company KPI and revenue estimates from web, app and transaction data | Not documented | Not documented | Institutional | No | -| [Revelio Labs](https://www.reveliolabs.com/data) | Workforce composition and flows, job postings, employee sentiment, layoff notices | Workforce data from 2007 to 2008; postings from 2021 | Not documented | Paid; academic via WRDS | No | -| [LinkUp](https://www.linkup.com/data) | Job postings collected directly from employer websites | From 2007 | Each posting is timestamped when found, updated and removed | Institutional | No | +| [Revelio Labs](https://www.reveliolabs.com/data) | Workforce composition and flows, job postings, employee sentiment, layoff notices | Workforce composition from 2007, transitions from 2008, postings from 2021 | Not documented | Paid; academic via WRDS | No | +| [LinkUp](https://www.linkup.com/data) | Job postings collected directly from employer websites | From 2007 | New, updated and removed listings are captured nightly | Institutional | No | --- @@ -111,11 +111,11 @@ so ask how the vendor weights the panel and whether history is reweighted when i | Source | What it measures | History | Access | ml4t-data | |--------|------------------|---------|--------|-----------| -| [Placer.ai](https://www.placer.ai/products/api) | Visits to stores and venues from a mobile-device panel | From January 2017 | Free tools, paid platform, enterprise API | No | -| [Advan Research Patterns+](https://advanresearch.com/products/patternsplus) | Foot traffic to points of interest; took over SafeGraph's Patterns product | Weekly data from January 2017 | Institutional | No | -| [Planet Labs](https://www.planet.com/industries/education-and-research/) | Satellite imagery: near-daily medium resolution and tasked high resolution | Archive from 2009 | Commercial; education and research program for university users | No | -| [RS Metrics](https://rsmetrics.com/) | Asset-level signals from satellite imagery: metal stockpiles, industrial activity | Operating since 2009 | Institutional; academic program | No | -| [Kpler](https://www.kpler.com/product/commodities) | Commodity flows from ship tracking (AIS), satellite, customs and port data; now owns MarineTraffic and Spire Maritime | MarineTraffic AIS archive from 2010 | Institutional | No | +| [Placer.ai](https://www.placer.ai/products/api) | Visits to stores and venues from a mobile-device panel | Not documented | Free tools, paid platform, enterprise API | No | +| [Advan Research Patterns+](https://advanresearch.com/products/patternsplus) | Foot traffic to points of interest, US and Canada; took over SafeGraph's Patterns product | Weekly updates; history from 2019 | Institutional | No | +| [Planet Labs](https://www.planet.com/industries/education-and-research/) | Satellite imagery: near-daily medium resolution and tasked high resolution | Archive from 2009 (RapidEye, 2009 to 2020) | Commercial; education and research program for university users | No | +| [RS Metrics](https://rsmetrics.com/) | Asset-level signals from satellite imagery: metals production at smelters, retail parking lots | Not documented | Institutional; data for research | No | +| [Kpler](https://www.kpler.com/product/commodities) | Commodity flows from ship tracking (AIS), satellite, customs and port data; now owns MarineTraffic and Spire Maritime | Not documented | Institutional | No | --- @@ -126,10 +126,10 @@ history as a sequence of vintages, and check which methodology produced each val | Source | What it measures | History and methodology | Access | ml4t-data | |--------|------------------|-------------------------|--------|-----------| -| [MSCI ESG Ratings](https://www.msci.com/data-and-analytics/sustainability-solutions/esg-ratings) | Industry-relative ESG ratings (AAA to CCC), about 17,000 issuers | Ratings since 2007; methodology revised several times since | Institutional; free public lookup for a subset | No | -| [Morningstar Sustainalytics ESG Risk Ratings](https://www.sustainalytics.com/esg-data) | Unmanaged ESG risk score (0 to 100), 16,000+ companies | A score change log exists; vintage policy not documented publicly | Institutional | No | -| [LSEG ESG Scores](https://www.lseg.com/en/media-centre/press-releases/2026/lseg-launches-new-suite-esg-scores-sustainability-analytics) | ESG scores from 220+ indicators, plus controversies | New suite launched March 2026; the earlier Refinitiv-era scores are being retired | Institutional | No | -| [S&P Global ESG Scores](https://www.marketplace.spglobal.com/en/datasets/s-p-global-esg-scores-(171)) | Scores from the annual Corporate Sustainability Assessment, about 13,000 companies | Annual assessments; history from 2013 | Free public scores for a subset; full data paid | No | +| [MSCI ESG Ratings](https://www.msci.com/data-and-analytics/sustainability-solutions/esg-ratings) | Industry-relative ESG ratings (AAA to CCC), about 17,000 issuers | Time series since 2007 | Institutional; free public lookup for a subset | No | +| [Morningstar Sustainalytics ESG Risk Ratings](https://www.sustainalytics.com/esg-data) | Unmanaged ESG risk score, 16,000+ companies | A score change log exists; vintage policy not documented publicly | Institutional | No | +| [LSEG ESG Scores](https://www.lseg.com/en/media-centre/press-releases/2026/lseg-launches-new-suite-esg-scores-sustainability-analytics) | ESG scores from 220+ indicators, plus controversies | New suite launched March 2026 on 220 standardised indicators; how it relates to earlier LSEG scores is not documented publicly | Institutional | No | +| [S&P Global ESG Scores](https://www.marketplace.spglobal.com/en/datasets/s-p-global-esg-scores-(171)) | Scores from the annual Corporate Sustainability Assessment | Annual assessments; history not documented publicly | Institutional | No | | [RepRisk](https://www.reprisk.com/insights/resources/methodology) | Reputational and conduct risk from news and stakeholder sources | Daily series with a consistent methodology since January 2007 | Institutional | No | --- @@ -138,10 +138,10 @@ history as a sequence of vintages, and check which methodology produced each val | Source | What it measures | History | Access | ml4t-data | |--------|------------------|---------|--------|-----------| -| [FactSet Supply Chain Relationships](https://www.factset.com/marketplace/catalog/product/factset-supply-chain-relationships) (formerly Revere) | Customer, supplier, partner and competitor links between companies | Not documented | Institutional; academic via WRDS | No | +| [FactSet Supply Chain Relationships](https://www.factset.com/marketplace/catalog/product/factset-supply-chain-relationships) (formerly Revere) | Customer, supplier, partner and competitor links between companies | North America from 2003; other regions from 2011 to 2016 | Institutional; academic via WRDS | No | | [Bloomberg Supply Chain (SPLC)](https://professional.bloomberg.com/institutions/corporations/supply-chain/) | Supplier and customer relationships with revenue and cost exposure, 100,000+ companies | From 2006 | Bloomberg Terminal and enterprise feed | No | -| [S&P Global Panjiva](https://www.marketplace.spglobal.com/en/datasets/panjiva-supply-chain-intelligence-(22)) | Shipment-level customs records, 2 billion+ records | US data from 2007 | Limited free search; full data institutional | No | -| [ImportGenius](https://www.importgenius.com/pricing) | US and international bill-of-lading records | US imports from 2006, exports from 2014 on the full tier | Retail paid tiers | No | +| [S&P Global Panjiva](https://www.marketplace.spglobal.com/en/datasets/panjiva-supply-chain-intelligence-(22)) | Shipment-level customs records, 2 billion+ records | Not documented | Institutional | No | +| [ImportGenius](https://www.importgenius.com/pricing) | US and international bill-of-lading records | US imports from 2006, exports from January 2014 on the Pro and Enterprise tiers | Self-serve subscriptions; enterprise plans | No | --- @@ -153,8 +153,8 @@ listed and resolved over time, so a set of markets picked today is a survivor sa | Source | What it measures | History | Access | ml4t-data | |--------|------------------|---------|--------|-----------| -| [Kalshi](https://docs.kalshi.com/welcome) | CFTC-regulated event contracts: economics, politics, weather, sports | From the 2021 launch | Public market-data API; trading needs a verified account | Yes: [`KalshiProvider`](kalshi.md) | -| [Polymarket](https://docs.polymarket.com/) | Event contracts on a crypto platform; a CFTC-regulated US exchange (QCX) relaunched in 2025 to 2026 | From 2020 | Public market data | Yes: [`PolymarketProvider`](polymarket.md) | +| [Kalshi](https://docs.kalshi.com/welcome) | CFTC-regulated event contracts: economics, politics, weather, sports | From the July 2021 launch | Public market-data API; trading needs a verified account | Yes: [`KalshiProvider`](kalshi.md) | +| [Polymarket](https://docs.polymarket.com/) | Event contracts on a crypto platform; open to US users again since December 2025 through QCEX, a CFTC-licensed exchange acquired in 2025 | From 2020 | Public market data | Yes: [`PolymarketProvider`](polymarket.md) | --- @@ -164,16 +164,16 @@ Marketplaces simplify procurement; they do not validate point-in-time behavior o | Source | What it offers | Access | |--------|----------------|--------| -| [Nasdaq Data Link](https://data.nasdaq.com/) (formerly Quandl) | Catalog of financial and alternative datasets from 400+ publishers, some free | Free account for free datasets; paid for premium | +| [Nasdaq Data Link](https://data.nasdaq.com/) (formerly Quandl) | Catalog of financial, economic and alternative datasets, some free | Free account for free datasets; paid for premium | | [Snowflake Marketplace](https://docs.snowflake.com/en/collaboration/collaboration-marketplace-about) | Datasets shared into a Snowflake account and queried in place | Free, trial and paid listings | | [AWS Data Exchange](https://aws.amazon.com/data-exchange/) | Datasets delivered into AWS, billed through AWS | Per-product subscriptions | | [Eagle Alpha](https://www.eaglealpha.com/) | Alternative data discovery and advisory, 2,500+ dataset profiles | Institutional | -| [Neudata](https://www.neudata.co/) | Alternative data discovery and research, about 2,800 datasets tracked | Institutional | -| [Datarade](https://datarade.ai/platforms) | Directory of data providers across industries | Free to browse; each provider sets its own terms | +| [Neudata](https://www.neudata.co/) | Alternative data discovery and research, 7,000+ datasets in its catalog | Institutional | +| [Datarade](https://datarade.ai/platforms) | Directory of data providers and data marketplaces | Free to browse; each provider sets its own terms | Of the free datasets that made Quandl popular, the WIKI prices table is frozen at April 2018 -(`ml4t-data` wraps it as [Wiki Prices](wiki_prices.md)), and the Zillow real-estate database is -no longer in the catalog. +(`ml4t-data` wraps it as [Wiki Prices](wiki_prices.md)), and the Zillow real-estate database +no longer has a catalog page and was last refreshed in July 2025. --- @@ -190,17 +190,17 @@ needs its own client. | Then | Now | |------|-----| -| data.ai (formerly App Annie) | Part of Sensor Tower since March 2024 | -| Earnest Analytics | Part of Consumer Edge since 2025 | +| data.ai (formerly App Annie) | Part of Sensor Tower since 2024 | +| Earnest Analytics | Part of Consumer Edge | | SafeGraph Patterns (foot traffic) | Sold to Advan Research; SafeGraph now sells places, geometry and spend data | -| Orbital Insight | Acquired by Privateer, April 2024 | +| Orbital Insight | Acquired by Privateer, May 2024 | | MarineTraffic | Acquired by Kpler, March 2023 | | Spire Maritime | Acquired by Kpler, April 2025 | -| Refinitiv ESG scores | Replaced by the LSEG ESG Scores suite, March 2026 | -| Reddit and X APIs | Paid access (Reddit since June 2023; X pay-per-use since February 2026) | +| Refinitiv ESG scores | LSEG ESG data; a new LSEG ESG Scores suite launched March 2026 | +| Reddit and X APIs | Paid access (Reddit since July 2023; X pay-per-use since February 2026) | | Stocktwits API | Closed to new developers | -| Polymarket (outside the US since 2022) | US exchange via the 2025 acquisition of CFTC-licensed QCEX | -| PredictIt | Operating under an amended CFTC no-action letter (2025) | +| Polymarket (closed to US users 2022 to 2025) | US access since December 2025 through QCEX, a CFTC-licensed exchange acquired in 2025 | +| PredictIt | CFTC approval to operate as a regulated exchange, September 2025 | | Quandl | Nasdaq Data Link (September 2021) | ## See Also diff --git a/docs/providers/market_data.md b/docs/providers/market_data.md index 547c181..2b67c84 100644 --- a/docs/providers/market_data.md +++ b/docs/providers/market_data.md @@ -8,7 +8,7 @@ than repeating it. !!! note "Verified September 2026" Vendor coverage, history, access terms and ownership change often. Every entry below was - checked against the vendor's own documentation or a published announcement in September 2026. + checked in September 2026 against the vendor's own documentation or a published announcement. Where a vendor does not document a property, the table says "not documented" instead of guessing. Confirm the current terms before you build a dataset on any of them. @@ -78,12 +78,12 @@ samples. | Source | Coverage and history | Delisted | Adjustments | Finest granularity | Point-in-time index membership | Access | ml4t-data | |--------|----------------------|----------|-------------|--------------------|--------------------------------|--------|-----------| -| [CRSP US Stock Databases (on WRDS)](https://wrds-www.wharton.upenn.edu/pages/about/data-vendors/center-for-research-in-security-prices-crsp/) | NYSE from December 1925, NASDAQ from December 1972 | Yes, keyed by PERMNO and PERMCO | Returns with and without dividends; full corporate-action history | Daily and monthly | Yes, index constituent files on WRDS | Academic and institutional, mainly via WRDS | No | -| [Norgate Data](https://norgatedata.com/data-content-tables.php) | US, Australian and Canadian stocks and ETFs; US stocks essentially complete from 1992 | Yes, on the higher tiers | Four modes: unadjusted, capital-reconstruction adjusted, plus special distributions, total return | Daily | Yes, on the higher tiers | Retail paid subscription | No | -| [Sharadar Equity Prices (SEP)](https://data.nasdaq.com/databases/SEP) | About 21,000 US equities and funds, from 1998 | Yes | Unadjusted, split-adjusted, and split, dividend and spin-off adjusted | Daily | Separate S&P 500 constituents table | Paid; direct at sharadar.com since July 2026, or Nasdaq Data Link | No | +| [CRSP US Stock Databases (on WRDS)](https://wrds-www.wharton.upenn.edu/pages/about/data-vendors/center-for-research-in-security-prices-crsp/) | NYSE monthly from December 1925 and daily from July 1962; NASDAQ from December 1972 | Yes, keyed by PERMNO and PERMCO | Returns with and without dividends; full corporate-action history | Daily and monthly | Yes, index constituent files on WRDS | Academic and institutional, mainly via WRDS | No | +| [Norgate Data](https://norgatedata.com/data-content-tables.php) | US, Australian and Canadian stocks and ETFs; US listed and delisted coverage essentially complete from late 1992 | Yes, on the Platinum and Diamond tiers | Four modes: unadjusted, capital-reconstruction adjusted, plus special distributions, total return | Daily | Yes, on the Platinum and Diamond tiers | Retail paid subscription | No | +| [Sharadar Equity Prices (SEP)](https://data.nasdaq.com/databases/SEP) | About 21,000 active and delisted US tickers, from 1998 | Yes | OHLCV adjusted for splits and stock dividends; an unadjusted close; a close also adjusted for cash dividends and spin-offs | Daily | Separate S&P 500 constituents table (`SHARADAR/SP500`) | Paid; direct at sharadar.com since July 2026, or Nasdaq Data Link | No | | [FirstRate Data](https://firstratedata.com/about/FAQ) | 16,000+ US tickers from 2000, including 7,000+ delisted | Partly: included where the vendor could source them | Unadjusted, split-adjusted, and split and dividend adjusted | 1-minute bars | No | Retail paid, one-off purchase | No | | [Kibot](https://www.kibot.com/faq.html) | US stocks and ETFs; minute bars from 1998 | Partly: active and delisted lists, stated as incomplete | Split and dividend adjusted series plus an adjustments API | Tick with bid and ask | No | Free guest access (daily only), paid tiers | No | -| [Alpaca](https://docs.alpaca.markets/us/docs/about-market-data-api) | US equities and ETFs, from 2016 | Not documented | `adjustment` parameter: raw, split, dividend or all | Trades and quotes | No | Free (IEX feed, delayed SIP); paid for real-time SIP | Yes: `AlpacaProvider` | +| [Alpaca](https://docs.alpaca.markets/us/docs/about-market-data-api) | US equities and ETFs, from 2016 | Not documented | `adjustment` parameter: raw, split, dividend, spin-off or all | Trades and quotes | No | Free (IEX feed, delayed SIP); paid for real-time SIP | Yes: `AlpacaProvider` | The free wrapped sources (Yahoo Finance, Tiingo, EODHD free tier) carry currently listed securities and vendor-adjusted prices; treat them as prototyping data, not as a survivorship-free @@ -95,13 +95,13 @@ panel. Wiki Prices stopped updating in April 2018. | Source | Coverage and history | Delisted | Adjustments | Finest granularity | Point-in-time index membership | Access | ml4t-data | |--------|----------------------|----------|-------------|--------------------|--------------------------------|--------|-----------| -| [NYSE TAQ](https://wrds-www.wharton.upenn.edu/pages/about/data-vendors/nyse-trade-and-quote-taq/) (Daily TAQ) | All US exchanges from 1993; millisecond timestamps from 2003 | Not applicable: daily files of every security traded that day | Raw prices; adjust with CRSP factors | Trades and NBBO quotes | No | Academic and institutional via WRDS | No | -| [AlgoSeek](https://algoseek.com/us-equities-package/) | About 27,500 US securities from 1998 (SIP); adjustment factors from 2007 | Yes | Raw and adjusted prices plus adjustment-factor files | Trades and quotes with nanosecond timestamps; bars from 1 second | Yes, Index Components dataset | Institutional; free sandbox with up to one year of data | No | +| [NYSE TAQ](https://wrds-www.wharton.upenn.edu/pages/about/data-vendors/nyse-trade-and-quote-taq/) | All US exchanges; Monthly TAQ 1993 to 2014, Daily TAQ (millisecond timestamps) from September 2003 | Not applicable: daily files of every security traded that day | Raw prices; adjust with CRSP factors | Trades and NBBO quotes | No | Academic and institutional via WRDS | No | +| [AlgoSeek](https://algoseek.com/us-equities-package/) | Every listed US equity from 2007 (SIP), including delisted securities; adjustment factors from 2007 | Yes | Raw and adjusted prices plus adjustment-factor files | Trades and quotes with nanosecond timestamps; bars from 1 second | Yes, Index Components dataset | Institutional; free sandbox with up to one year of data | No | | [Databento](https://databento.com/datasets/XNAS.ITCH) | US equities across exchanges and ATSs; Nasdaq TotalView-ITCH from 2018 | Not applicable: raw exchange feeds | Unadjusted; corporate actions sold as separate reference data | Full order book (market by order) | No | Usage-based; see the [Databento page](databento.md) | Yes: `DataBentoProvider` | -| [LSEG Tick History](https://www.lseg.com/en/data-analytics/market-data/data-feeds/tick-history) (formerly Refinitiv Tick History) | Cross-asset, 500+ venues, from 1996 | Not documented | Not documented | Trades, quotes and order-book depth | No | Institutional; some universities license it | No | +| [LSEG Tick History](https://www.lseg.com/en/data-analytics/market-data/data-feeds/tick-history) (formerly Refinitiv Tick History) | Cross-asset, 580+ venues and contributors, from January 1996 | Not documented | Not documented | Trades, quotes and order-book depth | No | Institutional; some universities license it | No | | [Nasdaq TotalView-ITCH samples](https://emi.nasdaq.com/ITCH/Nasdaq%20ITCH/) | Selected full-day files for Nasdaq-listed stocks | Not applicable | Raw messages | Every order add, cancel and execution | No | Free samples; the full archive is a paid Nasdaq product | Yes: [NASDAQ ITCH provider](nasdaq_itch.md) | | [DTN IQFeed](https://www.iqfeed.net/dev/) | Equities, futures, options, FX; tick history rolling 180 days, minute bars from 2005 to 2007 depending on asset | Not documented | Not documented | Trades and quotes | No | Retail paid | No | -| [Kinetick](https://ninjatrader-live.ninjatrader.com/support/helpguides/nt8/data_by_provider.htm) | Market data service for NinjaTrader users; equities, futures, FX | Not documented | Not documented | Tick | No | Retail; free end-of-day tier | No | +| [Kinetick](https://kinetick.com/) | Market data service bundled with NinjaTrader; stocks, futures, FX | Not documented | Not documented | Tick | No | Retail; free end-of-day tier | No | --- @@ -109,10 +109,10 @@ panel. Wiki Prices stopped updating in April 2018. | Source | Coverage and history | Expired contracts | Continuous series | Finest granularity | Access | ml4t-data | |--------|----------------------|-------------------|-------------------|--------------------|--------|-----------| -| [CME DataMine](https://www.cmegroup.com/datamine.html) | CME, CBOT, NYMEX and COMEX futures and options; history varies by dataset | Not documented | No, individual contracts | Market depth and time and sales | Paid per dataset; academic discount | No | -| [Databento CME Globex (GLBX.MDP3)](https://databento.com/datasets/GLBX.MDP3) | CME Group futures and options from June 2010 | Yes, per contract | Continuous symbology by roll rule | Full order book | Usage-based | Yes: `DataBentoProvider.fetch_continuous_futures()` | -| [Norgate Data](https://norgatedata.com/data-content-tables.php) | Global futures and cash commodities | Yes | Back-adjusted continuous contracts | Daily | Retail paid | No | -| [FirstRate Data](https://firstratedata.com/about/FAQ) | Futures, about 15 years intraday | Not documented | Unadjusted, absolute-adjusted and ratio-adjusted continuous series | 1-minute bars | Retail paid | No | +| [CME DataMine](https://www.cmegroup.com/datamine.html) | CME, CBOT, NYMEX and COMEX futures and options; history varies by dataset, some back to the 1970s | Yes, data is per contract | No, individual contracts | Market depth and time and sales | Paid per dataset; academic discount | No | +| [Databento CME Globex (GLBX.MDP3)](https://databento.com/datasets/GLBX.MDP3) | CME Group futures and options from June 2010; full order-by-order depth from March 2017 | Yes, per contract | Continuous symbology by roll rule | Full order book | Usage-based | Yes: `DataBentoProvider.fetch_continuous_futures()` | +| [Norgate Data](https://norgatedata.com/futurespackage.php) | Global futures and cash commodities | Yes, individual contract histories | Unadjusted and back-adjusted continuous contracts | Daily | Retail paid | No | +| [FirstRate Data](https://firstratedata.com/it/futures) | 130 most active futures, intraday from 2007 to 2008 | Yes, individual contracts | Unadjusted, absolute-adjusted and ratio-adjusted continuous series | 1-minute bars | Retail paid | No | --- @@ -121,10 +121,10 @@ panel. Wiki Prices stopped updating in April 2018. | Source | Coverage and history | Delisted underlyings | Analytics | Finest granularity | Access | ml4t-data | |--------|----------------------|----------------------|-----------|--------------------|--------|-----------| | [OptionMetrics IvyDB](https://optionmetrics.com/united-states/) | US equity and index options from 1996; Europe from 2002 | Yes | Implied volatility, Greeks, volatility surfaces; adjusts for dividends, splits and spin-offs | Daily; a separate intraday product exists | Academic via WRDS, institutional | No | -| [ORATS](https://orats.com/data-api) | US equity, ETF and index options (no futures options); end of day from 2007, 1-minute from August 2020 | Yes | Smoothed implied volatility and Greeks | 1-minute snapshots | Retail paid, enterprise | No | -| [Cboe DataShop](https://datashop.cboe.com/documentation) (LiveVol) | Options quotes, trades and end of day from 2010; VIX futures from 2004 | Not documented | Calculated implied volatility and Greeks in some datasets | Trades and quotes | Paid, with trial | No | -| [Databento OPRA](https://databento.com/datasets/OPRA.PILLAR) | All US equity options from March 2023 | Not applicable | None: raw feed | Trades and quotes | Usage-based | Yes: `DataBentoProvider.fetch_option_chain()`, `fetch_option_quotes()` | -| [AlgoSeek](https://algoseek.com/us-equities-package/) | US options as an add-on to equities | Not documented | Vendor-computed analytics on some products | Trades and quotes | Institutional | No | +| [ORATS](https://orats.com/data-api) | US equity, ETF and index options; end of day from 2007, 1-minute from August 2020 | Not documented | Smoothed implied volatility and Greeks | 1-minute snapshots | Retail paid, enterprise | No | +| [Cboe DataShop](https://datashop.cboe.com/documentation) (LiveVol) | Options trades, quotes and end of day from January 2012; VIX futures trades and quotes from April 2004 | Not documented | Greeks and implied volatility (Cboe Hanweck) as separate products | Trades and quotes | Usage-based | No | +| [Databento OPRA](https://databento.com/datasets/OPRA.PILLAR) | All US equity options from April 2013 | Not applicable | None: raw feed | Trades and quotes | Usage-based | Yes: `DataBentoProvider.fetch_option_chain()`, `fetch_option_quotes()` | +| [AlgoSeek](https://algoseek.com/us-equities-package/) | US equity options, full OPRA feed | Not documented | Greeks and implied volatility dataset | Trades and quotes | Institutional | No | --- @@ -136,7 +136,7 @@ dealer's or one venue's prices, and volumes are not comparable across sources. | Source | Coverage and history | Finest granularity | Access | ml4t-data | |--------|----------------------|--------------------|--------|-----------| | [Dukascopy](https://www.dukascopy.com/swiss/english/marketwatch/historical/) | FX, CFDs, commodities, indices from one Swiss bank's platform | Tick | Free download | No | -| [TrueFX](https://www.truefx.com/truefx-historical-downloads-2/) | Major and cross pairs, top of book | Tick, millisecond timestamps | Free historical downloads with registration; real-time is institutional | No | +| [TrueFX](https://www.truefx.com/truefx-historical-downloads-2/) | Major and cross pairs, top of book | Tick, millisecond timestamps | Free historical downloads with registration | No | | [LSEG Tick History](https://www.lseg.com/en/data-analytics/market-data/data-feeds/tick-history) | Contributed and venue FX quotes from 1996 | Tick | Institutional | No | | [Oanda](oanda.md) | One broker's quotes | Seconds-level candles | Account required | Yes: `OandaProvider` | @@ -151,7 +151,7 @@ survivorship reason as delisted stocks. | Source | Coverage and history | Delisted pairs and venues | Finest granularity | Access | ml4t-data | |--------|----------------------|---------------------------|--------------------|--------|-----------| | [Kaiko](https://www.kaiko.com/products/l1-l2-data) | 100+ centralized exchanges plus DeFi; trades from 2010, order-book snapshots from 2015 | Not documented | Trades, best bid and offer, full order book | Institutional | No | -| [Tardis.dev](https://docs.tardis.dev/historical-data-details/overview) | 50+ exchanges, spot, derivatives and options; most from March 2019 | Yes: closed venues such as FTX and BitMEX history retained | Raw exchange messages, full order-book depth | Paid, including an academic tier | No | +| [Tardis.dev](https://docs.tardis.dev/historical-data-details/overview) | 50+ exchanges, spot, derivatives and options; most from March 2019 | Yes: closed venues such as FTX and BitMEX history retained | Raw exchange messages, full order-book depth | Paid; academic, solo and pro tiers cover the last four years, business tiers from March 2019 | No | | [CoinAPI](https://www.coinapi.io/products/market-data-api) | 400+ exchanges; spot, futures, perpetuals, options | Not documented | Trades, quotes, order books | Free credits, paid, enterprise | No | | Binance, Binance Public, OKX, CoinGecko, CryptoCompare | Venue or aggregator data; see the [provider comparison](index.md) | Venue-specific | Bars; funding rates and premium index for perpetuals | Free or free tier | Yes | @@ -161,7 +161,7 @@ survivorship reason as delisted stocks. | Then | Now | |------|-----| -| CRSP (University of Chicago) | Owned by Morningstar since February 2, 2026; CRSP indexes renamed Morningstar indexes | +| CRSP (University of Chicago) | Owned by Morningstar since February 2, 2026; the CRSP Market Indexes were renamed Morningstar Market Indexes in 2026 | | Thomson Reuters / Refinitiv Tick History | LSEG Tick History | | Polygon.io | Massive (October 30, 2025); `api.polygon.io` still answers alongside `api.massive.com` | | Sharadar on Nasdaq Data Link only | Also sold direct at sharadar.com since July 2026 | From f0c1f72fd3afb70ab6b7fbcd8eea7f9cbbfd409f Mon Sep 17 00:00:00 2001 From: Stefan Jansen Date: Wed, 23 Sep 2026 16:45:39 -0400 Subject: [PATCH 3/3] docs: state Polymarket's US status as its press release does, and cite PredictIt The cited Polymarket release (2025-11-25) reports a CFTC amended order of designation for intermediated US access with launch to follow; it names no QCEX and no December access date, so the page no longer claims either. PredictIt's September 2025 approval now links the Bloomberg report. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01F6TR4nQDjiPtanA2n4F3Gw --- docs/providers/alternative_data.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/providers/alternative_data.md b/docs/providers/alternative_data.md index 7bda693..e1ea582 100644 --- a/docs/providers/alternative_data.md +++ b/docs/providers/alternative_data.md @@ -154,7 +154,7 @@ listed and resolved over time, so a set of markets picked today is a survivor sa | Source | What it measures | History | Access | ml4t-data | |--------|------------------|---------|--------|-----------| | [Kalshi](https://docs.kalshi.com/welcome) | CFTC-regulated event contracts: economics, politics, weather, sports | From the July 2021 launch | Public market-data API; trading needs a verified account | Yes: [`KalshiProvider`](kalshi.md) | -| [Polymarket](https://docs.polymarket.com/) | Event contracts on a crypto platform; open to US users again since December 2025 through QCEX, a CFTC-licensed exchange acquired in 2025 | From 2020 | Public market data | Yes: [`PolymarketProvider`](polymarket.md) | +| [Polymarket](https://docs.polymarket.com/) | Event contracts on a crypto platform; CFTC approval for intermediated US access, November 2025 | From 2020 | Public market data | Yes: [`PolymarketProvider`](polymarket.md) | --- @@ -199,8 +199,8 @@ needs its own client. | Refinitiv ESG scores | LSEG ESG data; a new LSEG ESG Scores suite launched March 2026 | | Reddit and X APIs | Paid access (Reddit since July 2023; X pay-per-use since February 2026) | | Stocktwits API | Closed to new developers | -| Polymarket (closed to US users 2022 to 2025) | US access since December 2025 through QCEX, a CFTC-licensed exchange acquired in 2025 | -| PredictIt | CFTC approval to operate as a regulated exchange, September 2025 | +| Polymarket (closed to US users from 2022) | [Amended CFTC order of designation](https://www.prnewswire.com/news-releases/polymarket-receives-cftc-approval-of-amended-order-of-designation-enabling-intermediated-us-market-access-302625833.html) for intermediated US access, November 2025 | +| PredictIt | [CFTC approval to operate a regulated exchange](https://www.bloomberg.com/news/articles/2025-09-05/predictit-gains-cftc-approval-to-launch-regulated-exchange), September 2025 | | Quandl | Nasdaq Data Link (September 2021) | ## See Also