A compact, offline search engine for the Brazilian CNPJ open data published monthly by
Receita Federal. One Rust binary downloads the release, turns its 37 ZIPs (~29 GB of
CSV, ~224 million rows) into a single compressed, indexed .ocnpj file, verifies it
against the sources, and serves CNPJ lookups, name search, filters, statistics and
exports over HTTP, with accounts, plans and quotas, an admin suite (web and native
desktop) and automatic monthly updates without downtime. No PostgreSQL, Redis or search
cluster.
The previous Go + PostgreSQL + Redis + React implementation is archived at tag
legacy-go-final.
cargo build --release
B=target/release/opencnpj
$B download --dest data/raw # latest complete release
$B build --input data/raw/2026-10 --output data/2026-10.ocnpj
$B verify data/2026-10.ocnpj --sources data/raw/2026-10 # row-for-row check
$B get data/2026-10.ocnpj 33.000.167/0001-01
$B search data/2026-10.ocnpj --q "padaria sao jo" --filter uf=SP --limit 5
$B accounts --data-dir data create --username admin --email [email protected] --role admin --key-file admin.key
$B serve --data-dir data # API :8080, admin http://127.0.0.1:8081/admin
# With the web application and the Portuguese assistant (see web/README.md):
# $B serve --data-dir data --web web/dist --assistant-agg data/intelligence/agg/2026-10.ocagg
K=$(cat admin.key)
curl -H "Authorization: Bearer $K" 'localhost:8080/api/v1/cnpj/00000000000191'
curl -H "Authorization: Bearer $K" 'localhost:8080/api/v1/search?cnae=G&uf=SP&data_inicio_atividade_de=2026-01-01&ordenar=data_inicio_atividade:desc'Users register with a username and an e-mail address and receive their login key by
e-mail (POST /api/v1/auth/register); serve --auth off runs without accounts.
Monthly operation (download → rebuild only if sources changed → verify → atomic switch, previous snapshot kept for rollback):
$B update --data-dir data --profile balanced
$B serve --data-dir data # picks up data/current automatically, and checks RFB for new releases
$B rollback --data-dir data- Lossless semantic storage. Every row and value of empresas, estabelecimentos,
sócios and Simples/MEI is recoverable exactly; lines that do not fit the layout are
kept verbatim.
verify --sourcesproves it per table against the ZIPs. - Numeric and alphanumeric CNPJ (2026 format) with check-digit validation.
- Search: accent-insensitive multi-word name search with prefix matching, typo tolerance (edit distance + swapped letters, automatic retry) and ranking; CNPJ and CNPJ prefix, CPF (MEI/EI owners exactly, partners by their masked digits), partner CNPJ; filters by UF, município, CNAE (principal/secundária/either, by section, division, group or class), situação, motivo, porte, natureza, Simples/MEI (and active), matriz/filial, CEP prefix, DDD, published e-mail/phone; date and capital ranges (opening, status, Simples/MEI opt-in and exclusion); sorting by date or capital; exact counts, statistics per field or per year, streaming export.
- Accounts: one login key per user (Argon2id), registration and key recovery by e-mail, sessions, roles (admin, moderator, analyst, user) with permissions, suspension and bans, plans with rate limits and daily/monthly quotas, per-user usage and search log, audit log (docs/ACCOUNTS.md).
- Assistant and web application: questions in Portuguese ("dentistas em Cuiabá com telefone", "novas empresas em Sinop") interpreted deterministically, without an LLM, and answered from the data with the filters applied shown; a clean web front end with chat, filter search, CNPJ records, the user's plan and usage, and staff administration of accounts and search limits (web/README.md, docs/intelligence/17-PROSPECTING.md).
- API + admin suite: HTTP API with snapshot distribution (resumable downloads); a web
dashboard and a native desktop app (
opencnpj-admin, Linux and Windows) with live latency histograms, user management and moderation, plans, usage, logs, query explain, snapshot switching, cache control, self-test benchmarks and background jobs. - Serving at scale: shared response cache with request coalescing and ETags, compression, separate lanes for lookups and heavy queries; 5,000 simultaneous authenticated users at p99 under 1 ms in steady state on one machine; binary upgrades and monthly data updates without dropping a request (docs/DEPLOYMENT.md).
- Integrity: checksums on every block and section, SHA-256 of sources and snapshot, strict bounds checks; corrupted or truncated files are rejected.
- Operations: one static binary for Linux and Windows, atomic publication, hot reload, timeouts, concurrency limits, Prometheus metrics, privacy switches.
223,628,880 rows from 7.81 GB of ZIPs (28.86 GB of CSV), i7-13620H laptop, NVMe. Every
snapshot below passed verify --sources (all rows reproduced exactly).
| snapshot | size | build (end to end) | CNPJ lookup p50 / p95 | name search p95 |
|---|---|---|---|---|
| 1.2 balanced — every index: names, CPF, partners, CNAE, CEP, DDD, contacts, dates, capital | 8.33 GB | 8 min 2 s | 0.166 / 0.200 ms | 37–76 ms |
| 1.0 fast (names + basic filters) | 7.63 GB | ~5 min | 0.033 / 0.050 ms | 36–69 ms |
| 1.0 balanced | 6.39 GB | 9 min 6 s | 0.158 / 0.189 ms | 40–83 ms |
| 1.0 compact | 5.75 GB | ~11.7 min | 1.50 / 1.92 ms | 61–100 ms |
Through HTTP, the 1.2 snapshot served 36.4k CNPJ lookups/s at p95 0.68 ms, CPF lookups at 893/s and selective filters at 1.3k/s (16 connections, loopback). The legacy stack imported 217 M rows into PostgreSQL in 27 min 9 s. Methodology, the experiment matrix, baselines and known bottlenecks are in docs/BENCHMARKS.md.
| Doc | Content |
|---|---|
| docs/ARCHITECTURE.md | Pipeline, crates, design decisions |
| docs/FORMAT.md | .ocnpj byte layout, codecs, index, compatibility |
| docs/API.md | HTTP endpoints, parameters, limits, legacy migration |
| docs/ACCOUNTS.md | Keys, recovery, roles, plans, quotas, admin API |
| docs/DEPLOYMENT.md | Production setup, capacity, zero-downtime updates and upgrades |
| crates/opencnpj-desktop | Native admin app: build, run, install |
| docs/BENCHMARKS.md | Methodology, results, experiments, Pareto front |
| docs/DATA_POLICY.md | Provenance, chain of trust, LGPD |
| docs/intelligence/ | Plan for a lightweight pt-BR data assistant (no LLM): requirements, research, design, roadmap |
| docs/legacy/ | Archived implementation and how to restore it |
| docs/CONTRIBUTING.md | Development workflow |
MIT OR Apache-2.0, at your option. See LICENSE, LICENSE-APACHE-2.0 and NOTICE. The CNPJ data belongs to Receita Federal do Brasil.