Today the engine is single-process: runProbe opens+tears down one browser per probe, and B3 caps concurrency within one process. Horizontal scale = launch N instances manually.
Scope (explore first)
- A bounded pool of warm browser processes reused across probes (crawlee BrowserPool model:
retireBrowserAfterPageCount, maxOpenPagesPerBrowser).
- Decide: in-process pool of contexts vs multi-process workers.
- Must preserve per-probe identity/proxy isolation and stealth.
Acceptance
- Sustained throughput higher than launch-per-probe without unbounded memory.
- Opt-in; default behavior unchanged.
Risk — touches the core probe lifecycle; needs careful design + benchmarks. Not a quick win.
Today the engine is single-process:
runProbeopens+tears down one browser per probe, and B3 caps concurrency within one process. Horizontal scale = launch N instances manually.Scope (explore first)
retireBrowserAfterPageCount,maxOpenPagesPerBrowser).Acceptance
Risk — touches the core probe lifecycle; needs careful design + benchmarks. Not a quick win.