You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Enable Storm dynamic producer batching in the archetype configurations #2131
The crawler-conf.yaml generated by the archetypes leaves Storm's inter-executor batching at its default (topology.producer.batch.size: 1), so no batching happens between the spout, fetcher, parser, indexer and status updater executors.
Storm 3.0 adds topology.producer.batch.dynamic: when enabled together with a topology.producer.batch.size greater than 1, the effective batch size adapts at runtime between 1 and the configured size with AIMD, staying small under light load to keep latency low and growing under sustained load to preserve throughput. That removes the usual objection to batching in a crawl topology, where traffic is bursty and latency to the status updater matters.
Proposal: enable it in the archetype configurations with a conservative batch size below topology.max.spout.pending, and document the Storm topology settings that affect a crawl (topology.message.timeout.secs, topology.max.spout.pending, the producer batch settings and the flush interval), which are currently not covered in the configuration documentation.
Activity
apache#2131 Document Storm topology settings affecting a crawl
jnioche commentedon Nov 1, 2026
See discussion on #2261 (comment)
Closing for now unless someone has evidence that it would make a difference. Thanks @GGraziadei