Skip to content

Enable Storm dynamic producer batching in archetype configs and docum… #2261

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
GitMehdi-sys wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Enable Storm dynamic producer batching in archetype configs and docum… #2261

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions docs/src/main/asciidoc/configuration.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,22 @@ If one of the keys is not present in your YAML file, the default value will be t
Note: Some configuration options may not be applicable depending on the specific components and
features you are using in your Apache StormCrawler topology. Some external modules might define additional options not listed here.

==== Storm Topology Settings

The following are Storm settings rather than StormCrawler ones, but they have a direct effect on how a crawl behaves. They are set in the same `crawler-conf.yaml`. The values below are Storm's defaults. The best values depend on your cluster resources, parallelism and topology, so measure before changing them.

[cols="2,1,6", options="header"]
|===
| Key | Default | Description
| topology.message.timeout.secs | 30 | Time after which a tuple which has not been fully acked is failed and replayed from the spout. It must be larger than the worst-case time for a URL to go through the fetcher, parser and indexer, including time spent waiting in queues.
| topology.max.spout.pending | not set | Maximum number of tuples in flight per spout task. It bounds memory use and the number of URLs being processed at any time. Keep it larger than `topology.producer.batch.size`.
| topology.producer.batch.size | 1 | Maximum number of tuples an executor accumulates before sending them to the next executor. 1 means no batching.
| topology.producer.batch.dynamic | false | When true, the effective batch size adapts at runtime between 1 and `topology.producer.batch.size`: it stays small under light load to keep latency low and grows under sustained load. It has no effect unless `topology.producer.batch.size` is greater than 1. Available since Storm 3.0.
| topology.batch.flush.interval.millis | 1 | How long (in milliseconds) a partially filled batch waits before being flushed, which bounds the latency batching can add.
|===

Batching reduces per-tuple overhead between the spout, fetcher, parser, indexer and status updater. Time a tuple spends waiting in a batch or behind a slow component counts against `topology.message.timeout.secs`.

==== Fetching and Partitioning

[cols="1,1,3", options="header"]
Expand Down