diff --git a/docs/src/main/asciidoc/configuration.adoc b/docs/src/main/asciidoc/configuration.adoc index e5d888ccc..1d2b456a0 100644 --- a/docs/src/main/asciidoc/configuration.adoc +++ b/docs/src/main/asciidoc/configuration.adoc @@ -169,6 +169,22 @@ If one of the keys is not present in your YAML file, the default value will be t Note: Some configuration options may not be applicable depending on the specific components and features you are using in your Apache StormCrawler topology. Some external modules might define additional options not listed here. +==== Storm Topology Settings + +The following are Storm settings rather than StormCrawler ones, but they have a direct effect on how a crawl behaves. They are set in the same `crawler-conf.yaml`. The values below are Storm's defaults. The best values depend on your cluster resources, parallelism and topology, so measure before changing them. + +[cols="2,1,6", options="header"] +|=== +| Key | Default | Description +| topology.message.timeout.secs | 30 | Time after which a tuple which has not been fully acked is failed and replayed from the spout. It must be larger than the worst-case time for a URL to go through the fetcher, parser and indexer, including time spent waiting in queues. +| topology.max.spout.pending | not set | Maximum number of tuples in flight per spout task. It bounds memory use and the number of URLs being processed at any time. Keep it larger than `topology.producer.batch.size`. +| topology.producer.batch.size | 1 | Maximum number of tuples an executor accumulates before sending them to the next executor. 1 means no batching. +| topology.producer.batch.dynamic | false | When true, the effective batch size adapts at runtime between 1 and `topology.producer.batch.size`: it stays small under light load to keep latency low and grows under sustained load. It has no effect unless `topology.producer.batch.size` is greater than 1. Available since Storm 3.0. +| topology.batch.flush.interval.millis | 1 | How long (in milliseconds) a partially filled batch waits before being flushed, which bounds the latency batching can add. +|=== + +Batching reduces per-tuple overhead between the spout, fetcher, parser, indexer and status updater. Time a tuple spends waiting in a batch or behind a slow component counts against `topology.message.timeout.secs`. + ==== Fetching and Partitioning [cols="1,1,3", options="header"]