docs: Refresh docs against current code

AGENTS.md

@@ -0,0 +1,48 @@
+# Snuba
+
+Snuba is Sentry's query and ingestion service on top of ClickHouse. The Python service is in `snuba/`, and the Rust Kafka consumers are in `rust_snuba/`.
+
+## Commands
+
+Run these from the repository root.
+
+- `devenv sync` syncs `.venv` and installs pre-commit hooks. `make install-rs-dev` builds `rust_snuba` into `.venv`.
+- `devservices up` starts ClickHouse, Redis, and Kafka. The tests need them.
+- `make test` runs the Python tests. Run one file with `SNUBA_SETTINGS=test pytest tests/path/to/test_file.py`.
+- `make test-rust` runs the Rust tests.
+- `make lint-check` and `make typecheck` run ruff and mypy. `pre-commit run --files <files>` runs the commit hooks.
+- `make lint-rust` and `make format-rust` run clippy and rustfmt.
+- `make validate-configs` validates the dataset YAML in `snuba/datasets/configuration/`.
+- `make generate-config-docs` and then `make snubadocs` build the Sphinx docs. Warnings fail the build.
+- `make listen-metrics` prints the DogStatsD metrics Snuba emits locally.
+
+## Invariants
+
+- After you change Rust code, rerun `make install-rs-dev` before Python code or tests can use it.
+- Never edit `docs/source/configuration/*.md` by hand. `make generate-config-docs` writes them from `snuba/datasets/configuration/json_schema.py`.
+- Add every new page under `docs/source/` to a toctree, or the docs build fails.
+- Snuba assumes UTC everywhere.
+
+## Docs
+
+| Topic | Doc |
+| --- | --- |
+| Local setup and running from source | [docs/source/contributing/environment.rst](docs/source/contributing/environment.rst) |
+| Services Snuba needs, settings | [docs/source/getstarted.rst](docs/source/getstarted.rst) |
+| Architecture and data flows | [docs/source/architecture/overview.rst](docs/source/architecture/overview.rst) |
+| Datasets, entities, storages | [docs/source/architecture/datamodel.rst](docs/source/architecture/datamodel.rst) |
+| Query processing pipeline | [docs/source/architecture/queryprocessing.rst](docs/source/architecture/queryprocessing.rst) |
+| Python and Rust consumers, healthchecks | [docs/source/architecture/consumer.rst](docs/source/architecture/consumer.rst) |
+| Rust consumer pipeline and batching | [rust_snuba/ARCHITECTURE.md](rust_snuba/ARCHITECTURE.md) |
+| Building and testing `rust_snuba` | [rust_snuba/README.md](rust_snuba/README.md) |
+| Dataset YAML schemas | [docs/source/configuration/overview.rst](docs/source/configuration/overview.rst) |
+| HTTP query API, subscriptions | [docs/source/query/overview.rst](docs/source/query/overview.rst), routes in [snuba/web/views.py](snuba/web/views.py) |
+| SnQL and MQL | [docs/source/language/snql.rst](docs/source/language/snql.rst), [docs/source/language/mql.rst](docs/source/language/mql.rst) |
+| Running and writing migrations | [docs/source/migrations/overview.md](docs/source/migrations/overview.md) |
+| Local and distributed migration modes | [docs/source/migrations/modes.rst](docs/source/migrations/modes.rst) |
+| ClickHouse topology, schema design, versions | [docs/source/clickhouse/](docs/source/clickhouse/) |
+| Data slicing | [docs/source/architecture/slicing.rst](docs/source/architecture/slicing.rst) |
+| Allocation policy config overrides | [snuba/query/allocation_policies/OVERRIDES.md](snuba/query/allocation_policies/OVERRIDES.md) |
+| Admin UI | [snuba/admin/README.md](snuba/admin/README.md) |
+| GoCD pipelines | [gocd/README.md](gocd/README.md) |
+| Profiling | [docs/source/profiler.rst](docs/source/profiler.rst) |

README.rst

@@ -1,6 +1,8 @@
 .. image:: https://github.com/getsentry/snuba/raw/master/snuba/admin/dist/snuba.svg
     :width: 280
 
+.. intro-start
+
 Snuba is a service that provides a rich data model on top of ClickHouse
 together with a fast ingestion consumer and a query optimizer.
 
@@ -9,15 +11,38 @@ Redis to search and provide aggregated data on Sentry errors.
 Since then it has evolved into the current form where it supports most
 time series related Sentry features over several data sets.
 
-Click `here <https://getsentry.github.io/snuba/>`_ for the full documentation.
-
 Features:
 ---------
 
-- Provides a database access layer to the ClickHouse distributed data store.
-- Provides a graph logical data model the client can query through the SnQL language which provides functionalities similar to those of SQL.
-- Support multiple separate data sets in a single installation.
-- Provides a rule based query optimizer.
-- Provides a migration system to apply DDL changes to ClickHouse both in a single node and distributed environment.
-- Ingest data directly from Kafka
-- Supports both point in time queries and streaming queries.
+* Provides a database access layer to the ClickHouse distributed data store.
+* Provides a graph logical data model the client can query through the SnQL
+  language which provides functionalities similar to those of SQL.
+* Supports multiple separate data sets in a single installation.
+* Provides a rule based query optimizer.
+* Provides a migration system to apply DDL changes to ClickHouse both in a
+  single node and distributed environment.
+* Ingests data directly from Kafka.
+* Supports both point in time queries and streaming queries.
+
+.. intro-end
+
+Get started
+-----------
+
+To run Snuba in a Sentry development environment, see
+`Getting started <https://getsentry.github.io/snuba/getstarted.html>`_.
+To query it, see `Querying Snuba <https://getsentry.github.io/snuba/query/overview.html>`_.
+
+The full documentation is at https://getsentry.github.io/snuba/.
+
+Get help
+--------
+
+Report bugs and request features in
+`GitHub issues <https://github.com/getsentry/snuba/issues>`_.
+
+License
+-------
+
+Snuba is licensed under the Functional Source License, Version 1.1, Apache 2.0
+Future License. See `LICENSE.md <LICENSE.md>`_.

docs/source/architecture/consumer.rst

@@ -27,9 +27,9 @@ Message processors
 Each storage in Snuba defines a conversion function mapping the layout of a
 Kafka message to a list of ClickHouse rows.
 
-Message processors are defined in :py:mod:`snuba.dataset.processors`, and
+Message processors are defined in :py:mod:`snuba.datasets.processors`, and
 need to subclass from
-:py:class:`snuba.dataset.processors.DatasetMessageProcessor`. Just by
+:py:class:`snuba.datasets.processors.DatasetMessageProcessor`. Just by
 subclassing, their name becomes available for reference in
 ``snuba/datasets/configuration/*/storages/*.yaml``.
 

docs/source/architecture/datamodel.rst

@@ -30,7 +30,7 @@ A Dataset is a name space over Snuba data. It provides its own schema and
 it is independent from other datasets both in terms of logical model and
 physical model.
 
-Examples of datasets are, discover, outcomes, sessions. There is no
+Examples of datasets are discover, outcomes, and replays. There is no
 relationship between them.
 
 A Dataset can be seen as a container for the components that define its