diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000000..bfeebf7ee1e --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,48 @@ +# Snuba + +Snuba is Sentry's query and ingestion service on top of ClickHouse. The Python service is in `snuba/`, and the Rust Kafka consumers are in `rust_snuba/`. + +## Commands + +Run these from the repository root. + +- `devenv sync` syncs `.venv` and installs pre-commit hooks. `make install-rs-dev` builds `rust_snuba` into `.venv`. +- `devservices up` starts ClickHouse, Redis, and Kafka. The tests need them. +- `make test` runs the Python tests. Run one file with `SNUBA_SETTINGS=test pytest tests/path/to/test_file.py`. +- `make test-rust` runs the Rust tests. +- `make lint-check` and `make typecheck` run ruff and mypy. `pre-commit run --files ` runs the commit hooks. +- `make lint-rust` and `make format-rust` run clippy and rustfmt. +- `make validate-configs` validates the dataset YAML in `snuba/datasets/configuration/`. +- `make generate-config-docs` and then `make snubadocs` build the Sphinx docs. Warnings fail the build. +- `make listen-metrics` prints the DogStatsD metrics Snuba emits locally. + +## Invariants + +- After you change Rust code, rerun `make install-rs-dev` before Python code or tests can use it. +- Never edit `docs/source/configuration/*.md` by hand. `make generate-config-docs` writes them from `snuba/datasets/configuration/json_schema.py`. +- Add every new page under `docs/source/` to a toctree, or the docs build fails. +- Snuba assumes UTC everywhere. + +## Docs + +| Topic | Doc | +| --- | --- | +| Local setup and running from source | [docs/source/contributing/environment.rst](docs/source/contributing/environment.rst) | +| Services Snuba needs, settings | [docs/source/getstarted.rst](docs/source/getstarted.rst) | +| Architecture and data flows | [docs/source/architecture/overview.rst](docs/source/architecture/overview.rst) | +| Datasets, entities, storages | [docs/source/architecture/datamodel.rst](docs/source/architecture/datamodel.rst) | +| Query processing pipeline | [docs/source/architecture/queryprocessing.rst](docs/source/architecture/queryprocessing.rst) | +| Python and Rust consumers, healthchecks | [docs/source/architecture/consumer.rst](docs/source/architecture/consumer.rst) | +| Rust consumer pipeline and batching | [rust_snuba/ARCHITECTURE.md](rust_snuba/ARCHITECTURE.md) | +| Building and testing `rust_snuba` | [rust_snuba/README.md](rust_snuba/README.md) | +| Dataset YAML schemas | [docs/source/configuration/overview.rst](docs/source/configuration/overview.rst) | +| HTTP query API, subscriptions | [docs/source/query/overview.rst](docs/source/query/overview.rst), routes in [snuba/web/views.py](snuba/web/views.py) | +| SnQL and MQL | [docs/source/language/snql.rst](docs/source/language/snql.rst), [docs/source/language/mql.rst](docs/source/language/mql.rst) | +| Running and writing migrations | [docs/source/migrations/overview.md](docs/source/migrations/overview.md) | +| Local and distributed migration modes | [docs/source/migrations/modes.rst](docs/source/migrations/modes.rst) | +| ClickHouse topology, schema design, versions | [docs/source/clickhouse/](docs/source/clickhouse/) | +| Data slicing | [docs/source/architecture/slicing.rst](docs/source/architecture/slicing.rst) | +| Allocation policy config overrides | [snuba/query/allocation_policies/OVERRIDES.md](snuba/query/allocation_policies/OVERRIDES.md) | +| Admin UI | [snuba/admin/README.md](snuba/admin/README.md) | +| GoCD pipelines | [gocd/README.md](gocd/README.md) | +| Profiling | [docs/source/profiler.rst](docs/source/profiler.rst) | diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md deleted file mode 100644 index 59dccc90715..00000000000 --- a/DEVELOPMENT.md +++ /dev/null @@ -1,6 +0,0 @@ -# Snuba Development Guide - -- see for API [snuba/web/views.py](snuba/web/views.py) -- Most documentation lives in [docs/](docs/) -- **[Admin README](snuba/admin/README.md)** — Building and running the Snuba admin UI locally. -- `make listen-metrics` for — Viewing datadog metrics locally. diff --git a/MIGRATIONS.md b/MIGRATIONS.md deleted file mode 100644 index 804d4d1bc80..00000000000 --- a/MIGRATIONS.md +++ /dev/null @@ -1,144 +0,0 @@ -# Snuba migrations field guide - -Snuba provides a set of tools to help manage ClickHouse database schemas and state. - -:warning: The migration system in Snuba is still under development and is subject to change. In particular, automated migrations for clustered deployments of ClickHouse (distributed and replicated tables) are not currently supported, but may still be discussed here and partially supported in some parts of the system. - -### Quick links: - -- [Overview](#overview) - -- [Getting started](#getting-started) - -- [Running migrations](#running-migrations) - -- [Writing migrations](#writing-migrations) - -## Overview - -Migrations are the mechanism through which database changes are defined and applied so that ClickHouse schemas can be evolved as the Snuba codebase changes. - -### Types of migrations - -There are two types of migrations with distinct functionality: - -1. ClickHouse node migrations. These execute SQL statements on one or many ClickHouse machines. These are typically used to change the ClickHouse schema in some way, such as adding or altering tables. Most migrations fall into this category. - -2. Global migrations. A more flexible type of migration that can execute any provided Python functions. Used for data migrations or more custom behavior that cannot be expressed as pure SQL. These are less common. - -### Migration groups - -Migrations follow a strict sequence to ensure that changes are applied in order, and any dependencies between migrations are met. - -Migrations in Snuba are organized into groups, which typically correspond with features or sets of related tables in Snuba. Each migration is applied in order by group, and the groups are themselves ordered. All of the migrations of the first group are applied first, then all of the migrations of the second group, and so on. Hence migrations in subsequent groups can depend on changes in earlier groups being already applied but not vice versa. The `system` group is the one that creates the ClickHouse table used by the migrations system itself, and must always remain as the first group. - -In this codebase, each group is represented by a folder in the `snuba.snuba_migrations` module, and all of the migrations for that group are stored within that folder. - -Migration groups can be defined as either optional or mandatory. Most groups in Snuba are mandatory, however optional groups can provide a mechanism to test experimental features in Snuba that should not be rolled out to most users yet. Optional groups can be toggled by the user via `settings.SKIPPED_MIGRATION_GROUPS`. In most cases, the default in settings.py should not be changed, and doing so may result in unexpected behavior. - -## Getting started - -Snuba supports running ClickHouse in either a single node or multi node configuration. Currently only the single node configuration is fully supported through the migration system. - -### Configure settings.CLUSTERS - -The mapping in `settings.CLUSTERS` defines the relationship between storage sets (groups of tables that must be colocated), and ClickHouse clusters. Every storage set must be assigned to one cluster. - -For each cluster, `single_node` can be set to either True or False. If False, `cluster_name` and `distributed_cluster_name` should also be provided. These should align to the ClickHouse cluster names in `system.clusters` where you want the data to be stored. - -For a single node cluster, do not change the default configuration. - -## Running migrations - -The `snuba migrations` CLI tool should be used to manage migrations. - -### List migrations - -`snuba migrations list` - -- Lists all migrations and their statuses - -### Run all migrations - -`snuba migrations migrate --force` - -- Runs all pending migrations and brings your database to the latest state. -- Running with the --force flag means that any migrations marked blocking (generally because they contain a data migration) will also be executed. Blocking migrations may take some time to complete. Running with --force assumes that any consumers filling the corresponding table are stopped and no new data is being written to the table as the migration is taking place. -- Running this without the --force flag, will only execute the migrations if none are blocking. -- If you are running `snuba devserver`, this command automatically run when the devserver is started and there is no need to manage migrations manually. - -### Run a single migration - -`snuba migrations run --group --migration-id ` - -- Runs a single migration -- Only allowed if there are no prior migrations in that group that have not been completed -- The `--dry-run` option can be used to simply print migration's SQL to stdout (note: not for Python migrations) - -### Reverse a single migration - -`snuba migrations reverse --group --migration-id ` - -- Reverses a single migration -- Only allowed if there are no subsequent completed migrations in that group -- Force must be used if the migration is already marked completed -- The `--dry-run` option can be used to simply print migration's SQL to stdout (note: not for Python migrations) - - -### Fake a migration(s) - -`snuba migrations migrate --fake` - -* will update the status for all migrations in the `migrations_local` table to be completed -* if faking a blocking migration, use `--force` as well -* if faking a specific migration, use the `snuba migrations run` command with `--fake`, and it will only update the status for that migration - - -## Writing migrations - -### Auto-generating migrations -If you want to write a migration for the following common use cases, you can skip the rest of this document and have it autogenerated for you: -* Add a column to an existing table - -Instructions: -1. First locate the relevant `storage.yaml` file for the table you want to update. If you dont already know, this is basically just a schema definition for the table. This file will be located in `snuba/datatsets/configuration//storages/.yaml` ex. `.../group_attributes/storages/group_attributes.yaml`. -2. Make the desired modifications to the storage.yaml. This will likely look something like adding a new column to `schema.columns` -3. run the command `snuba migrations generate path/to/storage.yaml`. This will generate the migration based on the modifications to the storage.yaml file. -4. You're basically done, now you just have to commit, push, and merge your migration! - -### Overview -In order to add a new migration, first determine which migration group the new migration should be added to. You must prefix migration IDs with a number matching the position of the migration in the group, i.e. the 4th migration in that group will be prefixed with `0004_`. Add a file which will contain the new migration at `/migrations/snuba_migrations//.py`. - -If you need to create a new group, add the group to `migrations.groups.MigrationGroup` and a loader to `migrations.group_loader` for the group defining the path to the directory where that group's migrations will be located. Register these to `migrations.groups._REGISTERED_MIGRATION_GROUPS` - note the position of the group in this list determines the order the migrations will be executed in. For a new MigrationGroup, the `readiness_state` should be set to `limited` which means the migrations will only be automatically executed in CI and the local environment. - -The new migration should contain a class called `Migration` which inherits from one of two types: -`ClickhouseNodeMigration` or `CodeMigration` - -### Adding an SQL migration (ClickhouseNodeMigration) - -A ClickhouseNodeMigration is the most common type of migration and determines the SQL to be run on each ClickHouse node. You must define two methods - `forwards_ops` and `backwards_ops` in order to provide the DDL for all ClickHouse layouts that a user may have. These functions return a sequence of sql ops for the forwards and backwards (reverse) migrations. - -Each operation in the sequence takes in a `target` parameter in its `__init__` to define if the operation is to be applied on a local or distributed table. `OperationTarget.LOCAL` for local tables, `OperationTarget.DISTRIBUTED` for distributed. By default, `target=OperationTarget.UNSET`. Your migration will not be applied if it is left unset and the runner will throw an error. - -For some SQL operations, such as `AddColumn` and `CreateTable`, it's important to apply them on local nodes before distributed. For others, such as `DropColumn`, it's important to apply them on the distributed first. For example, if your migration involves adding a column `col1` to a local and distributed table, your `forward_ops` should return a sequence: -``` -[ - AddColumn(target=OperationTarget.LOCAL ...., Column('col1') ), - AddColumn(target=OperationTarget.DISTRIBUTED ...., Column('col1') ) -] -``` - -Doing operations in the wrong order would risk the distributed table momentarily referencing a non-existent column. To assist in preventing these ordering issues, there is a function `validate_migration_order` in the `snuba.migrations.validator` that checks for ordering conflicts. By default, this function is called for all migrations in `tests/migrations/test_validator.py` - -For each forwards method, you should provide the sequence of operations to be run on that node. In case the forwards methods fail halfway, the corresponding backwards methods should also restore the original state so the migration can be retried. -For example if a temporary table is created during the forwards migration, the backwards migration should drop it. - -Once the migration has been added, the Snuba schema for that storage should also be updated accordingly. A test will fail if any columns in the schema are missing or inconsistent with what is found in the database post migration. - -### Adding a Python migration (CodeMigration) - -A CodeMigration can be used if the migration requires more complex behavior and needs to run custom Python code. You should provide the `forwards_global` and `backwards_global` methods. - -### The `blocking` flag - -A migration that can not immediately complete (i.e. requires data to be rewritten) should be marked as `blocking`. Typically this indicates that the migration needs to be run with the relevant consumer stopped or an alternate strategy (e.g. dual writing employed to ensure no downtime as the migration is running. diff --git a/README.rst b/README.rst index fbf6e1ee236..390797b5ea9 100644 --- a/README.rst +++ b/README.rst @@ -1,6 +1,8 @@ .. image:: https://github.com/getsentry/snuba/raw/master/snuba/admin/dist/snuba.svg :width: 280 +.. intro-start + Snuba is a service that provides a rich data model on top of ClickHouse together with a fast ingestion consumer and a query optimizer. @@ -9,15 +11,38 @@ Redis to search and provide aggregated data on Sentry errors. Since then it has evolved into the current form where it supports most time series related Sentry features over several data sets. -Click `here `_ for the full documentation. - Features: --------- -- Provides a database access layer to the ClickHouse distributed data store. -- Provides a graph logical data model the client can query through the SnQL language which provides functionalities similar to those of SQL. -- Support multiple separate data sets in a single installation. -- Provides a rule based query optimizer. -- Provides a migration system to apply DDL changes to ClickHouse both in a single node and distributed environment. -- Ingest data directly from Kafka -- Supports both point in time queries and streaming queries. +* Provides a database access layer to the ClickHouse distributed data store. +* Provides a graph logical data model the client can query through the SnQL + language which provides functionalities similar to those of SQL. +* Supports multiple separate data sets in a single installation. +* Provides a rule based query optimizer. +* Provides a migration system to apply DDL changes to ClickHouse both in a + single node and distributed environment. +* Ingests data directly from Kafka. +* Supports both point in time queries and streaming queries. + +.. intro-end + +Get started +----------- + +To run Snuba in a Sentry development environment, see +`Getting started `_. +To query it, see `Querying Snuba `_. + +The full documentation is at https://getsentry.github.io/snuba/. + +Get help +-------- + +Report bugs and request features in +`GitHub issues `_. + +License +------- + +Snuba is licensed under the Functional Source License, Version 1.1, Apache 2.0 +Future License. See `LICENSE.md `_. diff --git a/docs/source/_static/architecture/sessions_deployment.png b/docs/source/_static/architecture/sessions_deployment.png deleted file mode 100644 index 03423b46d93..00000000000 Binary files a/docs/source/_static/architecture/sessions_deployment.png and /dev/null differ diff --git a/docs/source/architecture/consumer.rst b/docs/source/architecture/consumer.rst index bcf732794da..a2fdd144725 100644 --- a/docs/source/architecture/consumer.rst +++ b/docs/source/architecture/consumer.rst @@ -27,9 +27,9 @@ Message processors Each storage in Snuba defines a conversion function mapping the layout of a Kafka message to a list of ClickHouse rows. -Message processors are defined in :py:mod:`snuba.dataset.processors`, and +Message processors are defined in :py:mod:`snuba.datasets.processors`, and need to subclass from -:py:class:`snuba.dataset.processors.DatasetMessageProcessor`. Just by +:py:class:`snuba.datasets.processors.DatasetMessageProcessor`. Just by subclassing, their name becomes available for reference in ``snuba/datasets/configuration/*/storages/*.yaml``. diff --git a/docs/source/architecture/datamodel.rst b/docs/source/architecture/datamodel.rst index d54906546e4..4681ef393b7 100644 --- a/docs/source/architecture/datamodel.rst +++ b/docs/source/architecture/datamodel.rst @@ -30,7 +30,7 @@ A Dataset is a name space over Snuba data. It provides its own schema and it is independent from other datasets both in terms of logical model and physical model. -Examples of datasets are, discover, outcomes, sessions. There is no +Examples of datasets are discover, outcomes, and replays. There is no relationship between them. A Dataset can be seen as a container for the components that define its diff --git a/docs/source/architecture/overview.rst b/docs/source/architecture/overview.rst index ceb3b300056..aa82d71aa8b 100644 --- a/docs/source/architecture/overview.rst +++ b/docs/source/architecture/overview.rst @@ -96,11 +96,6 @@ Errors and transaction: .. image:: /_static/architecture/errors_transactions_deployment.png - -Sessions: - -.. image:: /_static/architecture/sessions_deployment.png - Outcomes: .. image:: /_static/architecture/outcomes_deployment.png @@ -113,44 +108,35 @@ for the ``Events`` and ``Transactions`` Entities. These two entities serve most issue/errors related features in Sentry and the whole Performance product. -There is only one Kafka topic (``events``) shared between errors and transactions -that feeds this pipeline. This topic contains both error messages and transaction -messages. - -The Errors consumers consumes the ``events`` topic, writes messages in the Clickhouse -``errors`` table. Upon commit it also produces a record on the ``snuba-commit-log`` -topic. +Errors and transactions have separate Kafka topics. Sentry produces errors on +the ``events`` topic and transactions on the ``transactions`` topic. -Alerts on Errors are generated by the Errors Subscription Consumer. This is synchronized -consumer that consumes both the main ``events`` topic and the ``snuba-commit-log`` topic -so it can proceed in lockstep with the main consumer. +The Errors consumer consumes the ``events`` topic and writes messages to the +Clickhouse ``errors`` table. Upon commit it also produces a record on the +``snuba-commit-log`` topic. -The synchronized consumer then produces alerts by querying Clickhouse and produces -the result on the result topic. +Alerts on Errors are generated by the subscriptions scheduler and executor. +The scheduler consumes the ``snuba-commit-log`` topic, so it proceeds in +lockstep with the main consumer, and schedules subscription queries on the +``scheduled-subscriptions-events`` topic. The executor runs those queries +against Clickhouse and produces the results on the +``events-subscription-results`` topic. -An identical but independent pipeline exists for transactions. +An identical but independent pipeline exists for transactions, with its own +``snuba-transactions-commit-log`` topic. -The Errors pipeline has an additional step: writing to the ``replacements`` topic. -Errors mutations (merge/unmerge/reprocessing/etc.) are produced by Sentry on the -``events`` topic. They are then forwarded to the ``replacements`` topic by the -Errors Consumer and executed by the Replacement Consumer. +The Errors pipeline has an additional step: writing to the ``event-replacements`` +topic. Errors mutations (merge/unmerge/reprocessing/etc.) are produced by Sentry +on the ``events`` topic. They are then forwarded to the ``event-replacements`` +topic by the Errors Consumer and executed by the Replacement Consumer. The ``events`` topic must be partitioned semantically by Sentry project id to allow in order processing of the events within a project. This, as of today, is a requirement for alerts and replacements. -Sessions and Outcomes -===================== - -``Sessions`` and ``Outcomes`` work in very similar and simpler way. Specifically -``Sessions`` power Release Health features, while ``Outcomes`` mainly provide -data to the Sentry ``stats`` page. - -Both pipelines have their own Kafka topic, Kafka consumer and they write on their -own table in Clickhouse. - -Change Data Capture pipeline -============================ +Outcomes +======== -This pipeline is still under construction. It consumes the ``cdc`` topic and fills -two independent tables in Clickhouse. +``Outcomes`` mainly provide data to the Sentry ``stats`` page. The pipeline +has its own Kafka topic and Kafka consumer, and it writes to its own table in +Clickhouse. diff --git a/docs/source/architecture/queryprocessing.rst b/docs/source/architecture/queryprocessing.rst index 8caca1c69fd..97bba46f0e9 100644 --- a/docs/source/architecture/queryprocessing.rst +++ b/docs/source/architecture/queryprocessing.rst @@ -3,7 +3,7 @@ Snuba Query Processing ====================== Snuba has a query processing pipeline that starts with the parsing of the -Snuba query language (legacy and SnQL) into an AST ands with a SQL query +Snuba query language (SnQL and MQL) into an AST and ends with a SQL query being executed on Clickhouse. Between these two phases, several passes on the AST to apply query processing transformations are executed. @@ -27,19 +27,17 @@ for the phases discussed above. .. image:: /_static/architecture/queryprocessing.png -Legacy and SnQL Parsers ------------------------ +SnQL and MQL Parsers +-------------------- -Snuba supports two languages, the legacy JSON based one and the new one named -SnQL. With the exceptions of joins and composite queries which are not supported -by the legacy language, the query processing pipeline does not change whether -one or the other language is used. +Snuba supports two languages, SnQL and MQL. The query processing pipeline does +not change whether one or the other language is used. They both produce a logical query AST which is represented by `this data structure `_. -The legacy parser is `here `_, -while the SnQL parser is `in this module `_. +The SnQL parser is `in this module `_, +and the MQL parser is `in this module `_. Query Validation ---------------- @@ -53,12 +51,13 @@ validation. General validation is composed by a set of checks that are applied to each query right after the Query is produced by the parser. This happens -`in this function `_. +`in this module `_. This includes validations like preventing alias shadowing and function signature validation. Each entity can provide some validation logic as well in the form of required -columns. This happens `in this class `_. +columns. This happens in validators such as +`EntityRequiredColumnValidator `_. This allows the query processing to reject queries that do not have a condition on project_id or that do not have a time range. @@ -66,11 +65,12 @@ Logical Query Processors ------------------------ Query processors are stateless transformations that receive a Query object (with -its AST) and transform it in place. `This `_ +its AST) and transform it in place. `LogicalQueryProcessor `_ is the interface to implement for logical processors. In the logical phase each entity provides the query processors to be applied in sequence. Common use -cases are custom functions like `apdex `_, -or time bucketing like the `time series processor `_. +cases are custom functions like ``apdex`` in the +`custom function processor `_, +or time bucketing like the `time series processor `_. Query processors are not supposed to depend on other processors to be executed before or after and should be independent from each other. @@ -85,7 +85,7 @@ for performance reasons as some can respond to some queries faster. At the end of the logical processing phase (which is entirely based on the entity) the storage selector can inspect the query and pick the appropriate storage for the query. Storage selectors are defined in the entity data model and implement -this `interface `_. +the `QueryStorageSelector `_ interface. An example is the Errors entity, which has two storages, one is for consistent queries (they are routed to the same nodes where events are written) and the other only includes replicas we do not write onto to serve most queries. This @@ -108,7 +108,8 @@ Contrarily to query processors, translation rules do not have full context on the query and can only translate an individual expression. This allows us to compose translation rules easily and reuse them across entities. -`These `_ +The ``translation_mappers`` in +`transactions.yaml `_ are the translation rules for the transactions entity. Physical Query Processors @@ -117,7 +118,7 @@ Physical Query Processors Physical query processors work in a very similar way compared to the Logical query processors. Their interface is very similar and the semantics is the same. The difference is that they operate on the physical query and, as such, they -are mainly designed for optimizations. For example `this processor `_ +are mainly designed for optimizations. For example `this processor `_ finds equality conditions on tags and replace them with the equivalent condition on a tags hashmap (where we have a bloom filter index) making the filtering operation faster. diff --git a/docs/source/contributing/environment.rst b/docs/source/contributing/environment.rst index 5d5a620db53..2b52026468f 100644 --- a/docs/source/contributing/environment.rst +++ b/docs/source/contributing/environment.rst @@ -2,77 +2,79 @@ Snuba development environment ============================= -This section explains how to run snuba from source and set up a development -environment. - -In order to set up Clickhouse, Redis, and Kafka, please refer to :doc:`/getstarted`. +This section explains how to run Snuba from source. To set up ClickHouse, +Redis, and Kafka, see :doc:`/getstarted`. Prerequisites ------------- -It is assumed that you have completed the steps to set up the `sentry dev environment `_. -You will need an installation of Rust to develop Snuba. Go `here `_ to get one:: +Set up the `Sentry development environment `_ +first. - curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh +Install ``uv``. ``devenv sync`` no longer installs it:: -Make sure to follow installation steps and configure your shell for cargo (Rust's build sytem and package manager). + brew install uv -If you are using Homebrew and a M1 Mac, ensure the development packages you've installed with Homebrew are available -by setting these environment variables:: +If you use Homebrew on an Apple Silicon Mac, make the Homebrew development +packages visible to the Rust build:: export CPATH=/opt/homebrew/include export LIBRARY_PATH=/opt/homebrew/lib -Install / Run -------------- +Install and run +--------------- -Clone this repo into your workspace:: +Clone the repository:: git clone git@github.com:getsentry/snuba.git + cd snuba -These commands set up the Python virtual environment:: +Set up the environment. ``direnv allow`` puts ``.venv`` and ``rustup`` on your +``PATH``. ``devenv sync`` installs ``rustup``, ``cmake``, and ``protobuf`` from +``Brewfile``, syncs the Python virtual environment, and installs the +pre-commit hooks:: - cd snuba + direnv allow devenv sync -This builds rust_snuba (it's expensive, so is kept out of `devenv sync`):: +Build ``rust_snuba``. ``devenv sync`` skips it because the build is slow:: - make install-rs-dev (one time) - make watch-rust-snuba (or watch and rebuild) + make install-rs-dev -This command starts the Snuba api, which is capable of processing queries:: +To rebuild ``rust_snuba`` on every change, run ``make watch-rust-snuba`` instead. + +Start the Snuba API, which serves queries:: snuba api -This command starts the api and Snuba consumers to ingest -data from Kafka:: +Start the API and the consumers that ingest data from Kafka:: snuba devserver -Running tests -------------- +Run tests +--------- -This command runs unit and integration tests:: +Run the Python unit and integration tests:: - devenv sync (if you have not run it already) make test -Running sentry tests against snuba -++++++++++++++++++++++++++++++++++ +Run the Rust tests:: -This section instead runs Sentry tests against a running Snuba installation + make test-rust -Make sure there is no snuba container already running:: +Run Sentry tests against Snuba +++++++++++++++++++++++++++++++ + +Make sure no Snuba container is running:: docker ps -a | grep snuba -Start your local snuba api server:: +Start your local Snuba API:: git checkout your-snuba-branch - source .venv/bin/activate snuba api -and, in another terminal:: +In another terminal, update Sentry and start Snuba's dependencies:: cd ../sentry git checkout master @@ -80,12 +82,10 @@ and, in another terminal:: devservices up devservices toggle snuba local -This will get the most recent version of Sentry on master, and bring up all snuba's dependencies. - -You will want to run the following Sentry tests:: +Then run the Sentry tests that cover your change. For example, the Sentry tests +under ``tests/snuba/`` query Snuba:: - make test-acceptance - make test-snuba - make test-python + .venv/bin/pytest tests/snuba -These tests do not use Kafka due to performance reasons. The snuba test suite does test the kafka functionality +These tests do not use Kafka, for performance. The Snuba test suite covers +the Kafka paths. diff --git a/docs/source/getstarted.rst b/docs/source/getstarted.rst index cf2bcd5e1f1..970bc2d8d9b 100644 --- a/docs/source/getstarted.rst +++ b/docs/source/getstarted.rst @@ -2,60 +2,65 @@ Getting started with Snuba ========================== -This is a guide to quickly start Snuba up in the context of a Sentry -development environment. +This guide starts Snuba in a Sentry development environment. Requirements ------------ -Snuba assumes: +Snuba needs: -1. A Clickhouse server endpoint at ``CLICKHOUSE_HOST`` (default ``127.0.0.1``). -2. A redis instance running at ``REDIS_HOST`` (default ``127.0.0.1``). On port - `6379` -3. A Kafka cluster running at ``127.0.0.1`` on port `9092`. +1. A ClickHouse server at ``CLICKHOUSE_HOST`` (default ``127.0.0.1``). +2. A Redis instance at ``REDIS_HOST`` (default ``127.0.0.1``) on ``REDIS_PORT`` + (default ``6379``). +3. A Kafka cluster at ``DEFAULT_BROKERS`` (default ``127.0.0.1:9092``). -A quick way to get these services running is to set up sentry, and add the following line -in ``~/.sentry/sentry.conf.py``:: +Snuba assumes that everything runs on UTC. Other timezones cause timezone +mismatches. - SENTRY_EVENTSTREAM = "sentry.eventstream.kafka.KafkaEventStream" +Start Snuba with Sentry +----------------------- -Then start the services and switch Snuba to local development:: +With a Sentry development environment set up, start the services from the +Sentry repository:: devservices up - devservices toggle snuba local -To switch Snuba back to a containerized deployment, use:: +This runs Snuba in a container. To run Snuba from your Snuba checkout instead, +switch it to local mode:: - devservices toggle snuba containerized + devservices toggle snuba local -Note that Snuba assumes that everything is running on UTC time. Otherwise -you may experience issues with timezone mismatches. +To switch back to the container, run:: + devservices toggle snuba containerized -Sentry + Snuba --------------- +Start only Snuba's dependencies +------------------------------- -Add/change the following lines in ``~/.sentry/sentry.conf.py``:: +From the Snuba repository, start ClickHouse, Redis, and Kafka:: - SENTRY_SEARCH = 'sentry.search.snuba.EventsDatasetSnubaSearchBackend' - SENTRY_TSDB = 'sentry.tsdb.redissnuba.RedisSnubaTSDB' - SENTRY_EVENTSTREAM = 'sentry.eventstream.snuba.SnubaEventStream' + devservices up -Run:: +Then start Snuba itself as described in :doc:`/contributing/environment`. - devservices up +Query ClickHouse directly +------------------------- -Access raw clickhouse client (similar to psql):: +To open a ClickHouse client (similar to ``psql``), run:: docker exec -it snuba-clickhouse-1 clickhouse-client -Data is written into the table `sentry_local`: `select count() from sentry_local;` +Errors are written to the ``errors_local`` table:: + + select count() from errors_local; Settings -------- -Settings are found in ``settings.py`` +Settings are in ``snuba/settings/__init__.py``. To override them, set +``SNUBA_SETTINGS`` to a short name such as ``test``, which loads +``snuba/settings/settings_test.py``, or to an absolute path to a settings file. -- ``CLUSTERS`` : Provides the list of clusters and the hostname, port, and storage sets that should run on each cluster. Local vs distributed is also set per cluster. -- ``REDIS_HOST`` : The host redis is running on. +- ``CLUSTERS``: The ClickHouse clusters, with the host, port, and storage sets + for each. Each cluster also sets whether it is single node or distributed. +- ``REDIS_HOST``: The host Redis runs on. diff --git a/docs/source/index.rst b/docs/source/index.rst index b5b9ba14db2..e63624f940e 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -16,6 +16,7 @@ Contents: query/overview language/snql language/mql + migrations/overview migrations/modes contributing/environment clickhouse/topology diff --git a/docs/source/intro.rst b/docs/source/intro.rst index 80f331c6d37..11fe1cad873 100644 --- a/docs/source/intro.rst +++ b/docs/source/intro.rst @@ -1,23 +1,6 @@ -Snuba is a service that provides a rich data model on top of Clickhouse -together with a fast ingestion consumer and a query optimizer. - -Snuba was originally developed to replace a combination of Postgres and -Redis to search and provide aggregated data on Sentry errors. -Since then it has evolved into the current form where it supports most -time series related Sentry features over several data sets. - -Features: ---------- - -* Provides a database access layer to the Clickhouse distributed data store. -* Provides a graph logical data model the client can query through the SnQL - language which provides functionalities similar to those of SQL. -* Support multiple separate data sets in a single installation. -* Provides a rule based query optimizer. -* Provides a migration system to apply DDL changes to Clickhouse both in a - single node and distributed environment. -* Ingest data directly from Kafka -* Supports both point in time queries and streaming queries. +.. include:: ../../README.rst + :start-after: .. intro-start + :end-before: .. intro-end Some use cases in Sentry: @@ -27,7 +10,4 @@ Some use cases in Sentry: functionality is powered by Snuba as well as all the aggregations. * The ``discover`` data set powers all the Performance Monitoring related features. -* The ``sessions`` data set powers the Releases feature. Specifically this - data set ingests a much higher volume of data points and stores pre-aggregated - data to allow fast queries over higher volume of data. * The ``outcomes`` data set powers the Stats page. diff --git a/docs/source/migrations/modes.rst b/docs/source/migrations/modes.rst index cabc5055be0..008ac55ed3a 100644 --- a/docs/source/migrations/modes.rst +++ b/docs/source/migrations/modes.rst @@ -1,52 +1,39 @@ ====================== -Snuba Migration Modes +Snuba migration modes ====================== -This document outlines a way to try out distributed migrations. -Note that this is experimental, and should be used only for development -purposes at the moment. Distributed mode is not supported when testing yet. -Local mode for migrations is currently fully supported. +Snuba runs migrations against a single node ClickHouse (local mode) or a +replicated, sharded cluster (distributed mode). The mode comes from +``settings.CLUSTERS``. See :doc:`overview` for the settings. -If you are running ClickHouse via Sentry's devservices, the main "switch" -between the two modes for running data migrations (local and distributed) is -``SENTRY_DISTRIBUTED_CLICKHOUSE_TABLES``. Its default is defined in -`src/sentry/conf/server.py `_, -and you can override it in ``~/.sentry/sentry.conf.py``. +Local mode +========== -Once this boolean variable is set, one of two ClickHouse Docker volumes will be -used for data storage, depending on the mode (distributed or local). Whenever a user -wants to switch between the two modes, they must "turn off" the running ClickHouse -container, alter the environment variable mentioned above, and then "turn on" the -same container to be in the new mode. +Local mode is the default. The ClickHouse container that ``devservices up`` +starts is a single node, and the default settings use it:: -More information on migrations in general can be found `here `_. + devservices up + snuba migrations migrate --force -Enabling Local Mode -===================== +Distributed mode +================ -In ``~/.sentry/sentry.conf.py``, set ``SENTRY_DISTRIBUTED_CLICKHOUSE_TABLES`` -to False. This is the default setting, so configuration is already -set up for local mode migrations. Start up the corresponding ClickHouse -container (``devservices up clickhouse``). +Distributed mode needs a ClickHouse cluster with ZooKeeper. The devservices +ClickHouse is a single node, so it cannot run distributed migrations. -Now, run migrations as expected (``snuba migrations migrate --force``). +To run the distributed migration tests locally, run:: + make test-distributed-migrations -Enabling Distributed Mode -============================ +The target builds a Snuba image and starts the ``multi_node`` profile of +``docker-compose.gcb.yml``. That profile runs ZooKeeper, a query node, and four +storage nodes, and runs the tests in ``test_distributed_migrations/``. -In ``~/.sentry/sentry.conf.py``, set ``SENTRY_DISTRIBUTED_CLICKHOUSE_TABLES`` -to True. Start up the corresponding ClickHouse container (``devservices up clickhouse``). -Make sure that the Zookeeper container is also running; without it, distributed migrations -will not work properly. +To run migrations against your own cluster, set ``SNUBA_SETTINGS=distributed``. +This loads ``snuba/settings/settings_distributed.py``, which assigns every +storage set to the one-shard cluster ``cluster_one_sh``:: -Set up cluster connection details (for example, which storage is assigned to -which cluster) in -`snuba/settings/settings_distributed.py `_. -This is needed only for distributed migrations. The default configuration uses -a one-shard cluster and supports migrations for all storages. More information -about distributed tables is available in the -`ClickHouse documentation `_. + SNUBA_SETTINGS=distributed snuba migrations migrate --force -Now, run migrations with the ``SNUBA_SETTINGS`` environment variable pointing to distributed mode. -This can be done as follows: ``SNUBA_SETTINGS=distributed snuba migrations migrate --force``. +More information about distributed tables is in the +`ClickHouse documentation `_. diff --git a/docs/source/migrations/overview.md b/docs/source/migrations/overview.md new file mode 100644 index 00000000000..01f4e783cb6 --- /dev/null +++ b/docs/source/migrations/overview.md @@ -0,0 +1,111 @@ +# Snuba migrations + +Migrations define and apply ClickHouse schema changes as the Snuba codebase changes. You manage them with the `snuba migrations` CLI. + +## Types of migrations + +1. ClickHouse node migrations (`ClickhouseNodeMigration`) run SQL statements on one or many ClickHouse nodes. Most migrations change the schema, such as adding or altering tables, and use this type. +2. Code migrations (`CodeMigration`) run Python functions. Use them for data migrations or custom behavior that SQL cannot express. They are less common. + +## Migration groups + +Migrations are organized into groups that usually correspond to a feature or a set of related tables. Groups are ordered, and every migration in a group runs before any migration in the next group. A migration can depend on changes from earlier groups, but not from later ones. The `system` group creates the table the migration system itself uses, and it must stay first. + +Each group is a directory in `snuba/snuba_migrations/`, except `system`, which lives in `snuba/migrations/system_migrations/`. The groups and their order are registered in `_REGISTERED_MIGRATION_GROUPS` in `snuba/migrations/groups.py`. + +Groups are mandatory by default. Groups listed in `OPTIONAL_GROUPS` in `snuba/migrations/groups.py` can be skipped with `settings.SKIPPED_MIGRATION_GROUPS`. In most cases, keep the default in `snuba/settings/__init__.py`. + +## Configure settings.CLUSTERS + +`settings.CLUSTERS` maps storage sets (groups of tables that must be colocated) to ClickHouse clusters. Every storage set must be assigned to one cluster. + +For each cluster, set `single_node` to `True` or `False`. If it is `False`, also set `cluster_name` and `distributed_cluster_name` to the ClickHouse cluster names in `system.clusters` where the data should be stored. For a single node cluster, keep the default configuration. For distributed mode, see {doc}`modes`. + +## Run migrations + +### List migrations + +`snuba migrations list` lists all migrations and their statuses. + +### Run all migrations + +`snuba migrations migrate --force` runs all pending migrations and brings the database to the latest state. + +- `--force` also runs migrations marked blocking, which usually contain a data migration. Blocking migrations can take a long time. Stop any consumers that write to the affected table before you run them. +- Without `--force`, the command runs only if no pending migration is blocking. +- `snuba devserver` runs this command on startup, so you do not need to run it yourself. + +### Run a single migration + +`snuba migrations run --group --migration-id ` + +- The command is allowed only if every earlier migration in that group is completed. +- `--dry-run` prints the migration's SQL to stdout. It does not work for code migrations. + +### Reverse a single migration + +`snuba migrations reverse --group --migration-id ` + +- The command is allowed only if no later migration in that group is completed. +- If the migration is already marked completed, add `--force`. +- `--dry-run` prints the migration's SQL to stdout. It does not work for code migrations. + +### Fake migrations + +`snuba migrations migrate --fake` marks every migration as completed in the `migrations_local` table without running it. + +- If you fake a blocking migration, add `--force`. +- To fake one migration, run `snuba migrations run` with `--fake`. Only that migration's status changes. + +## Write migrations + +### Generate a migration + +`snuba migrations generate` writes a migration for you. It supports only adding a column to an existing table. + +1. Find the storage file for the table, at `snuba/datasets/configuration//storages/.yaml`. For example, `snuba/datasets/configuration/group_attributes/storages/group_attributes.yaml`. +2. Edit the file. To add a column, add it to `schema.columns`. Leave the change uncommitted, because the command diffs the working tree against `HEAD`. +3. From the repository root, run `snuba migrations generate path/to/storage.yaml`. The command writes the migration to `snuba/snuba_migrations//`. +4. Commit the storage change and the generated migration together. + +### Add a migration by hand + +1. Pick the group for the migration. +2. Add a file at `snuba/snuba_migrations//.py`. Prefix the migration ID with its zero-padded position in the group. For example, the fourth migration in a group starts with `0004_`. +3. In the file, define a class named `Migration` that inherits from `ClickhouseNodeMigration` or `CodeMigration`. + +To create a new group: + +1. Add the group to `MigrationGroup` in `snuba/migrations/groups.py`. +2. Add a loader for the group to `snuba/migrations/group_loader.py`. The loader names the directory that holds the group's migrations. +3. Register the group in `_REGISTERED_MIGRATION_GROUPS`. Its position in that dict sets the order in which groups run. +4. Set `readiness_state` to `ReadinessState.LIMITED`, so the migrations run automatically only in CI and local environments. + +### Add an SQL migration (ClickhouseNodeMigration) + +Define `forwards_ops` and `backwards_ops`. Each returns a sequence of SQL operations for the forwards and backwards (reverse) migration, covering every ClickHouse layout a user may have. + +Each operation takes a `target` argument that says whether it applies to the local or the distributed table: `OperationTarget.LOCAL` or `OperationTarget.DISTRIBUTED`. The default is `OperationTarget.UNSET`, and the runner raises an error for an operation with an unset target. + +Apply `AddColumn` and `CreateTable` to local tables before distributed tables. Apply `DropColumn` to distributed tables first. Otherwise, the distributed table can briefly reference a column that does not exist. For example, to add `col1` to a local and a distributed table, `forwards_ops` returns: + +```python +[ + AddColumn(target=OperationTarget.LOCAL, ..., Column("col1")), + AddColumn(target=OperationTarget.DISTRIBUTED, ..., Column("col1")), +] +``` + +`validate_migration_order` in `snuba/migrations/validator.py` checks for these ordering conflicts. `tests/migrations/test_validator.py` runs it for all migrations. + +If the forwards operations fail halfway, the backwards operations must restore the original state so you can retry the migration. For example, if the forwards migration creates a temporary table, the backwards migration drops it. + +After you add the migration, update the storage schema to match. A test fails if a column in the schema is missing from the database or differs from it after migrating. + +### Add a Python migration (CodeMigration) + +Define `forwards_global` and `backwards_global`. + +### Mark a migration blocking + +Mark a migration `blocking` if it cannot complete immediately, for example because it rewrites data. Run a blocking migration with the relevant consumer stopped, or use another strategy, such as dual writing, to avoid downtime while it runs. diff --git a/docs/source/query/overview.rst b/docs/source/query/overview.rst index 1c5686158ae..fc792165f89 100644 --- a/docs/source/query/overview.rst +++ b/docs/source/query/overview.rst @@ -170,8 +170,9 @@ to build Snuba queries. Once a ``Query`` object has been created the Snuba client api provided by Sentry can and should be used to send the query to Snuba. -The api is in `this module `_. -It takes care of caching, retries and allows bulk queries. +Use ``raw_snql_query`` or ``bulk_snuba_queries`` in +`sentry.utils.snuba `_. +They take care of caching and retries, and ``bulk_snuba_queries`` sends several queries at once. The method returns a dictionary that contains the data in response and additional metadata:: diff --git a/rust_snuba/ARCHITECTURE.md b/rust_snuba/ARCHITECTURE.md index 53cdfd2c824..e33d829392e 100644 --- a/rust_snuba/ARCHITECTURE.md +++ b/rust_snuba/ARCHITECTURE.md @@ -17,37 +17,17 @@ sources of truth for config. ## Steps -`rust_snuba` then uses the Rust port of Arroyo to define its own consumer. The following steps (processing strategies) exist (see `factory.rs`): - -1. Healthcheck step (`--health-check arroyo|commit-progress|partition-stall`), - touches a file as part of the main event loop to signal to k8s that the - consumer is still alive. -2. `SetJoinTimeout` step, resets the join timeout to 0 for the next steps. This - marks a section where it's ok to drop in-flight progress during rebalancing. - - * Message transformation (i.e. JSON parsing) has no side-effects, can be safely retried. - * The reduce/batching step can be retried because even though it opens a - connection to ClickHouse, the INSERT is not committed yet. You can open a - socket to ClickHouse and start writing rows, and for as long as you don't - explicitly close the request with an empty chunk (which happens in - `ClickhouseWriterStep`), the rows will not be visible. -3. Message transformers, basically a threadpool to parse messages, validate - them and turn them into rows. This can call both back into Python code for - when the MessageProcessor hasn't been ported to Rust yet, but most datasets - have a Rust-native processor. -4. `Reduce` opens batches by opening a connection to clickhouse, and writes - rows into them one-by-one. There is only one batch at a time, and the size - of a batch is constrained by `--max-batch-time-ms` and `--max-batch-size`. -5. `SetJoinTimeout` is reset to "no timeout" here. Between this and the - previous section, everything will be dropped by Arroyo during rebalancing as - parsing and validating messages has no side effects. Even sending data to - ClickHouse is idempotent for as long as the batch is not closed. -5. `ClickhouseWriterStep` is responsible for closing batches. It takes a batch - from the previous step, signals to ClickHouse that the batch is done, and - waits for ClickHouse to return 200 OK. -6. COGS and commit log are two topics we produce metadata into once we're done - writing to ClickHouse, but they are typically not relevant for performance. - +`rust_snuba` uses the Rust port of Arroyo to define its own consumer. `ConsumerStrategyFactoryV2` in `src/factory_v2.rs` builds these processing strategies, in the order a message passes through them: + +1. Healthcheck step, only with `--health-check-file`. `--health-check arroyo|commit-progress|partition-stall` picks the strategy. It touches a file to signal to Kubernetes that the consumer is alive. See [Healthchecks](../docs/source/architecture/consumer.rst). +2. `DlqByAge`, only when the storage has a DLQ. It sends a configurable share of too-old messages to the DLQ so a backlogged consumer can catch up. +3. `SetJoinTimeout`, set to `--join-timeout-ms`. During a rebalance, in-flight work in the next steps is dropped after this timeout instead of awaited. Dropping is safe for parsing and batching, which have no side effects. +4. Message transformers. A thread pool parses messages, validates them, and turns them into rows. Most datasets have a Rust-native processor. With `--use-python-processor`, or when no Rust processor exists, the step calls back into the Python `MessageProcessor`. Processors that emit replacements are followed by `ProduceReplacements`. +5. `Reduce` accumulates rows into one in-memory `BytesInsertBatch`. `--max-batch-size` (rows, or bytes with `--max-batch-size-calculation=bytes`) and `--max-batch-time-ms` bound a batch. Empty batches are flushed too, so offsets still get committed. +6. The ClickHouse writer, `JsonWriterStep` or, with `--use-row-binary`, `RowBinaryWriterStep` (in `src/strategies/clickhouse/writer_v2.rs`). It sends each batch as one HTTP `INSERT` and waits for ClickHouse to respond. Up to `--clickhouse-concurrency` batches (default 2) are in flight at once. +7. `SetJoinTimeout` again, set to `--join-timeout-ms`, for the steps after the write. +8. `RecordCogs` produces COGS data to the accountant topic, and `ProduceCommitLog` produces to the commit log topic. Each runs only when configured. They are typically not relevant for performance. +9. `CommitOffsets` commits Kafka offsets every second. ## Batching @@ -61,46 +41,17 @@ insert once every 12 seconds (`--max-batch-time-ms=12000`), so that on average there is one insert per second. In the ops repo this parameter is abstracted away with `batch_time_per_replica`, which is typically 1000 (ms). -At a lower level, batching is done using a custom reduce state. Every time a -new batch is opened, a new `BytesInsertBatch` is constructed, which -opens a connection to ClickHouse when the first row is written. - -The `HttpBatch` internally uses the `reqwest::Client` to stream data into -ClickHouse. `reqwest` asks for a stream type, so we create a bounded channel. -The receiver belongs to reqwest and we use the sender in `write_rows` to write -chunks of many rows into the channel. This indirection exists mostly because it -is more ergonomic for us to be able to call a function to write more data to -ClickHouse, but reqwest wants an interface where it can pull data out of. - -This design allows the consumer to open very large batches that do not fit into -consumer memory. Even in lower-scale datasets the memory savings are pretty -huge. - -Known issues: - -* If the `ClickhouseWriterStep` exerts backpressure on the previous step, it - can theoretically happen that `Reduce` is unable to flush its own `HttpBatch` - in a reasonable timeframe. This means that there is an idle connection - lingering around in `Reduce`, and ClickHouse may close it because of that - idleness, causing the consumer to crash after e.g. 30 seconds of it being - stuck. -* There can be backpressure between writing a row into the channel and sending - it to ClickHouse, meaning the channel will "fill up". If this backpressure - continues for a very long time, the consumer will panic instead of exerting - backpressure further. So far this has never happened. +`Reduce` holds the whole batch in consumer memory. The writer step lz4-compresses the encoded rows and sends them in one HTTP request, retrying according to the ClickHouse write retry policy in `src/options.rs`. Size `--max-batch-size` and `--clickhouse-concurrency` so that the in-flight batches fit in memory. ```mermaid flowchart TD - D1[HealthCheck Step] - D1 --> D2[SetJoinTimeout to 0 ms - Enter the idempotency Zone] - D2 --> E[Message Transformers] - E --> F[Reduce Step - Open & Stream Rows to ClickHouse] - F --> W[MPSC channel, one per batch] - W --> W1[writer task, one per batch, drives reqwest client that pulls from channel] - W1 --> H - F --> G[SetJoinTimeout to ∞ ms - Exit the idempotency Zone] - G --> H[ClickhouseWriterStep - Finalize & Confirm Batch] - - H --> I2[Produce to COGS Topic] - I2 --> I3[Produce to Commit Log Topic] + A[HealthCheck step] --> B[DlqByAge, if a DLQ is configured] + B --> C[SetJoinTimeout] + C --> D[Message transformers] + D --> E[Reduce - accumulate rows into a batch] + E --> F[JsonWriterStep or RowBinaryWriterStep - INSERT the batch] + F --> G[SetJoinTimeout] + G --> H[RecordCogs, if configured] + H --> I[ProduceCommitLog, if configured] + I --> J[CommitOffsets] ``` diff --git a/rust_snuba/README.md b/rust_snuba/README.md index 0a57c907751..6d5d97616f5 100644 --- a/rust_snuba/README.md +++ b/rust_snuba/README.md @@ -1,10 +1,18 @@ -# Rust impl of consumers (Experimental) +# rust_snuba -Rust consumers are an experimental project in Snuba. Most users should not be running this code! +`rust_snuba` is the Rust implementation of Snuba's Kafka consumers. It is compiled with PyO3 and maturin into a Python wheel, and `snuba rust-consumer` launches it from the Python CLI. -The goal of this project is to provide a feature complete Rust equivalent to the `snuba consumer` and `snuba multistorage-consumer` functionality that is currently written in Python code. +For the processing pipeline and batching, see [ARCHITECTURE.md](ARCHITECTURE.md). For how the Rust and Python consumers relate, see [Snuba consumers](../docs/source/architecture/consumer.rst). -## How to run +## Build -1. Run `make watch-rust-snuba`. -2. `snuba rust-consumer` can now be used to run a simple Rust consumer that currently does not insert into clickhouse. +Run these from the repository root after `devenv sync`: + +- `make install-rs-dev` builds `rust_snuba` into `.venv` once. +- `make watch-rust-snuba` rebuilds it on every change. + +## Test and lint + +- `make test-rust` runs `cargo test --workspace`. +- `make lint-rust` runs clippy with warnings as errors. +- `make format-rust` runs `cargo fmt --all`. diff --git a/snuba/cli/migrations.py b/snuba/cli/migrations.py index 04b8cba2b18..db97b66e8ad 100644 --- a/snuba/cli/migrations.py +++ b/snuba/cli/migrations.py @@ -354,7 +354,7 @@ def generate(storage_path: str, name: str | None = None) -> None: The generated migration will be written into the local directory. The user is responsible for making the commit, PR, and merging. - see MIGRATIONS.md in the root folder for more info + see docs/source/migrations/overview.md for more info """ expected_pattern = r"(.+/)?snuba/datasets/configuration/.*/storages/.*\.(yml|yaml)" if not re.fullmatch(expected_pattern, storage_path): diff --git a/snuba/query/parser/README.md b/snuba/query/parser/README.md deleted file mode 100644 index 66967665483..00000000000 --- a/snuba/query/parser/README.md +++ /dev/null @@ -1,259 +0,0 @@ -Querying Snuba - -#### selected_columns, groupby - -`groupby` is a list of columns (or column aliases) that will be translated into -a SQL `GROUP BY` clause. These columns are automatically included in the query -output. `selected_columns` is a list of additional columns that should be added -to the `SELECT` clause. - -Trying to use both of these in the same query will probably result in an invalid -query, as you cannot select a bare column, while grouping by another column, as -the value of the extra selected column for a given output row (group) would be -ambiguous. - -#### aggregations - -This is an array of 3-tuples of the form: - - [function, column, alias] - -which is transformed into the SQL: - - function(column) AS alias - -Some aggregation function are generated by other functions, eg topK. so an -example query would send: - - ["top5", "environment", "top_five_envs"] - -To produce the SQL: - - topK(5)(environment) AS top_five_envs - -Count is a somewhat special case, it doesn't have a column argument, and is -specified as "count()", not "count". - - ["count()", null, "item_count"] - -Aggregations are also included in the output columns automatically. - - -#### conditions - -Conditions are used to construct the WHERE clause, and consist of an -array of 3-tuples (in their most basic form): - - [column_name, operation, literal] - -Valid operations: - - ['>', '<', '>=', '<=', '=', '!=', 'IN', 'NOT IN', 'IS NULL', 'IS NOT NULL', 'LIKE', 'NOT LIKE'], - -For example: - - [ - ['platform', '=', 'python'], - ] - - platform = 'python' - -Top-level sibling conditions are `AND`ed together: - - [ - ['w', '=', '1'], - ['x', '=', '2'], - ] - - w = '1' AND x = '2' - -The first position (column_name) can be replaced with an array that -represents a function call. The first item is a function name, and nested -arrays represent the arguments supplied to the preceding function: - - [ - [['fn1', []], '=', '1'], - ] - - fn1() = '1' - -Multiple arguments can be provided: - - [ - [['fn2', ['arg', 'arg']], '=', '2'], - ] - fn2(arg, arg) = '2' - -Function calls can be nested: - - [ - [['fn3', ['fn4', ['arg']]], '=', '3'], - ] - - - fn3(fn4(arg)) = '3' - -An alias can be provided at the end of the top-level function array. This -alias can then be used elsewhere in the query, such as `selected_columns`: - - [ - [['fn1', [], 'alias'], '=', '1'], - ] - - (fn1() AS alias) = '1' - -To do an `OR`, nest the array one level deeper: - - [ - [ - ['w', '=', '1'], - ['x', '=', '2'], - ], - ] - - (w = '1' OR x = '2') - -Sibling arrays at the second level are `AND`ed (note this is the same -as the simpler `AND` above): - - [ - [ - ['w', '=', '1'], - ], - [ - ['x', '=', '2'], - ], - ] - - (w = '1' AND x = '2') - -And these two can be combined to mix `OR` and `AND`: - - [ - [ - ['w', '=', '1'], ['x', '=', '2'] - ], - [ - ['y', '=', '3'], ['z', '=', '4'] - ], - ] - - (w = '1' OR x = '2') AND (y = '3' OR z = '4') - -#### from_date / to_date - -#### granularity - -Snuba provides a magic column `time`, that you can use in groupby or filter -expressions. This column gives a floored time value for each event so that -events in the same minute/hour/day/etc. can be grouped. - -`granularity` determines the number of seconds in each of these time buckets. -Eg, to count the number of events by hour, you would do - - { - "aggregations": [["count()", "", "event_count"]], - "granularity": 3600, - "groupby": "time" - } - -#### having - -#### issues - -#### project - -#### sample - -Sample is a numeric value. If it is < 1, then it is interpreted to mean "read -this percentage of rows". eg. - - "sample": 0.5 - -Will read 50% of all rows. - -If it is > 1, it means "read up to this number of rows", eg. - - "sample": 1000 - -Will read 1000 rows maximum, and then return a result. - -Note that sampling does not do any adjustment/correction of aggregates. so if -you do a count() with 10% sampling, you should multiply the results by 10 to -get an approximate value for the 'real' count. For sample > 1 you cannot do -this adjustment as there is no way to tell what percentage of rows were read. -For other aggregations like uniq(), min(), max(), there is no adjustment you -can do, the results will simply deviate more and more from the real value in an -unpredictable way as the sampling rate approaches 0. - -Queries with sampling are stable. Ie the same query with the same sampling -factor over the same data should consistently return the exact same result. - - -### Groups / Issues - -Snuba provides a magic column `group_id` that can be used to group events by issue. - -Because events can be reassigned to different issues through merging, and -because snuba does not support updates, we cannot store the issue id for an -event in snuba. If you want to filter or group by `group_id`, you need to pass a -list of `group_ids` into the query. This list is a mapping from issue ids to the -event `primary_hash`es in that issue. Snuba automatically expands this mapping -into the query so that filters/grouping on `group_id` will just work. - -### Tags - -Event tags are stored in one of 2 ways. Promoted tags are the ones we expect to -be queried often and as such are stored as top level columns. The list of -promoted tag columns is defined in settings and is somewhat fixed in the -schema. The rest of an event's tags are stored as a key-value map. In practice -this is implemented as 2 columns of type Array(String), called `tags.key` and -`tags.value` - -The snuba service provides 2 mechanisms for abstracting this tiered tag -structure by providing some special columns that will be resolved to the -correct SQL expression for the type of tag. These mechanisms should generally -not be used in conjunction with each other. - -#### When you know the names of the tags you want. - -You can use the `tags[name]` anywhere you would use a normal column name in an -expression, and it will resolve to the value of the tag with `name`, regardless -of whether that tag is promoted or not. Use this syntax when you are looking -for a specific named tag. eg. - - # Find all events in production with user_custom_key defined. - "conditions": [ - ["tags[environment]", "=", "prod"], - ["tags[custom_user_tag]", "IS NOT NULL"] - ], - - - # Find the number of unique environments - "aggregations": [ - ["uniq", "tags[environment]", "unique_envs"], - ], - -#### When you don't know the name, or want to query all tags. - -These are virtual columns that can be used to get results when the names of the -tags are not explicitly known. Using `tags_key` or `tags_value` in an -expression will expand all of the promoted and non-promoted tags so that there -is one row per tag (an array-join in Clickhouse terms). For each row, the name -of the tag will be in the `tags_key` column, and the value in the `tags_value` -column. - - # Find the top 5 most often used tags - "aggregations": [ - ["top5", "tags_key", "top_tag_keys"], - ], - - - # Find any tags whose *value* is `bar` - "conditions": [ - ["tags_value", "=", "bar"], - ], - - -Note, when using this expression. the thing you are counting is tags, not events, so if you -have 10 events, each of which has 10 tags, then a `count()` of `tags_key` will return 100. diff --git a/snuba/snuba_migrations/README.md b/snuba/snuba_migrations/README.md index 67fe51cb86d..150243c5c87 100644 --- a/snuba/snuba_migrations/README.md +++ b/snuba/snuba_migrations/README.md @@ -1,9 +1,7 @@ -If you don't know anything about snuba migrations see `MIGRATIONS.md` in the root folder +# Migration groups -Each folder in here represents a migration group. see `snuba/migrations/group_loader.py` +Each directory here is a migration group. Loaders in `snuba/migrations/group_loader.py` map groups to directories. -Each migration (ex. `events/0001_events_initial.py`) needs to follow the naming scheme `xxxx_migration_name.py` -where `xxxx` is the 0 padded migration number. Migrations are applied in order of migration number. See `snuba/migrations/group_loader.py` for more info. +Name each migration `xxxx_migration_name.py`, where `xxxx` is the zero-padded migration number, for example `events/0001_events_initial.py`. Migrations in a group run in number order. -## Migration Auto-Generation -Who wants to write their own migrations by hand? Certainly not me! See `MIGRATIONS.md` to learn how you can have them generated for you. +To run, write, or generate migrations, see [the migrations guide](../../docs/source/migrations/overview.md).