Skip to content

Storage.md

Latest commit

 

History

History
36 lines (27 loc) · 5.47 KB

Storage.md

File metadata and controls

36 lines (27 loc) · 5.47 KB
Raw

Storage

Each S3-Object is split into chunks which are then split into shards, the shards are then encoded with reed-solomon
encoding and stored across multiple backends. The Objects are split in a way that guarantees a variable amount of backend failures before data is lost.

Chunks

A chunk is a contiguous segment of an S3-Object. Chunks are the units of data that are split into shards and distributed across backends.

Shards

A shard contains a part of a chunk and is stored on a backend. The shards are encoded with reed-solomon encoding to allow for backend failures.
Shards can be combined into a blob for a single backend to reduce the number of api calls to each backend. The maximum size of a blob is determined by the maximum file size of the backend.

Backends

Backend instances are declared in backends.yaml (path configurable via SHARDS3_BACKENDS_CONFIG_PATH):
each entry has an id, a driver kind (e.g. telegram, discord, file), whether it's enabled, and kind-specific non-secret settings. Secrets (bot tokens, API keys, etc.) are never stored in YAML - they are read from environment variables named SHARDS3_BACKEND_<ID>_<KEY> (id and key upper-cased), so a backend's non-secret configuration and its secrets are managed separately. Disabled backends are skipped entirely at startup, including their secret lookups.

New backend kinds register themselves with the storage layer via an init() call (interfaces.RegisterKind); adding one does not require touching existing backend code.

Metadata

Metadata is stored in a simple SQLite Database. The metadata contains information about each S3-Object as well as information about each chunk and shard.

Chunking and Sharding

Example

The following diagram shows how an S3-Objects is split and redundantly stored.
In this example there are three backends:

  • Backend 1. Allows file size of up to 40MiB
  • Backend 2. Allows file size of up to 25MiB
  • Backend 3. Allows file size of up to 200Mib

Since the smallest allowed size if 25MiB, each encoded shard must be 25MiB. The number of allowed backends to fail is one, which is why there are three parity shards. If any of the backends fail, the data is still available. If more than one backend fails, the data is not recoverable.

The sharding is calculated automatically by an algorithm in the storage module. While this looks pretty good, it can be optimized further by merging the shards into larger blobs up the the max file size of each backend. This reduces the number of api calls for each backend.
Final Result: