Skip to content
glamourdbPublic

About

GlamDB is an open-source columnar key-value storage engine in early development, built in C++23 on Apache Arrow, Parquet, and Iceberg, with a SQLite catalog. It focuses on durable writes and consistent snapshot reads.

Resources

Contributing

Security policy

Stars

7 stars

Watchers

0 watching

Forks

Latest commit

 

History

124 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

GlamDB

GlamDB

GlamDB is an open-source columnar database written in C++23. It stores typed records in Parquet files, manages snapshots with Apache Iceberg, and uses SQLite for its catalog and write outcomes. Use it as an embedded C++ library or through the HTTP server, browser workbench, command-line client, and Python SDK.

GlamDB is a software project developed for a hackathon at ITMO University and is not intended for production use. Presentation about the project

GlamDB is in early development. The engine supports append and explicit key-value collections on local disk, with one owning process, one serialized writer per store, and concurrent snapshot readers. Power-loss durability qualification is still pending.

Features

  • Typed collections. Fixed schemas with strings, booleans, int64, float64, UTC microsecond timestamps, exact decimal128 amounts and calendar dates, and nullable fields. See scalar types and compatibility. Appends preserve every row, including duplicate keys. Explicit KV collections provide full-row upserts, safe keyed deletes, fixed-snapshot batched lookups, bounded streaming bulk upsert and locality-preserving compaction.
  • Snapshot reads. Query the latest committed view or a retained historical snapshot. Select columns, limit results, and combine comparisons, membership, and NULL predicates with AND, OR, and NOT; scans prune candidate files and Parquet row groups conservatively and read only needed column ranges.
  • Explicit write outcomes. Persistent operation IDs support exact retries and outcome lookup after restart. Writes distinguish committed, definitely not committed, and unknown outcomes.
  • Open columnar formats. Arrow batches in memory, immutable Parquet data files, and Iceberg v2 tables with retained snapshot history. Collection and per-column Parquet compression, levels, encodings and dictionary policies are configurable through the SDK and workbench.
  • Data inspection. Browse schemas, snapshot history, statistics, files, Parquet row groups, scan plans, and execution profiles.
  • Native compute contract. Negotiate versions 1.0–1.5, operation support by append/KV mode and effective limits through the installed C++ API. See the storage contract. Contract 1.1 adds bounded native read views for repeated scans across a coherent per-store snapshot vector. Contract 1.2 adds bounded, repeatable parallel scan splits. Contract 1.3 adds scan pushdown negotiation with explicit exact/necessary-condition/residual filters and safe global limits. Contract 1.4 adds durable write sessions for one-pass append/KV output through a separate bounded staging journal and the existing Engine publisher. Contract 1.5 adds the snapshot archive (hot window, archived reads, maintenance), verified-checkpoint open and explicit full verification. Remote sessions and read views, stable C ABI and Flight remain unsupported.
  • Connected tools. Manage multiple databases through one authenticated server. Read and write through the UI, CLI, or Python SDK; retrieve scan results as JSON or Arrow IPC with fixed-snapshot pagination.

Queries use a typed predicate API; there is no SQL query interface. Collections are unpartitioned with immutable append or KV mode. Schema evolution, general garbage collection, object storage and replication remain outside the supported profile. The embedded C++ API supports explicit snapshot retention maintenance for append collections; history is retained forever unless a caller opts in. KV collections preserve all history and refuse snapshot expiration.

Open the public demo — anyone can try it. The organization workbench uses Keycloak sign-in, OAuth2 Proxy sessions and Cerbos policies for isolated database access. See the deployment and operating guide for HTTPS hosting, native account management and database ownership workflows. The combined-login guide also documents the explicit temporary hello/world setup for an existing demonstration account. The static Pages demo contains disposable sample data and remains a separate publication; it does not connect to protected database servers.

Get started

Clone the repository and start a local server:

git clone --recurse-submodules https://github.com/glamourdb/glamdb.git
cd glamdb
export GLAMDB_TOKEN="$(openssl rand -hex 32)"
./scripts/glamdb-local.sh up
./scripts/glamdb-local.sh status

The helper builds the server and its pinned dependencies, then starts it in the background. On Linux, it uses the native toolchain described under Build from source; on macOS, it uses a running Docker daemon. Allow 30–60 minutes for the first dependency build.

Open http://127.0.0.1:8080 and enter the value of GLAMDB_TOKEN. The server starts with no databases. Initialized managed databases can be explicitly removed through the SDK or glamdb-cli drop-database using their recorded store UUID; glamdb-cli bench tpch clean --state RUN_STATE supports dedicated benchmark cleanup. See the deletion and retry contract. Create a database, create a collection with its schema, then append and query rows in the workbench. The same UI provides snapshot selection, storage inspection, and scan explain views.

Data survives server restarts. The default location is ~/.local/share/glamdb/data on Linux, or the glamdb-data Docker volume. Set GLAMDB_DATA to choose another directory or volume and GLAMDB_PORT to change the port. Writes require free space of at least the larger of 1 GiB or 5% of the data filesystem, plus a 512 MiB reserve.

./scripts/glamdb-local.sh logs -f  # Follow server logs
./scripts/glamdb-local.sh down     # Stop the server and keep its data

Keep the token for subsequent starts, or omit GLAMDB_TOKEN and let the helper generate and reuse a token file; status shows its location. The service token grants access to all databases. Remote connections use HTTPS through a TLS proxy; see the server guide for deployment and API details.

Use GlamDB

The examples below use the local server and token from the quick start. Run commands from the repository root.

Command line

glamdb-cli is a standalone HTTP/HTTPS client. The local helper runs it with the server's address and token. Create a database, define a collection, append two rows, and read them back:

./scripts/glamdb-local.sh cli request --path /databases --body - <<'JSON'
{"name":"cli-example","database_id":"11111111-1111-4111-8111-111111111111"}
JSON

COLLECTION=/databases/11111111-1111-4111-8111-111111111111/collections
./scripts/glamdb-local.sh cli request --path "$COLLECTION" --body - <<'JSON'
{
  "collection_id": "22222222-2222-4222-8222-222222222222",
  "name": "events",
  "schema": {"fields": [
    {"name": "key", "type": "string", "nullable": false},
    {"name": "value", "type": "int64", "nullable": true}
  ]}
}
JSON

COLLECTION=$COLLECTION/22222222-2222-4222-8222-222222222222
./scripts/glamdb-local.sh cli request --path "$COLLECTION/append" --body - <<'JSON'
{
  "operation_id": "33333333-3333-4333-8333-333333333333",
  "rows": [{"key": "a", "value": "1"}, {"key": "b", "value": "2"}]
}
JSON

printf '%s' '{}' | ./scripts/glamdb-local.sh cli request \
  --path "$COLLECTION/scan" --body -

Int64 values use decimal strings in JSON to preserve precision. The request command exposes every HTTP route; --body implies POST. Scan responses include next_cursor when another page is available. For scripts that consume complete results, the Python SDK handles pagination automatically.

Named commands also cover create-collection, collections, resolve-collection, collection, schema, append, scan, operation, snapshots, files, and file. Use --url and --database-id with glamdb-cli. The optional glamdb-admin offers the same commands offline with --root and --store-id, or remotely with --url. Named scans complete HTTP pages within a 10,000-row, 16 MiB result bound. See the CLI guide for commands, input formats and the distinct failure/unresolved exit codes.

Databases default to eight collections and accept a resource-backed opt-in limit up to 64. Use create-store --max-collections 64 or the managed SDK/HTTP limits object; managed limits survive restarts. Collection inventories support fixed-view pages. See collection profiles for configuration and inventory semantics.

Python SDK

Install the SDK from the checkout with Python 3.10 or later:

python3 -m venv .venv
. .venv/bin/activate
python -m pip install ./sdk/python
import os
from uuid import UUID

from glamdb_sdk import Client, Field, Schema, col

url = "http://127.0.0.1:8080"
token = os.environ["GLAMDB_TOKEN"]
schema = Schema([Field("key", "string", False), Field("value", "int64", True)])
collection_id = UUID("44444444-4444-4444-8444-444444444444")
operation_id = UUID("55555555-5555-4555-8555-555555555555")

with Client(url, token=token) as admin:
    database = admin.create_database("python-example")

with Client(url, token=token, database_id=database["database_id"]) as client:
    created = client.create_collection("events", schema, collection_id=collection_id)
    if created.outcome != "committed":
        raise RuntimeError(created.raw)

    request = client.prepare_append(
        collection_id,
        [{"key": "a", "value": 1}, {"key": "b", "value": 2}],
        operation_id=operation_id,
        schema=schema,
    )
    outcome = client.append(request)
    if outcome.outcome != "committed":
        raise RuntimeError(outcome.raw)

    result = client.scan_json(collection_id, predicate=col("value", "int64").ge(2))
    print(result.rows)  # [{"key": "b", "value": 2}]

    with client.scan_batches(collection_id) as batches:
        for batch in batches:
            print(batch.to_pylist())

These fixed UUIDs make the example repeatable. In an application, generate and save a collection ID and an operation ID before submitting each new create or append. Preserve the original IDs and payload if a response is lost; look up the outcome before deciding to retry. The SDK never retries writes implicitly. See the SDK guide for historical reads, metadata, streaming, and recovery handling.

Bulk ingestion and benchmarks

Streaming bulk append accepts replayable batches as one operation and snapshot through C++, HTTP, Python and the CLI. Ordinary append keeps its separate 10,000-row/16 MiB limits. Bulk upsert and append-to-KV migration are embedded C++ operations.

The optional workload package provides complete TPC-H and ClickBench append/KV lifecycles, exact verification, saved-request recovery and benchmark cleanup. Scans use the typed API; reductions run in the client, including bounded DuckDB staging for ClickBench. These are not native SQL or official benchmark scores.

Embedded C++

The installed glamdb::core CMake target provides CreateStore, OpenExisting, Engine::CreateCollection, Engine::AppendDurable, and Engine::Scan through <storage/engine.h>. Input and output use Arrow batches. The embedded library and HTTP server share the same engine; an embedded owner must have exclusive ownership of its store directory.

Build from source

Native builds require Linux, CMake 3.28+, Ninja, Python 3.10+, OpenSSL and zlib development headers, and a C++23 compiler with std::expected support: GCC 14+ or Clang 18+ with a matching standard library. Build dependencies and GlamDB with the same compiler and C++ ABI.

git submodule update --init --recursive
cmake --preset deps-local
cmake --build --preset deps-local --parallel 2
cmake --preset local
cmake --build --preset local
ctest --preset local

To run the resulting server directly, with GLAMDB_TOKEN set:

./build/local/apps/server/glamdb --data-dir "$PWD/build/data" --ui-root "$PWD/ui"

For a smaller shared memory profile, --http-readers 1 --http-readers-per-database 1 queues heavy metadata reads without restricting control or publishing calls. Defaults remain four readers globally and two per database; see the HTTP service profile for the finite memory settings and admission limits.

The build also produces build/local/apps/cli/glamdb-cli. Use the release or asan-ubsan preset instead of local for release or sanitizer builds. cmake --install build/local installs binaries, UI assets, public headers, and the CMake package into build/install/local.

For the full native test suite in Docker, including on macOS:

./scripts/build-deps-in-docker.sh
./scripts/test-storage-in-docker.sh asan-ubsan

See the build guide for toolchain setup, standalone client builds, and embedding the installed library. For four-state rules and native Telegram setup, see the alerting guide.

For metrics, Grafana dashboards and deployment steps, see the monitoring guide.

For a single Linux server, the complete observability quickstart provides one command for Prometheus, Grafana, Loki, Alloy, Alertmanager and host/container collectors. Real application endpoints and a dedicated Loki filesystem are inputs; repeated deployment preserves owned data and credentials.

New databases default to byte-bound retained-history budgets (the former 4,096-operation/snapshot horizon is gone; persisted databases keep their limits). The linked SF=10 recipe uses ordinary appends and preserves permanent outcomes, duplicate rows, historical views and exact retries.

Client and UI tests

The workbench development environment requires Python 3.13, Node.js 22+, and OpenSSL. It includes the SDK, Python tests, Node tests, and Playwright browser tests:

bash scripts/bootstrap-workbench.sh
npm --prefix ui exec -- playwright install chromium
bash scripts/check-workbench.sh

On Linux, add --with-deps to the Playwright install command to install browser system dependencies. Tests use localhost sockets and disposable stores.

See the documentation index for API references and operator guides.

Contributing

Bug reports, documentation improvements, tests, and code contributions are welcome. Use issues to report a reproducible problem or discuss a proposed feature.

  1. Fork glamourdb/glamdb and create a branch in your fork.

  2. Make a focused change, add tests for behavior changes, and update affected docs.

  3. Run the relevant tests. For C++ changes, also run the quality checks with LLVM 18 clang-format and clang-tidy:

    cmake --build --preset local --target format-check tidy-check quality-negative-check
  4. Open a pull request against glamourdb/glamdb:main, describing the change and how you tested it. Storage changes should address recovery and compatibility.

Pull requests require passing checks and approval from another maintainer. See CONTRIBUTING.md for review and merge requirements. Report security vulnerabilities privately through the security policy.

License

GlamDB is licensed under the Apache License 2.0.

About

GlamDB is an open-source columnar key-value storage engine in early development, built in C++23 on Apache Arrow, Parquet, and Iceberg, with a SQLite catalog. It focuses on durable writes and consistent snapshot reads.

Resources

Contributing

Security policy

Stars

7 stars

Watchers

0 watching

Forks

Used by

Contributors

Languages