Skip to content

feat(database)!: replace postgres with sqlite - #61

Merged
Vuks69 merged 1 commit into
Neon-Genesis-Linux:mainfrom
analeis:migrate-sqlite
Nov 10, 2026
Merged

Vuks69 merged 1 commit into
Neon-Genesis-Linux:mainfrom
analeis:migrate-sqlite

Conversation

@analeis

@analeis analeis commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Replaces the PostgreSQL backend with SQLite, measured against the database server it replaces.

Both sides were benchmarked with the internal/db package taken from the respective commit, driven by the same harness over the same public API, on the same 8-core host. PostgreSQL 18.3 ran locally over a unix socket, so it was not charged for network latency. Figures are the median of three runs against a freshly created database, after a warm-up run excluded the cold-start checkpoint storm.

Query latency

Query latency, p50 (lower is better)

  GetGuild, p50                PostgreSQL █████████████████        60 µs
                               SQLite     ████        15 µs

  GetUser, p50                 PostgreSQL ██████████████████████████        89 µs
                               SQLite     ██████        22 µs

  GetModerationLogs page, p50  PostgreSQL ████████████████████████████████       110 µs
                               SQLite     █████████████        46 µs

  UpsertGuild, p50             PostgreSQL ██████████████████████████████████       118 µs
                               SQLite     █████████        30 µs

  LogModeration insert, p50    PostgreSQL █████████████████████        72 µs
                               SQLite     ████████        29 µs

  Median of three runs, 8-core host, 5k users and 50k moderation logs.
Operation PostgreSQL SQLite
GetGuild, mean 75 µs 15 µs 5.00x
GetGuild, p50 60 µs 15 µs 4.00x
GetUser, mean 92 µs 23 µs 4.00x
GetUser, p50 89 µs 22 µs 4.05x
GetModerationLogs page, mean 112 µs 48 µs 2.33x
GetModerationLogs page, p50 110 µs 46 µs 2.39x
UpsertGuild, mean 114 µs 34 µs 3.35x
UpsertGuild, p50 118 µs 30 µs 3.93x
LogModeration insert, mean 79 µs 32 µs 2.47x
LogModeration insert, p50 72 µs 29 µs 2.48x

Startup and bulk load

Startup and bulk load (lower is better)

  Startup (open + migrate)  PostgreSQL █         8 ms
                            SQLite     █         1 ms

  Seed 5k users + 50k logs  PostgreSQL ██████████████████████████████████     4,609 ms
                            SQLite     ████████████████     2,109 ms

  Median of three runs against a freshly created database.
PostgreSQL SQLite
Startup, open and migrate 8 ms 1 ms 8.00x
Seed 5k users and 50k moderation logs 4,609 ms 2,109 ms 2.19x

Concurrent load

Concurrent mixed load (higher is better)

  Concurrent mixed (8 goroutines)  PostgreSQL ███████████████ 13,640 ops/s
                                   SQLite     ██████████████████████████████████ 31,309 ops/s

  One third reads, two thirds writes. No errors in either engine.
PostgreSQL SQLite
Throughput 13,640 ops/s 31,309 ops/s 2.30x
Errors 0 0

SQLite serialises writes, so this is one writer against seven readers. The write-ahead log and a busy timeout keep the readers off the writer's path, and the pool is small on purpose.

Footprint

Data on disk (lower is better)

  Data on disk  PostgreSQL ██████████████████████████████████     91.1 MiB
                SQLite     ████      9.5 MiB

  After the same 5k users and 50k moderation logs were written.
PostgreSQL SQLite
Data on disk 91.1 MiB 9.5 MiB 9.59x
Server process ~31 MiB resident, separate container none
Containers per instance 2 1
Images to pull app + postgres:16-alpine (297 MiB) app only (3 MiB)
Credentials DB_USER, DB_PASSWORD, POSTGRES_PASSWORD none
Healthcheck required none
Bot binary 11.0 MiB 14.1 MiB +3.1 MiB

The binary grows because the SQLite engine is a pure Go port rather than a C library, which is what keeps the build at CGO_ENABLED=0 and the runtime image unchanged.

AI Usage disclosure

OpenCode with MiniMax M3.1 was used to migrate the backend, write the tests and the documentation, and produce the benchmarks above. The driver choice was settled by measuring the three candidates against the placeholder binding this codebase relies on: mattn/go-sqlite3 and ncruces/go-sqlite3 both bind $1-style placeholders by order of appearance rather than by index, which would have put the wrong values in the wrong place in the queries that build their argument list.

The implementation was reviewed or redirected atleast three times after the initial prompt for a thorough evaluation and feasibility of the migration.

sqlc was also evaluated and rejected due to being otherwise feasible but still in beta for SQLite, especially $ values.

Tested with go test ./..., custom-gcl including NilAway, and the Sphinx -W build, plus the container image built and run against a named volume.

The bot stores its data for a single guild, and each instance was given
its own postgres database to hold it. That cost a database server per
instance: a second container to pull and run, credentials to configure,
a healthcheck to wait on, and around 45 MiB of resident memory that the
bot did not need. The data itself is a handful of rows per guild, so a
file is enough.

The database is now SQLite, through modernc.org/sqlite. The driver is
pure Go, which keeps the build at CGO_ENABLED=0 and the runtime image
unchanged. It binds $1-style placeholders by index the way postgres
does, so the query text carried over without being rewritten, and it
binds them correctly: mattn/go-sqlite3 and ncruces/go-sqlite3 both
bind by order of appearance, which would have put the wrong values in
the wrong place in the queries that build their argument list.

Timestamps are stored as Unix milliseconds in INTEGER columns. SQLite
has no timestamp type, and the obvious alternative is silently wrong:
the driver writes a time.Time as text while a column default writes an
integer, so the column holds two kinds of value and SQLite orders it by
kind rather than by instant. Storing integers and letting the driver
convert keeps the column and the comparisons exact. The subtleties of
that representation are written down in AGENTS.md, since the two ways of
getting it wrong are a query that returns rows out of order and one that
fails to scan.

The schema uses INTEGER PRIMARY KEY AUTOINCREMENT. The AUTOINCREMENT
matters: without it SQLite reuses an id after a delete, and the
pagination cursor compares (created_at, id), so a reused id would make
a page skip rows. The moderation log index ends with id for the same
reason, and without it SQLite sorts every page again with a temporary
B-tree over the table. A test asserts the query plan, so the sort
cannot come back unnoticed.

The migration runner hands a file to the driver as one statement
instead of splitting it on semicolons. The driver knows the quoting
rules, so a semicolon inside a string literal no longer cuts a
statement in half. Two indexes that the unique constraints and the
composite index already covered are gone.

DB_PATH names the file, and DB_BOT_INSTANCE_ID names it when DB_PATH
is unset, so instances sharing a directory still get one file each.
The instance id is filtered down to characters that are safe in a file
name. The image creates the data directory owned by the user it runs
as, so a named volume mounted over it is writable without anyone
having to fix its permissions by hand.

Breaking: DATABASE_URL, DB_HOST, DB_PORT, DB_USER, DB_PASSWORD,
DB_NAME and POSTGRES_PASSWORD are gone, replaced by DB_PATH and
DB_BOT_INSTANCE_ID. compose.yaml no longer starts a database service.

There is no migration path for an existing postgres database. Nothing
was deployed against it, so there is nothing to carry over.

docs/OPERATIONS/DATABASE.md covers where the file lives, how to back
it up, and how to read it. The backup uses VACUUM INTO, because the
write-ahead log beside the file holds writes that a plain copy would
miss.

@Vuks69 Vuks69 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no concerns from me

@Vuks69
Vuks69 merged commit 95c383c into Neon-Genesis-Linux:main Nov 10, 2026
2 checks passed
@analeis
analeis deleted the migrate-sqlite branch November 10, 2026 22:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants