Repository navigation
models: enable pool_pre_ping by default - #158
Merged
Merged
Conversation
Every morning ~06:00-06:45 UTC, unattended-upgrades on 1.shaman restarts
postgresql (via needrestart) when a dependent package is upgraded. The
SQLAlchemy pools keep the now-dead connections, so the first /api/search
requests after the restart fail with
psycopg2.OperationalError: SSL connection has been closed unexpectedly
which surfaces as HTTP 500 to clients. Teuthology jobs querying package
versions during that window die with "Failed to fetch package version"
(https://tracker.ceph.com/issues/59193).
pool_pre_ping tests a pooled connection on checkout and transparently
replaces it if the database went away, at the cost of one trivial
round-trip per checkout. Use setdefault so a config file can still
opt out with pool_pre_ping: False.
Assisted-by: Claude Fable 5
Signed-off-by: David Galloway <[email protected]>
dmick
reviewed
Sep 10, 2026
| def _engine_from_config(configuration): | ||
| configuration = dict(configuration) | ||
| url = configuration.pop('url') | ||
| # Test pooled connections on checkout and transparently replace dead |
Member
There was a problem hiding this comment.
What does "on checkout" mean here?
Contributor
Author
There was a problem hiding this comment.
SQLAlchemy keeps a pool of open DB connections. When a request needs one it checks it out (borrows it from the pool), uses it, then returns it. pool_pre_ping=True means: at that borrow moment, before handing the connection to your code, SQLAlchemy fires a trivial SELECT 1 on it. If that fails, the connection is discarded and a fresh one is opened, transparently.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every morning ~06:00–06:45 UTC, unattended-upgrades on 1.shaman restarts postgresql (via needrestart) when a dependent package is upgraded. The SQLAlchemy pools keep the now-dead connections, so the first
/api/searchrequests after the restart fail with:which surfaces as HTTP 500 to clients. Teuthology jobs querying package versions during that window die with "Failed to fetch package version" — this is the genuine-infra slice of https://tracker.ceph.com/issues/59193 (confirmed against 1.shaman's journal: every recorded 500 lines up with a
postgresql@16-mainrestart fromapt-daily-upgrade.service, most recently 2026-09-02 06:41 UTC).pool_pre_pingtests a pooled connection on checkout and transparently replaces it if the database went away, at the cost of one trivial round-trip per checkout.setdefaultkeeps it overridable from a config file (pool_pre_ping: False).