Pooled sqlalchemy connections are implicitly returned to the pool when their object is garbage collected by python. From Constructing a Pool:
The proxy also returns its contained DBAPI connection to the pool when it is garbage collected, though it’s not deterministic in Python that this occurs immediately (though it is typical with cPython). This usage is not recommended however and in particular is not supported with asyncio DBAPI drivers.
But we shouldn't rely on that (indeed the docs right above say not to) and under certain circumstances we will hold DB connections open longer than they are needed and exhaust the pool. We should explicitly .close() the connections after we are done with them and return them to the pool.
In particular, GutenbergDatabaseDublinCore.load_from_database() will always hold two database connections because at the end it calls load_files_from_database() which opens its own database connection without the first one falling out of scope and thus garbage collected.
This will be more important when we have multiple processes each with their own pool and each needing to make better use of them.
Pooled sqlalchemy connections are implicitly returned to the pool when their object is garbage collected by python. From Constructing a Pool:
But we shouldn't rely on that (indeed the docs right above say not to) and under certain circumstances we will hold DB connections open longer than they are needed and exhaust the pool. We should explicitly
.close()the connections after we are done with them and return them to the pool.In particular,
GutenbergDatabaseDublinCore.load_from_database()will always hold two database connections because at the end it callsload_files_from_database()which opens its own database connection without the first one falling out of scope and thus garbage collected.This will be more important when we have multiple processes each with their own pool and each needing to make better use of them.