Share PostgreSQL with Substrate

[iplay88keys]

Important

This PR depends on agent-substrate/substrate#1752 and kagent-dev/substrate#32. After both merge and release, this branch still needs its Substrate image/chart dependency bumped and locked.

Summary

• Default embedded Substrate to Kagent's selected PostgreSQL database while keeping Substrate objects in a dedicated substrate schema.
• Disable Substrate's bundled PostgreSQL so the default installation deploys only Kagent's bundled database.
• Create a release-scoped Substrate connection Secret from Kagent's inline external URL or bundled PostgreSQL URL.
• Replace database.postgres.urlFile with database.postgres.secretRef.{name,key} for Kagent 1.x.
• Mount Kagent's selected Secret as an updatable volume and pass POSTGRES_DATABASE_URL=@file:... to the controller.
• Reread file-backed credentials and TLS material whenever pgx opens a new physical connection.
• Add database.postgres.role / POSTGRES_DATABASE_ROLE for a stable Kagent PostgreSQL role.
• Run Kagent pools and migrations as the stable role so grants and ownership survive generated-login rotation.
• Add --db-role and POSTGRES_DATABASE_ROLE support to kagent db migrate.
• Expose substrate.postgres.runtimeRole and substrate.postgres.ddlRole for embedded Substrate.
• Wire the existing pool settings into the controller, including database.postgres.pool.maxConnLifetime.
• Support bundled sharing, an existing shared Secret, and separate Kagent/Substrate runtime/DDL credentials.

Database layouts

Bundled database shared by default

substrate:
  enabled: true

Kagent and Substrate use Kagent's bundled PostgreSQL instance. Kagent uses its normal schema and Substrate uses the dedicated substrate schema.

The bundled database retains its existing static username behavior. Stable role settings are primarily for external systems that generate a new login during rotation.

Existing external Secret shared by Kagent and Substrate

Helm cannot dynamically copy one subchart value into another. Set the same Secret reference on both sides:

database:
  postgres:
    bundled:
      enabled: false
    secretRef:
      name: shared-postgres
      key: connectionString
    role: kagent_app
    pool:
      maxConnLifetime: 10m

substrate:
  enabled: true
  postgres:
    enabled: false
    schema: substrate
    connectionStringSecretRef:
      enabled: true
      name: shared-postgres
      key: connectionString
    runtimeRole: substrate_runtime
    ddlRole: substrate_ddl
    pool:
      maxConnLifetime: 10m

Because the same generated login is used by all three pools in this layout, it must be a member of kagent_app, substrate_runtime, and substrate_ddl.

Separate Kagent, Substrate runtime, and Substrate DDL credentials

database:
  postgres:
    bundled:
      enabled: false
    secretRef:
      name: kagent-postgres
      key: connectionString
    role: kagent_app
    pool:
      maxConnLifetime: 10m

substrate:
  enabled: true
  postgres:
    enabled: false
    schema: substrate
    connectionStringSecretRef:
      enabled: true
      name: substrate-postgres
      key: runtimeConnectionString
    ddlConnectionStringSecretRef:
      enabled: true
      name: substrate-postgres
      key: ddlConnectionString
    runtimeRole: substrate_runtime
    ddlRole: substrate_ddl
    pool:
      maxConnLifetime: 10m

All connections may point to the same PostgreSQL server and database. The dedicated substrate schema prevents Substrate's runtime grants from covering Kagent tables.

The stable DDL role owns and migrates the Substrate schema and performs outbox partition maintenance. The stable runtime role receives only the table and sequence permissions ateapi needs.

Omitting the Substrate DDL connection preserves single-connection operation. When both the DDL connection and ddlRole are omitted, Substrate uses the runtime source and role for both.

ddlConnectionStringSecretRef requires an explicit name. Kagent generates only the runtime connection Secret, so an unnamed DDL reference would resolve to a Secret without the DDL key.

Credential rotation

Kubernetes updates mounted Secret volumes in place. Kagent and Substrate reread their selected files before opening each new physical pgx connection; an application reload or pod restart is not required.

Existing connections retain their current credentials until retired. Set both database.postgres.pool.maxConnLifetime and substrate.postgres.pool.maxConnLifetime to bound the rotation window, and keep outgoing credentials valid during that overlap.

Password and TLS rotation work without stable roles if the username remains unchanged.

Username-changing rotation requires stable roles:

• Kagent generated logins must be members of database.postgres.role.
• Substrate runtime logins must be members of substrate.postgres.runtimeRole.
• Substrate DDL logins must be members of substrate.postgres.ddlRole.
• The credential provisioner must grant membership before publishing the updated Secret.

Each new connection authenticates as the generated login and then assumes its stable role. PostgreSQL grants and object ownership therefore remain attached to persistent roles rather than disposable users.

Without a configured stable role, Kagent or Substrate rejects a username change and requires a restart. If the incoming login lacks membership, SET ROLE fails and the connection does not enter the pool.

Host, port, fallback hosts, and database remain part of the pool identity and require a restart when changed. Inline URLs remain startup-static.

This supports credentials rotated into Kubernetes Secrets. It does not mint RDS IAM tokens in-process.

Migration and DDL credentials

database.postgres.role is used by both Kagent's runtime pool and in-process startup migrations. When startup migrations are enabled, that stable role must have the required DDL privileges.

Deployments that require separate Kagent runtime and migration privileges can continue to migrate out of band:

kagent db migrate up \
  --db-url @file:/path/to/migrator-connection-string \
  --db-role kagent_ddl

Then configure the controller with its restricted stable runtime role and set:

database:
  postgres:
    role: kagent_runtime
    skipMigrations: true

kagent db migrate also reads POSTGRES_DATABASE_ROLE when --db-role is omitted.

Substrate's independent DDL/runtime split is provided by agent-substrate/substrate#1752. Both credentials remain in ateapi because partition maintenance is still performed in process.

Database prerequisites

Neither chart creates external PostgreSQL group roles or grants rotating login memberships. Provision the stable roles before installation and make the credential rotator grant every incoming login the corresponding membership.

For example:

CREATE ROLE kagent_app NOLOGIN;
CREATE ROLE substrate_runtime NOLOGIN;
CREATE ROLE substrate_ddl NOLOGIN;

GRANT CONNECT ON DATABASE kagent
  TO kagent_app, substrate_runtime, substrate_ddl;
GRANT CREATE ON DATABASE kagent TO substrate_ddl;

GRANT kagent_app TO kagent_login_20260922;
GRANT substrate_runtime TO substrate_runtime_login_20260922;
GRANT substrate_ddl TO substrate_ddl_login_20260922;

The exact login names are generated by the external credential system. On the next rotation, it creates new login users and grants the same stable memberships before updating the Secrets.

ateapi creates the configured Substrate schema through the DDL role. It grants the stable runtime role access to migrated tables and sequences after each migration.

Breaking change

database.postgres.urlFile and POSTGRES_DATABASE_URL_FILE are removed for Kagent 1.x. Use database.postgres.secretRef in Kubernetes, or pass an @file:/absolute/path value through POSTGRES_DATABASE_URL when running the binary directly.

The chart fails with migration guidance if the removed Helm value is supplied.

Testing

• Kagent database and migration integration tests pass against PostgreSQL.
• Rotation coverage verifies that a new login is adopted while current_user remains the stable role.
• Username changes without stable roles fail closed.
• Focused application and migration CLI tests pass.
• The controller Helm suite passes all 68 tests.
• The standalone Substrate chart passes all 20 tests.

The full Kagent Helm suite currently has one unrelated existing failure caused by the packaged Substrate dependency not containing the branch's template-resync argument. The focused controller suite and new role test pass.

[smeegoan]

Three gaps after this PR.

--postgres-schema defaults to public, and atepg.go assigns RuntimeParams["search_path"] unconditionally after parsing the DSN. In the shared-Secret setup this PR documents, a connection string carrying search_path=agents is therefore overridden to public on the ate-api-server side, silently, unless the flag is set to match.

Migrations cannot use a separate identity. RunUp runs in-process on the same connection string before the pool opens, so the role serving traffic must also hold DDL rights on its own schema, and every migrated object ends up owned by it. A schema owner distinct from its runtime consumer is not expressible.

Rotation has no input left. POSTGRES_DATABASE_URL is an env var, frozen for the life of the process. urlFile was the only input a rotation system could write to, and passfile is no substitute because ParseConfig resolves that once too. Is rotation meant to work another way?

[iplay88keys]

@smeegoan, thanks for taking a look! Just updating this PR and reviewing your comment. Just a heads up that Codex helped put together this response, though I reviewed it.

--postgres-schema defaults to public, and atepg.go assigns RuntimeParams["search_path"] unconditionally after parsing the DSN. In the shared-Secret setup this PR documents, a connection string carrying search_path=agents is therefore overridden to public on the ate-api-server side, silently, unless the flag is set to match.

This PR defaults substrate.postgres.schema to substrate, which becomes the explicit --postgres-schema=substrate argument. We intentionally do not rely on the DSN’s search_path; users can override the Substrate schema value.

Migrations cannot use a separate identity. RunUp runs in-process on the same connection string before the pool opens, so the role serving traffic must also hold DDL rights on its own schema, and every migrated object ends up owned by it. A schema owner distinct from its runtime consumer is not expressible.

Kagent already supports this operationally. An operator can create the schema/grants and run kagent db migrate up --db-url <migrator-url> with privileged credentials, then configure the controller with restricted runtime credentials and database.postgres.skipMigrations=true. The chart does not provision those roles or grants. RunUp is skipped in that mode.

On the Substrate side, even if we added a way to pre-apply and skip startup migrations, that would not provide complete DDL/DML separation. ateapi also performs runtime DDL to create and remove outbox partitions, so its runtime role must retain those privileges.

Operators can configure Substrate with a separate connection Secret and role, even when it points to the same PostgreSQL server and database. This keeps Substrate’s DDL privileges off Kagent’s runtime role, but Substrate currently has no supported mechanism for separating its own migration/DDL and runtime/DML identities. Additional Substrate work would be required for that stricter separation.

Rotation has no input left. POSTGRES_DATABASE_URL is an env var, frozen for the life of the process. urlFile was the only input a rotation system could write to, and passfile is no substitute because ParseConfig resolves that once too. Is rotation meant to work another way?

urlFile was read only once during startup, so changing it did not rotate an active pool. A Secret-backed environment variable has the same restart requirement. Live rotation would require explicit reload and pool reconnection support.

So the PR does not remove an existing Kagent credential-separation or live-rotation capability. The real follow-up is adding equivalent migration-role separation to Substrate if that is required for shared production databases.

[iplay88keys]

@smeegoan, I see now how urlFile can be used to rotate credentials. In order to maintain that functionality, while still getting rid of the urlFile convention, I have updated this PR (as well as the required upstream Substrate PR) to now mount the secretRef as a volume and use the db url env to instead pass a path to that file.

As such, pgx's beforeConnect will re-read file-backed credentials and TLS material before each new physical connection. I also added the maxConnLifetime to substrate so the same mechanism for rotation can exist there.

So the updated design preserves a live Kubernetes Secret input and supports credential rotation without a pod restart, while replacing the arbitrary file-path urlFile value with explicit Secret references.

[smeegoan]

Thank you @iplay88keys, the @file: plus BeforeConnect design closes the rotation gap.

One thing stops us adopting it as is. sameConnectionIdentity treats user as part of connection identity, so a rotation that changes the username fails every new acquire with database connection identity changed; restart required, while existing connections keep serving until maxConnLifetime expires. Our rotation changes the username by design: each cycle creates a new user with the new password and leaves the previous user able to log in until the following cycle, so that two credentials authenticate at once and no consumer fails while the secret store promotes the new version. That is a property of the database's rotation, not something a single consumer opts into. New dials should authenticate as the incoming user while older connections finish on the outgoing one, which is what the pool already does.

Proposal: move User out of the identity comparison and into the set the refresh adopts, next to Password. Host, port, database and fallback host and port stay fenced, since those are the pool's endpoint and changing them mid-life does need a restart. That removes the user comparison rather than adding anything.

[iplay88keys]

@smeegoan, thanks for taking another look. I'm glad this approach overall will work for you. I have updated both sides of this (Substrate/Kagent) to allow for rotating the user to allow for the flow you laid out.