Allow nvcf-api / nvct-api to trust more than one JWT issuer

[sparve-nv]

What we need

We run a self-managed deployment and we want external callers — operators, automation and
partner clients — to authenticate to nvcf-api and nvct-api with JWTs issued by our own
identity provider, so every request carries a real per-principal identity instead of a
shared admin token. The IdP is whatever the deployment already uses for SSO; the point is
that it is external to the stack.

We cannot do this today without breaking internal service-to-service auth.

Current state

The ncp profile points each API's resource server at OpenBao, with a single issuer:

# cloud-functions/nvcf-service/src/main/resources/application-ncp.yaml:26-27
issuer-uri:  http://api.nvcf.svc.cluster.local
jwk-set-uri: http://openbao-server.vault-system.svc.cluster.local:8200/v1/services/nvcf-api/jwt/jwks

# cloud-tasks/nvct-service/src/main/resources/application-ncp.yaml:26-27
issuer-uri:  http://nvct-api.nvcf.svc.cluster.local
jwk-set-uri: http://openbao-server.vault-system.svc.cluster.local:8200/v1/services/nvct-api/jwt/jwks

Every internal caller obtains an OpenBao-signed token from services/<audience>/jwt/sign/<role>
via vault-agent, and the audience validates it against OpenBao's JWKS. Self-contained, and it
works well — we are not asking to change it.

The self-managed stack already exposes overrides for these two values
(deploy/stacks/self-managed/global.yaml.gotmpl: api.jwt.issuerUri / api.jwt.jwkSetUri,
and nvctApi.jwt.*). The problem is what those overrides do.

What happens if we repoint the issuer

Setting issuer-uri / jwk-set-uri at an external IdP does not add an issuer, it
replaces the only one. Every OpenBao-signed internal token is then rejected, because no
authentication manager is registered for iss: http://api.nvcf.svc.cluster.local.

Calls into nvcf-api that stop authenticating

Eight services hold a signer role on the nvcf-api JWT mount
(services/nvcf-api/jwt/sign/<role>). Every one of their authenticated calls into nvcf-api
is rejected:

Caller service	Signer role	Call that fails
invocation-service	invocation-api	HTTP invocation path → nvcf-api
grpc-proxy	grpc-proxy-proxy	gRPC invocation path → nvcf-api
llm-api-gateway	llm-api-gateway	LLM invocation front door → nvcf-api
llm-request-router	llm-request-router	LLM routing → nvcf-api
ratelimiter	ratelimiter-api	rate-limit enforcement → nvcf-api
function autoscaler	nvcf-autoscaler-service	scale up/down decisions → nvcf-api
nvcf-state-metrics	nvcf-state-metrics	state/metrics collection → nvcf-api
nvct-api (cloud-tasks)	nvct-api	task orchestration → nvcf-api

The first four cover every invocation path the stack serves, so in practice invocations stop
working. Autoscaling, rate limiting and task orchestration go with them.

(Evidence: these roles are created by migrations/openbao/migrations/{11,12,13,16,17,18,20,21}_setup_*.sh,
each granting a signer role under NVCF_API_SECRET_BASE_PATH.)

Calls into nvct-api

nvct-api is configured with the same single-issuer pattern
(iss: http://nvct-api.nvcf.svc.cluster.local), but the blast radius today is different: no
service in the tree mints a token for the nvct-api audience — there is no
NVCT_API_SECRET_BASE_PATH in any migration script, so no signer role exists on that mount.
Its inbound traffic is external callers (NGC/KAS) and nvapi- task-scoped keys.

So repointing nvct-api's issuer breaks fewer things right now, but it has the identical
limitation: the deployment still has to choose one issuer, and any internal caller added
later — or any operator token minted from that mount — would break the moment an external IdP
is configured. We are asking for the same fix on both services so they do not diverge.

Note this affects JWT-authenticated callers only. nvapi- API keys are short-circuited to
the api-keys introspector before issuer resolution
(AuthManagerResolverConfiguration.java:52-56), so they are unaffected — which is why the
failure mode is "internal control-plane traffic dies while key-based invokes still look fine"
and is easy to misdiagnose.

Why an issuer cannot simply be added

JwtAuthManagerConfiguration binds the two properties as scalars and publishes exactly
one entry:

// nvcf-core/.../configuration/JwtAuthManagerConfiguration.java:41-53
public JwtAuthManagerConfiguration(
        @Value("${spring.security.oauth2.resourceserver.jwt.issuer-uri}") String issuerUri,
        @Value("${spring.security.oauth2.resourceserver.jwt.jwk-set-uri}") String jwkSetUri,
        ...

@Bean
IssuerAuthenticationManagerEntry jwtAuthManager() {
    return new IssuerAuthenticationManagerEntry(issuerUri, jwtAuthenticationManager());
}

The only other producer is NotaryAuthManagerConfiguration, pinned to notary.base-url. So a
deployment can trust the external IdP or OpenBao, never both.

The consumer side is already plural. AuthManagerResolverConfiguration injects a list and
builds a JwtIssuerAuthenticationManagerResolver from it:

// nvcf-core/.../configuration/AuthManagerResolverConfiguration.java:45
private final List<IssuerAuthenticationManagerEntry> jwtAuthManagers;

So the resolver already supports N issuers. Only the producer is single-valued. This looks
like a config-surface change rather than an architectural one: emit one
IssuerAuthenticationManagerEntry per configured issuer instead of exactly one. The same
change applies to nvct-api (nvct-core/.../configuration/AuthManagerResolverConfiguration.java).

Proposal

Keep the built-in OpenBao issuer as the primary, and let deployments add trusted issuers
alongside it — matching the shape already used by ICMS (see Precedent):

security:
  jwt:
    trusted-issuers:
      - issuer-uri:  https://<external-idp>
        jwk-set-uri: https://<external-idp>/.well-known/jwks.json

with the existing spring.security.oauth2.resourceserver.jwt.issuer-uri / jwk-set-uri
retained as the primary issuer, so current deployments are unaffected.

We would also value the list being refresh-scoped, so an issuer can be added or rotated
without restarting the API — ICMS already does this.

Precedent — ICMS has already solved this

This is not a new design. The branch feat/improve/multi-issuer/refresh implements exactly
this pattern for ICMS:

• icms-core/.../configuration/security/TrustedJwtIssuerProperties.java
• icms-core/.../configuration/security/AuthManagerResolver.java (incl. admin-issuer-uri)
• icms-core/.../security/TrustedIssuerRefreshIntegrationTest.java (refresh-scoped)
• icms-service/src/main/resources/application.yaml:

# issuer-uri (and optional admin-issuer-uri). Empty here: only the primary
# (and admin) issuer is trusted. A deployment can trust extra issuers by
# supplying entries at runtime (remote config / env), e.g.:
#   security:
#     jwt:
#       trusted-issuers:
#         - issuer-uri: <ISS_CLAIM>
#           jwk-set-uri: <JWKS_URL>
security:
  jwt:
    trusted-issuers: []

Note ICMS deliberately did not overload
spring.security.oauth2.resourceserver.jwt; it kept the Spring property as the primary and
put extras in a separate security.jwt.trusted-issuers[] list. We would rather match that
than introduce a second, different convention.

Ask: extend the existing ICMS multi-issuer pattern to nvcf-api and nvct-api.

What we evaluated and ruled out

admin-token-issuer-proxy — we checked whether this is the intended bridge from an
external IdP into OpenBao-signed tokens. It is not:

• It performs no inbound authentication. internal/handlers.Keys checks only the HTTP
  method before reading the Vault token and signing. Its own SECURITY.md lists this as
  threat doc: Updated issue templates #1 and states the compensating assumption: "The gateway authenticates and authorizes
  callers before forwarding POST /v1/admin/keys."
• It mints tokens from the issuer the APIs already trust; it never validates a JWT. So it
  cannot extend trust to a new issuer.
• Even fronted by a gateway that validates an external JWT, its output is a shared admin
  token — which removes the per-principal identity that is the entire point of this request.

Its README also describes it as a temporary shim "until NAK/API natively supports this
functionality", so we would rather not build on it.

Note on nvcf-cli

The client side already handles both worlds: OpenBao-signed admin tokens
(internal/openbao/client.go, signing at services/nvcf-api/jwt/sign/nvcf-api-admin) and
OAuth2 client-credentials against an external IdP (NVCF_OAUTH2_CLIENT_ID / _SECRET /
_TOKEN_ENDPOINT). Only the server accepts one issuer at a time.

Workaround today

Stay on the OpenBao issuer and mint operator tokens from OpenBao
(openbao/tools/generate-admin-token.sh, or admin-token-issuer-proxy). Internal auth keeps
working, but callers lose per-principal identity from their own IdP.

Acceptance criteria

1. nvcf-api and nvct-api accept JWTs from the built-in OpenBao issuer and one or more
   configured external issuers, simultaneously.
2. Existing deployments that set only issuer-uri / jwk-set-uri are unchanged.
3. A token from an unconfigured issuer is still rejected.
4. nvapi- API-key auth is unaffected.
5. Trusted issuers can be added/rotated without restarting the service (as in ICMS).

[balajinvda]

🎉 This issue has been resolved in src/control-plane-services/cloud-tasks/v1.68.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

[balajinvda]

🎉 This issue has been resolved in src/control-plane-services/cloud-functions/v1.23.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀