feat(config): Add Vault-backed configuration

[magdang]

TL;DR

Resolve complete vault://mount/path#key configuration values directly from Vault KV v2.

Changes

• Add optional [vault] bootstrap configuration and async resolution after configuration source precedence is merged.
• Poll KV v2 metadata, renew renewable tokens, and atomically retain the last-known-good configuration on failures.
• Migrate Keystone and keystone-manage startup to the asynchronous loader.
• Add mock Vault coverage for validation, deduplication, redaction, renewal, reloads, and last-known-good retention.

Tests

• Config: 106 unit tests and 2 doctests passed.
• Keystone manage: 95 tests passed.
• Keystone: 659 tests passed.
• Targeted Clippy, cargo fmt, pre-commit, and git diff check passed.

Closes #502

[magdang]

For the parts of #502 that were not fully specified, I took the following approach:

• Support token authentication and KV v2 only.
• Resolve complete vault:///# values after configuration source precedence is merged.
• Keep [vault] as bootstrap configuration and exclude it from secret resolution.
• Poll KV v2 metadata for new versions because KV v2 secrets do not provide renewable leases.
• Renew renewable Vault tokens halfway through their TTL.
• Rebuild and validate the complete configuration before applying an update.
• Retain the last-known-good configuration if a runtime refresh fails.
• Keep synchronous loaders for configurations without Vault references and require the async loader when references are present.

I kept AppRole, Kubernetes authentication, dynamic secret engines, and Fernet key-source integration out of scope.

[gtema]

very nice, need small changes though

[gtema]

nit: 31_536_000s (1yr) magic sentinel for "no active vault runtime, never fire" deadline in tokio::select!. Prefer std::future::pending() for the no-vault branch — clearer intent, no magic number.

Something like this can be used, but feel free to use different approach

// 1. Define the future dynamically before the select block
let vault_deadline_fut = match &vault_runtime {
    Some(runtime) => {
        let deadline = runtime.next_deadline();
        futures::future::Either::Left(tokio::time::sleep_until(deadline))
    }
    None => futures::future::Either::Right(std::future::pending::<()>()),
};

tokio::select! {
    event = sync_rx.recv() => {
        // ... (your existing sync_rx logic)
    }
    // 2. Await the dynamic future directly
    _ = vault_deadline_fut => {
        // Safe to unwrap here because this branch is unreachable if vault_runtime was None
        let runtime = vault_runtime.as_mut().unwrap(); 
        
        if runtime.renew_if_due().await.is_err() {
            error!("Vault token renewal failed; retrying while retaining current configuration");
        }
        // ... (rest of your renewal / polling logic)
    }
}

[magdang]

Done. Dropped the sentinel. The select! now arms sleep_until(runtime.next_deadline()) only when a Vault runtime is present, and falls back to std::future::pending() otherwise:

let vault_deadline = vault_runtime.as_ref().map(vault::VaultRuntime::next_deadline);
let vault_tick = async {
    match vault_deadline {
        Some(deadline) => tokio::time::sleep_until(deadline).await,
        None => std::future::pending::<()>().await,
    }
};

Used a plain async block instead of future::Either to avoid pulling in futures.

[gtema]

I am not fan of such logic (raising error on sync call requiring async. This requires callers to know how to deal with the error with none of the user having any logic for this. Basically it renders the feature right now as useless. Let's make it async-only accepting wider interface change (mostly only adding .await for callers)

[magdang]

Agreed. Made the loaders async-only. Config::new and Config::load_all are now async and resolve Vault references directly, so the AsyncLoaderRequired error path is gone from the file loaders. Removed the redundant load_all_async and folded it into load_all, with a shared resolve_vault_references helper. Callers just gained .await.

The only remaining sync guard is on TryFrom<ConfigBuilder>, an in-memory test-only conversion that can't be made async. Happy to drop it too if you prefer.