Add support for multiple upgrade artifact source URIs

[lorienhu]

What is the problem this PR solves?

Currently, only one source for upgrade artifacts can be specified, meaning upgrades fail if that source is down.

How does this PR solve the problem?

Adds support for sending multiple upgrade artifact source URIs in an upgrade action with a new sources that deprecates source_uri. Backwards compatibility is maintained.

Design Checklist

• I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer.
• I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected.
• I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc.

Checklist

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool

Related issues

• Closes [Fleet Server] Allow multiple artifact URLs to be configured for a single policy #2586
• Relates [Elastic Agent] Allow multiple artifact URLs to be configured for a single policy elastic-agent#4752
• Relates Add support for multiple upgrade artifact source URIs elastic-agent#16097

[ycombinator]

Could we add a test case for where neither sources nor source_uri is set?

[ycombinator]

Also, this is more of a naming nit, but would source_uris be more consistent than sources?

[lorienhu]

Also, this is more of a naming nit, but would source_uris be more consistent than sources?

I picked sources because the naming for sourceURI vs actual sourceURI was colliding in downloadArtifact before with the old name, since it's technically the base path/base URI.

[lorienhu]

Could we add a test case for where neither sources nor source_uri is set?

Added anyway for clarity, but this just means neither is in the action, following the previously existing behaviour.

[github-actions]

TL;DR

The failing MacOS unit-test job never reaches test execution: Buildkite fails in checkout because it pins commit 676b0ca... and then cannot resolve it in the fetched PR ref (fatal: reference is not a tree). Immediate action is to rerun the PR build on the current head commit and avoid stale commit pinning during PR checkout.

Remediation

• Re-run Buildkite for PR Add support for multiple upgrade artifact source URIs #7682 so checkout uses current PR head (c8561be8bb0231096cba445533d2a0f271dd364f).
• In CI checkout logic, avoid hard-checking out a stale status-event SHA for PR builds; use the fetched PR head (FETCH_HEAD) or fetch the specific SHA before checkout.

Investigation details

Root Cause

This failure is in checkout/bootstrap, not in Fleet Server code or unit tests.

The failed step fetches refs/pull/7682/head successfully, then attempts:

• git checkout -f 676b0ca995253e4e72bfd929a08df9bc7ba59254
• and fails with fatal: reference is not a tree

That indicates the job is trying to check out a SHA that is not present in the local fetched object graph for that PR ref at execution time.

PR context confirms head is now c8561be8bb0231096cba445533d2a0f271dd364f and commit history has moved, which is consistent with stale commit pinning in the CI checkout step.

Evidence

• Build: https://buildkite.com/elastic/fleet-server/builds/16774
• Job/step: :smartbear-testexecute: Run unit tests: MacOS 13
• Key log excerpt:
  • git fetch -v --prune -- origin refs/pull/7682/head
  • # FETCH_HEAD is now c8561be8bb0231096cba445533d2a0f271dd364f
  • git checkout -f 676b0ca995253e4e72bfd929a08df9bc7ba59254
  • fatal: reference is not a tree: 676b0ca995253e4e72bfd929a08df9bc7ba59254

Verification

• Not run locally; conclusion is based on direct Buildkite log evidence and PR metadata.

Follow-up

• If this repeats after rerun, treat as CI checkout configuration/race issue (event SHA drift after force-push) rather than a product-code regression.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.