Skip to content

[9.5](backport #7794) refactor: clone ParsedPolicy in processPolicy to prevent shared-state races - #7801

Merged
ycombinator merged 2 commits into
9.5from
mergify/bp/9.5/pr-7794
Sep 10, 2026
Merged

[9.5](backport #7794) refactor: clone ParsedPolicy in processPolicy to prevent shared-state races#7801
ycombinator merged 2 commits into
9.5from
mergify/bp/9.5/pr-7794

Conversation

@mergify

@mergify mergify Bot commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Adds ParsedPolicy.Clone() which returns a fully independent copy — every slice, map, and pointer field gets its own backing storage: SecretKeys and Policy.Namespaces via slices.Clone; Inputs, Roles (including RoleT.Raw bytes), and Outputs (including Output.Role pointers) deep-copied element by element; Agent and Fleet via maps.Clone; Policy.Data via model.ClonePolicyData.
  • Also updates model.ClonePolicyData to clone its Agent and Fleet maps, OutputPermissions bytes, and OTel section maps via cloneOTelSection — necessary because prepareOTelExporters mutates per-component maps in-place; nil Inputs is preserved as nil.
  • Moves the clone to the dispatch site in monitor.go: s.ch <- policy.pp.Clone() instead of &policy.pp. Each channel send now transfers an exclusively-owned *ParsedPolicy to its subscriber, so processPolicy receives a copy it already owns and can mutate freely — no implicit contract that callers must clone.
  • processPolicy is unchanged in behaviour; the clone just moves upstream to the natural ownership-transfer boundary.
  • Adds TestParsedPolicyCloneIsolation to verify that mutating each field of a clone does not affect the original.

References

Relates #7739

🤖 Generated with Claude Code


This is an automatic backport of pull request #7794 done by Mergify.

… races (#7794)

* refactor: clone ParsedPolicy at dispatch site to prevent shared-state races

Move the ParsedPolicy clone from processPolicy to the monitor dispatch
loop, establishing ownership at the point where the policy transitions
from the shared monitor cache to an individual subscriber. Each channel
send now transfers an exclusively-owned *ParsedPolicy to its subscriber,
so processPolicy can treat its argument as its own.

Changes:
- monitor.go: compute cloned := policy.pp.Clone() before the select so
  it is not embedded in the send expression (Go evaluates all select
  case expressions on entry; doing clone inside the send ran it even
  when ctx.Done/default fired under the held mutex)
- handleCheckin.go: processPolicy no longer needs to clone pp
- handleCheckin_test.go: pass pp.Clone() per goroutine in the concurrent
  regression test, mirroring what the monitor now does at dispatch time
- ParsedPolicy.Clone() deep-copies every slice, map, and pointer field:
  SecretKeys and Policy.Namespaces via slices.Clone; Inputs, Roles
  (including RoleT.Raw bytes), and Outputs (including Output.Role
  pointers) element by element; Agent and Fleet via maps.Clone;
  Policy.Data via model.ClonePolicyData
- model.ClonePolicyData now also clones Agent/Fleet maps,
  OutputPermissions bytes, and output/OTel section maps via
  deepCloneMapAny — a new recursive helper that deep-clones
  map[string]any and []any trees; necessary because ProcessOutputSecret
  and prepareOTelExporters mutate nested map entries in-place;
  nil Inputs is preserved as nil
- TestParsedPolicyCloneIsolation verifies that mutating each field of a
  clone does not affect the original; SecretKeys check mutates an
  existing element (not append) to detect shared backing storage;
  Outputs check mutates both top-level and nested (ssl.key) paths to
  catch shallow-clone regressions

Relates #7739

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

* fix: skip Clone() when ctx is already cancelled in dispatchPending

Guard policy.pp.Clone() with a ctx.Err() check so the potentially
expensive clone does not run while holding m.mut when the context was
already cancelled between m.limit.Wait and the clone site.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

* refactor: narrow m.mut critical section in dispatchPending

Hold m.mut only for queue/map operations (popFront, pushFront,
policies lookup). Release it before m.limit.Wait, Clone(), and
channel sends so Subscribe/Unsubscribe/updatePolicy are not blocked
during those potentially slow operations.

policyT is a value type so the map lookup copies the struct, making
it safe to use policy.pp after the lock is released.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

* refactor: use direct assignment for Agent/Fleet/Inputs in Clone()

maps.Clone on Agent and Fleet, and per-element maps.Clone on Inputs,
were shallow clones that implied deeper isolation than they provided.
processPolicy never accesses Agent or Fleet, and never mutates Inputs
elements — only overwrites the slice header via Policy.Data.Inputs.

Replace with direct assignment (Agent, Fleet) and slices.Clone (Inputs)
and document the invariant. Remove the now-incorrect Agent isolation
assertion from TestParsedPolicyCloneIsolation.

Also drops the unused "maps" import.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

* fix: restore Agent/Fleet aliasing invariant in ParsedPolicy.Clone()

NewParsedPolicy assigns pp.Agent = p.Data.Agent and pp.Fleet =
p.Data.Fleet, so the two fields alias the same map. Clone() was
leaving clone.Agent/Fleet pointing at the original's maps while
clone.Policy.Data had freshly-cloned maps from ClonePolicyData,
breaking the invariant.

Set clone.Agent and clone.Fleet from the cloned Policy.Data after
ClonePolicyData runs, preserving the alias on the clone side.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

* test: restore Agent isolation assertion after Clone() aliasing fix

After 0f62096 clone.Agent is derived from the cloned Policy.Data
(not a shared reference), so the stale comment is removed and the
top-level isolation assertion is restored.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

---------

Co-authored-by: Claude Sonnet 4.6 <[email protected]>
(cherry picked from commit c907276)

# Conflicts:
#	internal/pkg/api/handleCheckin.go
#	internal/pkg/api/handleCheckin_test.go
@mergify mergify Bot added backport conflicts There is a conflict in the backported pull request labels Sep 10, 2026
@mergify
mergify Bot requested a review from a team as a code owner September 10, 2026 21:10
@mergify
mergify Bot requested review from swiatekm and ycombinator and removed request for a team September 10, 2026 21:10
@mergify

mergify Bot commented Sep 10, 2026

Copy link
Copy Markdown
Contributor Author

Cherry-pick of c907276 has failed:

On branch mergify/bp/9.5/pr-7794
Your branch is up to date with 'origin/9.5'.

You are currently cherry-picking commit c907276.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   internal/pkg/model/ext.go
	modified:   internal/pkg/policy/monitor.go
	modified:   internal/pkg/policy/parsed_policy.go
	modified:   internal/pkg/policy/parsed_policy_test.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   internal/pkg/api/handleCheckin.go
	both modified:   internal/pkg/api/handleCheckin_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

@github-actions github-actions Bot added bug Something isn't working Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team labels Sep 10, 2026
@github-actions github-actions Bot mentioned this pull request Sep 10, 2026
@github-actions

Copy link
Copy Markdown
Contributor

TL;DR

Buildkite failed because this backport branch still contains unresolved merge-conflict markers in internal/pkg/api/handleCheckin.go and internal/pkg/api/handleCheckin_test.go, which causes Go parsing to fail in every packaging lane.

Remediation

  • Resolve the cherry-pick conflict blocks and remove <<<<<<<, =======, >>>>>>> markers in both files, keeping the 9.5-compatible processPolicy(...) / Prepare(...) call signatures consistently.
  • Re-run CI (check-ci + package lanes) after conflict resolution to confirm parse/build success.
Investigation details

Root Cause

This is a code bug introduced by an incomplete conflict resolution during backporting. The PR diff itself shows conflict markers still present in processPolicy and its related test hunk:

  • internal/pkg/api/handleCheckin.go around @@ -1067... (conflict between HEAD and commit c907276)
  • internal/pkg/api/handleCheckin_test.go around @@ -1927... (same unresolved conflict pattern)

Because of those markers, the Go compiler/parser errors immediately before any functional tests can run.

Evidence

  • Build: https://buildkite.com/elastic/fleet-server/builds/16767
  • Failing jobs: Run check-ci, Package x86_64, Package x86_64 FIPS, Package aarch64, Package aarch64 FIPS
  • Key log excerpts:
    • internal/pkg/api/handleCheckin.go:1082:1: expected statement, found '<<' (check-ci)
    • internal/pkg/api/handleCheckin_test.go:1930:1: expected statement, found '<<' (check-ci)
    • internal/pkg/api/handleCheckin.go:1082:1: syntax error: unexpected <<, expected } (all package lanes)

Verification

  • Not run locally in this detective pass; analysis is based on Buildkite failure logs and PR diff content.

Follow-up

  • After resolving markers, if any lane still fails, re-check for branch-specific signature mismatches in processPolicy(...) and policyOutput.Prepare(...) introduced by the backport merge.

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

Conflict markers from the Mergify auto-backport of #7794 are resolved:
- handleCheckin.go: use pp.Policy.Data.Outputs (9.5 Prepare has no options param)
- handleCheckin_test.go: call processPolicy with pp.Clone() (no nil collector arg)

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
@ycombinator
ycombinator enabled auto-merge (squash) September 10, 2026 21:35
@ycombinator
ycombinator merged commit 14ae085 into 9.5 Sep 10, 2026
12 checks passed
@ycombinator
ycombinator deleted the mergify/bp/9.5/pr-7794 branch September 10, 2026 22:39
ebeahan added a commit that referenced this pull request Sep 11, 2026
…Policy to prevent shared-state races (#7801)" (#7804)

This reverts commit 14ae085.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport bug Something isn't working conflicts There is a conflict in the backported pull request Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant