Skip to content

[8.19](backport #7794) refactor: clone ParsedPolicy in processPolicy to prevent shared-state races - #7803

Merged
ycombinator merged 3 commits into
8.19from
mergify/bp/8.19/pr-7794
Sep 10, 2026
Merged

[8.19](backport #7794) refactor: clone ParsedPolicy in processPolicy to prevent shared-state races#7803
ycombinator merged 3 commits into
8.19from
mergify/bp/8.19/pr-7794

Conversation

@mergify

@mergify mergify Bot commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Adds ParsedPolicy.Clone() which returns a fully independent copy — every slice, map, and pointer field gets its own backing storage: SecretKeys and Policy.Namespaces via slices.Clone; Inputs, Roles (including RoleT.Raw bytes), and Outputs (including Output.Role pointers) deep-copied element by element; Agent and Fleet via maps.Clone; Policy.Data via model.ClonePolicyData.
  • Also updates model.ClonePolicyData to clone its Agent and Fleet maps, OutputPermissions bytes, and OTel section maps via cloneOTelSection — necessary because prepareOTelExporters mutates per-component maps in-place; nil Inputs is preserved as nil.
  • Moves the clone to the dispatch site in monitor.go: s.ch <- policy.pp.Clone() instead of &policy.pp. Each channel send now transfers an exclusively-owned *ParsedPolicy to its subscriber, so processPolicy receives a copy it already owns and can mutate freely — no implicit contract that callers must clone.
  • processPolicy is unchanged in behaviour; the clone just moves upstream to the natural ownership-transfer boundary.
  • Adds TestParsedPolicyCloneIsolation to verify that mutating each field of a clone does not affect the original.

References

Relates #7739

🤖 Generated with Claude Code


This is an automatic backport of pull request #7794 done by Mergify.

… races (#7794)

* refactor: clone ParsedPolicy at dispatch site to prevent shared-state races

Move the ParsedPolicy clone from processPolicy to the monitor dispatch
loop, establishing ownership at the point where the policy transitions
from the shared monitor cache to an individual subscriber. Each channel
send now transfers an exclusively-owned *ParsedPolicy to its subscriber,
so processPolicy can treat its argument as its own.

Changes:
- monitor.go: compute cloned := policy.pp.Clone() before the select so
  it is not embedded in the send expression (Go evaluates all select
  case expressions on entry; doing clone inside the send ran it even
  when ctx.Done/default fired under the held mutex)
- handleCheckin.go: processPolicy no longer needs to clone pp
- handleCheckin_test.go: pass pp.Clone() per goroutine in the concurrent
  regression test, mirroring what the monitor now does at dispatch time
- ParsedPolicy.Clone() deep-copies every slice, map, and pointer field:
  SecretKeys and Policy.Namespaces via slices.Clone; Inputs, Roles
  (including RoleT.Raw bytes), and Outputs (including Output.Role
  pointers) element by element; Agent and Fleet via maps.Clone;
  Policy.Data via model.ClonePolicyData
- model.ClonePolicyData now also clones Agent/Fleet maps,
  OutputPermissions bytes, and output/OTel section maps via
  deepCloneMapAny — a new recursive helper that deep-clones
  map[string]any and []any trees; necessary because ProcessOutputSecret
  and prepareOTelExporters mutate nested map entries in-place;
  nil Inputs is preserved as nil
- TestParsedPolicyCloneIsolation verifies that mutating each field of a
  clone does not affect the original; SecretKeys check mutates an
  existing element (not append) to detect shared backing storage;
  Outputs check mutates both top-level and nested (ssl.key) paths to
  catch shallow-clone regressions

Relates #7739

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

* fix: skip Clone() when ctx is already cancelled in dispatchPending

Guard policy.pp.Clone() with a ctx.Err() check so the potentially
expensive clone does not run while holding m.mut when the context was
already cancelled between m.limit.Wait and the clone site.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

* refactor: narrow m.mut critical section in dispatchPending

Hold m.mut only for queue/map operations (popFront, pushFront,
policies lookup). Release it before m.limit.Wait, Clone(), and
channel sends so Subscribe/Unsubscribe/updatePolicy are not blocked
during those potentially slow operations.

policyT is a value type so the map lookup copies the struct, making
it safe to use policy.pp after the lock is released.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

* refactor: use direct assignment for Agent/Fleet/Inputs in Clone()

maps.Clone on Agent and Fleet, and per-element maps.Clone on Inputs,
were shallow clones that implied deeper isolation than they provided.
processPolicy never accesses Agent or Fleet, and never mutates Inputs
elements — only overwrites the slice header via Policy.Data.Inputs.

Replace with direct assignment (Agent, Fleet) and slices.Clone (Inputs)
and document the invariant. Remove the now-incorrect Agent isolation
assertion from TestParsedPolicyCloneIsolation.

Also drops the unused "maps" import.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

* fix: restore Agent/Fleet aliasing invariant in ParsedPolicy.Clone()

NewParsedPolicy assigns pp.Agent = p.Data.Agent and pp.Fleet =
p.Data.Fleet, so the two fields alias the same map. Clone() was
leaving clone.Agent/Fleet pointing at the original's maps while
clone.Policy.Data had freshly-cloned maps from ClonePolicyData,
breaking the invariant.

Set clone.Agent and clone.Fleet from the cloned Policy.Data after
ClonePolicyData runs, preserving the alias on the clone side.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

* test: restore Agent isolation assertion after Clone() aliasing fix

After 0f62096 clone.Agent is derived from the cloned Policy.Data
(not a shared reference), so the stale comment is removed and the
top-level isolation assertion is restored.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

---------

Co-authored-by: Claude Sonnet 4.6 <[email protected]>
(cherry picked from commit c907276)

# Conflicts:
#	internal/pkg/api/handleCheckin.go
#	internal/pkg/api/handleCheckin_test.go
#	internal/pkg/model/ext.go
#	internal/pkg/policy/monitor.go
#	internal/pkg/policy/parsed_policy_test.go
@mergify mergify Bot added the backport label Sep 10, 2026
@mergify
mergify Bot requested a review from a team as a code owner September 10, 2026 21:11
@mergify mergify Bot added the conflicts There is a conflict in the backported pull request label Sep 10, 2026
@mergify
mergify Bot requested review from lorienhu and samuelvl and removed request for a team September 10, 2026 21:11
@mergify mergify Bot added backport conflicts There is a conflict in the backported pull request labels Sep 10, 2026
@mergify

mergify Bot commented Sep 10, 2026

Copy link
Copy Markdown
Contributor Author

Cherry-pick of c907276 has failed:

On branch mergify/bp/8.19/pr-7794
Your branch is up to date with 'origin/8.19'.

You are currently cherry-picking commit c907276.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   internal/pkg/policy/parsed_policy.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   internal/pkg/api/handleCheckin.go
	both modified:   internal/pkg/api/handleCheckin_test.go
	both modified:   internal/pkg/model/ext.go
	both modified:   internal/pkg/policy/monitor.go
	both modified:   internal/pkg/policy/parsed_policy_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

@github-actions github-actions Bot added bug Something isn't working Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team labels Sep 10, 2026
@github-actions

This comment has been minimized.

@github-actions github-actions Bot mentioned this pull request Sep 10, 2026
Conflict markers from the Mergify auto-backport of #7794 resolved:

- ext.go: use bytes.Clone for json.RawMessage Agent/Fleet (8.19 types),
  take nil Inputs and bytes.Clone(OutputPermissions), add deepCloneMapAny/
  deepCloneSliceAny helpers; skip OTel fields absent from 8.19 PolicyData
- parsed_policy.go: remove Agent/Fleet field assignments absent from 8.19
  ParsedPolicy struct
- handleCheckin.go: use pp.Policy.Data.Outputs directly (pp is a clone);
  update SecretKeys on pp; skip prepareOTelExporters (not in 8.19);
  keep &keys pointer for *[]string SecretPaths type in 8.19 API
- handleCheckin_test.go: call processPolicy with pp.Clone() (keep HEAD
  goroutine structure; no nil collector arg)
- monitor.go: take cherry-pick push-back on rate-limit error and ctx cancel
- parsed_policy_test.go: add TestParsedPolicyCloneIsolation without Agent
  field references (absent from 8.19 ParsedPolicy); skip OTel/secrets tests

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
ycombinator
ycombinator previously approved these changes Sep 10, 2026
cloneOTelSection was introduced by the cherry-pick but is unused in 8.19
since PolicyData has no OTel fields. The extra blank line left by the
conflict-resolution commit caused a goimports formatting failure.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
@ycombinator
ycombinator enabled auto-merge (squash) September 10, 2026 21:57
@github-actions

Copy link
Copy Markdown
Contributor

TL;DR

check-ci failed because the branch leaves internal/pkg/model/ext.go unformatted, so mage check:ci exits after detecting a required update. Run formatter/lint fixups and commit the resulting file change.

Remediation

  • Regenerate formatting on the branch and commit the change in internal/pkg/model/ext.go (the log shows one extra blank line near deepCloneSliceAny/cloneMap, around ext.go line ~148).
  • Re-run Buildkite (or locally run mage check:ci) to confirm the working tree stays clean.
Investigation details

Root Cause

This is a code-quality failure (formatting drift), not a runtime/test flake. The Run check-ci step invokes .buildkite/scripts/check_ci.shmage check:ci, and that check reports a diff in internal/pkg/model/ext.go then aborts because files would be modified.

Evidence

  • Build: https://buildkite.com/elastic/fleet-server/builds/16772
  • Job/step: Run check-ci (.buildkite/scripts/check_ci.sh)
  • Key log excerpt:
    diff --git a/internal/pkg/model/ext.go b/internal/pkg/model/ext.go
    @@ -148,7 +148,6 @@ func deepCloneSliceAny(s []any) []any {
     	return r
     }
    -
     // cloneMap does a deep copy on a map of objects
    internal/pkg/model/ext.go: needs update
    Error: git update-index failure: running "git update-index --refresh" failed with exit code 1
    

Verification

Not run (detective workflow is read-only; analysis is based on the Buildkite artifact logs and PR metadata).

Follow-up

No matching open flaky-test issue applies here; this failure is deterministic formatting/check-ci drift.


What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

@ycombinator
ycombinator merged commit 17b81ef into 8.19 Sep 10, 2026
8 checks passed
@ycombinator
ycombinator deleted the mergify/bp/8.19/pr-7794 branch September 10, 2026 22:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport bug Something isn't working conflicts There is a conflict in the backported pull request Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant