fix: bound pending DKG message retention per proTxHash

[PastaPastaPasta]

Issue being fixed or feature implemented

The DKG pending-message retention quota in CDKGPendingMessages is keyed by NodeId, which resets on every reconnect. A single masternode identity can therefore retain an effectively unbounded number of pending DKG messages on a victim masternode by reconnecting between sends.

This extracts the DoS-relevant part of #7557 so it can ship in the release; the deserialize-once/framing intake redesign remains in #7557 for post-RC.

What was done?

• Rekeyed the retention quota to the sender's MNAuth-verified proTxHash (locally produced messages charge this node's own proTxHash). The quota is cumulative for the round and not refunded on pop, so neither reconnects nor queue drains regain retention slots. The MNAuth gate pins quota keys to registered masternode identities.
• The duplicate-hash check now runs before the quota charge so resent hashes don't burn budget, and quota-dropped hashes are not marked seen, so another peer with remaining budget can re-deliver the message.
• PopAndDeserializeMessages and the framing/deserialize-once changes from perf: deserialize DKG messages once (framing-only intake) #7557 are intentionally not included.

How Has This Been Tested?

• New unit tests in llmq_dkg_tests: quota survives NodeId changes, quota is per-proTx, duplicates are rejected before charging the quota, own-message enqueue path shares the quota.
• Extended feature_llmq_dkg_intake.py with a late-message scenario: a reconnecting identity cannot exceed its budget across 2*llmq_size fresh NodeIds, over-quota drops are silent (banscore 0), a distinct proTx has its own budget, and round-start clearing discards retained messages without scoring.
• make -j5, test_dash --run_test=llmq_dkg_tests, test_runner.py feature_llmq_dkg_intake.py, lint-python, lint-whitespace.

Breaking Changes

None. Honest masternodes send one message per type per round, far under the unchanged size * 2 budget; the only behavior change is that a reconnecting identity can no longer refill its retention quota.

Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have added or updated relevant unit/integration/functional/e2e tests
• I have made corresponding changes to the documentation
• I have assigned this pull request to a milestone (for repository code-owners and collaborators only)

[thepastaclaw]

🔍 Review in progress — actively reviewing now (commit cf85283)
Stage: Codex precheck starting
ETA: complete ~17:12 UTC (median 17m across 30 recent reviews)
Running 4m · Last checked: 2026-08-12 17:00 UTC

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 8f263fe9-c5de-4931-a1d7-9f967283628e

📥 Commits

Reviewing files that changed from the base of the PR and between 5e46608 and cf85283.

📒 Files selected for processing (1)

• src/test/llmq_dkg_tests.cpp

🚧 Files skipped from review as they are similar to previous changes (1)

• src/test/llmq_dkg_tests.cpp

---

Walkthrough

CDKGPendingMessages now enforces cumulative retention quotas per sender proTxHash. It rejects duplicate hashes before quota accounting and clears per-sender counters during cleanup. DKG paths pass proTxHash values for locally generated and authenticated peer messages. Unit and functional tests cover reconnects, independent identities, duplicate messages, queue draining, and round resets.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant DKGPeer
  participant ProcessMessage
  participant CDKGPendingMessages
  participant DKGSession
  DKGPeer->>ProcessMessage: send authenticated DKG message
  ProcessMessage->>CDKGPendingMessages: enqueue payload with sender proTxHash
  DKGSession->>CDKGPendingMessages: enqueue locally generated message with session proTxHash
  CDKGPendingMessages->>CDKGPendingMessages: reject duplicate or over-quota hash
  CDKGPendingMessages-->>DKGSession: retain accepted message

Loading

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely summarizes the main change: limiting pending DKG message retention per sender proTxHash.
Description check	✅ Passed	The description directly explains the proTxHash-based quota change, its purpose, implementation details, and test coverage.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/llmq/dkgsessionhandler.cpp`:
- Around line 40-44: Update the sender_protx validation in the DKG session
handler to use a direct null check and return immediately for a null identity,
avoiding Assume/assertion_fail abort behavior. Add a unit test covering a null
sender_protx and confirming the request is dropped without aborting.

In `@src/test/llmq_dkg_tests.cpp`:
- Around line 109-114: Add a third sender with available quota in the
pending-message test, submit hash5 through that sender after confirming
protx_b’s quota rejection, and assert that hash5 is retained/enqueued. Update
the expected PopPendingMessages(5) count to include the successfully accepted
redelivery while preserving the existing HasSeen assertions for the rejected
attempt.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 89d4a6d7-efef-4f73-a157-b36a14eff395

📥 Commits

Reviewing files that changed from the base of the PR and between c30e901 and 5e46608.

📒 Files selected for processing (5)

• src/llmq/dkgsessionhandler.cpp
• src/llmq/dkgsessionhandler.h
• src/llmq/net_dkg.cpp
• src/test/llmq_dkg_tests.cpp
• test/functional/feature_llmq_dkg_intake.py

[PastaPastaPasta]

LGTM; this simplifies the remaining / other PR