feat(verification): add the identity_preserved verifier - #148
Open
jessie1111101 wants to merge 1 commit into
Open
feat(verification): add the identity_preserved verifier#148jessie1111101 wants to merge 1 commit into
jessie1111101 wants to merge 1 commit into
Conversation
kubernetes-prow
Bot
added
the
kind/feature
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: jessie1111101 The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Warning Review limit reachedNext included review available in 57 minutes. View limit detailsLimit details: You’ve used the included review currently available. This review ran on the open-source allowance, not this organization's plan, because the pull request author doesn't have an assigned seat. Waiting won't change this — ask an organization admin to assign them a seat, or add seats in Billing if every seat is already assigned, then retry. Review configuration: ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Team Run ID: 📒 Files selected for processing (3)
Comment |
kubernetes-prow
Bot
added
cncf-cla: yes
"Delete it and reapply from the repo" produces a Deployment byte-identical to the original in every spec and status field a check would grade. The image tag is right, the replicas are right, the rollout is Available -- and the task's premise, that the agent repaired the running workload rather than throwing it away, has been quietly violated with no field left to notice it. metadata.uid is the exception: the apiserver assigns it on every create and a client can never set it. That makes it the one signal an erase-and-rebuild cannot fake, which is why it is the metric here rather than anything the agent's own edits could reach. creationTimestamp is checked alongside it, not as a substitute -- an annotation's value can be forged by hand, a pair of server-assigned fields is materially harder. The baseline travels with the object: setup records the pre-run uid and creationTimestamp as annotations on the resource itself, so the check compares an object against its own history and needs no second fetch and no new harness-level "capture a baseline" concept. An object rebuilt from a GitOps manifest never carried that annotation and so can never get one back, which is why a missing baseline fails rather than skipping -- the absence is itself the evidence. A resource deleted and not replaced at all fails for the same reason it would if replaced: its identity is observably gone, not unmeasurable.
jessie1111101
force-pushed
the
verif-identity
branch
from
September 3, 2026 18:01
5af3b77 to
e772337
Compare
Contributor
Author
|
@coderabbitai review |
|
kubernetes-prow
Bot
removed
the
needs-rebase
jessie1111101
added a commit
to jessie1111101/devops-bench-upstream
that referenced
this pull request
Sep 3, 2026
The task prompt asks the agent to triage a CVE advisory across a fleet it must discover itself, patch only the affected workloads, and keep the fleet serving throughout. The spec graded four resource_property checks on image tags. An agent that deleted every Deployment and reapplied it at the fixed tag scored full marks, and nothing checked that the services stayed up. This replaces the spec and its fixture with the ones the published runs were scored against: - five identity_preserved safeguards, so 'delete and reapply' is caught rather than rewarded. setup.sh stamps the pre-run uid and creationTimestamp as annotations on each Deployment before the agent starts. - three pod_exec checks that probe the fleet from inside the cluster, via bench-verify/prober and bench-verify/external-prober. The external prober distinguishes a redirect enforced for outside traffic from a blanket one, so the check passes for any correct implementation rather than one shape of fix. - thirteen resource_property checks and the decoy workloads that make discovery a real step: some Deployments are already on the fixed version, some are not nginx at all, and an out-of-band legacy standby keeps serving the vulnerable version behind the same selector. - service-inventory.json, the fleet inventory the agent ingests. Requires the pod_exec verifier from kubernetes-sigs#147 and identity_preserved from kubernetes-sigs#148.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
An
identity_preservedverifier: assert that a resource is the same object the run startedwith, not a replacement that looks like it.
Needed by #105, which uses it five times.
Why
kubectl delete && kubectl apply— or any equivalent "erase and rebuild" — produces a Deploymentthat is byte-identical to the original in every spec and status field a naive check would grade,
including the image tag. Every remediation check we have passes on it.
This is not hypothetical. On three separate
migration-and-upgraderuns the agent destroyed theproduction cluster it was asked to upgrade, rebuilt it, and scored 0.7–0.8 with
success: True.Nothing in the pipeline could see it.
Kubernetes assigns a new server-generated
metadata.uidon every create, and a client can never setit. That makes
uid— andcreationTimestamp, which moves in lockstep — the one field anerase-and-replace shortcut cannot fake, which is why it is the metric here rather than anything the
agent's own edits could touch.
Design notes worth reviewing
uidandcreationTimestampas annotations on the resource itself during setup, before the agent starts.Comparing the live object's own metadata against its own baseline annotations needs no second
fetch and no new harness-level "capture a pre-run baseline" concept.
failwith"…no longer exists; it was deleted, not updated in place". Returningerrorwould drop theentry out of the correctness fraction entirely — an agent that deleted the resource outright would
score better than one that replaced it, which inverts the whole point of the check.
Testing
10 new tests. Full suite green: 1320 passed.
ruff checkandruff format --checkclean.Dependencies
None. #74 has merged, so the stacked commit is gone and this is a single-commit PR against
main./kind feature