Skip to content

fix(roadmap): stage includes perimeter tests for network policies #3972

New issue
New issue
Open
Open
fix(roadmap): stage includes perimeter tests for network policies#3972
Labels
effort/mediumEffort Mediumpriority/p1-highPriority P1: high-impact work for the next planned milestonescope/roadmapThe roadmap jobs: intake, refine, stage, implement and the Service that runs themsource/agentFiled by an agentstatus/readyStatus Ready: problem, acceptance criteria and an answer to each key questiontype/bugDefect, regression, or incorrect runtime behaviorvalue/highValue High: the delivery of the item matters much

Description

@dan-petty
dan-petty
opened on Nov 7, 2026

Problem Description

The stage planning job omits existing test files when a plan changes perimeter network policies. For example, adding an egress rule to k8s/logging/networkpolicy.yaml breaks tests/commands/k8s/test_k8s_logging_stack.py::test_logging_stack_security_and_scoping, which asserts len(egress_rules) == 2. The stage job did not include that test in the Changes: list of the plan. During the implement job, the runner cannot modify unlisted test files because the diff_scope constraint forbids it. The runner repeatedly runs tests and attempts workarounds until it exhausts its token budget of 8,000,000 tokens. The job then fails with LimitReached and marks the item status/blocked.

Reproduction Steps

  1. Stage an item that adds an egress rule to k8s/logging/networkpolicy.yaml.
  2. Inspect tasks.md and observe that tests/commands/k8s/test_k8s_logging_stack.py is absent from Changes:.
  3. Run the implement job on the item.
  4. Observe the runner fail the egress rule count assertion and enter an edit loop until it exhausts the token budget.

Expected vs Actual Behavior

Expected: The stage job identifies existing tests that inspect the modified manifests and adds them to Changes: in tasks.md.

Actual: The stage job omits the test files from Changes:, creating a deadlock between diff_scope and test assertions during implementation.

Environment Diagnostics

devops-cli 0.79.0, Linux x86_64, 2026-11-07.

Proposed Solution & Architectural Design

  1. Extend the stage probe or stage paths analyzer to find existing tests that reference the target manifest paths.
  2. Ensure tests with explicit rule counts or structural assertions on target manifests are included in task group changes.
  3. Add a check or early-stop guard in the implement runner when an invariant test fails on a file outside Changes:.

Existing Tool Considered

The existing stage_probe and stage_paths modules in src/devops_cli/roadmap/stage/.

Proposed design

Problem statement

The stage planning job leaves out existing tests that assert on a changed perimeter network policy. The implement job cannot edit a test outside its Changes: list because diff_scope forbids it. The runner then loops until it spends its token budget and marks the item blocked. This change makes the stage path check require those tests in the plan.

Acceptance criteria

Requirement: Perimeter test required

The stage path check SHALL refuse a plan that changes a YAML manifest under k8s/ and omits a test file that references the manifest.

Scenario: Logging egress plan omits the logging test

  • WHEN The plan lists k8s/logging/networkpolicy.yaml in Changes: and omits tests/commands/k8s/test_k8s_logging_stack.py, which asserts len(egress_rules) == 2 and contains the literals logging and networkpolicy.yaml.
  • THEN path_problems returns a problem that names k8s/logging/networkpolicy.yaml and tests/commands/k8s/test_k8s_logging_stack.py.

Requirement: Included perimeter test accepted

The stage path check SHALL accept a plan that lists every test file that references a changed manifest.

Scenario: Plan lists the logging perimeter test

  • WHEN The plan lists k8s/logging/networkpolicy.yaml and tests/commands/k8s/test_k8s_logging_stack.py in Changes:.
  • THEN path_problems returns no problem that names tests/commands/k8s/test_k8s_logging_stack.py.

Requirement: Path components identify a reference

The stage path check MUST recognize a test that builds a manifest path from separate string literals.

Scenario: Test builds the logging manifest path from components

  • WHEN A test file contains the string literals logging and networkpolicy.yaml.
  • THEN The check classifies that test as a reference to k8s/logging/networkpolicy.yaml.

Requirement: Unrelated test not required

The stage path check SHALL NOT require a test file that lacks the manifest basename or the parent directory name.

Scenario: A logging test without the network policy basename

  • WHEN A test file contains logging but not networkpolicy.yaml.
  • THEN The check does not name that test file as a missing perimeter test for k8s/logging/networkpolicy.yaml.

Requirement: Stage eval score task

The change's tasks.md MUST hold a task that runs uv run devops roadmap eval stage --replay with the words score, before and after.

Scenario: Stage path change plan

  • WHEN The plan changes src/devops_cli/roadmap/stage/stage_paths.py.
  • THEN check_proposal accepts the plan only when one group task holds that command and those three words.

Existing Tool Considered

No alternatives check has run. No open, maintained tool covers the need: this is an internal plan check in devops-cli. The change extends src/devops_cli/roadmap/stage/stage_paths.py and src/devops_cli/roadmap/stage/stage_change.py. The written cost is about 150,000 tokens to build, keep and fix the code, and about 2 engineer-days of elapsed time.

Key questions

  • Where should the stage job enforce inclusion of perimeter tests?: Add the check to path_problems in src/devops_cli/roadmap/stage/stage_paths.py. (best practice) [sources: src/devops_cli/roadmap/stage/stage_paths.py, openspec/changes/3307-the-plan-check-names-a-caller/design.md:5, docs/roadmap/package-layout.md:11]
    • Rejected: stage_probe. It reads the commit for the prompt and does not hold the plan-path refusal rule.
    • Chosen: stage_paths.path_problems. It already refuses a plan that leaves out a caller file, and check_proposal calls it.
    • Rejected: new module. It splits one plan-path rule across two modules.
  • How should the check find a test that inspects a changed manifest?: Collect string literals from each test file under tests/ with ast and match the manifest basename and its parent directory name. (best practice) [sources: tests/commands/k8s/test_k8s_logging_stack.py, k8s/logging/networkpolicy.yaml]
    • Rejected: exact full-path string. The logging test builds the path from REPO_ROOT, k8s, logging and networkpolicy.yaml, so it holds no full-path string.
    • Rejected: raw text search. It matches comments and imports and gives false references.
    • Chosen: AST string literals with parent and basename. It finds the logging test, ignores comments, and needs no live tree.
  • Which changed paths trigger the perimeter test check?: Every changed YAML file under k8s/. (best practice) [sources: k8s/logging/networkpolicy.yaml, tests/commands/k8s/test_k8s_logging_stack.py]
    • Rejected: only k8s/**/networkpolicy.yaml. Tests also assert on other manifests, so the same deadlock can return.
    • Chosen: every YAML file under k8s/. It covers the logging case and other manifests without touching source files.
    • Rejected: every changed file. It overlaps the caller check and can require tests for source files.
  • Should the stage path check refuse the plan or only warn?: Refuse the plan in path_problems. (best practice) [sources: docs/ROADMAP_WORKFLOW.md:387]
    • Rejected: warning only. The implement job still cannot edit the test because diff_scope forbids it.
    • Chosen: hard refusal. The plan must list the test before staging can finish.
  • Does a change to stage_paths.py need a stage eval score task?: Yes. One group task must run uv run devops roadmap eval stage --replay and state the score before and after. [sources: docs/ROADMAP_WORKFLOW.md:314, src/devops_cli/roadmap/stage/stage_change.py]
  • Should this change also add an implement runner early-stop guard?: No. Defer it to a separate issue. The stage gate removes the reported deadlock, and the guard is separate runtime behavior. (best practice) [sources: src/devops_cli/roadmap/implement/implement_step.py, src/devops_cli/roadmap/implement/implement_limits.py, docs/ROADMAP_WORKFLOW.md:387]
    • Rejected: include in this change. It adds a second runtime path and is not needed to meet the expected stage behavior.
    • Chosen: defer to a separate issue. The stage gate fixes the reported deadlock, and the guard needs its own tests.

Fit

Fits in one pull request.

Suspected block

None.

Dependencies

None.

Activity

  1. dan-petty
    added
    source/agentFiled by an agent
    type/bugDefect, regression, or incorrect runtime behavior
    scope/roadmapThe roadmap jobs: intake, refine, stage, implement and the Service that runs them
    status/newStatus New: the item is not ready yet
    value/highValue High: the delivery of the item matters much
    effort/mediumEffort Medium
    on Nov 7, 2026

  2. dan-petty commented on Nov 7, 2026

    @dan-petty
    dan-petty
    on Nov 7, 2026
    OwnerAuthor

    Intake placed this item to the backlog: intake puts every new item in the backlog. A person plans the milestones.

    • Type: type/bug. The fix prefix of the title gives the type.
    • Scope: scope/roadmap. The (roadmap) scope of the title gives the scope.
    • Priority: P1-High. The omission causes implement jobs to exhaust 8M tokens and mark items blocked, blocking all network-policy changes.
    • Value: High. Every plan touching a manifest with structural assertions is blocked, wasting token budgets and halting work.
    • Effort: Medium. Extending the stage probe to scan for tests referencing target manifest paths is scoped to existing modules.
    • Related: fix(roadmap): refuse a behavior group that omits a test it changes #3414, which asks for a different change. Both address the stage job omitting existing tests from Changes, but #3414 refuses the plan while the candidate extends the probe to find and include the tests.
  3. dan-petty
    added
    priority/p1-highPriority P1: high-impact work for the next planned milestone
    on Nov 7, 2026
  4. devops-cli-service
    added
    status/readyStatus Ready: problem, acceptance criteria and an answer to each key question
    and removed
    status/newStatus New: the item is not ready yet
    on Nov 7, 2026

  5. devops-cli-service commented on Nov 7, 2026

    @devops-cli-service
    devops-cli-serviceBot
    on Nov 7, 2026
    Contributor

    Status set to Ready at dc276b0 (see proposed design in issue body).

    Moved to new items: #3983.

  6. devops-cli-service commented on Nov 8, 2026

    @devops-cli-service
    devops-cli-serviceBot
    on Nov 8, 2026
    Contributor

    Staged: branch fix/3972-stage-includes-perimeter-tests-for-network, draft pull request #4138. The pull request plans the item and closes no issue.

    The task groups, with the tokens of the inputs of each step, for a budget of 64000 tokens:

    1. Record the tests that a changed manifest reaches: 37990 tokens
    2. Refuse a manifest plan that omits its reference: 36982 tokens
    3. Record the references in the job and the replay: 35622 tokens
    4. State the rule of a manifest plan: 18523 tokens

    The default branch changed these paths that the plan names, after the commit that the planner read. Read the commits before you judge the plan:

    The runs of the planner and the review made 65 model request(s), and used 5089580 input tokens and 98051 output tokens. The tool calls, in order:

    1. list_files({"directory":"src/devops_cli/roadmap/stage"})
    2. list_files({"directory":"tests/roadmap/stage"})
    3. search_code({"text":"caller_problems"})
    4. search_code({"text":"roadmap eval stage --replay"})
    5. search_code({"text":"STAGE_EVAL_TRIGGER_PATHS"})
    6. search_code({"text":"ProposalGate"})
    7. read_file({"path":"src/devops_cli/roadmap/stage/stage_change.py","start_line":180,"end_line":240})
    8. read_file({"path":"src/devops_cli/roadmap/stage/stage_change.py","start_line":1000,"end_line":1200})
    9. read_file({"path":"src/devops_cli/roadmap/stage/stage_change.py","start_line":1168,"end_line":1300})
    10. search_code({"text":"_self_acceptance_problems"})
    11. read_file({"path":"src/devops_cli/roadmap/stage/stage_change.py","start_line":790,"end_line":900})
    12. search_code({"text":"_PLAN_CHECK_FILE"})
    13. search_code({"text":"stage_problem_missing_caller"})
    14. read_file({"path":"tests/roadmap/stage/test_roadmap_stage_paths.py"})
    15. read_file({"path":"tests/roadmap/stage/test_roadmap_stage_paths.py","start_line":189,"end_line":420})
    16. read_file({"end_line":600,"path":"tests/roadmap/stage/test_roadmap_stage_paths.py","start_line":420})
    17. read_file({"end_line":720,"path":"tests/roadmap/stage/test_roadmap_stage_paths.py","start_line":600})
    18. read_spec({"capability":"roadmap-stage"})
    19. search_code({"text":"search_evidence"})
    20. read_file({"end_line":180,"path":"src/devops_cli/roadmap/stage/stage_change.py","start_line":100})
    21. search_code({"text":"_MESSAGE_NAME"})
    22. search_code({"text":"_stale_test_path_problems"})
    23. read_file({"end_line":60,"path":"src/devops_cli/roadmap/stage/stage_eval_gates.py","start_line":1})
    24. read_file({"end_line":1360,"path":"src/devops_cli/roadmap/stage/stage_change.py","start_line":1300})
    25. read_file({"end_line":160,"path":"src/devops_cli/roadmap/stage/stage_eval_gates.py","start_line":60})
    26. search_code({"text":"gate_arguments"})
    27. search_code({"text":"record_search_evidence"})
    28. read_file({"path":"src/devops_cli/roadmap/stage/stage_request.py"})
    29. read_file({"end_line":1200,"path":"src/devops_cli/roadmap/stage/stage.py","start_line":1100})
    30. read_file({"end_line":230,"path":"src/devops_cli/roadmap/stage/stage_form.py","start_line":140})
    31. search_code({"text":"self._grep"})
    32. search_code({"text":"def metric_senders"})
    33. read_file({"end_line":920,"path":"src/devops_cli/roadmap/stage/stage.py","start_line":830})
    34. callers({"symbol":"check_gates"})
    35. list_files({"directory":"src/devops_cli/roadmap"})
    36. list_files({"directory":"tests/roadmap/stage"})
    37. read_file({"path":"src/devops_cli/roadmap/plan_recheck.py","start_line":250,"end_line":330})
    38. read_file({"path":"src/devops_cli/roadmap/stage/stage_change.py","start_line":1120,"end_line":1270})
    39. file_symbols({"path":"src/devops_cli/roadmap/plan_recheck.py"})
    40. file_symbols({"path":"src/devops_cli/roadmap/stage/stage_eval_gates.py"})
    41. file_symbols({"path":"src/devops_cli/roadmap/stage/stage_eval_cases.py"})
    42. file_symbols({"path":"src/devops_cli/roadmap/stage/stage_request.py"})
    43. read_file({"path":"src/devops_cli/roadmap/stage/stage_eval_cases.py","start_line":155,"end_line":215})
    44. read_file({"path":"src/devops_cli/roadmap/stage/stage_eval_gates.py","start_line":60,"end_line":135})
    45. read_file({"path":"src/devops_cli/roadmap/stage/stage_request.py","start_line":58,"end_line":120})
    46. read_file({"path":"src/devops_cli/roadmap/plan_recheck.py","start_line":229,"end_line":250})
    47. search_code({"text":"_TOKEN_COST"})
    48. search_code({"text":"def eval_replay_problems"})
    49. file_symbols({"path":"src/devops_cli/roadmap/stage/stage_form.py"})
    50. file_symbols({"path":"src/devops_cli/roadmap/stage/stage.py"})
    51. read_file({"path":"src/devops_cli/roadmap/stage/stage.py","start_line":1154,"end_line":1200})
    52. read_file({"path":"src/devops_cli/roadmap/stage/stage_form.py","start_line":161,"end_line":205})
    53. read_file({"path":"src/devops_cli/roadmap/stage/stage_change.py","start_line":1029,"end_line":1060})
    54. read_file({"path":"src/devops_cli/roadmap/stage/stage_change.py","start_line":105,"end_line":118})
    55. callers({"symbol":"gate_arguments"})
    56. callers({"symbol":"check_proposal"})
    57. find_issues({"terms":"stage_paths perimeter test"})
    58. read_file({"path":"src/devops_cli/roadmap/stage/stage_change.py","start_line":155,"end_line":225})
    59. search_code({"text":"def path_problems"})
    60. search_code({"text":"def check_gates"})
    61. search_code({"text":"def record_search_evidence"})
    62. search_code({"text":"def _record"})
    63. search_code({"text":"class CaseFacts"})
    64. search_code({"text":"def _problems"})
    65. read_file({"path":"src/devops_cli/roadmap/stage/stage_paths.py","start_line":240,"end_line":420})
    66. read_file({"path":"src/devops_cli/roadmap/stage/stage_change.py","start_line":1100,"end_line":1300})
    67. read_file({"path":"src/devops_cli/roadmap/stage/stage_form.py","start_line":120,"end_line":240})
    68. read_file({"path":"src/devops_cli/roadmap/plan_recheck.py","start_line":150,"end_line":360})
    69. read_file({"end_line":1260,"path":"src/devops_cli/roadmap/stage/stage.py","start_line":1080})
    70. read_file({"end_line":240,"path":"src/devops_cli/roadmap/stage/stage_eval_cases.py","start_line":140})
    71. search_code({"text":"_grep"})
    72. search_code({"text":"def stage_problem_missing_caller"})
    73. search_code({"text":"stage_problem_missing_caller"})
    74. read_file({"end_line":920,"path":"src/devops_cli/roadmap/stage/stage.py","start_line":820})
    75. search_code({"text":"grep ="})
    76. search_code({"text":"grep="})
    77. search_code({"text":"def metric_senders"})
    78. search_code({"text":"def grep_at"})
    79. read_file({"end_line":230,"path":"src/devops_cli/roadmap/stage/stage_overlap.py","start_line":170})
    80. list_files({"directory":"tests/roadmap/stage"})
    81. search_code({"text":"_self_acceptance_problems"})
    82. search_code({"text":"self_acceptance"})
    83. read_file({"end_line":195,"path":"src/devops_cli/roadmap/stage/stage_change.py","start_line":150})
    84. read_file({"end_line":850,"path":"src/devops_cli/roadmap/stage/stage_change.py","start_line":800})
    85. list_files({"directory":"tests/roadmap"})
    86. search_code({"text":"plan_recheck"})
    87. read_spec({"capability":"roadmap-stage"})
    88. search_code({"text":"\"docs\""})
    89. search_code({"text":"ci docs"})
    90. search_code({"text":"networkpolicy.yaml"})
    91. search_code({"text":"egress_rules"})
    92. read_file({"end_line":340,"path":"tests/commands/k8s/test_k8s_logging_stack.py","start_line":280})
    93. read_file({"end_line":60,"path":"tests/commands/k8s/test_k8s_logging_stack.py","start_line":1})
    94. search_code({"text":"\"logging\""})
    95. search_code({"text":"k8s/logging"})
    96. callers({"symbol":"gate_arguments"})
    97. search_code({"text":"check_gates("})
    98. search_code({"text":"check_proposal("})
    99. search_code({"text":"check_proposal"})
    100. read_file({"end_line":440,"path":"src/devops_cli/roadmap/implement/implement.py","start_line":370})
    101. read_file({"end_line":410,"path":"src/devops_cli/roadmap/plan_check.py","start_line":350})
    102. search_code({"text":"def eval_replay_problems"})
    103. read_file({"end_line":180,"path":"src/devops_cli/roadmap/stage/stage_change.py","start_line":196})
    104. read_file({"end_line":1060,"path":"src/devops_cli/roadmap/stage/stage_change.py","start_line":1020})
    105. search_code({"text":"def _stale_test_path_problems"})
    106. read_file({"end_line":1000,"path":"src/devops_cli/roadmap/stage/stage_change.py","start_line":900})
    107. read_file({"end_line":1360,"path":"src/devops_cli/roadmap/stage/stage_change.py","start_line":1300})
    108. search_code({"text":"def _delta_problems"})
    109. read_file({"end_line":560,"path":"src/devops_cli/roadmap/stage/stage_change.py","start_line":430})
    110. read_file({"end_line":760,"path":"src/devops_cli/roadmap/stage/stage_change.py","start_line":660})
    111. read_file({"end_line":660,"path":"src/devops_cli/roadmap/stage/stage_change.py","start_line":590})
    112. search_code({"text":"networkpolicy.yaml"})
    113. references({"name":"networkpolicy.yaml"})
    114. read_file({"path":"src/devops_cli/roadmap/stage/stage_references.py"})
    115. read_file({"end_line":240,"path":"src/devops_cli/roadmap/stage/stage_eval_gates.py","start_line":150})
    116. read_file({"end_line":400,"path":"docs/ROADMAP_WORKFLOW.md","start_line":300})
    117. read_file({"end_line":395,"path":"docs/ROADMAP_WORKFLOW.md","start_line":375})
    118. read_file({"end_line":340,"path":"tests/roadmap/stage/test_roadmap_stage_request.py","start_line":240})
    119. read_file({"end_line":140,"path":"docs/roadmap/plan-recheck.md","start_line":95})

    Tool calls: callers=4 (empty=0), file_symbols=6 (empty=0), find_issues=1 (empty=0), list_files=6 (empty=0), read_file=54 (empty=0), read_spec=2 (empty=0), references=1 (empty=0), search_code=45 (empty=1)

    The stage job read the open pull requests and their files in 1 GraphQL request(s).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    effort/mediumEffort Mediumpriority/p1-highPriority P1: high-impact work for the next planned milestonescope/roadmapThe roadmap jobs: intake, refine, stage, implement and the Service that runs themsource/agentFiled by an agentstatus/readyStatus Ready: problem, acceptance criteria and an answer to each key questiontype/bugDefect, regression, or incorrect runtime behaviorvalue/highValue High: the delivery of the item matters much

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions