Skip to content

AgentiGrid: MATPOWER reader/writer hardening (no silent misreads, stable round trips) - #138

Open
ahajdar wants to merge 5 commits into
ORNL:developfrom
ahajdar:amir/parser-hardening
Open

ahajdar wants to merge 5 commits into
ORNL:developfrom
ahajdar:amir/parser-hardening

Conversation

@ahajdar

@ahajdar ahajdar commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator

Merge request type

  • New feature
  • Resolves bug
  • Documentation
  • Other

Relates to

  • OPFLOW
  • SOPFLOW
  • SCOPFLOW
  • TCOPFLOW
  • CMake build system
  • Spack configuration
  • Manual
  • Web docs
  • Other (AgentiGrid)

This MR updates

  • Header files
  • Source code
  • CMake build system
  • Spack configuration
  • Web docs
  • Manual
  • Other (AgentiGrid tests, claude_code_changes.md)

Summary

Follow-up to #137 (}; terminator for cell arrays), as agreed: same two files, found while writing an independent spec for the MATPOWER reader and writer.

Depends on #137. Until that merges, this branch also shows PR-A's two commits; please review only the last two ("reject malformed input, stable round trips" and "exact near-max floats, idempotent rounding, baseMVA literals"). Once #137 is merged I'll rebase this branch onto develop so only these two commits remain.

Defects fixed

# Before After
1 A row with a non-numeric value was skipped with only a log warning, so a bus, generator or branch could silently disappear. Too-short rows crashed with IndexError. ValueError naming the section and row for non-numeric values, wrong column counts, and non-integral values in integer columns.
2 Valid MATLAB syntax was misread: two rows on one line became one garbage row, and comma-separated rows were all dropped. ValueError with a clear message. The same applies to ... continuations, unterminated blocks and a missing mpc.bus.
3 The writer crashed on Inf/NaN, which the reader accepts. Written as Inf/-Inf/NaN.
4 Every read/write round trip added another header line. Writing is idempotent.
5 Case version '1' was accepted even though ExaGO reads version 2. ValueError. A missing version still means '2'.

Also: CRLF input reads the same as LF and output is always LF. The function mpc = … line always carries casename.

Compatibility

All 29 case files in datafiles/ (from case5 to case_ACTIVSg10k, including the CRLF ones) still parse and round-trip to equivalent networks. Sections other than bus, gen, branch and gencost (areas, dcline, …) are kept as opaque text and are not validated.

Tests

The new tests/grid_files/test_matpower_hardening.py has 29 tests; 22 of them fail without the fix. The existing suite still passes, and PR-B applies cleanly on top.

ahajdar and others added 5 commits October 3, 2026 21:25
The MATPOWER parser stored mpc.gentype / mpc.genfuel / mpc.bus_name without
the trailing ";", and the writer wrote them back as a bare "}". ExaGO's reader
(src/ps/psreaddata.cpp) ends mpc.genfuel at the first line containing "};", so
for every case AgentiGrid rewrote, ExaGO ignored the fuel types: all units fell
back to the default fuel, ramp rates stayed 0 (SCOPFLOW cannot redispatch in
contingencies and reports local infeasibility on ACTIVSg200 at base load, where
the original file converges) and wind/solar units stayed at Pmin = Pmax.

The parser now keeps "};" / "];"; the writer also terminates raw sections from
networks parsed before this fix. tests/grid_files/test_matpower_cell_terminators.py
mimics ExaGO's genfuel detection on a round-tripped ACTIVSg200 (3 tests; all
fail without the fix).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…onger overwrites it

tests/fixtures/sample_opflow_output.txt is OPFLOW output for case_ACTIVSg200.m
as rewritten by AgentiGrid. It was produced by the writer fixed in the previous
commit and therefore listed all 49 generators as COAL; test_gen_189 asserted
COAL although the case file says nuclear for bus 189. Regenerated with the
fixed writer (25 coal, 17 ng, 6 wind, 1 nuclear) and updated the expected
values in tests/results/test_opflow_parser.py; test_fuel_mix_matches_case_file
guards against regenerating it with a broken writer again.

TestExecutorLive::test_opflow_run overwrote this tracked fixture on every run
with an opflow binary, so the parser tests depended on the machine and test
order (test_solve_time already failed against the committed fixture). It now
writes the fixture only with AGENTIGRID_REGENERATE_FIXTURES=1, and the class
skips without the binary like TestMPIGuard. tests/README.md documents the
regeneration; claude_code_changes.md has the entry for both commits.

Fast suite without ExaGO binaries: 1083 passed, 13 skipped (before: 2 failed).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…nd trips

Five defects found while specifying the parser and writer:

1. Rows with a non-numeric value were skipped with only a log warning, so a
   bus, generator or branch could silently disappear; too-short rows crashed
   with IndexError. Known sections (bus, gen, branch, gencost) now raise
   ValueError naming the section and row for non-numeric values, column
   counts outside the section's limits, and non-integral values in integer
   columns.
2. Valid MATLAB syntax was misread: two rows on one line became one garbage
   row and comma-separated rows were all dropped. These, '...' continuations,
   unterminated blocks and a missing mpc.bus now raise ValueError.
3. The writer crashed on Inf/NaN, which the reader accepts. They are written
   as Inf/-Inf/NaN; bus solution columns no longer lose precision to .4f.
4. Every read/write round trip added another "MATPOWER Case Format" header
   line. Writing is now idempotent.
5. Case version '1' was accepted although ExaGO reads version 2; it now
   raises. A missing version still means '2'.

CRLF input is normalised on read and output is always LF. The function line
always carries network.casename. All 29 case files in datafiles/ still parse
and round-trip to equivalent networks. New
tests/grid_files/test_matpower_hardening.py (29 tests, 22 fail without the fix).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…rounding, baseMVA literals

Three defects found by property-based round-trip testing:

1. A finite value near the float maximum was written with 10 significant
   digits that round past it and read back as Inf. The writer now falls back
   to the exact representation when rounding overflows.
2. A value that rounds to an integer (e.g. 9999999999.5) was written as
   "1e+10" the first time and "10000000000" after re-reading, so writing was
   not idempotent. The integer form is now decided on the rounded value.
3. mpc.baseMVA was read with a digits-and-dots pattern: "1e2" (and "1e+15",
   which the writer itself emits for large values) was silently read as 1,
   and a negative value was reported as missing. Any numeric literal (sign,
   exponent, Inf/NaN) on an mpc.baseMVA line is now accepted, text in
   comments is ignored, and a non-numeric value raises ValueError.

18 more tests in tests/grid_files/test_matpower_hardening.py (12 fail
without the fix). All 29 case files in datafiles/ still round-trip.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…accounting

Agent loop
- Each iteration runs in a guard: an exception in an action handler is
  discarded with feedback to the model and the run continues; sweep, boundary
  and reserve handlers tolerate a non-dict feasibility / candidate_set.
- The UI callback fires only when the iteration recorded a journal entry;
  otherwise a UI-only "discarded" card says why. The final re-emit of the last
  entry is removed (it duplicated the last card and, after rejected
  iterations, showed a stale lower-numbered card after the last one).
- journal.discarded_actions records iterations that produced no entry (kind
  rejected / api_error / truncated / internal, feedback, stop_reason, raw
  response capped at 4000 characters). Entries and statistics are unchanged.
- Responses cut at max_tokens get their own message and feedback; a modify
  that applies no command says so; modify entries keep skipped_commands.
- Optional stall breaker: AGENTIGRID_STALL_LIMIT=N stops after N consecutive
  iterations without a new entry (off by default).

Backends
- Anthropic prompt caching of the per-session system prompt (llm.prompt_cache,
  default on; billing only). prompt_tokens stays the total input; cache writes
  and reads are reported separately.
- UsageMeter counts every LLM call; totals are exported as journal llm_usage.
- Models that reject `temperature` are retried once without it.
- LLMResponse gains api_error, stop_reason, content_types and cache fields;
  AGENTIGRID_MAX_TOKENS overrides the output-token cap.

UI and reports
- Iterations shown as "N + base case" (summary_stats adds llm_iterations and
  has_base_case; total_iterations unchanged).
- FAILED only for solves that did not converge, not for analysis / complete /
  sweep / contingency / explore entries or discarded proposals.
- Sidebar checkbox for the stall breaker (unchecked by default).
- Default Anthropic model claude-sonnet-4-20250514 (retired) -> claude-sonnet-4-6.

Tests: new tests/llm/test_prompt_cache.py and tests/journal/test_iteration_count.py;
8 new tests in tests/agent_loop/test_agent_loop.py. Fast suite: 1103 passed,
12 skipped; the 2 failures are the existing OPFLOW fixture issues fixed by the
genfuel PR.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant