ESS Diagnostics: interactive transcript-diagnosis skill (replaces Python tool)

[RamananVr]

Summary

Replaces the standalone ESS Diagnostics Python tool with a single self-contained Claude skill (tools/ESS-Diagnostics-Skills/SKILL.md) that walks a Field Engineer (FDE) through a Copilot Studio / PVA transcript interactively and produces evidence-cited outputs. The entire Python pipeline (src/, tests/, pyproject.toml) is removed.

What the skill does

• Step 0 — hard gate: stops and asks the FDE to confirm the transcript file path and state the specific problem before parsing anything (won't proceed without a problem statement).
• Per-turn interactive walkthrough: for each turn it shows a structured summary (intent, search query + rewrite, and all search results with their source text) plus a one-line verdict strip, then hard-stops with a section menu. The FDE can drill into any section (full evidence + the check's reasoning), override a verdict (with reason, carried into the report), continue, or run all to finish without pausing.
• 5 checks: intent recognition, search query issued (+ rewrite), search-result topical relevance, knowledge grounding consistency, final answer vs. retrieved content/guardrails. Divergence = first failing check; a root cause is composed from it.
• Real-schema parse map: derived from real transcript exports — events are Trace discriminated by p.data.kind (LlmIntentRecognized, PluginStart, PluginResponse, AnalyticsAiMetricsSignalTraceData, KnowledgeTraceData); user turns are SynchronousIncomingActivity; grounding/rewrite come from completionState / rewrittenMessage / verifiedSearchResults[].rankScore. Missing fields degrade to N/A rather than guessing.

Outputs (three files, written to the OS temp dir — never the repo, since transcripts contain employee PII)

1. <name>-debug-report.md — human-readable diagnosis: per-turn verdicts with evidence, FDE overrides, divergence point, root cause.
2. <name>-normalized.json — compact diagnostic view (only the fields the checks use; no verdicts).
3. <name>-transcript.json — faithful, lossless full transcript converted to JSON (every event and field preserved, _turn/_index annotated), produced by the bundled helper scripts/transcript_to_json.py.

Transcript helper is Python (ported from Node.js)

The bundled faithful-dump helper is scripts/transcript_to_json.py — a stdlib-only Python script, so the toolkit stays Python-only with no Node.js dependency. It began as a Node.js helper (transcript-to-json.js) and was ported to Python within this PR as an exact behavioral drop-in: identical CLI contract (<transcript.txt> [outFile.json]), same _turn/_index annotation, same temp-dir output path, same Error:/exit-1 failures, and content byte-identical output (ensure_ascii=False preserves raw Unicode). The Node.js version has been removed and SKILL.md now invokes the Python helper.

How to invoke the skill

From a Claude Code session in the repo, either invoke it by name:

/ESS-Diagnostics-Skills:diagnose-transcript

or point Claude at it directly:

Use the ESS diagnostics skill on C:\path\to\Transcript_xxxx.txt

The skill then (1) stops and asks for the file path + the specific problem, (2) parses and walks each turn — pausing so you can drill in, override a verdict, continue, or run all, and (3) writes the three output files to your OS temp dir and reports their absolute paths.

To convert a transcript to faithful JSON directly (also used internally by the skill) — use python on Windows, python3 on POSIX:

python tools/ESS-Diagnostics-Skills/scripts/transcript_to_json.py "<path-to-transcript.txt>"

Validation

Dry-run against a real (redacted) transcript confirmed the Step 0 gate, per-turn hard stops, real-schema extraction, drill-down/override, and temp-dir outputs. The run surfaced a genuine search relevancy/ranking defect — the correct KB was retrieved but under-ranked (all rankScore=0) and not cited — validating that the checks + FDE override mechanism work. No automated test suite (the skill is prose); the Python helper was validated manually (run against a real transcript, output diffed byte-for-byte against the original Node.js helper, and error-path spot-checked).

Notes

• No PII-bearing artifacts are written to the repo; all diagnosis outputs go to the OS temp directory.
• Planning docs live under tools/ESS-Diagnostics-Skills/docs/plans/, consolidated into two files that each cover both efforts (the v4 skill and the Node→Python helper port) in a Part 1 / Part 2 split: 2026-09-17-ess-diagnostics-design.md (combined design) and 2026-09-17-ess-diagnostics-implementation.md (combined implementation plan).

🤖 Generated with Claude Code

[AdwaitRaut]

The per-turn walkthrough is solid. N/A-on-failed-precondition and the FDE override path are the right calls.

One fix before merge: the output contract disagrees with itself across the Rules block, Step 4, and the README (two files vs three). Open question on invocation plus a few nits below — none of those need to hold it up.