The AI Skill Analyzer

Test and grade Microsoft Fabric Data Agents automatically. Ask questions, compare answers to expected values, get a score and actionable fixes.

What It Does

1. Sends questions to your Fabric Data Agent (via the Assistants API)
2. Grades answers against expected values (5 match types: exact, numeric, contains, regex, semantic)
3. Diagnoses failures with root cause analysis (wrong measure, query error, empty result, synthesis error...)
4. Generates an action plan (fix instructions, add few-shots, create measures)

Quick Start

Everything below runs against your own Fabric tenant. No identifiers from any other environment are stored in this repository.

1. Install

git clone https://github.com/Statyx/Fab-Analyze-Data-Agent.git
cd Fab-Analyze-Data-Agent
pip install -r requirements.txt

2. Sign in to Azure

az login --tenant <your-entra-tenant-id>

The Data Agent API requires the Item.ReadWrite.All scope and at least Contributor on the workspace hosting the agent.

3. Create your global config

cp config.yaml.example config.yaml

Edit config.yaml and set tenant_id. config.yaml is gitignored — your IDs never leave your machine.

4. Point a profile at your Data Agent

Every profile ships a profile.yaml.example; the real profile.yaml is gitignored. Either reuse a bundled profile:

cp profiles/marketing360/profile.yaml.example profiles/marketing360/profile.yaml
# then fill in workspace_id, agent_id, semantic_model_id

…or scaffold a brand-new one:

python -m analyzer init my_agent
# creates profiles/my_agent/{profile.yaml,questions.yaml}

If profile.yaml is missing — or still holds a <YOUR_...> placeholder — the analyzer stops with an explicit error. It never falls back to another environment.

5. Validate, then run

python -m analyzer -p my_agent validate   # auth + workspace + agent + model
python -m analyzer -p my_agent run
python -m analyzer -p my_agent analyze --latest

Where to Find Your IDs

ID	Lives in	Where to find it
tenant_id	config.yaml	Azure portal → Microsoft Entra ID → Overview → Tenant ID
workspace_id	profiles/<name>/profile.yaml	Fabric portal → open the workspace → URL: app.fabric.microsoft.com/groups/<workspace_id>/...
agent_id	profiles/<name>/profile.yaml	Fabric portal → open the Data Agent → URL: .../groups/<ws>/aiskills/<agent_id> or GET /v1/workspaces/<ws>/items?type=AISkill
semantic_model_id	profiles/<name>/profile.yaml	Fabric portal → open the semantic model → Settings → URL or GET /v1/workspaces/<ws>/items?type=SemanticModel
semantic_model_name	profiles/<name>/profile.yaml	Display name of the semantic model (used in reports)

Files You Must Never Commit

File	Why
config.yaml	Holds your Entra tenant ID
profiles/*/profile.yaml	Holds your workspace / agent / model IDs

Both are gitignored, and the no-client-leak GitHub Actions workflow fails the build if a real GUID, a Fabric SQL endpoint, or a personal filesystem path reappears in a tracked file. Run the same check locally:

python scripts/check_no_client_leak.py

Profile Structure

Each agent gets a folder under profiles/:

profiles/my_agent/
  profile.yaml.example  # Versioned template with placeholders
  profile.yaml          # YOUR connection IDs — gitignored, never committed
  questions.yaml        # Test cases with expected answers
  instructions.md       # (optional) Agent instructions for reference
  fewshots.json         # (optional) Few-shot examples for reference

questions.yaml format

test_cases:
  - question: "What is the total revenue?"
    expected: "31200000"
    match_type: "numeric"     # exact | numeric | contains | regex | semantic
    tolerance: 5              # % tolerance (numeric only)
    tags: ["kpi", "revenue"]

Commands

Command	What it does
init <name>	Create a new profile with template files
validate	Check auth + workspace + agent + model connectivity
snapshot	Cache agent config and model schema locally
run	Run all questions, grade, produce diagnostics
rerun	Re-run only failed questions from a previous run
analyze	Offline analysis of an existing run (RCA + action plan)
diff	Compare two runs side by side
profiles	List available profiles

All commands accept -p <profile> to select which agent to test.

Output

Each run creates a timestamped folder:

runs/my_agent/20260506_143022/
  batch_summary.json          # Score, pass/fail counts, timing
  diagnostics/                # Per-question JSON with full trace
    Q1_full_diag_*.json
    Q2_full_diag_*.json

Authentication

The analyzer uses Azure CLI credentials (az login). Make sure you're logged into the right tenant:

az login --tenant <your-tenant-id>

The Data Agent API requires Item.ReadWrite.All scope and at least Contributor role on the workspace.

Typical Workflow

1. Create profile → init
2. Write 5-10 questions → questions.yaml
3. First run (no expected answers) → run → see what the agent returns
4. Fill in expected values for correct answers → questions.yaml
5. Re-run → get a score (e.g. 8/10 = 80%)
6. Read the action plan → fix instructions / add measures / add few-shots
7. Redeploy agent → re-run → compare with diff
8. Iterate until score is stable

Configuration

config.yaml

tenant_id: "<your-entra-tenant-id>"
default_profile: "my_agent"
snapshot_ttl_hours: 24
max_workers: 4
output_dir: "runs"

profile.yaml

workspace_id: "<YOUR_WORKSPACE_ID>"
agent_id: "<YOUR_AGENT_ID>"
semantic_model_id: "<YOUR_SEMANTIC_MODEL_ID>"
semantic_model_name: "My_Model"
stage: "production"           # or "sandbox" for draft agents

Placeholders wrapped in <...> are rejected at load time — replace all three before running.

Troubleshooting

Problem	Fix
Browser popup doesn't appear	Run az login manually first
"Profile ... has no profile.yaml"	cp profiles/<name>/profile.yaml.example profiles/<name>/profile.yaml and fill in your IDs
"still contains placeholder values"	Replace every <YOUR_...> value with a real ID from the Fabric portal
"Workspace not found" (404)	Check workspace_id, ensure Contributor access
"Agent not found" (404)	Check agent_id, ensure agent is published (not draft-only)
Slow responses / timeouts	Fabric capacity may be paused — resume it in the portal
Wrong answers	Check agent instructions, add few-shot examples, verify measure names

Contributing

1. Fork the repo
2. Create a profile for your agent
3. Run tests, iterate on quality
4. PRs welcome for core analyzer improvements

License

MIT