Repository navigation
Feat/mcp compass publish - #117
Open
operetz-rh wants to merge 18 commits into
Open
operetz-rh wants to merge 18 commits into
operetz-rh wants to merge 18 commits into
Conversation
operetz-rh
marked this pull request as draft
October 18, 2026 16:37
operetz-rh
force-pushed
the
feat/mcp-compass-publish
branch
2 times, most recently
from
October 18, 2026 16:51
5f252e4 to
5fff2ec
Compare
operetz-rh
marked this pull request as ready for review
October 19, 2026 09:03
operetz-rh
marked this pull request as draft
October 26, 2026 09:56
Publish Phase 1+2 evaluation results to Compass SoundCheck on dedicated track. Includes CLI runner, bootstrap script, Tekton task, and tests. Staging validated against real MCP server and Compass staging.
Integrate Phase 3 behavioral evaluation results (from PR #116) into Compass SoundCheck publishing. Adds BehavioralEvalSummary model, Phase 3 check mappings (tool-correctness, edge-case-handling, sensitive-data- redaction), and Compass track with 3 behavioral checks. - Support behavioral-eval-summary.json parsing and fact publishing - Add Phase 3 to bootstrap_compass_track (fixes 400 error on track update) - Fix timestamp consistency across all fact payloads (single evaluated_at) - Fix Phase 3 naming via lookup table (consistency with Phase 1/2) - Fix Phase 3 optional payload fields (False/0 vs None for schema) Validated: 31 unit tests pass, real openshift-mcp-server eval published to Compass staging (HTTP 200), track visible with all phases. Co-Authored-By: Claude Haiku 4.5 <[email protected]>
operetz-rh
force-pushed
the
feat/mcp-compass-publish
branch
from
October 26, 2026 10:06
563b12a to
1f70384
Compare
Rename second TestSummaryPath to TestSummaryPathHelper to avoid F811 redefinition.
operetz-rh
force-pushed
the
feat/mcp-compass-publish
branch
from
October 26, 2026 10:11
9fa28dc to
8b0f361
Compare
…lishing (dry-run + live modes). Live consume tool-name/oauth facts from API.
Refactors compass publish and bootstrap to support arbitrary phase counts, not just hardcoded phase1/2/3. Enables Phase 4+ by configuration only. Changes: - eval_result.py: Add phase3 to PHASE_CHECK_NAMES/SOURCE_DEFAULTS. Fix overall_score to default None pass scores to 1.0 not 0.0. - bootstrap_compass_track.py: Refactor to build checks/levels dynamically from PHASE_FACT_MAPS. Imports PHASE_FACT_MAPS instead of individual maps. - compass_publish.py: Already generic (uses .phases.items()). Fix doc reference phase2_compass_checks → conformance_eval_compass_checks. Fix evaluated_at to use passed param instead of hardcoding datetime.now(). - tests/test_mcp_compass_publish.py: Remove phase1_gate import/test, update bootstrap test for 3 levels. Tested live: 18 facts published to Compass staging (phase1/2/3 all present). All unit tests pass (36 compass + 76 phase tests). Co-Authored-By: Claude Haiku 4.5 <[email protected]>
- Updated all phase name references from phase1/2/3 to new naming - Fixed runners to output correct phase names - Updated PHASE_FACT_MAPS, PHASE_SUMMARY_FACTS with new keys - Updated PHASE_CHECK_NAMES, PHASE_SOURCE_DEFAULTS - Updated bootstrap track metadata and order - Updated test fixtures and assertions - Verified e2e: Phase 1 real scans, Phase 2 live probes, publish to Compass - 36/36 tests pass, all code aligned Co-Authored-By: Claude Haiku 4.5 <[email protected]>
…s-fresh Feat/mcp generic phases fresh
- Remove leftover conflict markers from rebase - Update example filenames to use new phase naming (static-eval, conformance-eval, behavioral-eval) - Include all 3 phases in example usage - Keep generic phase loader description (current state) Co-Authored-By: Claude Haiku 4.5 <[email protected]>
- Update static_eval_gate.py docstring with correct phase name - Update run_mcp_eval.sh to reference static-eval-summary.json consistently - Fix variable reference from P1_REPORTS to REPORTS_DIR Co-Authored-By: Claude Haiku 4.5 <[email protected]>
operetz-rh
marked this pull request as ready for review
October 28, 2026 13:35
operetz-rh
force-pushed
the
feat/mcp-compass-publish
branch
from
October 28, 2026 14:25
79eef23 to
7ebc9fa
Compare
…asks Replace monolithic mcp-compass-publish task with two-stage architecture: 1. generate-compass-facts task: Transform phase summaries to fact payloads - New fact_generator.py module with domain models and fact builders - Fact files contain only factRef + data (no entityRef) - Runs on Konflux cluster with no credential access 2. publish-to-compass task: Publish facts from target cluster - Splits into prepare-job and run-publish-job steps - Injects entityRef from Secret at publish time - Uses ConfigMap for fact file transfer - Supports parameterized secrets-namespace and configmap-name - Secret copying follows conformance task pattern Security: entityRef injection enforces two-cluster boundary - evaluation runs on target cluster with credentials, Konflux never sees entity mapping. Removed: - abevalflow/mcp/checks/compass_publish.py - abevalflow/mcp/eval_result.py - scripts/mcp/runners/compass_publish.py - scripts/mcp/bootstrap_compass_track.py (admin tool, not pipeline feature) - tests/test_mcp_compass_publish.py Added: - scripts/mcp/fact_generator.py (core fact generation logic) - scripts/mcp/runners/generate_compass_facts.py (CLI runner) - pipeline/tasks/konflux/generate-compass-facts.yaml - pipeline/tasks/konflux/publish-to-compass.yaml - templates/mcp-compass-publish-job.yaml (Kubernetes Job template) - schemas/ (JSON Schema validation files) Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
Update generate-compass-facts and publish-to-compass task bundles to use new image registry path with SHA256 digests for reproducibility. Add debug task to print facts and reports directory contents for troubleshooting. Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
Update all task bundle references to use :latest tag instead of pinned digests, eliminating the need to update the pipeline after each bundle push during development. Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
Replace all bundle references from :latest to :0.1.0 to fix MANIFEST_UNKNOWN errors. The bundles are published with version tags, not with :latest. Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
IlonaShishov
force-pushed
the
feat/mcp-compass-publish
branch
from
November 2, 2026 16:23
c1bef3b to
a6025fc
Compare
Update fetch-mcp-repo subPath and mcp-static-eval source-subdir from '_mcp-server-repo' to 'mcp-server-repo' to match the default expected by all MCP evaluation tasks (mcp-static-eval, deploy-mcp). This fixes: - 'Helm chart not found' error in deploy-mcp (was looking in wrong path) - Ensures consistent subdirectory naming across the pipeline Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
Update MCP_EVAL_IMAGE default from quay.io/ecosystem-appeng/abevalflow-mcp-eval:0.1 to quay.io/ecosystem-appeng/konflux-skill-evaluation/abevalflow-mcp-eval:0.1 This ensures the pipeline uses the image built from konflux-ai5-evaluation repo's Containerfile which includes the Python package __init__.py files required for task module imports. Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
Change COMPASS_DRY_RUN default from "true" to "false" to enable actual publishing of evaluation facts to Compass SoundCheck. Task default remains "true" for safety; PipelineRun now explicitly opts into live publishing. Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
The generate-compass-facts task now requires component-name for building the Compass entity ref. Wire it from parse-snapshot results. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the write side for MCP evaluation results — publish Phase 1+2 evaluation results to Compass SoundCheck on a dedicated track. Completes the reporting pipeline from local evaluation to centralized Compass dashboard for team visibility.
Changes
abevalflow/mcp/eval_result.py— backend-agnostic domain model (Phase 1/2/3) with summary loadersabevalflow/mcp/checks/compass_publish.py— Compass-specific publish logic with FactRef mappings and payload buildersscripts/mcp/runners/compass_publish.py— CLI runner with dry-run default (safety-first)scripts/mcp/bootstrap_compass_track.py— one-time track/check provisioning scriptpipeline/tasks/konflux/mcp-compass-publish.yaml— Tekton task for pipeline integrationtests/test_mcp_compass_publish.py— 20 unit tests covering models, loaders, payloads, publish flowscripts/mcp/README.md— documented staging flow, API shapes, commands, Phase 3 slottests/test_mcp_phase2.py— guards against raw-body SSE parsing bugTest plan
pytest tests/test_mcp_compass_publish.py— 20 tests passpytest tests/test_mcp_phase2.py— 46 tests pass (includes regression)openshift-mcp-server) against Compass staging:/api/soundcheck/facts/— all HTTP 200/api/soundcheck/results— HTTP 200https://compass.stage.redhat.com/soundcheck/tracks/abevalflow-mcp-evaluation/...