Repository navigation
ci: add /evals slash command to run Cloud MCP evals on a PR - #1395
Conversation
/evals dispatches airbytehq/cloud-mcp-evals' evals workflow for the PR's latest commit, limited to scenarios using MCP tools the PR changed; /evals full=true runs the full suite. Results are appended to the triggering comment. Eval failures are reported, not fatal. Co-Authored-By: Claude Opus 5.5 <[email protected]>
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. 💡 Show Tips and TricksTesting This PyAirbyte VersionYou can test this version of PyAirbyte using the following: # Run PyAirbyte CLI from this branch:
uvx --from 'git+https://github.com/airbytehq/PyAirbyte.git@calebevans/agentic-2355-evals-command' pyairbyte --help
# Install PyAirbyte from this branch for development:
pip install 'git+https://github.com/airbytehq/PyAirbyte.git@calebevans/agentic-2355-evals-command'PR Slash CommandsAirbyte Maintainers can execute the following slash commands on your PR:
📚 Show Repo GuidanceHelpful ResourcesCommunity SupportQuestions? Join the #pyairbyte channel in our Slack workspace. |
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Code coverage overviewLanguages: Python Python / code-coverage/pytest-no-creds
View coverage of the 10 most impacted files.
Python / code-coverage/pytest
View coverage of the 10 most impacted files.
Python / code-coverage/pytest-unit
Line coverage data for the main branch is not yet available. View coverage of the 10 most covered files.
ℹ️ Calculations represent line coverage. Results may change as calculations are updated. Updated |
|
Codex (@codex) review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2ceaa13fae
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
|
Codex (@codex) review |
|
Codex Review: Didn't find any major issues. Another round soon, please! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback". |
2 participants
Adds a
/evalsPR slash command that runs the Cloud MCP evals (airbytehq/cloud-mcp-evals, "PyAirbyte evals" workflow) against the PR's latest commit. Linear: AGENTIC-2355.Usage
/evals: spot check. Runs only the scenarios that use MCP tools this PR changed (changed_tools_only=true)./evals full=true: runs the full eval suite.Only users with write access can trigger it (enforced by the existing dispatcher's
permission: write).How it works
evals-command.ymldispatchesevals.ymlon cloud-mcp-evalsmainwithpyairbyte=#<pr>, finds the run it created (matching run title and creation time, up to 3 minutes), waits for it, then downloads theall-evidence-*artifact and counts trial statuses fromresults.json.Progress and results are appended to the triggering comment, like the other
*-command.ymlworkflows:Cloud MCP evals queued forabc1234(changed tools only). [Check job output.]Evals running: <run url>Cloud MCP evals onabc1234(full suite): **46 / 57 passed** (11 failed) · [summary]No evals ran: no scenario uses the MCP tools this PR changed. [summary]Evals run ended (cancelled) before producing results. [summary]Eval failures do not fail this job; some trials fail from model variance today, so the comment reports counts instead. Only infrastructure problems (dispatch failed, run not found, evidence listing or download failed, or a full run with no evidence) fail the job, and those append a failure line to the comment. "No evals ran" is only reported for a
/evalsspot check whose run uploaded no evidence.Targets
mainrather thanrelease-candidate/v1: the slash-command dispatcher runs the workflow from the default branch, so it only works once it is onmain, and it does not change the library.Before it works
permission-actions: write.workflow_dispatchruns the default-branch workflow.Testing
46 / 57 passed (11 failed)) and a spot check with nothing to run (the "No evals ran" message). Also checked that#138does not match a#1386run title, and a synthetic results file with errored, uncertain and missing trials.🤖 Generated with Claude Code