Skip to content

Add MiniMax Code Harbor evaluation profile - #2

Open
hetaoBackend wants to merge 1 commit into
frontier-harness-eval:feat/third-part-harnessfrom
hetaoBackend:feat/minimax-code-harbor
Open

Add MiniMax Code Harbor evaluation profile#2
hetaoBackend wants to merge 1 commit into
frontier-harness-eval:feat/third-part-harnessfrom
hetaoBackend:feat/minimax-code-harbor

Conversation

@hetaoBackend

Copy link
Copy Markdown

Summary

  • add a reproducible MiniMax Code profile that runs all 30 published task IDs through Harbor's mcode adapter
  • replace MCode's networked per-task install with a checkpointed, SHA-256-verified Node/MCode bundle while preserving Harbor's normal configuration, execution, and trajectory paths
  • pin Harbor, MiniMax Code, Node, Terminal-Bench, and DeepSWE inputs; overlay the repository's public task definitions onto the complete upstream task assets
  • add Fireworks, Moonshot, OpenRouter, and Together Kimi K3 mappings, checkpoint/run identity checks, evidence collection, reporting metadata, and focused regression tests

Reproducibility pins

  • Harbor 0.22.0
  • @minimax-ai/code 0.2.7
  • Node.js 22.23.2 (official Linux x64 archive SHA-256 verified)
  • Terminal-Bench 2.1 dataset sha256:7d7bdc1cbedad549fc1140404bd4dc45e5fd0ea7c4186773687d177ad3a0699a
  • DeepSWE commit 435ee89ec2f2e2289f33b0da4f992f0b7b7266b9

Every restored MCode checkpoint recomputes and compares the public-task tree, runnable Terminal-Bench and DeepSWE trees, MCode bundle, adapter source, and DeepSWE HEAD before a trial can run. An existing run ID also rejects changes to the checkpoint, harness/version, provider/model routes, timeout, or command identities.

Validation

  • skills/frontierharness-eval/tests/test-minimax-code.sh
  • bash -n skills/frontierharness-eval/scripts/*.sh skills/frontierharness-eval/tests/*.sh
  • node --check skills/frontierharness-eval/scripts/*.mjs
  • uvx ruff check skills/frontierharness-eval/agents/frontierharness_mcode.py
  • skill metadata validation with Codex's quick_validate.py
  • Harbor 0.22.0 resolution and full export of the pinned Terminal-Bench digest; verified all 21 selected Terminal-Bench and 9 DeepSWE overlays load from the resulting local task trees
  • offline adapter install contract test and Harbor custom-agent --print-config validation

I did not run a real Runta/Kimi K3 task or the full 30-task sweep from this environment: it has no Runta/provider credentials and its Docker daemon is unavailable. This PR adds the reproducible integration workflow, not benchmark result claims.

Stack

This PR is intentionally based on and targets #1 (feat/third-part-harness), which introduces the agent-neutral evaluation workflow that this MiniMax Code profile extends.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant