Skip to content

Use DeepSeek agent for BuffBench - #622

Closed
jahooma wants to merge 6641 commits into
mainfrom
jahooma/buffbench-base2-deepseek
Closed

Use DeepSeek agent for BuffBench#622
jahooma wants to merge 6641 commits into
mainfrom
jahooma/buffbench-base2-deepseek

Conversation

@jahooma

@jahooma jahooma commented May 8, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Switch BuffBench main runner from base2-free-evals to base2-free-deepseek.

Validation

  • Reviewed git diff origin/main...HEAD.

James Grugett and others added 30 commits April 19, 2026 17:33
Opus 4.7's tokenizer differs from most models; using 4.6 gives more
representative counts for open-source models routed through the
Anthropic count_tokens endpoint.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…on, remove redundant IS_FREEBUFF guard, extract END_SESSION_MESSAGE constant
@codebuff-team

ghost commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

The stated change — swapping base2-free-evals for base2-free-deepseek in evals/buffbench/main.ts — is a one-line, sensible change and clearly in scope.

However the diff is much larger than the description suggests. agents/base2/base-deep.ts and agents/base2/base2.ts also drop file-picker, file-picker-max, directory-lister, and glob-matcher from spawnableAgents, rewrite the system/step prompts to point at glob/list_directory/read_subtree/read_files instead, and get a bunch of unrelated prettier-style reformatting (multi-line function signatures, template literal indentation). None of that is mentioned in the PR body, and there's no rationale given for retiring those sub-agents or reason to believe it's tied to the DeepSeek swap.

As it stands this reads like two unrelated changes squashed together: a trivial eval config change, and a meaningful behavioral change to how base2/base-deep orchestrate sub-agents. The latter deserves its own PR with a description of why those agents are being removed and what testing (if any) shows the direct-tool approach performs as well. Please split these so each can be reviewed and ported on its own merits.

@codebuff-team codebuff-team added bot:triaged Classified by the community triage bot pr:needs-work Right idea, not mergeable as written labels Aug 20, 2026
@victorxheng
victorxheng force-pushed the jahooma/buffbench-base2-deepseek branch from 421a0a5 to d20fdc8 Compare August 31, 2026 21:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bot:triaged Classified by the community triage bot pr:needs-work Right idea, not mergeable as written

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants