feat: 策略生成成功率评测(RAG vs 无 RAG 对照) - #1
Merged
Merged
Conversation
- 生成器新增 skipRetrieval 开关:同链路跳过知识库检索,用于无 RAG 对照 - generate 重构:LLM 调用移出事务(仅持久化走 TransactionalExecutor), 新增 generateForUser 供评测等无 SecurityContext 场景调用 - 新增评测模块 generator/eval:env 门控 CommandLineRunner(APP_EVAL_ENABLED), 12 条内置用例 × RAG/no-RAG 双分支,python3 ast.parse 语法检查 + LLM 评委 rubric 评分,输出 eval-output/ JSON+MD 报告后自动退出 - LocalKbSeedRunner 加 @order(10) 保证评测在其后执行 Co-Authored-By: Claude Code <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
验证「生成策略代码的成功率」,并与不使用知识库的裸模型做对照。
改动内容
生成器
GenerateStrategyRequest新增skipRetrieval字段(null/false 走原链路,前端零改动)StrategyGeneratorService:generate重构:LLM 调用移出事务(修复「事务内调 LLM」规则违规),仅持久化走TransactionalExecutor,新增generateForUser(request, user)供评测等无 SecurityContext 场景调用评测模块(新包
generator/eval)APP_EVAL_ENABLED=true ./gradlew :app:bootRun启动评测,写报告后自动退出(退出码 0/1/2)resources/eval/strategy-eval-cases.json,4 市场 × 2 频率 × 3 难度,场景取自 PTrade 知识库文档),每用例跑 RAG / no-RAG 两分支ast.parse语法检查(不可用自动降级)+ LLM 评委七维 rubric 评分(≥70 通过,strategy-eval-judge.st含反注入指令)eval-output/(gitignored):JSON + Markdown + 控制台汇总vector_status),结束按 ID 精确清理评测产生的生成记录LocalKbSeedRunner加@Order(10)保证评测在其后执行测试
./gradlew :app:test通过运行方式
docker compose -f docker-compose.dev.yml up -d APP_EVAL_ENABLED=true APP_EVAL_JUDGE_PROVIDER=qwen APP_EVAL_GENERATE_PROVIDER=qwen \ APP_EVAL_VECTOR_WAIT_TIMEOUT=300s ./gradlew :app:bootRun # 产物: eval-output/eval-report.json + eval-report.md注:知识库删除功能探索发现已完整存在(后端端点/前端按钮/级联清理),按用户决策未改动。
🤖 Generated with Claude Code