[Fix] NLG judge 빈 분석 평가 배열 서버 정규화 - #222
Merged
Merged
Conversation
- review CSV에 question, answer, mainTasks, qualifications 컬럼 추가 - 입력 CSV 값을 변환 없이 검수용 review row에 복사 - 기존 replay 판정 및 summary 생성 로직은 유지 - review CSV 문맥 컬럼 복사 테스트 추가
- 1차 후보 프롬프트에 missingKeywordCandidates 전용 생성 규칙 추가 - 누락 키워드는 mainTasks 또는 qualifications의 명시 항목만 사용하도록 제한 - JD 밖 개념 확장 및 일반화 금지 규칙 추가 - relatedRequirement는 JD 원문 기반으로만 작성하도록 제한 - 답변에 이미 존재하는 역량은 누락 후보로 생성하지 않도록 명시 - precision 우선 및 애매한 경우 빈 배열 반환 규칙 추가 - 상세페이지/브랜드 운영/행정지원 negative few-shot 추가 - 후보 프롬프트 회귀 테스트 보강
- caseId mismatch 발생 시 expected/actual caseId와 response null 여부 로깅 - 빈 questionAnalyses에 대한 judge 평가 개수 불일치 로깅 - NLG judge 평가 예외 발생 시 stacktrace 포함 ERROR 로그 출력 - JsonProcessingException originalMessage 로깅 - IllegalArgumentException validationMessage 로깅 - 기존 judge 결과, 평가 기준, CSV 포맷은 유지
- Judge 응답 null, caseId mismatch, questionAnalysis 개수 불일치 원인 구분 로그 추가 - Judge 호출 및 Structured Output 역직렬화 실패 시 stacktrace 포함 ERROR 로그 출력 - Jackson path/reference chain과 실패 필드명 로깅 - NlgEvaluationErrorCode enum 역직렬화 실패 시 unknownValue와 targetEnum 로깅 - root cause 타입과 메시지 로깅 - 기존 Judge 프롬프트, DTO, enum, validation 기준, CSV failureStage 호환성 유지
- questionAnalysesJson에 명시된 분석 항목만 questionAnalysisEvaluations로 평가하도록 프롬프트 보강 - questionAnalysesJson이 빈 배열이면 questionAnalysisEvaluations도 반드시 빈 배열로 반환하도록 명시 - raw/sanitized/review candidate에서 분석을 추론하거나 복원하지 않도록 금지 - rejected 또는 removed candidate를 questionAnalysisEvaluation으로 변환하지 않도록 명시 - 빈 분석은 validation error가 아니며 noAnalysisAppropriateness로 평가하도록 정리 - 분석 N개 입력 시 입력 분석 N개에만 대응 평가하도록 출력 계약 추가 - NLG judge 프롬프트 회귀 테스트 추가 - DTO, enum, validation, CSV 포맷은 변경하지 않음
- questionAnalyses가 비어 있는데 judge가 questionAnalysisEvaluations를 생성한 경우만 1회 재시도 - 내부 JudgeValidationReason과 JudgeValidationResult로 validation 실패 사유 구분 - response null, caseId mismatch, 역직렬화 오류, judge 호출 실패는 재시도하지 않도록 유지 - 재시도 시작, 성공, 재실패 로그 추가 - 재시도 성공 시 두 번째 응답만 최종 결과로 사용 - 재시도 실패 시 기존 judge_validation_failed 결과 유지 - NLG judge 재시도 조건 및 비대상 실패 유형 테스트 추가 - DTO, enum, 프롬프트, CSV 포맷은 변경하지 않음
- questionAnalyses가 빈 입력이면 questionAnalysisEvaluations를 서버에서 빈 배열로 정규화 - 빈 입력에서 Judge가 생성한 문장 평가를 실패 처리하지 않고 구조적으로 제거 - noAnalysisAppropriateness, strengths, missingKeywords, overallUsefulness, errorCodes, rationale 보존 - response null 및 caseId mismatch는 기존 validation failure 유지 - question_analysis_count_mismatch 재시도 로직과 내부 validation reason 타입 제거 - 빈 입력 정규화 시 원본/정규화 평가 개수 WARN 로그 추가 - NLG judge 빈 입력 정규화 및 비대상 실패 회귀 테스트 보강 - DTO, enum, 프롬프트, CSV 포맷은 변경하지 않음
|
Warning Review limit reached
Next review available in: 47 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (6)
📝 WalkthroughWalkthroughNLG 평가의 프롬프트 계약, 응답 정규화, 예외 로깅이 강화되었습니다. Missing keyword 후보 생성 규칙이 구체화되었고, 리플레이 리뷰 CSV에 원본 입력 필드가 추가되었습니다. 관련 테스트도 각 변경 사항에 맞게 갱신되었습니다. ChangesNLG 평가 처리
Missing keyword 리뷰 CSV
Missing keyword 후보 프롬프트
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant NlgEvaluationBatchService
participant NlgEvaluationAiClient
participant OpenAI
NlgEvaluationBatchService->>NlgEvaluationAiClient: evaluate 호출
NlgEvaluationAiClient->>OpenAI: 평가 프롬프트 전송
OpenAI-->>NlgEvaluationAiClient: 구조화된 평가 응답 반환
NlgEvaluationAiClient-->>NlgEvaluationBatchService: 평가 결과 반환
NlgEvaluationBatchService->>NlgEvaluationBatchService: 빈 questionAnalysisEvaluations 정규화
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@src/main/java/com/jobdri/jobdri_api/domain/analysis/evaluation/MissingKeywordSanitizerReplayService.java`:
- Around line 53-59: Update REQUIRED_HEADERS in
MissingKeywordSanitizerReplayService to include "question" so missing question
headers fail fast during input validation. In
src/main/java/com/jobdri/jobdri_api/domain/analysis/evaluation/MissingKeywordSanitizerReplayService.java
lines 53-59, add the required header; in
src/test/java/com/jobdri/jobdri_api/domain/analysis/evaluation/MissingKeywordSanitizerReplayServiceTest.java
lines 154-163, add a regression test removing the question header and asserting
validation failure.
In
`@src/main/java/com/jobdri/jobdri_api/domain/analysis/evaluation/NlgEvaluationAiClient.java`:
- Around line 196-202: The duplicated rootCause(Throwable) traversal should be
centralized. In
src/main/java/com/jobdri/jobdri_api/domain/analysis/evaluation/NlgEvaluationAiClient.java
lines 196-202, remove the private rootCause method and use Spring’s
NestedExceptionUtils.getMostSpecificCause(...) or a shared package utility;
apply the same removal and shared call in
src/main/java/com/jobdri/jobdri_api/domain/analysis/evaluation/NlgEvaluationBatchService.java
lines 965-971, and consider moving findCause there into the same utility if it
serves the same purpose.
In
`@src/main/java/com/jobdri/jobdri_api/domain/analysis/evaluation/NlgEvaluationBatchService.java`:
- Around line 156-184: Update both validation-failure branches in
NlgEvaluationBatchService so each emits only one log.warn call. Add the
responseNull field to the existing reason-specific log messages, using true for
the null-response branch and false for the case-id-mismatch branch, then remove
the duplicate warn calls while preserving the existing failure returns.
- Around line 844-907: Refactor logEvaluationFailure so the shared “NLG Judge
evaluation failed” log is emitted once after determining the exception-specific
values, using one consistent exceptionType format (prefer the simple class
name). Preserve the additional JsonProcessingException detail log with
getOriginalMessage(), and retain the existing fields and root-cause information
for all exception types.
In
`@src/main/java/com/jobdri/jobdri_api/domain/analysis/service/ai/AnalysisAiClient.java`:
- Around line 675-678: Example 3의 `JD main_tasks`와 `Answer` 내용을 일치시키도록 수정하세요.
`사업 및 행정지원`을 유지하려면 답변에 사업 지원 수행 경험을 명시하고, 그렇지 않으면 JD를 `행정지원`으로 좁혀 행정 업무와 예산
관리만으로 전체 충족으로 판정되지 않게 하세요.
In
`@src/test/java/com/jobdri/jobdri_api/domain/analysis/evaluation/NlgEvaluationAiClientTest.java`:
- Around line 129-156: Extend NlgEvaluationAiClientTest with an evaluate()
regression test that stubs LlmConcurrencyLimiter.execute(any(), any()) to throw
a specific RuntimeException and asserts client.evaluate(...) rethrows the
identical instance without wrapping. Reuse the existing judge-input construction
helpers and mock setup, and cover the original exception propagation contract
rather than only prompt generation.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: c451b342-90b9-46e2-8fe4-bff02e480ab5
⛔ Files ignored due to path filters (8)
evaluation/evaluation_missing_keyword_sanitizer_review.csvis excluded by!**/*.csvevaluation/evaluation_nlg_judge_current.csvis excluded by!**/*.csvevaluation/evaluation_nlg_judge_current_comparison.csvis excluded by!**/*.csvevaluation/evaluation_nlg_judge_two_pass_candidate_prompt_fix.csvis excluded by!**/*.csvevaluation/evaluation_nlg_judge_two_pass_candidate_prompt_fix_comparison.csvis excluded by!**/*.csvevaluation/evaluation_nlg_judge_two_pass_candidate_prompt_fix_diagnostics.csvis excluded by!**/*.csvevaluation/evaluation_nlg_judge_two_pass_candidate_prompt_fix_empty_contract.csvis excluded by!**/*.csvevaluation/evaluation_nlg_judge_two_pass_candidate_prompt_fix_retry.csvis excluded by!**/*.csv
📒 Files selected for processing (8)
src/main/java/com/jobdri/jobdri_api/domain/analysis/evaluation/MissingKeywordSanitizerReplayService.javasrc/main/java/com/jobdri/jobdri_api/domain/analysis/evaluation/NlgEvaluationAiClient.javasrc/main/java/com/jobdri/jobdri_api/domain/analysis/evaluation/NlgEvaluationBatchService.javasrc/main/java/com/jobdri/jobdri_api/domain/analysis/service/ai/AnalysisAiClient.javasrc/test/java/com/jobdri/jobdri_api/domain/analysis/evaluation/MissingKeywordSanitizerReplayServiceTest.javasrc/test/java/com/jobdri/jobdri_api/domain/analysis/evaluation/NlgEvaluationAiClientTest.javasrc/test/java/com/jobdri/jobdri_api/domain/analysis/evaluation/NlgEvaluationBatchServiceTest.javasrc/test/java/com/jobdri/jobdri_api/domain/analysis/service/ai/AnalysisAiClientTest.java
- missing keyword replay 입력 필수 헤더에 question 추가 - question 헤더 누락 fail-fast 회귀 테스트 추가 - NLG judge root cause 추적을 Spring NestedExceptionUtils로 통일 - NLG judge validation 실패 WARN 중복 제거 - NLG judge 실패 로그를 공통 경로로 정리 - JsonProcessingException 상세 originalMessage 로그 유지 - missing keyword negative example의 JD와 답변 문맥 정합성 수정 - NLG judge evaluate 예외 전파 회귀 테스트 추가
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
✨ 어떤 이유로 PR를 하셨나요?
📋 세부 내용 - 왜 해당 PR이 필요한지 작업 내용을 자세하게 설명해주세요
📸 작업 화면 스크린샷
🚨 관련 이슈 번호 [#150 ]
Summary by CodeRabbit
개선 사항
테스트