Repository navigation
Assessment: Add support for Anthropic #932
Copy link
Copy link
Labels
enhancementNew feature or requestNew feature or request
Description
Is your feature request related to a problem?
The assessment module's batch pipeline only supports OpenAI and Gemini providers, preventing assessments configured with Anthropic models from running as batch jobs. This limitation hinders cost-efficient bulk evaluations of Claude models.
Describe the solution you'd like
- Implement an
AnthropicBatchProviderthat meets theBatchProviderinterface, using the Anthropic Message Batches API for batch creation, status polling, and result retrieval. - Add
build_anthropic_jsonlto format assessment items for the Anthropic batch request and route assessments to the new provider. - Include
map_kaapi_to_anthropic_paramsfor translating Kaapi model parameters to Anthropic API format. - Create
build_anthropic_attachment_partsto convert assessment attachments into Anthropic content blocks. - Extend output parsing to manage Anthropic batch responses, including error handling and text extraction.
- Provide unit tests for the new provider, mappers, and parsing in the relevant test directories.
Original issue
Describe the current behavior
The assessment module's batch pipeline (backend/app/core/batch/) only supports OpenAI and Gemini providers, so assessments configured with Anthropic models cannot run as batch jobs. This blocks cost-efficient bulk evaluation runs on Claude models and forces Anthropic-backed assessments through the synchronous path.
Describe the enhancement you'd like
- Add an
AnthropicBatchProviderimplementing theBatchProviderinterface (backend/app/core/batch/anthropic.py), covering batch creation via the Anthropic Message Batches API, status polling, and result retrieval. - Add
build_anthropic_jsonlinbackend/app/crud/assessment/batch.pyto format assessment items into Anthropic batch request format, and extendsubmit_assessment_batchto route Anthropic-provider assessments to the new provider. - Add
map_kaapi_to_anthropic_paramsinbackend/app/services/assessment/mappers.pyto translate Kaapi model parameters (includingmax_tokensdefaulting viaDEFAULT_ANTHROPIC_MAX_TOKENS) to Anthropic API format. - Add
build_anthropic_attachment_partsinbackend/app/services/assessment/utils/attachments.pyto convert assessment attachments into Anthropic content blocks. - Extend output parsing in
backend/app/crud/assessment/processing.pyto handle Anthropic batch responses, including error results and text extraction from content blocks. - Cover the new provider, mappers, and parsing with unit tests (
app/tests/core/batch/test_anthropic.py,app/tests/assessment/).
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request
Type
Projects
- StatusShow more project fieldsClosed
Activity
[-]Assessment: Add Claude batch processing support[/-][+]Assessment: Add support for Anthropic[/+]