Evaluation: Optional duplication factor #1105

Problem

duplication_factor is fixed at dataset upload time — stored in dataset_metadata (DATASET_META_DUPLICATION_FACTOR) and baked in when the dataset is created (backend/app/api/routes/evaluations/dataset_v2.py:41, backend/app/services/evaluations/dataset.py:35).

At run time, POST /evaluations (v2 trigger, backend/app/api/routes/evaluations/evaluation_v2.py:32) has no way to change it — _load_items_from_object_store in backend/app/services/evaluations/fast.py:91 always falls back to the dataset's stored factor:

metadata = dataset.dataset_metadata or {}
duplicate_at_runtime = bool(metadata.get(DATASET_META_DUPLICATE_AT_RUNTIME, False))
duplication_factor = (
    max(1, int(metadata.get(DATASET_META_DUPLICATION_FACTOR, 1)))
    if duplicate_at_runtime
    else 1
)

To re-run the same uploaded dataset at a different duplication factor, you currently have to re-upload it. Datasets are otherwise reusable across runs (different configs, models) — duplication factor shouldn't force a re-upload.

Proposal

Add an optional duplication_factor param to the v2 evals run endpoint:

• evaluate_v2 (backend/app/api/routes/evaluations/evaluation_v2.py): optional duplication_factor: int | None = Body(None, ...), threaded through validate_and_start_fast_evaluation.
• _load_items_from_object_store (backend/app/services/evaluations/fast.py:91): accept an optional override param; when passed, use it instead of metadata.get(DATASET_META_DUPLICATION_FACTOR, 1). When absent, fall back to current behavior (dataset's stored factor, or 1 if not runtime-duplicated).
• attach_cost/summary path (backend/app/crud/evaluations/fast.py:1141) reads duplication_factor from dataset metadata for the ai_summary math (repetition-aware averaging) — needs the same override threaded through so the summary reflects the factor actually used for that run, not the dataset's stored default.

Only applies to v2 (runtime-duplicated) datasets — v1 datasets are physically duplicated in the CSV already, factor forced to 1, override should be a no-op or rejected for those.

Why

Makes an uploaded v2 dataset reusable at different duplication factors without re-uploading — e.g. quick sanity check at 1x, full eval at 5x, same dataset either way.