Feat/benchmark metrics

.dockerignore

@@ -0,0 +1,14 @@
+.git
+.claude
+__pycache__
+**/__pycache__
+*.pyc
+.venv
+.pytest_cache
+.mypy_cache
+.ruff_cache
+benchmarks/results/*
+!benchmarks/results/sample/
+benchmarks/your_datasets/*_results/
+*.log
+*.egg-info

.gitignore

@@ -8,11 +8,17 @@ __old/*
 *.pdf
 *.json
 benchmarks/data/large_scale/*.json
+benchmarks/your_datasets/*.json
 benchmarks/results/*
 temp/*
 *.log
 benchmarks/*.parquet
 bin/*
 .venv/
 .env
-*.DS_Store
+*.DS_Store
+
+# Sample benchmark output committed as the benchmarking "how to" deliverable
+!benchmarks/results/sample/
+!benchmarks/results/sample/*.json
+!benchmarks/results/sample/*.csv

Dockerfile

@@ -0,0 +1,27 @@
+FROM python:3.11-slim
+
+ENV POETRY_HOME=/opt/poetry \
+    POETRY_NO_INTERACTION=1 \
+    POETRY_VIRTUALENVS_IN_PROJECT=true \
+    PATH="/opt/poetry/bin:$PATH"
+
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends build-essential curl \
+    && rm -rf /var/lib/apt/lists/*
+
+RUN curl -sSL https://install.python-poetry.org | python3 -
+
+WORKDIR /app
+
+# Copy dependency files first so the dependency layer is cached across code changes.
+COPY pyproject.toml poetry.lock README.md ./
+
+# Same groups as .github/workflows/ci.yml, plus "benchmarks" for running benchmarks/benchmark.py.
+RUN poetry install --with dev,benchmarks --no-root
+
+COPY . .
+
+RUN poetry install --with dev,benchmarks
+
+ENTRYPOINT ["poetry", "run"]
+CMD ["pytest", "tests/unit", "--import-mode=importlib"]

README.md

@@ -66,6 +66,18 @@ print(response)
 ```
 
 
+### Docker
+
+A `Dockerfile` is provided for a fully reproducible environment (same dependency groups as CI):
+
+```bash
+docker build -t vcache .
+docker run --rm vcache                                   # runs unit tests (default CMD)
+docker run --rm vcache pytest tests/integration           # integration tests
+docker run --rm -e OPENAI_API_KEY vcache python benchmarks/benchmark.py
+```
+
+
 ## How vCache Works
 
 vCache intelligently detects when a new prompt is semantically equivalent to a cached one, and adapts its decision boundaries based on your accuracy requirements.
@@ -156,7 +168,7 @@ You can find complete working examples in the [`playground`](playground/) direct
 
 
 ### Eviction Policy
-vCache supports FIFO, LRU, MRU, and a custom SCU eviction policy. See the [Eviction Policy Documentation](vcache/vcache_core/cache/eviction_policy/README.md) for further details.
+vCache supports FIFO, LRU, MRU, Cost-Aware, ARC, and a custom SCU eviction policy. See the [Eviction Policy Documentation](vcache/vcache_core/cache/eviction_policy/README.md) for further details.
 
 
 

benchmarks/ReadMe.md

@@ -125,6 +125,30 @@ You can benchmark vCache on your own datasets. The script supports `.csv` and `.
 
 Benchmark results are saved to the `benchmarks/results/` directory, organized by dataset, embedding model, and LLM. For each run, the output includes:
 - **JSON files** containing raw data on cache hits, misses, latency, accuracy metrics, and internal vCache statistics.
+- **CSV files** with the same per-query metrics in a flat, row-per-query table (`cache_hit`, `latency_direct`, `latency_vcache`, `cpu_percent`, `memory_mb`, `gpu_util_percent`, ...), convenient for spreadsheets or `pandas`.
 - **Plot images (`.png`, `.pdf`)** visualizing key trade-offs, such as cache hit rate vs. accuracy and latency savings.
 
-These metrics help assess the trade-offs between reliability, efficiency, and reuse across different semantic caching strategies.
+These metrics help assess the trade-offs between reliability, efficiency, and reuse across different semantic caching strategies.
+
+A sample run's output is committed at [`benchmarks/results/sample/`](results/sample/) so you can see the file format without running anything.
+
+
+### Resource & Throughput Metrics
+
+Alongside cache hit rate, accuracy, and latency, every run also records, per query:
+- `cpu_percent_list` / `memory_mb_list`: the benchmark process's CPU usage (%) and resident memory (MB), sampled via [`psutil`](https://pypi.org/project/psutil/) right after each query completes. `peak_memory_mb` is the run's maximum.
+- `gpu_util_list`: GPU utilization (%) of device 0, sampled via [`pynvml`](https://pypi.org/project/pynvml/). This is **best-effort**: it's `None` for every query unless you `pip install pynvml` and have a working NVIDIA driver — no error is raised either way.
+
+And for the run as a whole:
+- `elapsed_time_sec`: total wall-clock time for the benchmark loop.
+- `throughput_qps`: queries processed per second (`num_queries / elapsed_time_sec`).
+- `throughput_tps`: tokens processed per second, summing prompt + response tokens (counted with [`tiktoken`](https://pypi.org/project/tiktoken/)'s `cl100k_base` encoding when available, falling back to a whitespace word count otherwise).
+
+These are implemented in `benchmarks/common/resource_metrics.py` and wired into `Benchmark.update_stats` / `dump_results_to_json` / `dump_results_to_csv` in `benchmarks/benchmark.py`.
+
+
+## Continuous Integration
+
+`tests/integration/test_benchmark_smoke.py` and `tests/unit/Benchmark/test_resource_metrics.py` exercise the same metrics pipeline (`Benchmark.run_benchmark_loop`, `update_stats`, `dump_results_to_json`/`dump_results_to_csv`, and the resource-sampling helpers) against a small synthetic, fully offline dataset using `BenchmarkInferenceEngine`/`BenchmarkEmbeddingEngine`. These run automatically in the `test` job of `.github/workflows/ci.yml` on every commit.
+
+If you want to track real performance trends over time, periodically run `python benchmarks/benchmark.py` with a small `RUN_COMBINATIONS` entry and commit or archive the resulting JSON/CSV — see `benchmarks/results/sample/` for the expected format.