Skip to content

About

Reproducibility artifact for: Kozuchi Agent (manuscript under review)

Resources

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

ASE 2026 DOI arXiv Zenodo Python License: MIT Project Website

Kozuchi Agent — Reproducibility Artifact

Tip

🆕 New: Interactive Project Website

Explore the paper's headline results, architecture, and one-command reproduction guide on the new project website:

📄 Paper PDF · 🖼 ASE 2026 poster (A0) · 📊 Talk slides (PPTX) · 🎤 ASE 2026 presentation · DOI 10.1145/3832783.3834531 · 📚 arXiv:2608.15579 · 📓 Rendered walkthrough notebook · 🗃 Dataset on Zenodo (under review — preview access)

This directory is the self-contained artifact for the paper:

Kozuchi Agent: A Language-Agnostic Open-Weight Agent for Software Repair (Bahrami et al. — accepted for presentation at ASE 2026, the 41st IEEE/ACM International Conference on Automated Software Engineering, Munich, Germany, October 12–16, 2026)

📚 Preprint: arXiv:2608.15579 · PDF

What the paper is about

Kozuchi Agent is a software-repair agent: give it a bug report and a repository, and it produces a patch that makes the project's tests pass. The paper is an industry experience report on making such an agent work under real engineering constraints — a locally hosted, open-weight model (Qwen3.5-27B, no fine-tuning) instead of a proprietary API, and heterogeneous internal clusters instead of a single research machine.

The design rests on four ideas: an explicit eight-phase workflow (reproduce the issue, synthesize a test, localize code and tests, fix, verify, close, report) that keeps long repair trajectories auditable; persistent state on the filesystem so facts survive context compression; a model-independent action interface and small deterministic tool set, so switching model families does not mean rewriting the agent; and cross-agent test-time selection, which picks among 8 independent candidate patches using only tests the runs wrote themselves — never the benchmark's hidden tests.

Measured results: 374/500 (74.8%) on SWE-bench Verified, first among open-weight systems and 12th of 135 overall on the official evaluator. The same configuration, with no language-specific tuning, resolves 41/128 (32.03%) on Multi-SWE-bench Java — first among strict open-weight submissions and fourth of 42 overall. Peers at comparable scores use models an order of magnitude larger.

What this repository is for

This is the reproducibility artifact: the evidence behind those claims, packaged so a reviewer can check them without access to our infrastructure. It contains the evaluation outputs, agent trajectories, configurations, analysis scripts, and the paper source itself.

Everything cited in the paper — every numeric value, every figure, every table, every trajectory-bundle counter — is regenerable from files inside this folder only. Run reproduce.sh to rebuild them all, or read notebooks/reproduce.html for a narrative pass that prints each number next to the script that derives it and the place in the paper that cites it. It does not ship model weights or our internal cluster configuration; see Trust model and license boundary.

Quick links What it gives you
INDEX.md One row per paper number. Each row links to the deriving script, the cited section/figure/table, and the regenerated artifact under stats/ or figures/.
REFERENCES_INDEX.md Annotated bibliography indexing paper/references.bib: every cite key, URL, and why the paper uses it.
reproduce.sh One-command rebuild of every number and figure.
compress.sh / decompress.sh Pack / unpack the heavy trajectory and log directories under data/archives/.
notebooks/reproduce.ipynb Rendered, narrative walkthrough of every paper number.
notebooks/reproduce.html Static HTML render of the notebook (open in any browser, no Jupyter required).

1. Layout

paper/artifacts/
├── README.md               <- you are here
├── INDEX.md                <- paper-number ↔ script ↔ section ↔ output
├── REFERENCES_INDEX.md     <- bibliography index for paper/references.bib
├── reproduce.sh            <- regenerate every cited number / figure
├── compress.sh             <- pack data/trajectories + heavy logs to data/archives/
├── decompress.sh           <- restore data/archives/*.zst back into place
│
├── paper/                  <- LaTeX sources, figures, stats, build.sh
│   ├── main.tex            <- final paper source
│   ├── main.pdf            <- compiled paper (mirrored)
│   ├── references.bib      <- bibliography
│   ├── source-index.md     <- list of cited external sources
│   ├── figures/            <- figures used by main.tex
│   ├── stats/              <- LaTeX fragments / JSON used by main.tex
│   └── build.sh            <- rebuild PDF (latexmk / pdflatex)
│
├── scripts/                <- every reproduction script
│   ├── numbers/            <- compute_*.py + run_all.sh (paper numbers)
│   ├── figures/            <- plot_*.py figure scripts
│   ├── analysis/           <- richer analyses cited in §RQ1–RQ6
│   └── rq6_cross_track/    <- standalone RQ6 reproduction bundle
│
├── stats/                  <- regenerated stats (mirrored from paper/stats)
├── figures/                <- regenerated figures (mirrored from paper/figures)
├── plots/                  <- auxiliary plot outputs (Pareto frontier etc.)
├── notebooks/              <- rendered Jupyter notebook
│
└── data/                   <- self-contained inputs
    ├── archives/           <- *.tar.zst bundles for the heavy artifacts
    ├── configs/            <- agent_sota.yaml, model_*.yaml, ...
    ├── operational_metadata/ <- redacted CI / cluster / tool inventory CSVs
    │                          (replaces the old repo_meta/ and
    │                          swe_sota_agent/tools/ mirrors; see
    │                          operational_metadata/README.md)
    ├── trajectories/       <- per-instance trajectory bundle (small files
    │                          checked in; raw trajectories archived)
    ├── experiments/        <- Python (Verified) and Java (Multi-SWE-bench)
    │                          evaluation directories with src/, results/,
    │                          metadata.yaml, README.md, comparison.md
    ├── paper_comparison/   <- xlsx workbook + java leaderboard plots
    └── paper_java_src/     <- Java analysis CSVs (used by figures)

2. Quick-start

Download Trajectory Dataset

You may download datasets for both Python and Java from Zenodo with reserved DOI: 10.5281/zenodo.19941649 and save it under data/archives/. Alternatively, you may review pre-rendered notebooks.

2.1 Regenerate every paper number and figure

./reproduce.sh                # numbers + figures (stats/, figures/, plots/)
./reproduce.sh --paper        # same as above, plus rebuild paper/main.pdf
./reproduce.sh --skip-figures # numbers only (no matplotlib needed)

The script:

  1. Runs every script under scripts/numbers/ (see scripts/numbers/run_all.sh) which emits 30 files under stats/.
  2. Regenerates the RQ6 cross-track table and figure (scripts/rq6_cross_track/build_rq6.py).
  3. Regenerates the Python-vs-Java six-panel figure (scripts/figures/plot_python_vs_java.py).
  4. Regenerates the parameter-efficiency Pareto figure (scripts/figures/plot_broad_with_java.py).
  5. Optionally rebuilds the paper PDF (--paper).

2.2 What you need

  • Python 3.10+ (no third-party packages required for scripts/numbers/).
  • matplotlib, pandas, seaborn, numpy, openpyxl for the figure scripts. Install with pip install matplotlib pandas seaborn numpy openpyxl, or run reproduce.sh directly: it auto-uses uv run --with ... when uv is on $PATH.
  • pdflatex / latexmk (TeX Live) only if you want ./reproduce.sh --paper.

2.3 Heavy data (raw trajectories and harness logs)

The full 22 GB of agent trajectories and 4.5 GB of SWE-bench harness logs are not needed to recompute paper numbers — the small JSON reports under data/trajectories/.../runs/r0X/report.json are sufficient. The heavy directories are stored as compressed *.tar.zst archives under data/archives/. Restore them in place with:

./decompress.sh                                 # extract every archive (idempotent)
./decompress.sh --list                          # show recipe status
./decompress.sh --force                         # re-extract, overwriting the loose tree

To regenerate the archives after editing the loose data, or to clear the loose copies before git add, use the matching compress.sh:

./compress.sh                                   # build any missing/stale archives
./compress.sh --list                            # show recipe status
./compress.sh --force                           # rebuild every archive
./compress.sh --prune                           # archive AND delete the loose source

Both scripts are recipe-driven; the recipe table is the only source of truth and is duplicated identically in compress.sh and decompress.sh. Current recipes:

Archive (under data/archives/) Compressed Expanded Loose location
q35_verified500_tts8_75p2_submission_bundle_20260326-055715_merged500.tar.zst 533 MB 22 GB data/trajectories/q35_..._merged500/
python_evaluation_trajs.tar.zst 80 MB 2.7 GB data/experiments/evaluation/verified/.../trajs/
python_evaluation_logs.tar.zst 1.4 MB 33 MB data/experiments/evaluation/verified/.../logs/
java_evaluation_trajs.tar.zst 421 MB 1.2 GB data/experiments/java/.../trajs/
java_evaluation_logs.tar.zst 202 MB 308 MB data/experiments/java/.../logs/
java_evaluation_all_preds.jsonl.zst 197 MB 255 MB data/experiments/java/.../all_preds.jsonl

2.4 Submitting to git

There is no top-level "lite vs full" bundle — the artifact tree is checked into git as-is. Only data/archives/*.zst (≈1.4 GB) needs an out-of-band channel (e.g. Git LFS, OSF, Zenodo) if your repository host imposes file-size limits. Typical submission flow:

./compress.sh --prune    # produce every archive AND remove the loose source
git add -A               # commit the artifact (archives included or LFS-tracked)
# reviewers run:
./decompress.sh          # restore the loose trajectories where reproduction expects them

3. Walkthrough notebook

For a reviewer-friendly narrative pass that prints every paper number side-by-side with its source script and its location in main.tex:

  • No tools required — open notebooks/reproduce.html in any browser to see the fully rendered walkthrough with all outputs inlined.
  • Interactive re-execution — open notebooks/reproduce.ipynb in Jupyter and re-run every cell to regenerate numbers and figures from scratch:
jupyter notebook notebooks/reproduce.ipynb

Both files are committed in rendered form so that opening them on GitHub / GitLab shows every output without re-execution.


4. Trust model and license boundary

  • Numbers and figures are derivable from artifacts only. We never reach for the SWE-bench hidden test labels at selection time, nor do we use any closed-source verifier.
  • Model weights for Qwen/Qwen3.5-27B, Devstral, and the Nemotron backbones are not shipped here; they are gated by their providers' licenses. The model inventory under paper/stats/model_inventory.tex records which configs reference them.
  • Internal cluster paths, SLURM partitions, runner tokens, and the agent's tool source code are not shipped with this artifact. The CI / cluster / tool-inventory claims cited in the paper are reproduced from the redacted CSVs under data/operational_metadata/; the upstream .gitlab-ci.yml, scripts/module_load.sh, and swe_sota_agent/tools/*.py are intentionally omitted.
  • Source attribution for cited external work is in REFERENCES_INDEX.md and paper/source-index.md; each ties a bibliography key to a publicly resolvable URL.

5. Reading order

  1. INDEX.md — tour the paper through its numbers; every row links to the script that produces it and the section that discusses it.
  2. paper/main.pdf — the paper itself.
  3. notebooks/reproduce.html (static) or notebooks/reproduce.ipynb (live) — narrative reproduction with inlined data.
  4. scripts/numbers/ and scripts/figures/ — implementation.
  5. data/ — primary inputs (configs, CSVs, JSON reports, archives).

6. Building the paper PDF locally

cd paper
./build.sh --skip-figures

The build script writes paper/main.pdf (and a copy at paper/build/main.pdf). It uses the bundled ACM class files (paper/acmart.cls, etc.) so no internet access is required.


7. Citation

If you build on this artifact, please cite the paper (doi:10.1145/3832783.3834531). Session page: ASE 2026 Industry Showcase. A freely available preprint is on arXiv:2608.15579.

ACM Reference Format

Mehdi Bahrami, Kosaku Kimura, Satoshi Munakata, Satoshi Nakashima, Yu Ishikawa, Kosuke Maeda, Nao Soma, Kenichi Kobayashi, Keisuke Miyazaki, Keizo Kato, Shigeki Fukuta, Tatsuo Kumano, Nobutaka Imamura, Kevin Musgrave, Shahbaz Abdul Khader, Kwun Ho Ngan, Joe Townsend, Fayas Asharindavida, Matthieu Parizy, Akira Sakai, Yuma Ichikawa, Yang Zhao, Michiaki Takizawa, Taku Fukui, Hiroki Ohtsuji, Wei-Peng Chen, and Hiromichi Kobashi. 2026. Kozuchi Agent: A Language-Agnostic Open-Weight Agent for Software Repair. In Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering (ASE '26), October 12–16, 2026, Munich, Germany. ACM, New York, NY, USA, 13 pages. https://doi.org/10.1145/3832783.3834531

BibTeX

@inproceedings{kozuchi2026,
  author    = {Bahrami, Mehdi and Kimura, Kosaku and Munakata, Satoshi and
               Nakashima, Satoshi and Ishikawa, Yu and Maeda, Kosuke and
               Soma, Nao and Kobayashi, Kenichi and Miyazaki, Keisuke and
               Kato, Keizo and Fukuta, Shigeki and Kumano, Tatsuo and
               Imamura, Nobutaka and Musgrave, Kevin and Abdul Khader, Shahbaz and
               Ngan, Kwun Ho and Townsend, Joe and Asharindavida, Fayas and
               Parizy, Matthieu and Sakai, Akira and Ichikawa, Yuma and
               Zhao, Yang and Takizawa, Michiaki and Fukui, Taku and
               Ohtsuji, Hiroki and Chen, Wei-Peng and Kobashi, Hiromichi},
  title     = {Kozuchi Agent: A Language-Agnostic Open-Weight Agent for Software Repair},
  year      = {2026},
  isbn      = {979-8-4007-2882-2},
  publisher = {Association for Computing Machinery},
  address   = {New York, NY, USA},
  url       = {https://doi.org/10.1145/3832783.3834531},
  doi       = {10.1145/3832783.3834531},
  booktitle = {Proceedings of the 41st IEEE/ACM International Conference on
               Automated Software Engineering},
  numpages  = {13},
  keywords  = {AI agent, automated program repair, SWE-bench, experience report},
  location  = {Munich, Germany},
  series    = {ASE '26}
}

About

Reproducibility artifact for: Kozuchi Agent (manuscript under review)

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages