Tip
Explore the paper's headline results, architecture, and one-command reproduction guide on the new project website:
📄 Paper PDF · 🖼 ASE 2026 poster (A0) · 📊 Talk slides (PPTX) · 🎤 ASE 2026 presentation · DOI 10.1145/3832783.3834531 · 📚 arXiv:2608.15579 · 📓 Rendered walkthrough notebook · 🗃 Dataset on Zenodo (under review — preview access)
This directory is the self-contained artifact for the paper:
Kozuchi Agent: A Language-Agnostic Open-Weight Agent for Software Repair (Bahrami et al. — accepted for presentation at ASE 2026, the 41st IEEE/ACM International Conference on Automated Software Engineering, Munich, Germany, October 12–16, 2026)
📚 Preprint: arXiv:2608.15579 · PDF
Kozuchi Agent is a software-repair agent: give it a bug report and a repository, and it produces a patch that makes the project's tests pass. The paper is an industry experience report on making such an agent work under real engineering constraints — a locally hosted, open-weight model (Qwen3.5-27B, no fine-tuning) instead of a proprietary API, and heterogeneous internal clusters instead of a single research machine.
The design rests on four ideas: an explicit eight-phase workflow (reproduce the issue, synthesize a test, localize code and tests, fix, verify, close, report) that keeps long repair trajectories auditable; persistent state on the filesystem so facts survive context compression; a model-independent action interface and small deterministic tool set, so switching model families does not mean rewriting the agent; and cross-agent test-time selection, which picks among 8 independent candidate patches using only tests the runs wrote themselves — never the benchmark's hidden tests.
Measured results: 374/500 (74.8%) on SWE-bench Verified, first among open-weight systems and 12th of 135 overall on the official evaluator. The same configuration, with no language-specific tuning, resolves 41/128 (32.03%) on Multi-SWE-bench Java — first among strict open-weight submissions and fourth of 42 overall. Peers at comparable scores use models an order of magnitude larger.
This is the reproducibility artifact: the evidence behind those claims, packaged so a reviewer can check them without access to our infrastructure. It contains the evaluation outputs, agent trajectories, configurations, analysis scripts, and the paper source itself.
Everything cited in the paper — every numeric value, every figure,
every table, every trajectory-bundle counter — is regenerable from
files inside this folder only. Run reproduce.sh to
rebuild them all, or read
notebooks/reproduce.html for a narrative
pass that prints each number next to the script that derives it and the
place in the paper that cites it. It does not ship model weights or
our internal cluster configuration; see
Trust model and license boundary.
| Quick links | What it gives you |
|---|---|
INDEX.md |
One row per paper number. Each row links to the deriving script, the cited section/figure/table, and the regenerated artifact under stats/ or figures/. |
REFERENCES_INDEX.md |
Annotated bibliography indexing paper/references.bib: every cite key, URL, and why the paper uses it. |
reproduce.sh |
One-command rebuild of every number and figure. |
compress.sh / decompress.sh |
Pack / unpack the heavy trajectory and log directories under data/archives/. |
notebooks/reproduce.ipynb |
Rendered, narrative walkthrough of every paper number. |
notebooks/reproduce.html |
Static HTML render of the notebook (open in any browser, no Jupyter required). |
paper/artifacts/
├── README.md <- you are here
├── INDEX.md <- paper-number ↔ script ↔ section ↔ output
├── REFERENCES_INDEX.md <- bibliography index for paper/references.bib
├── reproduce.sh <- regenerate every cited number / figure
├── compress.sh <- pack data/trajectories + heavy logs to data/archives/
├── decompress.sh <- restore data/archives/*.zst back into place
│
├── paper/ <- LaTeX sources, figures, stats, build.sh
│ ├── main.tex <- final paper source
│ ├── main.pdf <- compiled paper (mirrored)
│ ├── references.bib <- bibliography
│ ├── source-index.md <- list of cited external sources
│ ├── figures/ <- figures used by main.tex
│ ├── stats/ <- LaTeX fragments / JSON used by main.tex
│ └── build.sh <- rebuild PDF (latexmk / pdflatex)
│
├── scripts/ <- every reproduction script
│ ├── numbers/ <- compute_*.py + run_all.sh (paper numbers)
│ ├── figures/ <- plot_*.py figure scripts
│ ├── analysis/ <- richer analyses cited in §RQ1–RQ6
│ └── rq6_cross_track/ <- standalone RQ6 reproduction bundle
│
├── stats/ <- regenerated stats (mirrored from paper/stats)
├── figures/ <- regenerated figures (mirrored from paper/figures)
├── plots/ <- auxiliary plot outputs (Pareto frontier etc.)
├── notebooks/ <- rendered Jupyter notebook
│
└── data/ <- self-contained inputs
├── archives/ <- *.tar.zst bundles for the heavy artifacts
├── configs/ <- agent_sota.yaml, model_*.yaml, ...
├── operational_metadata/ <- redacted CI / cluster / tool inventory CSVs
│ (replaces the old repo_meta/ and
│ swe_sota_agent/tools/ mirrors; see
│ operational_metadata/README.md)
├── trajectories/ <- per-instance trajectory bundle (small files
│ checked in; raw trajectories archived)
├── experiments/ <- Python (Verified) and Java (Multi-SWE-bench)
│ evaluation directories with src/, results/,
│ metadata.yaml, README.md, comparison.md
├── paper_comparison/ <- xlsx workbook + java leaderboard plots
└── paper_java_src/ <- Java analysis CSVs (used by figures)
You may download datasets for both Python and Java from Zenodo with reserved DOI: 10.5281/zenodo.19941649 and save it under data/archives/. Alternatively, you may review pre-rendered notebooks.
./reproduce.sh # numbers + figures (stats/, figures/, plots/)
./reproduce.sh --paper # same as above, plus rebuild paper/main.pdf
./reproduce.sh --skip-figures # numbers only (no matplotlib needed)The script:
- Runs every script under
scripts/numbers/(seescripts/numbers/run_all.sh) which emits 30 files understats/. - Regenerates the RQ6 cross-track table and figure
(
scripts/rq6_cross_track/build_rq6.py). - Regenerates the Python-vs-Java six-panel figure
(
scripts/figures/plot_python_vs_java.py). - Regenerates the parameter-efficiency Pareto figure
(
scripts/figures/plot_broad_with_java.py). - Optionally rebuilds the paper PDF (
--paper).
- Python 3.10+ (no third-party packages required for
scripts/numbers/). matplotlib,pandas,seaborn,numpy,openpyxlfor the figure scripts. Install withpip install matplotlib pandas seaborn numpy openpyxl, or runreproduce.shdirectly: it auto-usesuv run --with ...whenuvis on$PATH.pdflatex/latexmk(TeX Live) only if you want./reproduce.sh --paper.
The full 22 GB of agent trajectories and 4.5 GB of SWE-bench harness
logs are not needed to recompute paper numbers — the small JSON
reports under data/trajectories/.../runs/r0X/report.json are
sufficient. The heavy directories are stored as compressed *.tar.zst
archives under data/archives/. Restore them in
place with:
./decompress.sh # extract every archive (idempotent)
./decompress.sh --list # show recipe status
./decompress.sh --force # re-extract, overwriting the loose treeTo regenerate the archives after editing the loose data, or to clear
the loose copies before git add, use the matching compress.sh:
./compress.sh # build any missing/stale archives
./compress.sh --list # show recipe status
./compress.sh --force # rebuild every archive
./compress.sh --prune # archive AND delete the loose sourceBoth scripts are recipe-driven; the recipe table is the only source of
truth and is duplicated identically in compress.sh and
decompress.sh. Current recipes:
Archive (under data/archives/) |
Compressed | Expanded | Loose location |
|---|---|---|---|
q35_verified500_tts8_75p2_submission_bundle_20260326-055715_merged500.tar.zst |
533 MB | 22 GB | data/trajectories/q35_..._merged500/ |
python_evaluation_trajs.tar.zst |
80 MB | 2.7 GB | data/experiments/evaluation/verified/.../trajs/ |
python_evaluation_logs.tar.zst |
1.4 MB | 33 MB | data/experiments/evaluation/verified/.../logs/ |
java_evaluation_trajs.tar.zst |
421 MB | 1.2 GB | data/experiments/java/.../trajs/ |
java_evaluation_logs.tar.zst |
202 MB | 308 MB | data/experiments/java/.../logs/ |
java_evaluation_all_preds.jsonl.zst |
197 MB | 255 MB | data/experiments/java/.../all_preds.jsonl |
There is no top-level "lite vs full" bundle — the artifact tree is
checked into git as-is. Only data/archives/*.zst (≈1.4 GB) needs an
out-of-band channel (e.g. Git LFS, OSF, Zenodo) if your repository
host imposes file-size limits. Typical submission flow:
./compress.sh --prune # produce every archive AND remove the loose source
git add -A # commit the artifact (archives included or LFS-tracked)
# reviewers run:
./decompress.sh # restore the loose trajectories where reproduction expects themFor a reviewer-friendly narrative pass that prints every paper number
side-by-side with its source script and its location in main.tex:
- No tools required — open
notebooks/reproduce.htmlin any browser to see the fully rendered walkthrough with all outputs inlined. - Interactive re-execution — open
notebooks/reproduce.ipynbin Jupyter and re-run every cell to regenerate numbers and figures from scratch:
jupyter notebook notebooks/reproduce.ipynbBoth files are committed in rendered form so that opening them on GitHub / GitLab shows every output without re-execution.
- Numbers and figures are derivable from artifacts only. We never reach for the SWE-bench hidden test labels at selection time, nor do we use any closed-source verifier.
- Model weights for
Qwen/Qwen3.5-27B,Devstral, and the Nemotron backbones are not shipped here; they are gated by their providers' licenses. The model inventory underpaper/stats/model_inventory.texrecords which configs reference them. - Internal cluster paths, SLURM partitions, runner tokens, and the
agent's tool source code are not shipped with this artifact. The CI
/ cluster / tool-inventory claims cited in the paper are reproduced
from the redacted CSVs under
data/operational_metadata/; the upstream.gitlab-ci.yml,scripts/module_load.sh, andswe_sota_agent/tools/*.pyare intentionally omitted. - Source attribution for cited external work is in
REFERENCES_INDEX.mdandpaper/source-index.md; each ties a bibliography key to a publicly resolvable URL.
INDEX.md— tour the paper through its numbers; every row links to the script that produces it and the section that discusses it.paper/main.pdf— the paper itself.notebooks/reproduce.html(static) ornotebooks/reproduce.ipynb(live) — narrative reproduction with inlined data.scripts/numbers/andscripts/figures/— implementation.data/— primary inputs (configs, CSVs, JSON reports, archives).
cd paper
./build.sh --skip-figuresThe build script writes paper/main.pdf (and a copy at
paper/build/main.pdf). It uses the bundled ACM class files
(paper/acmart.cls, etc.) so no internet access
is required.
If you build on this artifact, please cite the paper (doi:10.1145/3832783.3834531). Session page: ASE 2026 Industry Showcase. A freely available preprint is on arXiv:2608.15579.
ACM Reference Format
Mehdi Bahrami, Kosaku Kimura, Satoshi Munakata, Satoshi Nakashima, Yu Ishikawa, Kosuke Maeda, Nao Soma, Kenichi Kobayashi, Keisuke Miyazaki, Keizo Kato, Shigeki Fukuta, Tatsuo Kumano, Nobutaka Imamura, Kevin Musgrave, Shahbaz Abdul Khader, Kwun Ho Ngan, Joe Townsend, Fayas Asharindavida, Matthieu Parizy, Akira Sakai, Yuma Ichikawa, Yang Zhao, Michiaki Takizawa, Taku Fukui, Hiroki Ohtsuji, Wei-Peng Chen, and Hiromichi Kobashi. 2026. Kozuchi Agent: A Language-Agnostic Open-Weight Agent for Software Repair. In Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering (ASE '26), October 12–16, 2026, Munich, Germany. ACM, New York, NY, USA, 13 pages. https://doi.org/10.1145/3832783.3834531
BibTeX
@inproceedings{kozuchi2026,
author = {Bahrami, Mehdi and Kimura, Kosaku and Munakata, Satoshi and
Nakashima, Satoshi and Ishikawa, Yu and Maeda, Kosuke and
Soma, Nao and Kobayashi, Kenichi and Miyazaki, Keisuke and
Kato, Keizo and Fukuta, Shigeki and Kumano, Tatsuo and
Imamura, Nobutaka and Musgrave, Kevin and Abdul Khader, Shahbaz and
Ngan, Kwun Ho and Townsend, Joe and Asharindavida, Fayas and
Parizy, Matthieu and Sakai, Akira and Ichikawa, Yuma and
Zhao, Yang and Takizawa, Michiaki and Fukui, Taku and
Ohtsuji, Hiroki and Chen, Wei-Peng and Kobashi, Hiromichi},
title = {Kozuchi Agent: A Language-Agnostic Open-Weight Agent for Software Repair},
year = {2026},
isbn = {979-8-4007-2882-2},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3832783.3834531},
doi = {10.1145/3832783.3834531},
booktitle = {Proceedings of the 41st IEEE/ACM International Conference on
Automated Software Engineering},
numpages = {13},
keywords = {AI agent, automated program repair, SWE-bench, experience report},
location = {Munich, Germany},
series = {ASE '26}
}