Reads your GitHub activity and writes it up. It listens to webhooks from a repository, stores every commit with its diff, and uses language models to produce short summaries, pull request descriptions, decision records, operating procedures, code reviews and weekly digests. You can also ask questions about the repository in plain English and get answers based on what was actually committed.
| Feature | How it works |
|---|---|
| Commit summaries | Every pushed commit gets a one-sentence summary and a vector embedding |
| Search and chat | Ask a question; matching commits are retrieved and answered from |
| PR descriptions | Written from all commits on a branch |
| Decision records (ADR) | Written from a pull request's discussion |
| Operating procedures (SOP) | Written from operational changes on a branch (CI, Dockerfiles, scripts, migrations) |
| Code review | A local code model reviews the diffs on a branch |
| Weekly digests | Scheduled summary of the week's work, optionally posted to Slack |
| Slack delivery | Any generated document can be posted to a project's Slack channel with one click, and weekly digests are delivered there automatically. The webhook URL is encrypted and never returned by the API |
| Secret redaction | Credentials in diffs and commit messages are removed before storage and before any model sees them |
Four models, each doing a different job.
| Job | Model | Runs on | Why |
|---|---|---|---|
| Summaries, chat, PR descriptions, ADR, SOP, digests | gemini-2.5-flash |
Google API | Fast and cheap, and these paths have a person waiting |
| Embeddings | nomic-embed-text |
Local Ollama | Runs on every commit; free and fast enough locally |
| Code review | qwen2.5-coder:7b-instruct |
Local Ollama | Runs in the background, so slower local inference costs nothing. A fine-tune of this model was tried and measured worse — see below |
| Evaluation judge | gemini-2.5-flash |
Google API | Held fixed so scores stay comparable between runs |
Which model serves which job is configuration, not code — see the
app.ai.tasks.* block in application.properties. Anything speaking the OpenAI
API can be added as a new app.ai.providers.<name>.* block.
Sign in with GitHub
Home
Project dashboard
Ask about the repository
PR review on a branch
Document shared to Slack
The goal was to stop depending on Gemini entirely and run every AI feature on local models.
Gemini is fast and cheap, but it is still a dependency with three costs. It needs a funded billing account — the free tier turned out to allow only 20 requests a day, which is not enough to summarise a single busy afternoon. It charges per call. And every diff leaves the machine and goes to a third party, which is a hard sell for a private repository.
Embeddings already ran locally on nomic-embed-text. Code review was the
natural next thing to move, because nobody sits and waits for it — it runs in
the background, so a model that takes 25 seconds instead of 10 costs nothing in
practice. The longer-term intent was to move document generation across too
(PR descriptions, decision records, operating procedures, digests), using
something like Llama 3, Qwen 3 or Gemma if the hardware could hold it.
Rather than ask one general local model to do everything, the plan was to fine-tune a model that does exactly one job — review a diff — and do it well.
| Base model | Qwen2.5-Coder-7B-Instruct — a code model for a code task |
| Framework | Unsloth, LoRA fine-tuning |
| Dataset | ronantakizawa/github-codereview |
| Training sample | 1,000 rows |
| Method | LoRA, rank 16, alpha 32, applied to all attention and MLP projections |
| Training precision | 4-bit quantised base (bnb-4bit) |
The resulting LoRA adapter was merged back into the base weights at 16-bit,
converted to GGUF, quantised to q4_K_M (about 4.7 GB), and served through
Ollama so the application could reach it over the same OpenAI-style API it uses
for everything else.
All six AI features are measured, each against two architectures: everything on
a paid cloud model, and everything on a local model with no paid API at all.
Full numbers, method and per-case detail are in
evals-reports/.
Commit summaries. A fixed set of 50 real commits was hand-labelled, then scored by a judge model that never changes between runs. Rewriting the summarisation prompt took output-rule compliance from 60% to 100% on the same 50 commits — markdown violations went from 36% to zero.
The more useful finding was uncomfortable: the judge scored 5.0 out of 5 for faithfulness while three vulnerabilities went unmentioned. Those summaries were accurate — an SSRF hole described as ordinary plumbing is still a true description. A high score on the wrong question tells you nothing, so "is it accurate" and "does it warn the reader" are scored separately and never averaged.
Can it run without a paid API? Yes, and mostly it should not:
| Cloud (Gemini 2.5 Flash) | Local (qwen3:8b) | |
|---|---|---|
| Summaries — follows the output rules | 100% | 92% |
| Summaries — discloses the vulnerability | 10/13 | 8/13 |
| PR descriptions — invented content | 1/12 | 4/12 |
| ADRs — conveyed the real decision | 5/9 | 3/9 |
| Digests — invented content | 0/7 | 1/6 |
| Whole 50-commit run | 8 min | 41 min |
On two branches the local model described a change that introduced a
vulnerability as one that fixed it — calling a new dangerouslySetInnerHTML "a
critical security fix mitigating an XSS vulnerability". That is worse than
missing the flaw: a reviewer reads it and believes the code got safer.
The free part works best. Before any model is called, two pure functions decide whether the input is worth spending on — is there real discussion behind this pull request, does this branch actually describe a procedure. They were right 42 times out of 42 and refused half the requests. No model, no cost.
Code review. 20 hand-labelled diffs — 12 that introduce a real flaw, 8 that only look alarming — run through three models, changing nothing else:
| Gemini 2.5 Flash | Qwen2.5-Coder 7B base | Qwen2.5-Coder 7B fine-tuned | |
|---|---|---|---|
| Found the flaw | 12/12 | 11/12 | 10/12 |
| Cited code that does not exist | 1/20 | 4/20 | 6/20 |
| Time per review | 11.6s | 22.4s | 27.0s |
Fine-tuning made the model worse than the base it was trained from — fewer flaws found, three times as much invented code, and slower. That is a negative result and it is published as one. The fine-tune learned the shape of a review without learning to stay inside the diff it was given, which is the failure that matters most: telling a developer to fix code that was never touched wastes more time than missing a flaw.
How much to trust these numbers. Sample sizes are small (12 flawed diffs, so one case is roughly 8 percentage points), the diffs are short, and each model was run once. Close results should be read as ties. The pull request discussions and operational diffs are written fixtures, not harvested from real repositories.
The harness was less reliable than the models. Seven defects were found in
the evaluation code itself, and every one made a model look worse than it was —
a judge shown only part of what the prompt saw reported 86% invented content
where the true figure was 0%; a token budget set below what the local model
spends thinking produced three empty documents that read as capability failure.
The judge-validation step, which exists to catch exactly this, caught none of
them: it checks the judge's reasoning, not whether it was handed the right
inputs. All seven are listed in evals-reports/ rather than
quietly fixed, because the failure mode generalises — the measurement is code
too, and nothing was measuring it.
Only 1,000 rows were used. That is a small sample for teaching a task. It is enough for a model to pick up the register and format of code review, and not enough to teach it the discipline underneath — which is probably why the output looks more like a review than the base model's while being less accurate.
The dataset may have taught the exact failure that was measured. Real GitHub
review comments are written by people who can see the whole repository, so they
routinely refer to code outside the diff — "same problem as in UserService",
"this duplicates the helper above". Training on comments like that, without the
surrounding code, teaches a model that confidently naming code it cannot see is
normal. The fine-tune cited absent code in 30% of reviews against the base
model's 20%, which fits that explanation.
LoRA rank 16 is a small amount of capacity to add to a 7B model, so the fine-tune sits close to the base while still being pulled away from it.
Training and serving used different precisions. Training ran against a
4-bit quantised base, then the adapter was merged into 16-bit weights and
re-quantised to q4_K_M for serving. Each conversion is lossy, and the model
that answered was never exactly the model that was trained.
None of these is confirmed. They are the hypotheses worth testing next, and the cheapest one is simply training on far more of the dataset.
The measurements decided the architecture rather than the other way round. The project now runs on three models:
| Job | Model | Where |
|---|---|---|
| Summaries, chat, PR descriptions, ADR, SOP, digests | gemini-2.5-flash |
Cloud |
| Code review | qwen2.5-coder:7b-instruct, the base model, not the fine-tune |
Local |
| Embeddings | nomic-embed-text |
Local |
The original ambition — everything local — is not met, and the honest reason is that a 7B model on consumer hardware is not yet good enough for the paths where a person is waiting. Chat has to prefill a large retrieval context before it can say anything, which takes tens of seconds locally against a few seconds on Gemini.
Where local does work is the background: embeddings on every commit, and code review on a branch. Those two run for free, keep diffs on the machine, and cost nothing in user-visible latency. That is the split the numbers support, so that is the split that shipped.
- Java 21+ and Docker
- A Supabase project (PostgreSQL with the
vectorextension) - A Google AI Studio API key with billing enabled
- Ollama running locally, with
nomic-embed-textpulled - Optional: a GitHub App, and a Slack incoming webhook
These live in backend-api/.env, which is not committed.
| Variable | What it is |
|---|---|
SUPABASE_DB_PASSWORD |
Database password, from Supabase → Project Settings → Database |
GEMINI_API_KEY |
Google AI Studio key. Needs billing enabled; the free tier allows only 20 requests a day |
APP_ENCRYPTION_KEY |
Base64 AES key used to encrypt stored credentials. Generate with openssl rand -base64 32. Changing it makes existing encrypted values unreadable |
GITHUB_APP_ID |
Your GitHub App's numeric id |
GITHUB_APP_PRIVATE_KEY_B64 |
The App's private key, base64 encoded: base64 -i app.pem | tr -d '\n'. Without it the app falls back to a personal access token |
SPRING_AI_OLLAMA_BASE_URL |
Defaults to http://localhost:11434 |
The frontend needs its own frontend/.env:
VITE_SUPABASE_URL=your_supabase_project_url
VITE_SUPABASE_ANON_KEY=your_supabase_anon_key
PostgreSQL with the vector extension. Nine tables:
| Table | Holds |
|---|---|
users |
Accounts, plus an encrypted GitHub token |
projects |
A tracked repository, its webhook secret and an encrypted Slack webhook |
project_members |
Per-project roles, synced from GitHub collaborators |
commit_logs |
Commits with diff, summary and a 768-dimension embedding |
pull_requests |
Pull requests |
pr_comments |
Review and issue comments |
documents |
Everything generated, with the model and prompt version that produced it |
chat_messages |
Chat history per project |
ai_calls |
One row per model call: tokens, latency, outcome |
commit_logs carries an HNSW index on the embedding and a full-text index over
message and summary; search fuses both rankings.
The schema is managed by hand, not generated by Hibernate. The initial schema
and each incremental change live in db/migrations/, applied in filename order.
- Create
backend-api/.envandfrontend/.envas above. - Apply everything in
db/migrations/to your Supabase database, in order. - Pull the embedding model:
ollama pull nomic-embed-text - Start it:
docker compose up -d --build - Open http://localhost:5173, sign in with GitHub, add a repository.
- In that repository's GitHub settings, add a webhook pointing at
/api/webhooks/githubusing the secret you set in the app. Use a tunnel such as ngrok when running locally.
Code review additionally needs its model in Ollama:
ollama pull qwen2.5-coder:7b-instruct
To run the fine-tune instead, import the merged Qwen2.5-Coder-7B, quantize to q4_K_M, and set num_predict and a
stop token in the Modelfile, or generation will not terminate.
cd backend-api && ./mvnw test
Tests that call live models are tagged eval and excluded by default. Run them
explicitly by name:
./mvnw test -Peval -Dtest=SummaryEvaluationTest
./mvnw test -Peval -Dtest=CodeReviewEvaluationTest
The first scores 50 hand-labelled commits and writes a report into
evals-reports/summary/. The second scores 20 hand-labelled diffs into
evals-reports/code-review/, using whichever model
app.ai.tasks.code-review points at — override it to compare models without
changing any code:
./mvnw test -Peval -Dtest=CodeReviewEvaluationTest -Dapp.ai.tasks.code-review=gemini
Both evaluations depend on a judge model, so check the judge still agrees with hand-written answers before trusting a comparison:
./mvnw test -Peval -Dtest=JudgeValidationTest
./mvnw test -Peval -Dtest=ReviewJudgeValidationTest
Retrain the code-review model properly. The fine-tune used 1,000 rows of the dataset. Training on the full set, and on examples where the reviewer can only see the diff, would test whether the failure was the method or the data.
Measure the rest of the features. Only commit summaries and code review have a golden set. PR descriptions, decision records, operating procedures and digests currently ship on inspection alone, so no claim about their quality means anything yet.
Move document generation local. Once a local model can hold its own on generation quality, the remaining Gemini dependency can go. Chat is the hardest case: it has to read a large retrieval context before answering, which is where local models are slowest.
Process webhooks after acknowledging them. Ingestion currently makes one blocking GitHub call per commit inside the request, so a large push risks timing out the delivery. Storing the payload first and processing it in the background fixes that.
Paginate the commit list. The dashboard fetches every commit with its full diff and polls every three seconds. Fine at a few hundred commits, not at thousands.
Rerank retrieved commits. Search fuses vector and keyword rankings, but nothing re-scores the fused set, so the top result is only as good as the fusion.
Rebuild the false-alarm metric. The current one counts real findings as errors because its "clean" diffs are only free of one kind of flaw. It needs hand-written, genuinely defect-free diffs to mean anything.
Contributions are welcome! If you want to contribute, please follow these steps:
- Fork the Repository: Create your own branch from main.
- Create a Feature Branch: git checkout -b feature/AmazingFeature
- Commit your Changes: Write clear commit messages.
- Push to the Branch: git push origin feature/AmazingFeature
- Open a Pull Request: Describe the changes you made and the problem they solve.
Before opening a pull request
./mvnw testmust pass, and changes should come with a test.- Every project-scoped endpoint must check access through
ProjectAccessService. Reads needVIEWER, anything that calls a model needsMAINTAINER, secrets needOWNER. - Refusals return 404, never 403, so the API does not confirm which projects exist to someone who cannot see them.
- Anything from a repository — diffs, commit messages, PR bodies, comments — is
untrusted. Wrap it with
UntrustedContent.fence()before it reaches a prompt. - Never log a credential. Log that one was found and what kind it was, not its value.
- Schema changes go in a new numbered file in
db/migrations/. Nothing is applied automatically. - Changing a prompt means bumping its
*_PROMPT_VERSIONconstant, so documents written by different prompts stay distinguishable afterwards.
If you find a bug or have a feature request, please use the GitHub Issues tab. Include the following in your report:
- A clear title.
- Steps to reproduce the bug.
- Expected vs. actual behavior.
- Screenshots or error logs if you have them.