Distill

Distill crawls a set of Atom/RSS feeds, stores their entries in a local SQLite database, and exposes a RESTful HTTP API for managing feeds and reading back their entries — including keyword search across an entry's title and summary.

Running

go run .

This opens (creating if necessary) feeds.db in the working directory and starts the API server on :8080. Feeds aren't crawled automatically; use the /feeds/refresh endpoints below to trigger a crawl.

Copy .env.example to .env first — API_KEY is required, and the server refuses to start without it (see Auth below). Fill in the API key(s) for whichever LLM provider(s) — and, optionally, email provider — you want to use for daily briefs. .env is gitignored and loaded automatically on startup.

See api.http for runnable examples (VS Code REST Client extension) covering the full flow: registering feeds, refreshing them, querying entries, and generating a digest.

API

Auth

Every request must carry Authorization: Bearer <API_KEY>, checked against the single static key in .env. A missing or wrong key gets a 401; if API_KEY itself isn't set, every request gets a 500 rather than silently running unauthenticated (though in practice the server won't even start in that case).

Endpoints

Feeds support full CRUD. Entries are read-only, since they're populated by crawling rather than authored directly.

Method	Path	Purpose
POST	/feeds	create a feed (409 on duplicate)
GET	/feeds	list feeds
GET	/feeds/{id}	get a feed
PUT	/feeds/{id}	update a feed's URL
DELETE	/feeds/{id}	delete a feed + its entries
DELETE	/feeds	delete every feed + their entries
POST	/feeds/refresh	crawl all feeds, persist entries
POST	/feeds/{id}/refresh	crawl one feed
GET	/feeds/{id}/entries	entries for one feed
GET	/entries	list entries, ?keyword= or ?after= filter
GET	/entries/{id}	single entry with full content
POST	/digest	generate (and optionally email) a daily brief from the last 24h of entries, ?extract_provider= / ?provider=claude|openai|deepseek (provider defaults to claude; extract_provider defaults to whatever provider resolves to) — the brief is persisted
GET	/digests	list previously generated digests
GET	/digests/{id}	single digest

?keyword= matches entries whose title or summary contains the given keyword (case-insensitive substring match). ?after= filters to entries published after the given RFC3339 timestamp.

/digest builds the brief in two LLM steps, each with its own provider:

1. Extract (?extract_provider=) — filters the last 24h of entries down to the ones that clear a baseline relevance bar and reduces each to a compact factual summary. High-volume but mechanical, so a cheap/fast model is a good fit here.
2. Reason (?provider=) — reads that filtered, summarized list and reasons about importance and impact to group, rank, and write the actual digest aimed at a lead technical engineer. This is where judgment calls happen, so it benefits most from a stronger model.

Each provider's API key is read from the environment, and each step's model can be overridden independently via <PROVIDER>_EXTRACT_MODEL / <PROVIDER>_REASON_MODEL (falling back to the provider-wide <PROVIDER>_MODEL) — see .env.example. extract_provider defaults to whatever provider resolves to, so ?provider=openai alone runs both steps on OpenAI, picking up its OPENAI_EXTRACT_MODEL / OPENAI_REASON_MODEL overrides to still use two different models. Pass both params (e.g. ?extract_provider=openai&provider=claude) to mix providers across steps instead.

If a recipient is available — ?to= or EMAIL_TO in .env — the brief is also emailed, rendered as simple HTML (headings/bold/links/lists) instead of raw markdown. ?email_provider=webde|gmail (default webde, or EMAIL_PROVIDER) picks the SMTP account to send from; email delivery failure is reported in the response (email_sent/email_error) but doesn't fail the request, since the brief itself was still generated.

Project layout

• main.go — loads .env, opens the database, and starts the API server.
• internal/feeds — fetches and parses Atom/RSS feeds.
• internal/database — SQLite persistence and queries for feeds, entries, and generated digests.
• internal/api — the HTTP API handlers built on top of internal/database.
• internal/llm — minimal clients for the Claude, OpenAI, and DeepSeek APIs.
• internal/digest — builds the daily-brief prompt from recent entries, calls an internal/llm.Client, and renders the result as HTML for email.
• internal/email — sends HTML email over SMTP, configured per provider (web.de, Gmail) via environment variables.
• internal/config — tiny .env file loader.