LlmBench

Kotlin Multiplatform workspace for using multiple AI services from one client without turning every provider into a separate installed app.

Status: early prototype. Android is the first client; shared domain/provider logic is moving to Kotlin Multiplatform so desktop and iOS clients can reuse the same core.

Rolling Android build: download the signed APK. The asset is replaced after each successful release build from main.

What it is

LlmBench is intended to combine three layers:

1. Web accounts — persistent WebView tabs for services where the user signs in with their normal account.
2. Free/native providers — a shared chat/compare surface for API-compatible free providers such as OpenRouter-style endpoints.
3. Optional AI tooling — reusable profiles/instructions from trvny/.ai without copying that repository into this one.

The first prototype already contains Compose UI, persistent per-provider WebViews, a native comparison chat, profile/instruction rendering, YAML editing and a small skills/docs browser.

On Android, LlmBench also appears in the system share sheet for text, images and application files. Shared content is routed to a chosen web provider; text uses the existing focused-empty-composer bridge, while granted content URIs are staged for the provider's next compatible file chooser and require an explicit one-time confirmation before delivery to the embedded page.

Initial provider targets

Account-backed WebViews

• ChatGPT
• Claude
• Gemini
• DeepSeek
• Kimi
• Mistral Vibe (formerly Le Chat)
• Qwen
• Microsoft Copilot
• Z.ai
• Grok
• Character.AI
• Venice
• Meta AI

Native / free-provider layer

The native/free-provider layer supports OpenRouter Free and AIHubMix through the shared OpenAI-compatible gateway adapter. Direct Gemini, OpenAI and Claude chats preserve bounded provider-scoped history with native message roles and stream text incrementally with cancellable requests; interrupted partial replies are never replayed. Their model pickers refresh from each gateway's live catalog and keep only zero-cost text models, with bundled models as an offline fallback. Both gateways stay outside the default All Models comparison to avoid duplicate aggregator routing. All Models dispatches only to direct providers with configured API keys and reports API failures without substituting simulated answers. Provider-specific details belong behind adapters rather than being spread through UI code.

Provider support matrix

Full means the LlmBench-side integration is implemented; provider-side login or page changes can still affect an embedded web client. Partial calls out a known limitation rather than hiding it.

Provider / surface	Status	Authentication	Uploads	Activity tracking	Notes
ChatGPT Web	Full	Provider page in WebView	Yes	Generating + unread	Persistent session, provider-scoped mobile/desktop mode
Claude Web	Full	Provider page in WebView	Yes	Generating + unread	Persistent session and provider-scoped tweaks
Gemini Chat Web	Partial	Google sign-in may be blocked in embedded user-agents	Yes	Generating + unread	The chat surface is integrated, but fresh Google OAuth inside WebView is not a supported flow
DeepSeek Web	Full	Provider page in WebView	Yes	Generating + unread	Persistent session
Kimi Web	Full	Provider page in WebView	Yes	Generating + unread	Persistent session
Mistral Vibe Web	Full	Provider page in WebView	Yes	Generating + unread	Tracks the locale-independent square stop control in the composer
Qwen Web	Partial	Qwen-owned email/password + Google/GitHub sign-in surface; embedded not verified	Provider-documented image upload; embedded not verified	Not yet	Qwen documents image upload in Qwen Chat; embedded sign-in/upload flow and provider-specific activity tracking still need verification
Microsoft Copilot Web	Partial	Microsoft auth hosts stay in-provider; Google/Apple embedded sign-in not verified	Provider-documented; embedded not verified	Not yet	Official Copilot and Microsoft 365 entry aliases are bounded in-WebView; unrelated HTTPS navigation with a user gesture opens externally, while navigation without a gesture is blocked; embedded sign-in/upload flow and provider-specific activity tracking still need verification
Z.ai Web	Partial	Provider-owned email plus Google/GitHub sign-in surface; embedded not verified	Page-driven	Not yet	chat.z.ai plus Google/GitHub auth hosts are bounded in-WebView; unrelated HTTPS navigation with a user gesture opens externally, while navigation without a gesture is blocked; embedded sign-in/upload flow and provider-specific activity tracking still need verification
Grok Web	Partial	Provider page in WebView	Provider-documented; embedded not verified	Not yet	xAI documents multi-file upload on the web; embedded upload flow and provider-specific activity tracking still need verification
Character.AI Web	Partial	Provider page in WebView	Provider-documented image attachments; embedded not verified	Not yet	Character.AI documents image attachments in chats; embedded sign-in/upload flow and activity probe still need verification
Venice Web	Partial	Provider page in WebView	Provider-documented; embedded not verified	Not yet	Venice documents file uploads in the chat input; embedded sign-in/upload flow and provider-specific activity tracking still need verification
Meta AI Web	Partial	Meta login surface is provider-owned	Not verified	Not yet	alpha.meta.ai is a verified provider-owned login alias; embedded sign-in, chat uploads and activity tracking still need verification
OpenRouter Free	Native gateway	API key	N/A	Native request state	Uses openrouter/free; excluded from default All Models compare
AIHubMix Free	Native gateway	API key	N/A	Native request state	Uses explicit -free models; excluded from default All Models compare

Verification references for the newer web providers: Microsoft Copilot entry points, Microsoft Copilot file upload, Grok files FAQ, Venice upload changelog, Character.AI image attachments, Qwen VLo image upload in Qwen Chat, the provider-owned Qwen sign-in surface, the provider-owned Z.ai sign-in surface, and the provider-owned Meta AI login surface. These verify provider capabilities or owned hosts, not Android WebView login compatibility or stable generation DOM selectors.

Google documents the embedded-user-agent restriction in its OAuth 2.0 policies.

WebView approach

Account sessions persist, but LlmBench must not keep every heavy provider SPA alive forever. The Android host uses a small LRU pool, pauses inactive WebViews and evicts them under memory pressure while cookies/session state remain provider-owned. Provider tweaks live in a small, auditable in-app registry: scripts are static, scoped to the matching provider host and applied after page load; remote userscript code is never fetched.

Provider diagnostics expose only the provider host, WebView package/version, capability counts, activity-tracking support and file-picker events. They never collect page text, full URLs, form values, file names, cookies or authentication tokens.

LlmBench must not scrape passwords, session cookies, OAuth tokens or other login credentials. Authentication remains between the embedded provider page and that provider.

Relationship to .ai

trvny/.ai remains the canonical portable AI configuration core. This repository contains the multiplatform LlmBench core plus platform clients; Android is the first shipping client.

trvny/.ai             reusable profiles / instructions / skills
      │
      └── optional consumption
              │
              ▼
twojstar/llmbench      shared KMP core + platform clients

Do not vendor a second copy of .ai here. If runtime integration becomes useful, consume a pinned/exported representation with an explicit boundary.

Architecture direction

shared (Kotlin Multiplatform)
├── provider/model registry
├── profile + prompt tools
└── portable domain logic

platform clients
├── Android app
│   ├── WebView host + file chooser
│   ├── provider tweaks / mobile performance
│   └── Custom Tabs / intents / Keystore
├── Desktop app (planned)
└── iOS app (planned)

Technical details for native provider transport, conversation state, Prompt Studio boundaries and API-level TODOs live in docs/provider-runtime.md. Product/workflow ideas collected from app research and sibling tools live in docs/product-ideas.md.

Backends are optional, not the default. If a feature truly needs one, prefer a tiny stateless service and evaluate Cloudflare, Google Cloud, AWS or Oracle free tiers based on the actual requirement rather than choosing infrastructure first.

No account wall: LlmBench itself must remain useful without an LlmBench account, cloud sync or hosted backup. Provider logins/API keys are required only for the providers the user explicitly chooses. Local chats, projects, files, tools, settings, import/export and manual backup/restore stay available locally. Any future LlmBench cloud/sync/integration account is additive and opt-in, never a prerequisite for local features.

Near-term roadmap

• remove generated/build-machine files from version control
• normalize app name, namespace and application ID to LlmBench
• harden WebView security while preserving provider login compatibility
• migrate portable domain/provider logic to KMP shared
• add mobile WebView LRU/memory-pressure handling for long chats
• implement reliable provider file uploads through the platform file picker
• apply rendered Studio instructions to an empty focused web composer with clipboard fallback
• add AIHubMix and OpenRouter-compatible free-provider gateways
• create a provider-tweak/userscript interface instead of hard-coded WebView hacks
• add privacy-safe provider diagnostics for embedded capability verification
• show generating and unread response status on ChatGPT, Claude, Gemini, DeepSeek, Kimi and Vibe web tabs
• add Qwen, Microsoft Copilot, Z.ai, Grok, Character.AI, Venice and Meta AI account-backed WebView entries
• verify embedded sign-in, embedded upload flows and provider-specific generation activity probes for Qwen, Copilot, Z.ai, Grok, Character.AI, Venice and Meta AI
• add identity-assisted provider onboarding with a preferred Google/GitHub/Microsoft sign-in path, verified provider capabilities and browser-backed auth where needed, without a mandatory LlmBench account or cookie/token copying
• add a local Markdown prompt vault with edit/import/export and chat-to-.md workflows
• add manually editable/importable SKILL.md assets with safe capability gating
• bring Docbench's format validation/repair, EOL normalization, local tokenizer and hidden-text/prompt-smuggling inspector into prompt/file tooling
• add first-party Docbench/Codebench tools/plugins for compatible chats, plus a persistent Streambench radio player as first-party companion UI
• modernize chat/navigation UX with current stable Material 3 Adaptive, measured Compose performance work and large-screen list-detail layouts
• add provider-aware chat/message widgets and notification replies, including real Direct Reply where the transport can safely send in background
• treat security/privacy as a cross-cutting release gate: sensitive storage/backup rules, WebView isolation, tool permissions, redacted widgets/notifications, safe exports and security regression tests
• add CI build/lint checks
• document which providers work fully, partially, or block embedded login

Development

Android targets API 35 with minSdk 26. The shared core uses Kotlin Multiplatform; platform UI remains Compose-first and moves into Compose Multiplatform only where it does not weaken native WebView, upload, authentication or secure-storage behavior.

Mobile UX is a product constraint: long chats must stay responsive, file upload must work, and provider tweaks should reduce wasted chrome/animation without breaking provider pages. Native API keys are encrypted at rest with an Android Keystore-backed AES-GCM key; credentials are never logged or exported.

License

ISC, see LICENSE.