codai phone

An AI assistant that can actually use your Android phone for you.

You type (or say) what you want in plain language — "open WhatsApp and tell me what Ana wrote", "set an alarm for 6:30", "find the PDF I downloaded yesterday" — and codai taps, types and reads the screen the way you would. It asks before doing anything risky, shows you every step, and the same conversation can be followed or steered from your other devices.

This repository is the public mirror of the Android app at apps/phone-android in the codai monorepo. The app is Apache-2.0; the codai cloud service it talks to is a separate, hosted product.

Screenshots are on their way — docs/img/ is a placeholder for now.

---

What can it do?

Everything below is in the app today. Anything not shipped yet is marked Planned.

• Use other apps for you. codai sees the screen the way a screen reader does (a list of buttons, texts and fields), then taps, types, scrolls and swipes. Say: "Open WhatsApp and tell me the last message from Ana."
• Answer questions about your phone. Read notifications, list alarms, search contacts, check calendar events, read recent SMS, tell you which app is in front, check Bluetooth or battery. Say: "Do I have any alarms set for tomorrow?"
• Do small chores. Set alarms and timers, add calendar events, open a link, share text, copy to the clipboard, reply to a notification, start a phone call. Say: "Add 'dentist' to my calendar on Friday at 10."
• Work with files. List, search, read, edit and move files in your storage. Say: "Find the invoice PDF I downloaded yesterday."
• Run commands in a terminal. A companion terminal app (codai.term) gives codai a real Linux-style shell (Termux-compatible) — handy for developers, invisible if you never need it.
• Work offline for small things. An optional on-device model (Gemma 4 E2B via LiteRT-LM, plus our fine-tuned codai-nano variants) can handle simple steps on the phone itself and hands over to the cloud when a task is too hard. You choose the mode: off, fast-lane or local-first.
• Ask before risky actions. Sending a message, paying, or touching a banking / settings / password-manager app pops up an ask card. Nothing is sent or bought behind your back.
• Choose how much freedom to give it. Three autonomy levels: strict (confirm every action), assisted (default — confirm sensitive ones), autopilot (only confirm sends and payments).
• Run several tasks at once. Multiple sessions, each in its own window (split-screen / pop-up on Samsung One UI). Tasks that need the screen wait politely in a visible queue; the others run in parallel.
• Follow along from another device. With Session sync on, the same conversation is mirrored live to your other phones and the web console. Devices that are not the one doing the work show a REMOTE badge, and you can be a viewer or an editor.
• Start a task on your phone from somewhere else. Dispatch hands a task to a named device; the phone wakes up via push (Firebase Cloud Messaging), picks up the task and does the work — even with the screen off. You can turn this off in Settings › Background ("Remote dispatch"). Builds without a Firebase google-services.json simply skip the push part.
• See what it cost. A "Cost · Receipt" sheet shows the server's receipt for a session, and you can set a per-session spending budget in Settings.
• Bring your own AI key. Use codai's cloud, or plug in OpenAI, Anthropic, Google Gemini, OpenRouter, GitHub Copilot or any OpenAI-compatible endpoint. Keys stay on the device and go only to the provider you picked.
• Extras. A floating bubble, a mini-chat, a home-screen widget, a Quick Settings tile, voice in/out (Android speech recognition + text-to-speech), and a daily digest.

How it works

Three steps — you, the app, and either your screen or the cloud:

flowchart LR
    You([You: "Open WhatsApp and read Ana's message"]) --> App[codai phone app]
    App -- reads & taps via Accessibility --> Screen[Your phone's screen & apps]
    App -- only when needed --> Gateway[(codai gateway<br/>or your own AI key)]
    Gateway --> App
    Screen --> App
    App --> You2([You: see each step, approve risky ones])

Loading

1. You ask. In the chat, by voice, from the bubble, or from another device.
2. codai thinks. Small steps can run on the on-device model; harder ones go to the codai gateway (or the AI provider you chose).
3. codai acts. It reads the screen, taps, types, checks the result, and repeats — showing you every step and stopping to ask before anything risky.

When Session sync is on, the same conversation is shared across your devices:

flowchart LR
    Phone[Your phone<br/>executor — does the work] <--> GW[(codai gateway<br/>session log + roles)]
    GW <--> Tablet[Your tablet<br/>REMOTE · viewer]
    GW <--> Web[Web console<br/>REMOTE · editor]
    Web -. "send / answer / cancel" .-> GW

Loading

Only one device — the executor — drives the phone at a time. Everyone else sees the live transcript with a REMOTE badge; editors can send messages, answer questions and cancel; viewers just watch.

Privacy & safety

What stays on the phone

• Your API keys and sign-in (stored in Android's encrypted preferences).
• The screen contents codai reads, unless a step needs the cloud model.
• Everything the on-device model handles in local-first / fast-lane mode.
• Per-turn trace files, if you enable them for debugging (written to your Downloads folder, never uploaded automatically).

What leaves the phone

• Your prompt, the screen summary and tool results go to the AI backend you configured — the codai gateway by default, or your own provider key.
• If Session sync is on (Settings → Advanced), the step-by-step transcript is mirrored to the codai gateway so your other devices can see it.
• If a build includes Firebase (google-services.json), the push token used to wake the phone for Dispatch.

What never happens

• No analytics SDK, no ad SDK, no crash reporter bundled. We do not sell data.
• No message is sent, no payment made, no settings changed in a bank / wallet / password app without an ask card you tap first. Messaging apps default to "ask every time"; you can set them to "block" entirely.
• You can hit Stop at any time; a turn also has a time budget (4 minutes by default) after which it stops on its own.

About the Accessibility permission — honestly

To tap and read other apps, Android requires an Accessibility Service. That permission is powerful: while it is on, codai can see what is on screen (including text in fields) and act on it. We only use it while a task is running, we ask before sensitive actions, and the whole code is here for you to inspect. If that is not a trade-off you want, you can still use codai as a plain chat app without granting it — you just lose phone control.

More: docs/faq.md · SECURITY.md.

Install

Step-by-step guide: docs/getting-started.md.

1. Download the latest .apk from Releases and open it. Android 11 or newer. Allow "install from this source" if asked.
2. Sign in with your codai account (email/password or Sign in with Google) — or skip and paste an API key under Settings → Providers.
3. Turn on the Accessibility service when onboarding asks (Settings → Accessibility → Installed apps → codai → On). Optional: floating bubble (overlay), notification access, battery exemption, Shizuku for extra tools.
4. Optional: download the on-device model (Settings → Models). About 2.6 GB for Gemma 4 E2B; works best on recent flagship phones.

Google Play distribution is Planned; today the app ships as an APK.

Shared sessions

The phone is one executor for a session protocol that any client can speak — the web console, the desktop app, a CLI or your own agent. Roles are owner / editor / viewer; sessions can be shared with your other devices, with a team, or via an expiring link. The protocol is documented publicly in codai-ro/codai-protocol; the phone depends only on that document, never on server internals.

For developers

Architecture, debugging and contribution details live in docs/: architecture.md · debugging.md · CONTRIBUTING.md.

Build from source

Requirements: JDK 17, Android SDK with compileSdk 37 (AGP 9.x has built-in Kotlin — do not apply org.jetbrains.kotlin.android), NDK 28.2 + CMake 3.22.1 for :terminal. Android Studio works; so does the command line.

# local.properties (gitignored)
sdk.dir=C:\\Users\\you\\AppData\\Local\\Android\\Sdk

.\gradlew.bat --no-daemon :app:assembleRelease      # APK
.\gradlew.bat --no-daemon :app:testReleaseUnitTest  # JVM unit tests

Without keystore.properties the release build signs with the debug key, so forks and CI build out of the box. One lint line about ExpiredTargetSdkVersion on :terminal is expected (targetSdk 28 is deliberate so the terminal can execute downloaded binaries).

local.defaults.properties holds two public OAuth client ids compiled into BuildConfig: CODAI_GITHUB_CLIENT_ID (Copilot device-flow sign-in) and CODAI_GOOGLE_SERVER_CLIENT_ID (Sign in with Google). Override with -P<name>=… or in local.properties. Self-hosting the auth server? Point the Google id at your own web client or Google sign-in is rejected server-side.

google-services.json (optional) enables push wake for Dispatch. Drop your own Firebase file at app/google-services.json; without it the build logs building without FCM push and everything else works.

Self-hosted gateway: Settings → Advanced → Gateway URL (default https://ai.codai.ro). Any server implementing OpenAI-compatible /v1/chat/completions plus the shared-sessions endpoints (/v1/sessions*, /v1/devices) works.

Project layout

Path (app/src/main/java/ro/codai/phone/)	What lives there
agent/	Turn loop, tools, risk/permission rules, time budgets, encrypted Settings
session/	SessionRegistry / AgentRun (N concurrent sessions), ScreenArbiter
sync/	Gateway mirror: events up, controls down, SSE resume, lease heartbeat, push
provider/	Wire adapters: OpenAI-compatible, Anthropic, Gemini, Copilot device flow
account/	Sign-in (email/password, Google), entitlements, billing
nano/	On-device LLM (LiteRT-LM), model store, local agent, DebugDrive
a11y/	The Accessibility service and notification listener
surface/	Bubble, mini-chat, widget, Quick Settings tile, voice, assistant entry points
ui/	Jetpack Compose: chat, sessions drawer, ask cards, receipt, settings hub
terminal/ (app) + :terminal (module)	Companion terminal client + the codai.term PTY/bootstrap APK

Roadmap

• End-to-end encrypted sessions with the HIDE protocol (hybrid post-quantum HPKE + signatures); the server would store only ciphertext. Planned — HIDE has no Android binding yet; that is the first step.
• Dispatch on the phone (wake via push, run a task started elsewhere). Planned; the server side is live.
• Desktop app over the same protocol: codai-ro/codai-desktop (Tauri 2).
• Offline voice (on-device speech-to-text and text-to-speech). Planned.
• Google Play listing. Planned.
• iOS — not planned at the moment; iOS has no equivalent of Android's Accessibility automation.

Contributing

Bug reports, ideas and pull requests are welcome — you do not need to be a developer to file a good bug. See CONTRIBUTING.md (DCO sign-off, conventional commits, run the unit tests) and the issue templates.

Security

Found a vulnerability? Please do not open a public issue. Email [email protected] — we acknowledge within 72 hours and follow a 90-day coordinated disclosure. Details in SECURITY.md.

License

Apache-2.0 — see LICENSE and NOTICE.

"codai" is a trademark of Dragos Catalin Vladulescu. You may fork this code under Apache-2.0, but a redistributed build must use its own name, icon and applicationId. The on-device models derive from Gemma 4 and are distributed under the Gemma Terms; they are downloaded at runtime, never bundled here.