✨ Chexsa

Your personal AI that can use your computer.

For a simple explanation of every project file, read file_use.md.

Chexsa is an open-source AI assistant that is being built to see what is happening on your computer and take actions for you.

Instead of only answering questions, Chexsa can work through a task step by step.

You tell Chexsa what you want.
Chexsa figures out what to do next and gets to work.

---

🌸 What can Chexsa do?

Right now, Chexsa can use your browser.

You can ask it things like:

Open Gemini and ask what the capital of the USA is.

Open YouTube and search for a Java tutorial.

Go to this website and fill out the form.

Chexsa looks at the page, decides what to do, performs the action, and looks again.

You
 ↓
Chexsa
 ↓
See what is there
 ↓
Decide what to do
 ↓
Do it
 ↓
Repeat

No separate script needs to be written for every website.

---

🖥️ Desktop control

Chexsa is also learning how to use normal Windows apps.

The goal is to support requests like:

Open Word and write a paragraph about AI.

Open my document and add a new section.

Chexsa can already read many Windows buttons, menus, text boxes, and windows. Desktop actions are still being developed.

---

🚧 Current progress

Feature	Status
Terminal chat	✅
Open websites	✅
Click and type	✅
Search websites	✅
Work with browser tabs	✅
Scroll pages	✅
Upload files	✅
Read webpage content	✅
"Hey Chexsa" voice commands	✅
Read Windows apps	🚧
Control Windows apps	🚧
Browser + desktop together	🔜
Choose chat, browser, or desktop from your request	✅
Check its own work	✅ Browser verification
Learn repeated tasks	🔜

Chexsa is still an early project, so some things will break while it grows.

---

💡 The idea

Most AI assistants work like this:

You ask something
      ↓
AI answers

Chexsa is trying to work like this:

You ask for something
        ↓
Chexsa understands the goal
        ↓
Chexsa uses your computer
        ↓
Task complete

The bigger goal is a personal AI that can work across websites, apps, and files instead of being limited to a chat box.

---

🌱 Where Chexsa is going

A future goal is for Chexsa to learn from tasks it has already completed.

Instead of figuring out the same workflow from scratch every time, it could save what worked and reuse it later.

First time
Think through the task
        ↓
Task succeeds
        ↓
Save what worked
        ↓
Next time
Reuse it

This could make repeated tasks faster and more reliable.

---

🛠️ Getting Started

Chexsa currently works best on Windows.

1. Clone it

git clone https://github.com/MashrafeeAryan/Chexsa.git
cd Chexsa

2. Install it

pip install -e .

3. Add your AI provider

Create a .env file.

For an OpenAI-compatible provider:

LLM_PROVIDER=openai_compatible
LLM_MODEL=your-model
LLM_API_KEY=your-api-key
LLM_BASE_URL=your-api-url

Or use Gemini:

LLM_PROVIDER=gemini
LLM_MODEL=your-model
GEMINI_API_KEY=your-api-key

Never upload your API keys to GitHub.

4. Start Chexsa

chexsa

Chexsa first waits for your request. Greetings and normal questions stay in chat. Browser tasks connect to Chrome; Windows app tasks use the desktop agent. Voice mode makes the same choice after hearing your command.

For a browser task, Chexsa reuses Chrome's debugging connection on port 9222. A normal Chrome window without that connection cannot be attached automatically. If needed, Chexsa starts a separate persistent profile under %LOCALAPPDATA%/Chexsa/chrome and skips Chrome's first-run setup.

Chrome account sign-in/sync is not required. Sign into Gmail or other websites once in the Chexsa profile; their saved login state is reused across sessions, subject to the website's normal expiry and security checks. This profile does not copy logins from your usual Chrome profile and does not use Guest mode.

Type a task at You >. Ctrl+C cancels a running task. Exit with exit or quit. Chrome stays open when Chexsa disconnects. If the selected tab closes, the next observation selects an open tab or creates a blank one. Old element refs are rejected, so a pending action cannot silently hit a different tab.

List saved tasks or resume after observing the current browser:

chexsa --tasks
chexsa --resume TASK_ID

The browser engine keeps large pages locally and sends focused regions to the model. It supports node-bound refs, search/expansion, visual fallback, verified results, and recovery without automatically repeating uncertain submissions. See ARCHITECTURE.md for settings and behavior.

---

🎙️ Voice Mode

Chexsa can listen for "Hey Chexsa" and route the rest of the sentence to chat, the browser, or the Windows desktop, just like terminal mode.

Install the optional voice tools:

pip install -e ".[voice]"

Start voice mode:

chexsa-voice

Or start Chrome and voice mode together:

.\scripts\start_voice.ps1

Then say:

Hey Chexsa, search JavaScript tutorials on YouTube.

You can also say only "Hey Chexsa" and Chexsa will listen once more for your command.

Voice mode must be running to hear the wake phrase. Speech is currently transcribed with Google Speech Recognition through the SpeechRecognition package.

---

✅ Tests

Run the fast unit tests:

python -m unittest discover -s tests -p "test_*.py" -v

Run the isolated Chrome integration fixtures (no model key or real email needed):

.\scripts\test_browser.ps1

GitHub Actions runs the unit and Chrome fixture tests on Python 3.11 and 3.12 for pushes and pull requests. The fixture suite includes 20 consecutive email workflows, delayed/refused/uncertain sends, large pages, frames, and visual targeting. It uses a scripted model; live Gmail and model-provider behavior require separate account-based validation. Version tags such as v0.1.0 also build release packages automatically.

---

🗺️ Roadmap

Browser control              ✅
        ↓
Windows app control          🚧
        ↓
Browser + desktop together
        ↓
Check whether browser actions worked ✅
        ↓
Recover when something fails
        ↓
Learn reusable tasks
        ↓
Personal memory

---

⚠️ Early Project

Chexsa is experimental.

It can interact with real websites and is being developed to control more of your computer. Avoid using it for anything where a wrong action could cause serious problems.

---

🤝 Contributing

Chexsa is open source and still growing.

Bug reports, ideas, experiments, and pull requests are welcome.

---

🌸 Chexsa

An AI that doesn't just tell you what to do.
It helps do it.