Offline Chit Chat: Running AI on a Potato

Device: Vivo 1820 (2GB RAM)
OS: Android 8.1 (Oreo)
Model: Gemma3 270M IT (Quantized)
PROCESSOR: Helio P22 (64-bit)

Model Download Link

Gemma 3 270M IT GGUF

Status: RUNNING Download the apk File

Live Demo

demo_video.mp4

---

The Story

Hi there! This isn't just a regular Flutter app. This represents a stubborn developer's refusal to accept "your device is too old" as an answer.

I wanted to run a Local Large Language Model (LLM) entirely offline. No internet, no APIs, just raw compute on my old Vivo 1820. I tried a lot of things. And oh boy, did it crash. A lot.

The Detective Work (The spy in my blood) ------

I started with the standard llama_cpp_dart package. It compiled, installed, and... SIGSEGV. Instant crash. The app would just vanish the moment I tried to load the model.

Most people would assume "out of memory" on a 2GB phone. But the logs showed something weirder. It was crashing before the model even loaded into RAM. It was crashing during the "handshake" between my Dart code and the C++ engine.

The "Blueprint" Bug ::)

I dug into the native C++ code (llama.h) and compared it with the Dart FFI (Foreign Function Interface) bindings. It was like comparing two blueprints for the same house, but one was missing a room.

The Dart code was allocating a memory block for llama_context_params that was too small. It didn't know about the new samplers field that the C++ engine expected. So when the C++ engine tried to write its default settings effectively...

1. It filled the struct.
2. It kept writing past the end of the allocated memory.
3. BOOM. Memory corruption.

The Fixes

I had to manually patch the library on my machine to fix this. Here's exactly what I did to make this potato fly:

1. Patched the FFI Bindings: I physically edited llama_cpp.dart to add the missing samplers and use_direct_io fields. This aligned the "blueprints" perfectly.
2. Forced 32-bit Architecture: My phone is old. Modern libraries default to 64-bit. I had to strictly filter the ndk configuration to armeabi-v7a and arm64-v8a to stop it from trying to load incompatible x86 libraries.
3. The "Android 8.1" Special: Older Androids handle shared libraries differently. I had to manually ensure libomp.so (OpenMP) was loaded explicitly before the main library, or the linker would give up.

Device Information

How to Run This

If you want to run this code (especially on an older device)like mine, you can't just flutter run. You need my specific patches.

The Patch

Check out the offline_llm_integration_guide.md in this repo for the exact lines of code you need to change in the llama_cpp_dart package.

The Result [End is the Start]

It's not ChatGPT-4o fast. It takes a few seconds to think. But it works. It's a neural network thinking inside a 7500 rupee phone from 2020.

And that, to me, is magic.

This project is running on Vivo 1820, proving that AI can run anywhere.

---

• By a curious 'me its me' who likes fixing things.*