Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 12 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,8 +73,10 @@ The touchscreen pages provide:

Mic records from the Mac only while the button is held. To change modes, open
**Menu → Switch Buddy**; the device restarts without erasing pairing data.
Codex plays distinct cues when an Agent needs input, completes, or errors. Use
**Menu → Sound** to mute or unmute all notification sounds.
Codex plays distinct cues when an Agent needs input, completes, or errors. A
needs-input notice closes by itself once you answer the Agent on the Mac;
complete and error notices stay until tapped. Use **Menu → Sound** to mute or
unmute all notification sounds.

### Codex troubleshooting

Expand Down Expand Up @@ -110,9 +112,9 @@ Claude mode includes:
face-down screen sleep. Claude error cues require an error-bearing turn event.

Place the device screen-up and flat once after startup to calibrate its
orientation. Touch or a permission request wakes the display. Use **Info →
Sound** to change the shared persisted mute setting, or **Info → Switch Buddy**
to return to the startup selector.
orientation. Touch, picking the device up, or a permission request wakes the
display. Use **Info → Sound** to change the shared persisted mute setting, or
**Info → Switch Buddy** to return to the startup selector.

Pairing is stored on the device and normally survives restarts. Pair again if
you forget the device on the computer, erase NVS, or change its BLE identity.
Expand All @@ -128,12 +130,13 @@ Claude mode uses an experimental developer API; see

Normal operation uses Bluetooth and does not require a USB cable.

After inactivity, both modes dim the backlight after 15 seconds and turn the
LCD panel off after one minute while keeping Bluetooth connected. Touch wakes
After inactivity, both modes dim the backlight to a low but readable level
after 15 seconds and turn the LCD panel off after one minute while keeping Bluetooth connected. Touch wakes
the display; the first touch from off is consumed to avoid activating a hidden
control. Permission prompts, pairing passkeys, waiting Claude sessions, and
Codex Agents requiring input immediately restore and hold normal brightness
until the action is resolved.
Codex Agents requiring input immediately restore normal brightness and hold it
for up to one minute while the action is unresolved; the display then turns off
as usual until the next touch.

## Development

Expand Down
15 changes: 9 additions & 6 deletions components/buddy_ui/buddy_ui.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -62,14 +62,17 @@ static void rect(Canvas *canvas, int x, int y, int width, int height,
canvas->pixels[row * Width + column] = color;
}

/* Rasterizes a filled circle into the canvas. */
/* Rasterizes a filled circle as one clipped horizontal span per row. */
static void circle(Canvas *canvas, int center_x, int center_y, int radius,
uint16_t color)
{
for (int y = -radius; y <= radius; ++y)
for (int x = -radius; x <= radius; ++x)
if (x * x + y * y <= radius * radius)
rect(canvas, center_x + x, center_y + y, 1, 1, color);
int half = radius;
for (int y = 0; y <= radius; ++y) {
while (half * half + y * y > radius * radius) --half;
rect(canvas, center_x - half, center_y + y, 2 * half + 1, 1, color);
if (y != 0)
rect(canvas, center_x - half, center_y - y, 2 * half + 1, 1, color);
}
}

/* Rasterizes a one-pixel line using an integer error accumulator. */
Expand Down Expand Up @@ -160,7 +163,7 @@ static void centered(Canvas *canvas, const char *value, int center_x, int y,
color);
}

/* Truncates text to a pixel width before drawing it into a bounded row. */
/* Truncates text to a character count before drawing it into a bounded row. */
static void clipped_text(Canvas *canvas, const char *source, int x, int y,
size_t characters, uint16_t color)
{
Expand Down
5 changes: 4 additions & 1 deletion components/claude_model/claude_model.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,10 @@ PetState petState(const Model &model, std::uint32_t nowMs) noexcept
{
if (model.connection != Connection::Connected) return PetState::Sleep;
if (model.faceDown) return PetState::Sleep;
if (model.transientUntilMs != 0 && nowMs < model.transientUntilMs)
/* Uptime milliseconds wrap after about 49.7 days; compare the signed
distance so a transient started before the wrap still expires. */
if (model.transientUntilMs != 0 &&
static_cast<std::int32_t>(model.transientUntilMs - nowMs) > 0)
return model.transientState;
if (model.promptActive || model.waitingSessions > 0) return PetState::Attention;
if (model.runningSessions > 0) return PetState::Busy;
Expand Down
4 changes: 3 additions & 1 deletion components/claude_protocol/claude_protocol.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -238,7 +238,9 @@ bool commandResponse(std::string_view json, const Parser &parser, int command,

void Decoder::reset() noexcept
{
line_.fill('\0');
/* line() is bounded by length_, so clearing the whole 6 KiB buffer after
every message would be wasted work. */
line_[0] = '\0';
length_ = 0;
discarding_ = false;
}
Expand Down
3 changes: 2 additions & 1 deletion components/codex_ble_transport/codex_ble_transport.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -230,7 +230,8 @@ esp_err_t sendJson(std::string_view json) noexcept
{
if (json.empty()) return ESP_ERR_INVALID_ARG;
if (!connected()) return ESP_ERR_INVALID_STATE;
std::array<buddy::codex::Report, 8> reports{};
std::array<buddy::codex::Report, buddy::codex::MaxReportsPerMessage>
reports{};
std::size_t report_count{};
const buddy::codex::Result encoded = buddy::codex::encodeReports(
json, reports, report_count);
Expand Down
22 changes: 12 additions & 10 deletions components/codex_controller/codex_controller.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -83,16 +83,18 @@ bool handleAction(Model &model, const Action &action, ActionPhase phase,
sendKey(transport, key, buddy::codex::KeyAction::Step);
}
const bool pressed = phase == ActionPhase::Press;
if (!sendKey(transport, key, pressed ? buddy::codex::KeyAction::Press
: buddy::codex::KeyAction::Release))
return false;
if (key == buddy::codex::Key::Mic) {
if (pressed) (void)applyEvent(model, Event{MicPressed{}});
else if (phase == ActionPhase::Cancel)
(void)applyEvent(model, Event{InputCancelled{}});
else (void)applyEvent(model, Event{MicReleased{}});
}
return true;
const bool sent = sendKey(
transport, key, pressed ? buddy::codex::KeyAction::Press
: buddy::codex::KeyAction::Release);
if (key != buddy::codex::Key::Mic) return sent;
if (pressed)
return sent && applyEvent(model, Event{MicPressed{}});
/* Always leave the local Listening state, even when the release
report is lost; otherwise its overlay would block all input. */
const bool changed = phase == ActionPhase::Cancel
? applyEvent(model, Event{InputCancelled{}})
: applyEvent(model, Event{MicReleased{}});
return sent || changed;
}
}
return false;
Expand Down
8 changes: 8 additions & 0 deletions components/codex_model/codex_model.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,14 @@ bool applyEvent(Model &model, const Event &event) noexcept
transitionOverlay != Overlay::None) {
model.notification = transitionOverlay;
model.notificationSlot = slot;
} else if (model.notification == Overlay::RequiresInput &&
model.notificationSlot == slot &&
current.status != SlotStatus::RequiresInput) {
/* The request was answered on the Mac, so the prompt to check
it is stale. Complete and Error notices stay until tapped
because they report an outcome the user may not have seen. */
model.notification = Overlay::None;
model.notificationSlot = NoSlot;
}
return true;
},
Expand Down
3 changes: 2 additions & 1 deletion components/codex_protocol/codex_protocol.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,8 @@ Result encodeReports(std::string_view json, std::span<Report> reports,

void Decoder::reset() noexcept
{
json_.fill('\0');
/* json() is bounded by length_, so only the terminator needs clearing. */
json_[0] = '\0';
length_ = 0;
scanOffset_ = 0;
depth_ = 0;
Expand Down
6 changes: 6 additions & 0 deletions components/codex_protocol/include/codex_protocol.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,12 @@ inline constexpr std::uint8_t ReportId = 6;
inline constexpr std::size_t ReportBodySize = 63;
inline constexpr std::size_t PayloadSize = 61;
inline constexpr std::size_t RpcBufferSize = 4096;
/* Longest device-to-host JSON message. The RPC response buffer and the BLE
transport's report array are both sized from this single limit. */
inline constexpr std::size_t MaxMessageSize = 511;
/* Reports needed for MaxMessageSize bytes plus the trailing newline. */
inline constexpr std::size_t MaxReportsPerMessage =
(MaxMessageSize + 1 + PayloadSize - 1) / PayloadSize;

enum class Key : std::uint8_t {
Agent1,
Expand Down
3 changes: 2 additions & 1 deletion components/codex_rpc/include/codex_rpc.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,8 @@ namespace buddy::codex {

inline constexpr const char *FirmwareVersion = "1.0.0-cores3";
inline constexpr std::size_t MaxEvents = SlotCount;
inline constexpr std::size_t ResponseSize = 512;
/* Holds the longest sendable message plus its null terminator. */
inline constexpr std::size_t ResponseSize = MaxMessageSize + 1;

struct RequestContext {
const Model *model{};
Expand Down
14 changes: 9 additions & 5 deletions components/codex_ui/codex_ui.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -58,14 +58,18 @@ static void fill_rect(Canvas *canvas, int x, int y, int width, int height,
canvas->pixels[row * Width + column] = color;
}

/* Rasterizes a filled circle with clipping delegated to the rectangle helper. */
/* Rasterizes a filled circle as one clipped horizontal span per row. */
static void fill_circle(Canvas *canvas, int center_x, int center_y,
int radius, uint16_t color)
{
for (int y = -radius; y <= radius; ++y)
for (int x = -radius; x <= radius; ++x)
if (x * x + y * y <= radius * radius)
fill_rect(canvas, center_x + x, center_y + y, 1, 1, color);
int half = radius;
for (int y = 0; y <= radius; ++y) {
while (half * half + y * y > radius * radius) --half;
fill_rect(canvas, center_x - half, center_y + y, 2 * half + 1, 1, color);
if (y != 0)
fill_rect(canvas, center_x - half, center_y - y, 2 * half + 1, 1,
color);
}
}

/* Builds a filled rounded rectangle from rectangular bands and corner circles. */
Expand Down
4 changes: 3 additions & 1 deletion components/platform_core_s3/include/platform_core_s3.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,9 @@ struct BatteryStatus {
/* Returns the shared 320 x 240 RGB565 framebuffer allocated in PSRAM. */
[[nodiscard]] std::span<std::uint16_t> framebuffer() noexcept;

/* Transfers the entire RGB565 framebuffer and waits until DMA is finished. */
/* Transfers the entire RGB565 framebuffer and waits until DMA is finished.
The buffer is left in panel byte order, so callers must redraw every pixel
before presenting again. */
[[nodiscard]] esp_err_t present() noexcept;

/* Coordinates the LCD panel with normal, dimmed, and off backlight levels. */
Expand Down
32 changes: 20 additions & 12 deletions components/platform_core_s3/platform_core_s3.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,16 @@
namespace buddy::platform {

static const char *TAG = "platform_core_s3";
static constexpr int NormalBrightness = 15;
static constexpr int DimmedBrightness = 2;
/* The CoreS3 backlight is powered by AXP2101 DLDO1, so brightness is set by
its output voltage: register step n gives 0.5 V + n * 0.1 V. The BSP's
usable range is steps 20-28 (2.5-3.3 V); it calls anything lower "too dark",
which is what the dimmed idle state wants. Normal keeps the previous 2.6 V.
Dimmed was previously 2.5 V, only one step lower and hardly visible, so it
now uses 2.4 V. Step 18 (2.3 V) saves more if the screen stays readable on
a given unit; step 20 or above saves very little. */
static constexpr std::uint8_t NormalBacklightStep = 21;
static constexpr std::uint8_t DimmedBacklightStep = 19;
static constexpr std::uint8_t MaximumBacklightStep = 28;
static esp_lcd_panel_handle_t s_panel;
static esp_lcd_panel_io_handle_t s_panel_io;
static esp_lcd_touch_handle_t s_touch;
Expand Down Expand Up @@ -89,22 +97,21 @@ static esp_err_t add_power_devices() noexcept
return i2c_master_bus_add_device(bus, &config, &s_io_expander);
}

static esp_err_t set_backlight(int brightness_percent) noexcept
/* Sets DLDO1 to a voltage step, or disables the LDO entirely for step 0. */
static esp_err_t set_backlight(std::uint8_t step) noexcept
{
if (s_pmu == nullptr) return ESP_ERR_INVALID_STATE;
if (brightness_percent <= 0) {
if (step == 0) {
esp_err_t error = update_register(
s_pmu, PmuLdoEnableRegister, PmuBacklightEnable, false);
if (error != ESP_OK) return error;
return write_register(s_pmu, PmuBacklightVoltageRegister, 0);
}
if (brightness_percent > 100) brightness_percent = 100;
if (step > MaximumBacklightStep) step = MaximumBacklightStep;
esp_err_t error = update_register(
s_pmu, PmuLdoEnableRegister, PmuBacklightEnable, true);
if (error != ESP_OK) return error;
const uint8_t voltage = static_cast<uint8_t>(
20 + 8 * brightness_percent / 100);
return write_register(s_pmu, PmuBacklightVoltageRegister, voltage);
return write_register(s_pmu, PmuBacklightVoltageRegister, step);
}

static esp_err_t set_speaker_hardware(bool enabled) noexcept
Expand Down Expand Up @@ -177,14 +184,13 @@ esp_err_t initialize() noexcept
s_panel_io, &callbacks, nullptr)) != ESP_OK ||
(error = bsp_display_brightness_init()) != ESP_OK ||
(error = esp_lcd_panel_disp_on_off(s_panel, true)) != ESP_OK ||
(error = bsp_display_brightness_set(NormalBrightness)) != ESP_OK ||
(error = bsp_touch_new(nullptr, &s_touch)) != ESP_OK) {
ESP_LOGE(TAG, "Display/touch initialization failed: %s",
esp_err_to_name(error));
return error;
}
if ((error = add_power_devices()) != ESP_OK ||
(error = set_backlight(NormalBrightness)) != ESP_OK) {
(error = set_backlight(NormalBacklightStep)) != ESP_OK) {
ESP_LOGE(TAG, "CoreS3 power initialization failed: %s",
esp_err_to_name(error));
return error;
Expand All @@ -201,7 +207,9 @@ std::span<std::uint16_t> framebuffer() noexcept
: std::span<std::uint16_t>{s_framebuffer, PixelCount};
}

/* Byte-swaps RGB565 in place, submits LCD DMA, waits, then restores byte order. */
/* Byte-swaps RGB565 in place, submits LCD DMA, and waits for completion. The
byte order is not restored: every renderer redraws the complete frame before
the next present(), so skipping a second 150 KB PSRAM pass is safe. */
esp_err_t present() noexcept
{
if (s_panel == nullptr || s_framebuffer == nullptr || s_transfer_done == nullptr)
Expand Down Expand Up @@ -239,7 +247,7 @@ esp_err_t setDisplayPower(DisplayPower power) noexcept
if (error == ESP_OK)
error = set_backlight(
power == DisplayPower::Normal
? NormalBrightness : DimmedBrightness);
? NormalBacklightStep : DimmedBacklightStep);
} else {
error = set_backlight(0);
if (error == ESP_OK) error = esp_lcd_panel_disp_on_off(s_panel, false);
Expand Down
26 changes: 17 additions & 9 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,10 +39,9 @@ address derived from the device Bluetooth MAC.

The modes also keep separate bonding data: Claude uses a dedicated Bluedroid
configuration store named `claude_bt`, while Codex uses the default store. This
is intended to let both desktop pairings survive mode switches without exposing
both services at once. The design is implemented, but its macOS pairing and
service-cache behavior remains a physical validation item in
[HARDWARE_TEST.md](HARDWARE_TEST.md).
lets both desktop pairings survive mode switches without exposing both services
at once. Switching between modes without re-pairing or macOS service-cache
confusion was confirmed on hardware; see [HARDWARE_TEST.md](HARDWARE_TEST.md).

Claude requires LE Secure Connections with MITM protection. The CoreS3 shows a
six-digit passkey, and its UART characteristics require an encrypted link.
Expand All @@ -51,8 +50,8 @@ needs a reviewed storage choice and may require a partition-layout change.

## Recommended reading order

Do not begin by reading all 700+ lines of `main/main.cpp` from top to bottom.
Choose one mode and follow its data path:
`main/main.cpp` only selects a mode; the interesting logic lives in the
runtime classes and components. Choose one mode and follow its data path:

### Codex path

Expand Down Expand Up @@ -159,18 +158,19 @@ sound, and persistence. Motion samples take a separate path through
| `codex_rpc` | Supported desktop methods and model events | BLE and drawing |
| `fixed_json` | Bounded JSON tokenization and object lookup | Protocol meaning, allocation |
| `buddy_layout` | Shared control rectangles and point containment | Rendering style, actions |
| `ui` | Codex pixel rendering | Hardware transfer and touch |
| `codex_ui` | Codex pixel rendering | Hardware transfer and touch |
| `codex_ble_transport` | Codex BLE HID lifecycle and reports | Application state |
| `claude_model` | Claude UI/session/permission state | Parsing, I/O, drawing |
| `claude_protocol` | Claude JSON framing and commands | BLE, NVS, RTC, sound |
| `buddy_ui` | Selector/Claude rendering and hit testing | Hardware transfer |
| `ui_font` | LVGL Montserrat glyph measurement and blending | LVGL display objects |
| `claude_ble_transport` | Claude BLE service, security, bonding | Protocol meaning |
| `claude_storage` | Persisted Claude identity and decision counters | Runtime policy |
| `buddy_settings` | Shared persisted preferences | Sound policy and UI state |
| `buddy_audio` | Mute state, persistence coordination, and guarded playback | Cue selection |
| `motion_detector` | Motion filtering and gesture events | IMU hardware access |
| `platform_core_s3` | LCD, touch, speaker, RTC, and IMU APIs | Mode-specific behavior |
| `main` | Composition, owned runtime classes, queues, side effects | Reusable domain logic |
| `main` | Composition, runtime classes, queues, display-power policy, side effects | Reusable domain logic |

## State and ownership rules

Expand All @@ -181,7 +181,13 @@ sound, and persistence. Motion samples take a separate path through
- BLE callbacks copy data into fixed-size queue events. The copied buffers must
remain large enough for the transport callback's maximum chunk/report.
- Renderers always draw a complete 320 x 240 RGB565 frame into the shared PSRAM
framebuffer. `buddy::platform::present()` transfers it to the LCD.
framebuffer. `buddy::platform::present()` byte-swaps it in place for the
panel and transfers it to the LCD, so a frame must be fully redrawn before
it is presented again.
- Display dimming and power-off timing live in
`main/display_power_policy.hpp`. Attention states (prompts, passkeys, waiting
sessions, Agents requiring input) hold normal brightness for up to one
minute.
- Most pure components have no ESP-IDF dependency. This is intentional: they
compile and run as strict C++20 host tests.

Expand Down Expand Up @@ -217,6 +223,8 @@ The files in `test/host/` show expected behavior without needing a CoreS3:
messages and owl state transitions.
- `test_codex_ui.cpp` and `test_buddy_ui.cpp` verify renderer behavior at selected pixels.
- `test_motion_detector.cpp` documents gesture thresholds through examples.
- `test_display_power_policy.cpp` shows the dim, off, and attention-hold
timeouts, including uptime wraparound.

Run all host tests with `bash test/host/run.sh`. Build the actual ESP-IDF
application with `pio run -e m5stack-cores3`; neither command flashes hardware.
Loading
Loading