Repository navigation
Conversation
…writes Fresh private fork of acheong08/ferroxide (itself a fork of emersion/hydroxide), MIT-licensed with full attribution retained. Focus: a hardened Proton Calendar CalDAV bridge for a client that does its own iTIP/iMIP scheduling. - Rename + re-module github.com/acheong08/ferroxide -> github.com/mathewcsims/protoxide; rename cmd and rebrand config-dir/usage strings. - SEQUENCE monotonicity fix: read prior value from Proton's signed (cleartext) shared card and force SEQUENCE >= old+1 on update / SEQUENCE:0 on create, addressing the known upstream write-corruption bug. - Pagination fix: follow Proton's calendar-event pagination to completion with a hard page cap, instead of returning only the first page (silent truncation). - New README documenting purpose, design intent, changes, and security model; upstream README preserved as README.upstream.md. Drop upstream release CI.
…ession Adds a session-import path so protoxide can be seeded from credentials a browser already holds (UID + access token, optional refresh token), avoiding the interactive SRP login. Reuses the existing key-unlock and encrypted-cache storage so the caldav/serve commands work unchanged afterward. - protonmail.Client.SetSession(uid, accessToken): seed a client with an existing session for API calls without a fresh login. - cmd: `protoxide import-session <username>` prompts for the session values and mailbox password, verifies the session (key-salts fetch + key unlock), and saves the encrypted cache.
Proton challenges third-party logins with a human-verification (CAPTCHA) step. This implements the same one-time flow Proton's own clients use: the account owner completes Proton's genuine challenge in their browser, and the resulting token is submitted with the login so it can proceed. Nothing solves or bypasses the challenge — the human completes it. After this one-time login, the session self-sustains via the existing refresh-token rotation. - protonmail: capture the 9001 challenge details; carry the completed verification token on subsequent requests. - cmd: on a 9001 during `auth`, open Proton's challenge for the user to complete, capture the token, and retry the login with it.
Events authored by the account owner can be returned with an empty Author
field. The code fetched public keys for that empty address, which the API
rejects ("The Email is required"), and the error failed the entire calendar
listing. Fall back to the account's own keyring when Author is empty.
The added ListAllCalendarEvents page loop could issue up to its page cap of sequential API calls when Proton doesn't decrement pages as expected, which looks like a hang on a calendar-query REPORT. Revert both listing paths to a single request; the time-range filter already bounds queries. Proper pagination is a TODO to revisit once the read/write loop is proven.
Re-enable fetching all event pages, but stop on a short page OR when a page returns no new event IDs — so it both reaches the true end and can't loop forever if the server ignores the Page parameter. Logs per-page counts to make the server's pagination behaviour observable.
Proton's events endpoint ignores time-range filtering and paginates oldest-first, so answering each client query by walking every page and decrypting every event is far too slow for a large calendar and times the client out. Add a per-calendar cache: - Fetch + decrypt the whole calendar once in the background; a cold request waits only briefly (well under the client timeout) then serves what has loaded so far, filling in over subsequent polls. - Serve list/query/get from the decrypted cache; filter time ranges client-side since Proton won't. Recurring/unparseable events are included rather than dropped. - Writes and deletes update the cache so changes show immediately. - Stale-while-revalidate refresh on a 30-minute TTL. Known follow-up: refresh is a full re-fetch on the TTL; a delta sync via Proton's event stream would make inbound changes near-instant.
Proton returns RecurrenceID as a number (a timestamp) for recurring-event
exceptions, but the field was typed as string, so any page containing such an
event failed to unmarshal and aborted the whole listing. Loosen the type to
interface{} (as with the sibling RRule/Exdates fields); it is not read anywhere.
Many old/imported events are signed with a key that no longer verifies, which was failing the whole event. For a personal read bridge, being able to decrypt the event matters more than verifying its signature — log the mismatch and serve it. (Events that fail to *decode* are still skipped.)
On-disk cache: the decrypted event cache is now written atomically to the config directory (0600) after every load and write, and restored on startup. Restarts are instant instead of re-walking every page — the expensive full load only runs when the disk cache is missing or past its TTL. This holds decrypted event data locally, the same trade-off Thunderbird's own cache makes. Event loop: the receiver was a no-op that never drained its channel, so the events poller blocked after the first event. Drain and log it. The core event loop is mail-only (Messages/Contacts) and carries no parsed calendar changes, so true push-delta needs Proton's separate calendar-sync endpoint; the logging is groundwork to determine what signal, if any, is available.
Creating an event from a CalDAV client PUTs to a path with the client's own
UID. UpdateCalendarEvent probes whether the event exists and treats Proton
error 2061 ("not a valid ID") as "does not exist yet -> create". But
GetCalendarEvent wraps its error with fmt.Errorf(%w), so the direct
err.(*APIError) type assertion never matched, the create branch was skipped,
and the PUT failed — the event never reached Proton and vanished on the next
sync. Match with errors.As so 2061 is recognised and the create proceeds.
Proton rejected event writes with [2001] "Provide data signed using the address key" because the event was signed with userKr[0] — the first key in the ring, which on a multi-address account is not necessarily the calendar member's own address. Select the keyring entity whose identity matches the calendar member's email and sign with that (falling back to the first key if none matches).
Refreshing on every startup discarded a still-valid access token and hit Proton's refresh endpoint, which is unreliable for third-party sessions and can bounce into human-verification. Set the session from the cached access token and attempt Unlock first; fall back to AuthRefresh/SRP only when that fails, letting the normal 401 handler refresh once the token has actually expired.
Two bugs prevented events created in a CalDAV client (Thunderbird) from round-tripping through Proton: 1. Create returned HTTP 500. makeUpdateData set SEQUENCE via go-ical's SetText, which emits "SEQUENCE;VALUE=TEXT:0". SEQUENCE is an INTEGER property (RFC 5545); Proton's server rejects the VALUE=TEXT form with an opaque 500 while lenient clients tolerate it. Emit a bare "SEQUENCE:0" integer property instead, normalising any tagged value the client sends. 2. Created events did not reappear without a full re-sync. CalDAV objects were keyed by Proton's opaque event ID, but Proton reassigns that ID on create, so the client's chosen href stopped resolving (GET -> 2061) and the event only surfaced after a full reload re-keyed everything to new paths. Derive the CalDAV path from the iCalendar UID (objectToken), which is stable across the write round-trip and across cache reloads. Resolve the UID back to a Proton event ID via the cache (findByToken) for update/delete, so an edit updates in place instead of creating a duplicate. Recurrence overrides sharing a UID are disambiguated by RECURRENCE-ID.
A full cache reload takes minutes and captures Proton's state at the moment it started, then swaps its snapshot in wholesale. An event written (or deleted) via CalDAV while a reload was in flight was therefore absent from the reload result and got dropped when the snapshot swapped in, reappearing only on the next 30-minute reload — so a just-created event vanished from the client until a manual full refresh. Keep a short-lived overlay of local writes/deletes (recentWriteTTL) and re-apply it on top of every reload swap, so a write survives a racing reload until Proton is consistent and a later reload naturally includes it. The overlay entry then expires.
Previously the cache only refreshed on a 30-minute timer via a minutes-long
full reload, so events created in Proton (or on another device) took up to
half an hour to reach a CalDAV client, and a client-created event could vanish
until the next reload.
Poll Proton's calendar event loop
(GET /calendar/v1/{calID}/modelevents/{cursor}) every 30s, the same mechanism
the Proton web client uses for live updates. Seed a cursor at the current head,
then patch only changed events into the cache: creates/updates are re-fetched
by ID for their encrypted cards and decrypted; deletes are dropped. Refresh
triggers a full reload; More paginates. One poller per calendar, started lazily
on first access. Falls back to periodic reload if the cursor cannot be seeded.
Adds Client.GetLatestCalendarModelEventID / GetCalendarModelEvents and the
CalendarModelEvents/CalendarModelEventChange types.
cacheStore/cacheDelete wrote the whole in-memory cache to disk on every mutation. A write (or poller-driven inbound change) landing while the initial load or a reload was still in flight therefore persisted a partial set, overwriting the full on-disk cache. On the next start only that partial set was restored, and every client refresh served a slowly-growing snapshot while the cache rebuilt — so a single refresh returned only a few events and a re-add needed many refreshes. Persist from cacheStore/cacheDelete only when the cache is fully loaded and not mid-reload. In-flight writes remain in memory and in the recent-writes overlay, and loadCache folds them into the full set it persists on completion, so nothing is lost and the on-disk cache is always complete.
The UID-based path change used the raw iCalendar UID as the object href. Most
UIDs are safe, but imported/Proton-side events can carry UIDs containing '/',
':', '|', quotes or spaces (e.g. "my-blood.co.uk/<uuid>", 'TKF/.../.../0/0',
a literal '"'). A '/' splits the href into extra path segments and the others
break the multistatus XML, so a client aborts parsing at the first bad href;
Go's randomized map iteration moved the break each refresh, so events dribbled
in over many refreshes and a re-add appeared to load only a handful at a time.
Encode path tokens to a safe set: pass through UIDs that are already URL/XML
safe (what clients generate, so client-created events still round-trip to the
same href) and hex-encode any UID with unsafe characters ("enc.<hex>"). The
token is matched by recomputation (findByToken), so the encoding need not be
reversible. Verified against the live cache: 3106 events -> 3106 unique, all
safe, zero collisions.
calendar-multiget calls GetCalendarObject once per requested href, and both it and findByToken scanned the entire cache each call, recomputing objectToken for every event. A full-calendar multiget was therefore O(n^2) — ~10M iCal parses for 3106 events — slow enough that the client timed out the REPORT partway and ingested only a partial, varying subset (different each refresh due to Go's randomized map iteration), so a re-add appeared to load a handful at a time. Maintain a byToken index (path token -> Proton event ID) alongside the objects map, updated on load, disk-restore, incremental publish, store and delete, so GetCalendarObject/findByToken are O(1). A multiget is now O(n) overall and completes in one pass. Also log per-request object counts for observability.
CalendarEventCardType.Encrypted() returned true only for CalendarEventCardEncryptedAndSigned (type 3), missing CalendarEventCardEncrypted (type 1) — the encrypted-but-unsigned card type. Cards of type 1 were therefore treated as cleartext and their base64 ciphertext was fed straight to the iCal parser, which failed with "malformed content line: missing colon". This dropped ~30 imported events (stored encrypted without a signature) from the calendar. Confirmed by dumping the rejected card data: it was an undecrypted tag-18 PGP packet, not iCalendar. Treat type 1 as encrypted so it is decrypted.
Two robustness fixes surfaced while chasing the skipped events: - sanitizeICS: a small number of imported events have a genuinely malformed decrypted body (an unescaped newline splits a value into a line with no name:value separator). On decode failure, repair and retry rather than dropping the event. - Cold-load handling: a full load fetches every page before decrypting, so the cache is legitimately empty for the whole multi-minute fetch. Serving that as an empty object list makes a client treat the calendar as emptied and then churn once events appear. cachedObjects now reports readiness, and the list/ query handlers return 503 (retry) until the first load completes, so a client waits and then loads everything in one pass. Also log per-request counts.
Without a change token, a CalDAV client re-syncs only on its full refresh interval, so events created in Proton (surfaced into the cache by the event-loop poller within ~30s) still take until the next interval to reach the client. Upstream go-webdav has no hook to advertise the CalendarServer getctag property, so vendor it (v0.6.0, MIT, via a replace directive) with a minimal additive patch: a Calendar.CTag field and getctag emission in propFindCalendar. See third_party/go-webdav/PROTOXIDE-PATCH.md. protoxide computes the CTag as an order-independent hash of every object's (path, etag) plus the object count, so it is stable across restarts for unchanged content and changes on any add/edit/delete. It is advertised on the calendar collection; a client can poll it cheaply and only run a full sync when it changes.
A client-created event did not appear in the client until a manual refresh. UpdateCalendarEvent mutates the event it is given while building Proton's encrypted cards — it strips the UID (and other properties). That event shares its underlying component with the calendar we hold, so after the write our copy had no UID: objectToken fell back to the Proton event ID, and we returned the new object (and its Location header) at a Proton-ID path instead of the UID path the client PUT to. The client couldn't match the created resource to its own optimistic copy and dropped it until a full resync re-fetched it at the new path. Hand UpdateCalendarEvent an isolated clone of the calendar and keep the original pristine, so the served data, the derived UID path, and the cache key all match the client's PUT URL. Confirmed via debug logging that the pre-write UID was present and only lost after the write.
Bring the README in line with what the bridge actually does now (working read/write round-trip, on-disk cache, event-loop inbound delta sync, CTag) and add a prominent, explicit "Limitations & caveats" section covering reliability/ trust, session durability, sync latency, cold-start cost, feature gaps (scheduling, attendees, recurring-exception edits, disabled crypto checks), data-at-rest exposure (decrypted on-disk cache; localhost plain HTTP), and the inherited-but-unsupported mail/IMAP/SMTP/CardDAV surface. Also add exact setup/ client-configuration steps and a "how it works" overview. Remove the leftover create-payload diagnostic log in UpdateCalendarEvent that was only needed while debugging the write 500.
- Write auth.json with 0600 instead of the default 0644 — the session file is encrypted at rest but is still a long-lived credential and must not be world-readable to other local users. - Add calcache-*.json (and stray local config dirs) to .gitignore. The decrypted calendar cache holds plaintext event data; the real cache lives outside the tree, but this is defensive against ever committing it if run from the repo.
Add a best-effort notify package that POSTs alerts to an Apprise endpoint (PROTOXIDE_APPRISE_URL → ntfy etc.). It is a no-op unless configured, never blocks the caller, and de-duplicates repeated alerts (10 min per kind) so a persistent fault doesn't flood. Wire it into the real failure points: CalDAV auth failure (most often an expired Proton session — the alert says to re-run auth), calendar event-loop poll failure/degradation, calendar write failure, a recovered handler panic, and the server exiting. A recover() is added to the CalDAV handler so a panic is alerted and 500'd instead of being silently swallowed by net/http. Also gate the chatty per-request logs (request line, object counts, cache miss) behind the existing -debug flag via caldav.Debug, so a long-running service logs only meaningful events and its log stays small.
Add service/ with a per-user macOS launchd setup so the CalDAV bridge runs headless, starts at login, restarts on crash (KeepAlive, throttled), and returns after a reboot: - install.sh — builds protoxide, installs the binary and a runner into ~/Library/Application Support/protoxide/bin, generates the LaunchAgent with the user's paths and Apprise config (prompted or from env), and loads it. The plist is written 0600 since it may carry an Apprise URL with a secret key. - protoxide-caldav-run.sh — runs the binary, appends to a size-rotated log, and alerts Apprise if the process is killed by a signal (a crash in-process alerting can't report). - uninstall.sh — unload/remove (optionally --purge the binary); never touches the session or calendar cache. Document the service and the Apprise alerting model in the README.
Match the self-hosted stack's notification convention (🚫/⚠️ /✅/🔔 by severity) so protoxide's alerts read consistently alongside the other notifiers feeding the shared Apprise -> Discord/ntfy endpoint.
Note that once installed the launchd service is the sole owner of 127.0.0.1:8081, and give the exact unload -> run -> load sequence (with -debug) for running a manual session without the two instances fighting over the port.
- Add SECURITY-AUDIT.md: comprehensive security audit assessing the repo as SAFE TO PUBLISH, with detailed findings across all security categories - Add CRYPTO-FIX-PROPOSAL.md: proposed solution (Option A) to re-enable the two disabled cryptographic checks with graceful fallback, maintaining backward compatibility while improving security posture
- Add protoxide-logo.png (512x512) with calendar icon and sync arrows - Add logo to top of README.md - Include generate_logo.go for reproducibility
- Regenerate protoxide-logo.png without text for cleaner profile image - Calendar grid with sync arrows, no text - Keep generate_logo.go for reproducibility
- Resize to 1280x640 pixels for GitHub profile image - Add fun, quirky elements: smiling sun, clouds, lightning bolts - Floating confetti dots background - Colorful event blocks in calendar - Bridge/cable sync indicator - Keep generate_logo.go for reproducibility
Request paths that don't sit under the calendar home set parsed into an
empty calendar ID: CutPrefix left the path untouched and Split then
yielded an empty first segment, so the backend keyed cache and poller
state on "" and called
GET /api/calendar/v1//modelevents/latest -> [404] Path not found
GET /api/calendar/v1//events/caldav -> [404] Path not found
for the life of the process. Worse, the spawned pollCalendar("") hit its
seed failure path and returned permanently, logging 'delta sync disabled'
and firing a degradation notice for a calendar that doesn't exist.
Reject such paths up front with a 404 in GetCalendarObject, PutCalendarObject,
DeleteCalendarObject, ListCalendarObjects and QueryCalendarObjects, and add
defensive empty-ID guards in pollCalendar and loadCache.
Adds a regression test covering the malformed shapes observed in production
logs.
The client used for all Proton API calls had no timeout, so a single connection that stalls server-side wedges the caller forever. Observed in production: the event loop's GetEvent hung silently for ~6.5 hours (last heartbeat 14:39, first error 21:14) on a stalled connection, and the ensuing retry storm OOM-killed a 2 GiB container twice in two days. Bound every API call — including reading the response body — with a 90 s timeout, applied to both the direct and the proxy-constructed client.
Author
|
Correction + notes from a deeper verification pass on our side (independent review of the branch):
|
|
Thanks for these fixes and the additional notes about the timeout behavior. I’m evaluating Protoxide for use with Thunderbird. One source-review question about the path guards: they discard the matched boolean returned by strings.CutPrefix, so a relative object path such as cal1/review-event.ics appears to pass the segment check even though it lacks the calendar home-set prefix. The vendored multiget handler also forwards href.Path directly. Would you prefer an explicit prefix check in this PR, or should stricter path validation be handled separately from the reported empty-calendar-ID bug? Our supplemental tests have not run yet, so this is a source-level question rather than a confirmed runtime finding. |
The five accessors discarded strings.CutPrefix's matched bool, so the guards were shape checks (two non-empty segments), not provenance checks. A relative path such as cal1/review-event.ics — reachable via calendar-multiget, whose vendored handler forwards href.Path from client XML straight into GetCalendarObject — passes the shape check and keys phantom cache + poller state on a crafted calendar ID: the same failure mode as the empty-ID bug. Direct HTTP routes cannot deliver relative paths (Go rejects non-absolute request targets), so multiget is the only exposure. Require the prefix match in all five accessors, and extend TestPathsOutsideHomeSetAreRejected with both relative shapes (both fail against the previous commit). Finding credit: @thunder-detector.
Author
|
Nice catch — you're right on both counts, and the shape is reachable. Confirmed at source level and with a failing test: the accessors discard One scope note: direct HTTP can't deliver a relative path — Go's server rejects non-absolute request targets — so calendar-multiget, which forwards Pushed 094f326: all five accessors now require the prefix to actually match ( I've deliberately left the vendored multiget handler untouched: the accessor boundary now rejects anything that isn't home-set-prefixed regardless of route, and I'd rather not diverge the vendored copy from upstream go-webdav. If you find a shape that still slips through, that would be a strong reason to harden the handler too. Happy to look at your supplemental tests when they've run — thanks for the careful read; this was a real gap in the original fix. |
|
Thanks, Matt — I independently retested 094f326 against ab13e30 on Linux amd64 with Go 1.26.8 and 1.27.1. Valid cached GETs and collection queries still pass, and mixed multiget requests preserve the valid object’s response and ETag. The rejection probes show no cache creation, poller registration or Proton API calls. I also verified the 90-second timeout configuration and cancellation during stalled headers and bodies through loopback direct/HTTP-proxy tests, using a shortened test timeout. All eight supplemental test groups pass under the race detector on both toolchains. Stock builds, root tests/race checks and nested go-webdav tests pass. Vet reports only the same two pre-existing warnings at exports/messages.go:30. |
lstmemery
force-pushed
the
fix/empty-calid-and-api-timeout
branch
from
October 11, 2026 00:17
094f326 to
3521395
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two fixes for failure modes observed running protoxide as a 24/7 bridge under systemd (LXC container, 2 GiB cap). Both are verified against a live bridge; a regression test is included.
1. Paths outside the home set parse to an empty calendar ID and poison backend state
GetCalendarObject/PutCalendarObject/DeleteCalendarObjectstrip the home-set prefix withstrings.CutPrefixand thenstrings.Splitthe remainder. For any path not under the home set (e.g./caldav/watchdog-probe),CutPrefixleaves the path unchanged,Splityields an empty first segment, and the backend keys cache + poller state on"":Proton answers these instantly and sanely (404 on the malformed, double-slash URL) — but the spawned
pollCalendar("")hits its seed-failure path and returns permanently, loggingdelta sync disabledand firing a degradation notice for a calendar that doesn't exist.ListCalendarObjects/QueryCalendarObjectshave the same exposure viacalID == "".Fix: reject non-home-set paths up front with a 404 in all five accessors, plus defensive empty-ID guards in
pollCalendarandloadCache.path_test.gocovers the shapes observed in production logs (including/caldav/watchdog-probe, an actual health-probe path we use). Extended after review (094f326): the guards now also require the home-set prefix to actually match — a relative href such ascal1/review-event.ics, deliverable via calendar-multiget, would otherwise pass the shape check and key phantom state on a crafted calendar ID.2. No timeout on the Proton API client
The
http.ClientinnewClient()(and the proxy variant) had no timeout. Observed in production: the event loop'sGetEventhung silently for ~6.5 hours on a stalled connection — journal shows the lastevent loop:heartbeat at 14:39:25 and the firstcannot receive event:error at 21:14, nothing in between — then the retry storm OOM-killed a 2 GiB container (twice in two days, ~9 h CPU runaway each time).Fix: bind a 90 s client timeout (covers the response-body read; calendar paging stays fine since each page is its own request) on both the direct and proxy-constructed clients.
Verification
Against a live bridge (same Proton account, ~112-event calendar):
GET /caldav/watchdog-probev1//404s +poller : could not seed cursorphantomrequest failedlines, no phantom pollerPUT→ Proton event id → readback →DELETE→ cache grep)poller <real-cal-id>: started, cursor seededon first requestgo build ./...andgo test ./caldav/are clean;go vet ./...has no new findings (two pre-existing unkeyed-struct warnings inexports/messages.go:30, present at base f332c8b).