Channels v2 (1/3): one visibility rule, one write path, delivery and question rows

[anukuljha]

PR 1 of 3 for channels v2. Builds on #993 (merged), and replaces both its schema and this PR's first version. 0057 and 0058 are edited in place because neither has been released.

Schema revised further up the stack. Two reviews of the whole stack were applied on #1004, which edits this PR's 0058 tables in place: support state moves to channel_cases, rev moves to a channel_changes log, pushes coalesce in channel_push_pending, deliveries gain reason_code/channel_id/seq, mentions, questions, threads and places get composite FKs, channel_sources.kind becomes PERSON/CHAT, and external_accounts is keyed by workspace_key + bot_key. Read #1004's "Data-model review" section for the final shape; the three PRs merge together.

What changes

Channels own every line a person or an agent writes, and everything that follows from it: who can see it, who is told, which agents answer, what it asks and how it is delivered. agent_surfaces keeps the platforms; the rules move into channels, agent and workflow.

• One visibility rule. domain/access.may_see and its SQL twin infrastructure/access.visible_to are used by every read: channel list and get, messages, threads, inbox, search, agent tools and background.

  • DIRECT is visible only to its member.
  • CONTACT is visible to members with contact.read.
  • GROUP is visible to the whole pod when public, and to its members when private.
  • A group mirrored from a platform is private when the platform keeps it to its members. A group made in Lemma is public by default.

• One write path. Timeline.post does everything in one transaction:

  • numbers the line (a gapless per-channel seq, under the channel lock) and seats the speaker;
  • records mentions and the question the line asks;
  • moves a contact's channel along: a member writing to the contact is assigned and the channel waits on the contact; the contact writing reopens it;
  • plans deliveries and pushes, and announces them.

  A line is written once per client_message_id, and once per platform message.

• Deliveries are rows. Every send outside Lemma is a channel_message_deliveries row, unique per line and target. That covers a line for a place and a push to a member. A new executor (agent_surfaces/services/channel_deliveries.py, kind surface.channel_delivery) sends each row:

  • it claims the row, applies the contact rules (STOP, WhatsApp's 24 hours, the daily limit counted once per line, the unsubscribe line on email), sends, and records SENT, SKIPPED with the reason, or FAILED;
  • a failed row is retried with backoff and becomes DEAD after 6 tries; a sweeper re-announces rows that are due;
  • a member is pushed where their notify_via says, otherwise where the sending agent can best reach them.

  This replaces member_posts, DirectMessages.deliver, the notification dispatch, said_in_run, line_went_out and opened.

• Questions are rows. channel_questions holds:

  • a ref unique per pod;
  • a state, OPEN → ANSWERING → ANSWERED | CANCELLED | EXPIRED;
  • an expiry, enforced by a sweeper.

  An answer is claimed before the asker takes it, so a second tap gets 409. Approvals are answered only by the person's own tap. An agent answering one gets 403, and an answer without a decision gets 422.

  • Askers register by prefix:
    • tool: covers agent pauses (agent/services/pause_questions.py), and multi-question ask_user is answered field by field;
    • wf: covers workflow forms (workflow/infrastructure/form_questions.py), whose ref is wf:{run}:{node}:{wait} so a looping graph can ask again;
    • msg: covers one member's agent asking another, and its batch wakes the asker once.

• Read state by seq. channel_read_state.last_read_seq; unread is last_seq − read_seq for joined channels.

• External accounts. A place belongs to an account rather than a surface: a Slack team, Teams tenant, Telegram bot, inbox or web widget, each from account_key_for. A reinstalled bot keeps its places. Every lookup is scoped to the asking surfaces' accounts. Only signed routing reads that prove no narrower scope look across a whole platform.

• In-app answering moves to agent. agent.channel_line and agent.channel_answer run one turn per (line, agent), and the answer lands in the line's thread.

  • Agents answer a line that names them, every line if they're set to, and in a member's own channel the pod's agent answers.
  • in_app_turns, pause_lines, asked_questions_work, question_facts and the workflow-notification adapter are deleted from agent_surfaces.

• Contacts' inbox fields are columns: agents_answer, hand_back_after_hours, assignee_user_id with assignee_last_line_at, status and attention_*. The settings JSON and handler are gone. The roster is channel_members with role, notify, notify_via, muted_until, and answers for agents.

• A member leaving the pod is taken out of its channels and unassigned, their questions are cancelled, and their own channel is archived.

API (/pods/{pod_id})

/channels GET (kind, archived, contact, assignee, status, joined; cursor) · POST (Idempotency-Key) · /channels/me GET · POST · /channels/{c} GET · PATCH (absent = unchanged, null = clear) · DELETE · /members POST · /members/me POST · PATCH · DELETE · /members/{kind}:{id} PATCH · DELETE · /read POST {seq} · /messages GET (before_seq/after_seq) · POST (client_message_id) · /messages/{m} PATCH · DELETE · /thread · /answer · /deliveries GET · /deliveries/{d}/retry · /inbox · /search. GETs never write.

SDKs (Python pod.channels, TypeScript channels), the CLI (lemma channels …, members, messages, deliveries, agent <id> on|off) and the existing frontend channel screens are ported to it. The full UI is PR 3.

Why

A review of the first version of this PR found about 100 problems. Most came from a missing model rather than from bugs:

• delivery and questions decided by each producer;
• three copies of the visibility rule;
• no idempotency key for sends;
• agent_surfaces owning in-app turns, questions, DM fan-out and contact policy.

The design and its decisions are agreed (D1 to D7: one visibility rule, posting and telling as one step, delivery rows, question rows, accounts, seq, and surfaces as transport only).

Next:

• PR 2: surfaces as transport, with agent replies on delivery rows and a full contacts policy.
• PR 3: one live socket, API conventions, the full UI, and hardening.

Bugs found and fixed on the way

• said_in_run read ChannelMessageModel.text; the column is body.
• /inbox: every call raised. The membership EXISTS subquery auto-correlated its own table away.
• Retrying a send:
  • a retried send to a contact spent the contact's daily limit, and a Redis outage gave up instead of retrying;
  • a push that failed because a platform was down was recorded as unreachable and never retried.
• Inbound reads: the chat and person indexes on channel_sources were missing, so every inbound message scanned every place on the platform (11 rows read grew to 411 in the plan test).
• Telegram: the member who brought the bot into a group owned it but wasn't in its channel. set_owner now seats them.
• Cold emails: a cold email's place was keyed by thread, while the reply is keyed by address, so each cold email left a duplicate place.
• A call for a person:
  • it was shown only on contact channels, so a call in a group was invisible;
  • a member writing in the group now answers it.
• DIRECT channels: a notice from one member to another seated the sender in the recipient's own channel. Only its owner is ever seated there now.
• Notes: a members-only note was pushed outside the app. It now waits in the app unless it names you.
• Questions: a second line asking an existing question ref silently got the first line's question. It now raises.
• client_message_id: a long provider call id overflowed the column (varchar(64)). Long keys are hashed.
• Generated TypeScript client: it took (channelId, podId) because pod_id came last in the handlers. They now declare it first.
• Edits: an edit could name someone who can't see the channel, which a post refuses.

How to verify

make quality                     # 32 quality gates passed
make quality-frontend            # pass
make format-check                # pass
cd lemma-backend
uv run pytest app -m "not e2e" -n 8 -q          # 11646 passed, 153 skipped
uv run pytest app/modules/channels/tests app/modules/contacts/tests/e2e \
  app/modules/workflow/tests/e2e/test_form_questions_e2e.py \
  app/modules/agent/tests/e2e/test_channel_answers_e2e.py \
  app/modules/agent/tests/e2e/test_channel_tools_e2e.py \
  app/modules/agent/tests/e2e/test_message_reply_delivery_e2e.py -n 4 -q   # 55 passed
uv run pytest app/modules/agent_surfaces/tests/e2e -n 4 -q   # 446 passed

Client suites: lemma-python 189 passed; lemma-cli 883 passed; lemma-typescript 463 passed and the build is clean; lemma-frontend 1556 passed.

New e2e tests:

• channels/tests/e2e/test_channels_v2_api_e2e.py (17): visibility on every read, idempotency, seq paging, read state, threads, deliveries, questions, roster, inbox;
• agent_surfaces/tests/e2e/test_channel_deliveries_e2e.py (15): pushes, notify prefs, mute and NONE, cold email and its reply, retry and backoff to DEAD, the daily cap counted once, outage retried, deleted lines;
• agent/tests/e2e/test_channel_answers_e2e.py (8) and test_channel_tools_e2e.py (4);
• workflow/tests/e2e/test_form_questions_e2e.py (3);
• test_channel_fold_e2e.py (contact merge renumbers seq) and test_channels_migration_e2e.py (0058 up, down, up).

Checklist

• Tests cover the behavioral change
• Lint, typecheck, and architecture gates pass locally
• Migrations: 0058 upgrade, downgrade and upgrade again tested (test_channels_migration_e2e.py); 0057 adds contact_suppressions
• API surface: OpenAPI spec, both SDKs and lemma-client.js regenerated and committed
• Compatibility: see below
• Security: visibility checked on every read path; approvals human-only; no new secret in logs or payloads
• Rollback: see below

Compatibility and rollback notes

• Breaking API changes, accepted on purpose because nothing here has been released:

  • removed: channel.mark_read (now channel.read {seq}), channel.message.search (now channel.search), channel.notify.update (now members/me), and channel.participant.add/remove (now members);
  • agent.notice.send no longer takes title, and no longer returns delivered_via/reason, because a push is sent asynchronously and its outcome is on the delivery row;
  • the agent tool channel_post lost platform: pushes follow the member's notify_via.

• Migrations: 0057 and 0058 are rewritten in place. They are unreleased, but a database that already ran the old 0058 has to be downgraded to 0057 and upgraded again.

• Baselines: four channels reads are recorded in unbounded-reads-baseline.json, each bounded in a way the AST pass can't see:

  • _with_question and _with_surface are statement builders, and every caller adds its own limit;
  • the flag in list_for is the membership subquery, under an outer limit;
  • the reads in InboxReads.unread are narrowed to the ids of the page being shown.

  The two notification-repository entries drop out with their file.

• Not in this PR: the platform's message id on delivery rows (PR 2, with the egress executor); moving /sources under /channels and the channel.* permission names (PR 3, API conventions).

• Rollback: revert the PR, then downgrade 0058 → 0057. Channels start empty again; nothing has been released, so there is no data to carry back.

🤖 Generated with Claude Code

[coderabbitai]

Walkthrough

Warning

Review details and warnings were omitted to fit the comment limit.

[github-actions]

CodeQL

No new findings on lines this pull request changed (as of eb86dd6).

_{Updated on every push. A finding judged safe goes in .codeql-allow.txt with its reason, or is dismissed in the Security tab. GitHub Code Quality posts its own review comments separately.}

[github-actions]

Module	Unit	E2E only	Combined
Overall	73.2%	82.1%	91.5%
agent	78.7%	83.3%	92.8%
agent_surfaces	73.0%	82.8%	90.8%
apps	75.7%	87.6%	92.4%
connectors	78.7%	73.6%	90.2%
datastore	69.0%	86.4%	90.9%
function	70.2%	87.6%	92.8%
icon	88.8%	77.1%	89.6%
identity	64.4%	86.1%	92.5%
pod	62.3%	89.5%	92.0%
pod_bundle	75.8%	69.3%	88.1%
schedule	63.5%	88.9%	91.7%
usage	74.4%	84.5%	91.9%
workflow	70.1%	82.9%	89.9%
workspace	79.4%	59.4%	86.8%

[deepak-jha-kgp]

Factory review: Needs a person's review

round 5 · reviewed at 54d6d8f · updated 19 Oct, 07:35 UTC · waiting on @deepak-jha-kgp · high risk

One rule and one write path replace groups, notifications and per-producer sends

• Before: A pod's groups were read through SpaceGroups and their lines written by channels.services.timeline.Timeline; a question waiting on a member was a notifications row, kept and dispatched by NotificationService and shown by a bell of its own. Each producer -- a member's post, a DM fan-out, a notification -- planned and sent its own deliveries, and nothing decided who may see a channel in one place.
• After: Every line is written through channels.services.posting.Timeline.post, which seats the speaker, records who it names, asks what it asks, plans one delivery row per target and announces the line in one transaction; one predicate decides who sees a channel, in Python (may_see) and in SQL (visible_to). A question is a row on the line it was asked on, claimed before the asker takes the answer.

What could break

• If any environment already applied revision 0058_channels, then alembic sees the schema as current and none of the new DDL runs there, and the app queries tables that do not exist (medium likelihood, high impact; guard: the commit says 0057/0058 are unreleased; nothing in code enforces it)
• If a member holds contact.read, then every contact's channel -- their DMs, emails and web chats -- is listed, searchable and readable to them, including conversations they never worked (medium likelihood, high impact; guard: contact.read is granted deliberately, and test_channels_v2_api_e2e.py:228 asserts the denial without it)
• If the deploy runs 0058 with an open notification in the table, then that member's pending ask or approval is deleted with the table and never appears as a waiting line, for them (high likelihood, medium impact; guard: none)

362 of 95,734 changed lines worth reading. The rest is mechanical, generated or lockfiles.

Decisions this PR makes

1. The migration drops the notifications table and the surface-group tables and carries nothing over — A member's open ask or approval is a notifications row today; dropping the table deletes it, and no backfill makes it a waiting line, so the deploy silently loses work that was waiting on a person (lemma-backend/migrations/versions/2026-10-17_channels_0058.py)
2. One visibility rule decides every read, in Python and in SQL — may_see and visible_to must agree on four cases -- DIRECT, CONTACT, public GROUP, private GROUP -- or a channel is listed and not readable, or readable and not listed (lemma-backend/app/modules/channels/domain/access.py)
3. Every line is written through Timeline.post, which plans deliveries and questions in one transaction — If any producer writes a line another way, it gets no delivery row, no question and no announcement, and the failure is invisible (lemma-backend/app/modules/channels/services/posting.py)
4. A question is claimed before the asker takes the answer — Without the claim and its commit, a tap on a platform and an answer in the app both reach the asking side, and a run resumes twice or a form is submitted twice (lemma-backend/app/modules/channels/services/questions.py)
5. Every send outside Lemma is a row, retried with backoff until it is given up — The row is the only record of a send (lemma-backend/app/modules/channels/infrastructure/deliveries.py)

Findings · 3 open · 3 settled

• A run paused when this deploys gets no line waiting for its member — lemma-backend/app/modules/agent/services/pause_questions.py:203
• 0058 is edited in place, so an environment that already applied it gets none of the new tables — lemma-backend/migrations/versions/2026-10-17_channels_0058.py:31
• Dropping the notifications table deletes every question that was still waiting on a member — lemma-backend/migrations/versions/2026-10-17_channels_0058.py:483

Worth fixing · 3 warnings

1. A run paused when this deploys gets no line waiting for its member — lemma-backend/app/modules/agent/services/pause_questions.py:203-228

sync_pause_questions is what turns a paused ask_user or request_approval into a channel line with a question row, and the only thing that calls it is the agent.pause_questions handler subscribed to AgentRunFinished (agent/events/channel_question_work.py:113-119). A run that is already paused at deploy has finished nothing since, so no channel_questions row is ever written for it and the member's inbox -- waiting_on (infrastructure/questions.py:241) and the waiting count (infrastructure/inbox.py) -- shows nothing. They can still answer from the conversation's own card, but the new inbox loses it.

Suggested fix:

On deploy, run the sync for every conversation with an undecided pause, or enqueue AGENT_PAUSE_QUESTIONS per paused run as part of the release.

2. 0058 is edited in place, so an environment that already applied it gets none of the new tables — lemma-backend/migrations/versions/2026-10-17_channels_0058.py:31-32

The revision id is unchanged from main -- revision = "0058_channels", down_revision = "0057_contacts_for_everyone" -- while the DDL inside it is rewritten: the old 0058 created the channel tables and kept notifications; this one drops notifications and the three agent_surface_* tables and creates channel_members, channel_read_state, channel_sources, channel_mentions, channel_questions and channel_message_deliveries. Alembic decides by revision id, so any database that already ran 0058 is considered current and none of this runs there: notifications stays, the new tables never appear, and the first request that touches channel_questions or channel_message_deliveries fails. Check alembic current on every environment before merging.

Suggested fix:

If any environment is at 0058_channels, add the new DDL as a new revision on top (0059) instead of editing 0058, or stamp those databases back to 0057 and re-run.

3. Dropping the notifications table deletes every question that was still waiting on a member — lemma-backend/migrations/versions/2026-10-17_channels_0058.py:483

upgrade() runs DELETE FROM work_items WHERE kind = 'notification.expire' and then op.drop_table("notifications") before creating the channel tables. Every row there is an ask, an approval or a form that a member has not answered -- the docstring calls them "what the app shows as waiting on them". Nothing reads them into channel_questions, and downgrade() recreates the table empty, so those rows are gone for good. The docstring says groups and web chat were unreleased and nothing is carried over; it does not say the same for notifications.

Suggested fix:

Before the drop, insert one channel line and `channel_questions` row per open notification (its recipient, kind, body, expiry), or confirm in writing that no environment holds open notifications.

• The Channels tab is always shown but its pane is behind a flag, so it opens an empty page — fixed
• One unsubscribed email address silences every address that contact has — fixed
• Two answers racing can both reach the asking side before either closes the line — fixed

Rounds

#	When	Commit	What	Verdict
1	18 Oct, 16:22 UTC	1550cd9	full read	Changes needed
2	19 Oct, 00:45 UTC	667ae8f	new commits	did not finish
3	19 Oct, 01:00 UTC	00e9311	full read	Needs a person's review
4	19 Oct, 07:13 UTC	63d4edb	full read	Needs a person's review
5	19 Oct, 07:35 UTC	54d6d8f	new commits	Needs a person's review

One comment, edited after every round. Last updated 19 Oct, 07:35 UTC.