Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 19 additions & 9 deletions docs/PI.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,10 @@ an executable).

A session needs a model Pi can reach, any of:

- **An account** connected on the phone (the agent's page → Accounts): an
API key for any of Pi's providers, or a sign-in to a subscription where
Pi has one (Claude Pro/Max, ChatGPT Plus/Pro, GitHub Copilot, xAI and
others). Pi's own code stores it, in its home (below), exactly where its
- **An account** connected on the phone (the agent's page → AI providers →
Connect a provider): an API key for any of Pi's providers, or a sign-in to
a subscription where Pi has one (Claude Pro/Max, ChatGPT Plus/Pro through
OpenAI Codex, GitHub Copilot, xAI and others). Pi's own code stores it, in its home (below), exactly where its
`/login` would. The model picker then lists that provider's models.
- **A provider profile** added on the phone (the agent's page → AI
providers): any OpenAI-compatible endpoint — a provider's own API or a
Expand All @@ -33,7 +33,7 @@ A session needs a model Pi can reach, any of:
levels and the longest answer. A size the endpoint states wins; a model Pi
does not know runs at 128k context without reasoning.
- **The bridge's environment**: the keys Pi reads itself, such as
`ANTHROPIC_API_KEY` or `OPENROUTER_API_KEY`. Accounts shows them as the
`ANTHROPIC_API_KEY` or `OPENROUTER_API_KEY`. The phone shows them as the
machine's, which only the machine can change.

## Accounts
Expand All @@ -45,10 +45,20 @@ by Pi only when a session uses it.
A sign-in gives a page to open on any device. Some providers then show a
code to enter there and finish on their own; the others send the browser
back to an address on the bridge's machine, which a phone cannot open: paste
the address it fails to load, and Pi finishes the sign-in with it. A
provider whose login asks something more than a key or a sign-in (a
resource name, a region) is refused with Pi's question — run Pi's own
`/login` on the machine for it.
the address it fails to load, and Pi finishes the sign-in with it.

What else a login asks is on the form, in Pi's own words: Cloudflare's
account and gateway IDs beside the key, how to sign in to Anthropic or
OpenAI Codex (a page, or a code for another device), a GitHub Enterprise
domain (blank for github.com). Pi asks these as its login goes, so the
bridge finds them when the phone lists the providers, by running each
login with every way off the machine blocked (the network, listening,
other programs) and its placeholder answers kept in a scratch file deleted
afterwards. Where a login offers several ways and only one works without
the machine's own setup (Amazon Bedrock's bearer token, Google Vertex's API
key, rather than an AWS profile or a credentials file), the phone offers
that one. A way that needs the machine itself is not offered — run Pi's
own `/login` there for it.

## Its own home

Expand Down
32 changes: 20 additions & 12 deletions packages/agent-host/src/drivers/pi/__tests__/fakePi.ts
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,11 @@ export class FakePi {
this.child.stdout.emit('data', Buffer.from(`${JSON.stringify(record)}\n`));
}

/** Write one raw line on stdout, as a Pi that printed something else would. */
print(line: string): void {
this.child.stdout.emit('data', Buffer.from(`${line}\n`));
}

private respond(id: unknown, command: string, data?: unknown): void {
this.emit({ ...(typeof id === 'string' ? { id } : {}), type: 'response', command, success: true, ...(data !== undefined ? { data } : {}) });
}
Expand Down Expand Up @@ -254,14 +259,16 @@ export class FakePi {
this.say('recovered', { retried: 2 });
return;
}
if (text === '/give-up') {
this.failedAttempt(1);
this.failedAttempt(2);
this.failedAttempt(3);
this.attemptError();
if (text === '/give-up' || text.startsWith('/give-up ')) {
// `/give-up <error>`: every attempt fails with that provider error.
const error = text.slice('/give-up'.length).trim() || RATE_LIMITED;
this.failedAttempt(1, error);
this.failedAttempt(2, error);
this.failedAttempt(3, error);
this.attemptError(error);
this.emit({ type: 'turn_end', message: {}, toolResults: [] });
this.emit({ type: 'agent_end', messages: [] });
this.emit({ type: 'auto_retry_end', success: false, attempt: 3, finalError: RATE_LIMITED });
this.emit({ type: 'auto_retry_end', success: false, attempt: 3, finalError: error });
this.emit({ type: 'agent_settled', aborted: false });
return;
}
Expand All @@ -270,24 +277,25 @@ export class FakePi {
this.failedAttempt(1);
return;
}
if (text === '/bad-request') {
// An error Pi does not retry: settled at once.
if (text === '/bad-request' || text.startsWith('/bad-request ')) {
// An error Pi does not retry: settled at once. `/bad-request <error>`
// names the provider's error.
this.emit({ type: 'agent_start' });
this.attemptError('400: invalid request');
this.attemptError(text.slice('/bad-request'.length).trim() || '400: invalid request');
this.settle(false);
return;
}
this.say(`echo: ${text}`);
}

/** One failed request that Pi then retries. */
private failedAttempt(attempt: number): void {
private failedAttempt(attempt: number, error = RATE_LIMITED): void {
if (attempt === 1) this.emit({ type: 'agent_start' });
this.attemptError();
this.attemptError(error);
this.emit({ type: 'turn_end', message: {}, toolResults: [] });
this.emit({ type: 'agent_end', messages: [] });
// Pi retries up to three times, each retry announced before it starts.
this.emit({ type: 'auto_retry_start', attempt, maxAttempts: 3, delayMs: 2000 * attempt, errorMessage: RATE_LIMITED });
this.emit({ type: 'auto_retry_start', attempt, maxAttempts: 3, delayMs: 2000 * attempt, errorMessage: error });
this.emit({ type: 'agent_start' });
}

Expand Down
120 changes: 112 additions & 8 deletions packages/agent-host/src/drivers/pi/__tests__/piAccounts.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,56 @@ real('Pi accounts, through the installed Pi', () => {
await expect(manager.change('openrouter', { kind: 'remove', connection }, noEnd)).rejects.toThrow(/only the machine/);
expect(existsSync(path.join(home, 'auth.json')) ? readFileSync(path.join(home, 'auth.json'), 'utf8') : '{}').not.toContain('sk-or-env');
}, 60_000);

it('lists what each login asks before it starts, and stores nothing while finding out', async () => {
const { accounts: manager, home } = accounts();
const all = await manager.list(true);
const method = (id: string, kind: string) => all.find((a) => a.id === id)?.methods?.find((m) => m.kind === kind);
// A key login with questions of its own.
expect(method('cloudflare-workers-ai', 'key')?.fields).toEqual([
{ key: 'Enter Cloudflare account ID', title: 'Cloudflare account ID', required: true },
]);
// A key login with several ways, of which only the key works from a phone.
expect(method('amazon-bedrock', 'key')).toBeDefined();
expect(method('amazon-bedrock', 'key')?.fields).toBeUndefined();
// Sign-ins that ask first: a choice of how, and a domain left blank for github.com.
expect(method('anthropic', 'sign-in')?.fields).toEqual([
expect.objectContaining({
key: 'Select Anthropic login method:',
required: true,
default: 'browser',
options: [expect.objectContaining({ id: 'browser' }), expect.objectContaining({ id: 'copy_code' })],
}),
]);
expect(method('github-copilot', 'sign-in')?.fields).toEqual([
{ key: 'GitHub Enterprise URL/domain (blank for github.com)', title: 'GitHub Enterprise URL/domain (blank for github.com)', placeholder: 'company.ghe.com' },
]);
// Finding out ran Pi's logins on placeholders: none of it reached Pi's home.
const auth = path.join(home, 'auth.json');
expect(existsSync(auth) ? JSON.parse(readFileSync(auth, 'utf8')) : {}).toEqual({});
expect(await manager.list(false)).toEqual([]);
}, 60_000);

it('stores a key with the answers its login asks for, and refuses it without them', async () => {
const { accounts: manager, home } = accounts();
await expect(
manager.change('cloudflare-workers-ai', { kind: 'key', method: 'key', key: 'cf-codedeck-test' }, noEnd),
).rejects.toThrow(/Cloudflare account ID is needed/);
await manager.change(
'cloudflare-workers-ai',
{ kind: 'key', method: 'key', key: 'cf-codedeck-test', answers: { 'Enter Cloudflare account ID': 'acct-1' } },
noEnd,
);
const stored = JSON.parse(readFileSync(path.join(home, 'auth.json'), 'utf8')) as Record<string, unknown>;
expect(stored['cloudflare-workers-ai']).toEqual({ type: 'api_key', key: 'cf-codedeck-test', env: { CLOUDFLARE_ACCOUNT_ID: 'acct-1' } });
}, 60_000);

it('takes the key way of a login that offers several', async () => {
const { accounts: manager, home } = accounts();
await manager.change('amazon-bedrock', { kind: 'key', method: 'key', key: 'bedrock-codedeck-test' }, noEnd);
const stored = JSON.parse(readFileSync(path.join(home, 'auth.json'), 'utf8')) as Record<string, unknown>;
expect(stored['amazon-bedrock']).toEqual({ type: 'api_key', key: 'bedrock-codedeck-test' });
}, 60_000);
});

/** A scripted helper: records what the driver sends, and answers with the
Expand Down Expand Up @@ -109,14 +159,37 @@ class ScriptedHelper {
}
}

function scripted(helper: ScriptedHelper, now = () => 1_000): PiAccounts {
/** The paths of a login that asks nothing but its key, and of a sign-in
* that goes straight to its page. */
const PLAIN = {
keyPaths: [{ prompts: [{ type: 'secret', message: 'Key' }], end: 'done' }],
signInPaths: [{ prompts: [], end: 'user' }],
};

/** A helper that lists one provider, as Pi's helper does before a login. */
function listing(provider: string, paths: Record<string, unknown> = PLAIN): ScriptedHelper {
const helper = new ScriptedHelper();
helper.onSend = (record) => {
if (record.op !== 'list') return;
setImmediate(() => {
helper.say({ type: 'list', providers: [{ id: provider, name: provider, key: 'Key', signIn: 'Sign in', ...paths }], credentials: [] });
helper.exit(0);
});
};
return helper;
}

/** A manager whose helpers are, in turn: the listing of `provider` a key or
* a sign-in reads first, then `helper`. */
function scripted(helper: ScriptedHelper, provider: string, paths?: Record<string, unknown>, now = () => 1_000): PiAccounts {
const home = mkdtempSync(path.join(tmpdir(), 'codedeck-pi-accounts-'));
const queue = [listing(provider, paths), helper];
return new PiAccounts({
home,
entryPoint: async () => path.join(home, 'dist', 'bundle', 'cli.js'),
env: () => ({}),
changed: () => {},
spawnFn: (() => helper.child) as never,
spawnFn: (() => (queue.shift() ?? helper).child) as never,
now,
});
}
Expand All @@ -141,7 +214,7 @@ describe('Pi sign-in', () => {
}
};
const ends: SignInState[] = [];
const manager = scripted(helper);
const manager = scripted(helper, 'anthropic');
const started = await manager.change('anthropic', { kind: 'sign-in', method: 'sign-in' }, (end) => ends.push(end));
expect('signIn' in started && started.signIn).toMatchObject({
status: 'started',
Expand All @@ -161,7 +234,7 @@ describe('Pi sign-in', () => {
helper.onSend = (record) => {
if (record.op === 'login') setImmediate(() => helper.say({ type: 'notify', event: { type: 'auth_url', url: 'https://example.com/a' } }));
};
const manager = scripted(helper);
const manager = scripted(helper, 'xai');
const started = await manager.change('xai', { kind: 'sign-in', method: 'sign-in' }, noEnd);
const attemptId = 'signIn' in started ? started.signIn.attemptId : '';
await manager.change('xai', { kind: 'finish', attempt: attemptId, callbackUrl: 'http://localhost:1/cb?code=z' }, noEnd);
Expand All @@ -179,7 +252,7 @@ describe('Pi sign-in', () => {
}
};
const ends: SignInState[] = [];
const manager = scripted(helper);
const manager = scripted(helper, 'github-copilot');
const started = await manager.change('github-copilot', { kind: 'sign-in', method: 'sign-in' }, (end) => ends.push(end));
expect('signIn' in started && started.signIn.attempt).toEqual(
expect.objectContaining({ url: 'https://github.com/login/device', instructions: expect.stringContaining('ABCD-1234'), expiresAt: 1_000 + 900_000 }),
Expand All @@ -198,7 +271,7 @@ describe('Pi sign-in', () => {
if (record.op === 'login') setImmediate(() => helper.say({ type: 'notify', event: { type: 'auth_url', url: 'https://example.com/a' } }));
};
const ends: SignInState[] = [];
const manager = scripted(helper);
const manager = scripted(helper, 'openai');
const started = await manager.change('openai', { kind: 'sign-in', method: 'sign-in' }, (end) => ends.push(end));
const attemptId = 'signIn' in started ? started.signIn.attemptId : '';
const cancelled = await manager.change('openai', { kind: 'cancel', attempt: attemptId }, noEnd);
Expand All @@ -207,6 +280,37 @@ describe('Pi sign-in', () => {
await tick();
expect(ends).toEqual([]);
});

it('answers the choice Pi asks before its page with the one made on the phone', async () => {
const how = 'Select Anthropic login method:';
const choice = (id: string) => ({
prompts: [{ type: 'select', message: how, options: [{ id: 'browser', label: 'Browser' }, { id: 'copy_code', label: 'Copy code' }], choice: id }],
end: 'user',
});
const helper = new ScriptedHelper();
helper.onSend = (record) => {
if (record.op === 'login') {
setImmediate(() => helper.say({ type: 'prompt', id: 'p1', prompt: { type: 'select', message: how } }));
}
if (record.op === 'answer') {
setImmediate(() => helper.say({ type: 'notify', event: { type: 'auth_url', url: 'https://claude.ai/oauth/authorize?code=true' } }));
}
};
const manager = scripted(helper, 'anthropic', { signInPaths: [choice('browser'), choice('copy_code')] });
const started = await manager.change('anthropic', { kind: 'sign-in', method: 'sign-in', answers: { [how]: 'copy_code' } }, noEnd);
expect(helper.sent).toContainEqual({ op: 'answer', id: 'p1', value: 'copy_code' });
expect('signIn' in started && started.signIn.status).toBe('started');
});

it('refuses a choice the phone could not have been offered', async () => {
const how = 'Sign in how:';
const helper = new ScriptedHelper();
const manager = scripted(helper, 'radius', {
signInPaths: [{ prompts: [{ type: 'select', message: how, options: [{ id: 'browser', label: 'Browser' }, { id: 'device-code', label: 'Code' }], choice: 'browser' }], end: 'user' }, { prompts: [{ type: 'select', message: how, choice: 'device-code' }], end: 'user' }],
});
await expect(manager.change('radius', { kind: 'sign-in', method: 'sign-in', answers: { [how]: 'carrier-pigeon' } }, noEnd)).rejects.toThrow(/Choose one/);
expect(helper.sent).toEqual([]);
});
});

describe('Pi key login', () => {
Expand All @@ -223,7 +327,7 @@ describe('Pi key login', () => {
});
}
};
await expect(scripted(helper).change('azure', { kind: 'key', method: 'key', key: 'k' }, noEnd)).rejects.toThrow(/Azure resource name/);
await expect(scripted(helper, 'azure').change('azure', { kind: 'key', method: 'key', key: 'k' }, noEnd)).rejects.toThrow(/Azure resource name/);
});

it('keeps the key out of an error Pi gives back', async () => {
Expand All @@ -237,7 +341,7 @@ describe('Pi key login', () => {
});
}
};
const failure = scripted(helper).change('openrouter', { kind: 'key', method: 'key', key: 'sk-secret-value' }, noEnd);
const failure = scripted(helper, 'openrouter').change('openrouter', { kind: 'key', method: 'key', key: 'sk-secret-value' }, noEnd);
await expect(failure).rejects.toThrow(/invalid key/);
await expect(failure).rejects.not.toThrow(/sk-secret-value/);
});
Expand Down
51 changes: 51 additions & 0 deletions packages/agent-host/src/drivers/pi/__tests__/piDriver.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -212,6 +212,57 @@ describe('PiSession startup', () => {
expect(h.fake.token).toBe('');
});

describe('a profile token quoted back by a failure', () => {
const TOKEN = 'sk-codedeck-profile-secret-0123456789';
// Any piece of it long enough to be worth anything.
const PIECE = TOKEN.slice(0, 16);
const provider = {
id: 'codedeck-fake',
label: 'Fake',
baseUrl: 'https://api.example.com',
authToken: TOKEN,
models: [{ id: 'fake-model-1', label: 'Fake model' }],
};

it('is not in the errors and retries the phone is shown', async () => {
const h = withDriver();
const ctx = await ready(h, { model: 'fake-model-1', provider });
// An endpoint that echoes the header it refused, the token straddling
// the point where a retry line is cut short (it starts at column 100).
await turn(h.session!, ctx, `/give-up 401 Unauthorized: ${'x'.repeat(59)} Authorization: Bearer ${TOKEN}`);
await turn(h.session!, ctx, `/bad-request invalid api key ${TOKEN}`);
const shown = JSON.stringify(entries(ctx));
expect(shown).toContain('401 Unauthorized');
expect(shown).toContain('invalid api key');
expect(shown).not.toContain(PIECE);
});

it('is not in what Pi printed, in the log or in the reason the session ended', async () => {
const h = withDriver({ fake: { exitOnStart: { code: 1, stderr: `Error: the endpoint refused ${TOKEN}` } } });
const ctx = start(h, { model: 'fake-model-1', provider });
const ended = await ctx.ended();
expect(ended).toMatchObject({ error: expect.stringContaining('the endpoint refused') });
expect(JSON.stringify(ended)).not.toContain(PIECE);
expect(h.logs.join('\n')).toContain('the endpoint refused');
expect(h.logs.join('\n')).not.toContain(PIECE);
});

it('is not in the log when an extension fails or Pi prints a line that is not a record', async () => {
const h = withDriver();
await ready(h, { model: 'fake-model-1', provider });
h.fake.emit({ type: 'extension_error', error: `registerProvider refused apiKey ${TOKEN}` });
// A line that starts with it: a parse error quotes the line's start,
// cut where the redaction could no longer find it.
h.fake.print(`${TOKEN} was not accepted`);
h.fake.emit({ type: 'response', success: false, error: `bad command near ${TOKEN}` });
const logged = h.logs.join('\n');
expect(logged).toContain('an extension failed');
expect(logged).toContain('dropped a line');
expect(logged).not.toContain(PIECE);
expect(logged).not.toContain(TOKEN.slice(0, 8));
});
});

it('refuses a provider profile with an insecure endpoint before it starts', () => {
const h = withDriver();
expect(() =>
Expand Down
Loading
Loading