Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 13 additions & 2 deletions docs/evidence-contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,8 +66,12 @@ EvidenceSource ──< EvidenceItem >── EvidenceClaim ──> Answer / Resea
- **authority 元数据只在实际已知时填写**(如监管备案 =
`{ primary: true, sourceClass: 'regulator' }`),投影从不猜测。
- **冲突与不可用有显式状态位**(`availability: 'conflicted' | 'unavailable'`)。
当前投影仅产出 `available`;金融数据的 `reconciliation` 只保留在
`source.providerMeta`,尚未映射到 `availability`。
结构化金融投影在 `reconciliation.state` 为 `material-conflict` 或
`incomparable` 时把每条事实标为 `conflicted`。选中的规范化数值保持原样,
候选值留在 `source.providerMeta.reconciliation.candidates`,**不会被平均**。
`agreement` / `within-tolerance` / 仅有 `providers`+`method` 的旧记录仍是
`available`。`freshness.delayed` 只在信封 `delayed === true` 时写入。
`provenance.envelopeId` 保留原始 `fe_*` id,供引用加入。
- **验证状态**首版恒为 `unverified`。现有
`packages/shared/src/research/claim-verifier.ts` 使用独立的 claim ID 与状态契约;
接入 claim 级验证(#13)或来源漂移检测(#20)时需要显式映射。
Expand All @@ -79,6 +83,13 @@ EvidenceSource ──< EvidenceItem >── EvidenceClaim ──> Answer / Resea
- **Source Inspector(#30)与引用检查器**尚未接入 bundle;当前 Inspector
仍由 `Message.financialEvidence` / `toolCalls` 组装。后续接线可使用 bundle
的来源类别、URL、摘录与溯源字段,但需显式处理现有 ID 空间的映射。
- **Copilot 引用溯源(#101 的一个切片)**由
`resolveCopilotCitationTrace` 按需计算,不新增持久化。它从已保存的问题、
回答文本和 `financialEvidence` 出发,把行内 `⟦cite:toolCallId⟧` 与已闭合
`folio-block` 的 `evidenceIds`(工具调用 id 或 `fe_*`)解析到同一条事实:
provider、instrument、metric、unit/currency、`asOf` 与 `retrievedAt`、
delayed / stale / fallback,以及冲突候选。无法解析的引用留在
`resolution: 'unmapped'`,不会被静默丢掉,也不会用实时行情重算。
- **评测(#14/#15)**后续可基于 claimId/evidenceId 统计引用覆盖率。

## 集成示例
Expand Down
81 changes: 81 additions & 0 deletions packages/core/src/answer-segments.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
import { ANSWER_BLOCK_FENCE_LANG } from './answer-blocks.ts';

/**
* One segment of a Copilot answer: either Markdown text or a typed
* `folio-block` fence body. Untyped fences stay in text segments so ordinary
* code blocks keep rendering through Markdown.
*/
export type AnswerSegment =
| { kind: 'text'; text: string }
| { kind: 'block'; body: string; closed: boolean };

const FENCE_OPEN = /^[ \t]{0,3}(`{3,}|~{3,})(.*)$/;
const FENCE_CLOSE = /^[ \t]{0,3}(`{3,}|~{3,})[ \t]*$/;

// `split('\n')` leaves the carriage return on CRLF lines. Strip it only for
// delimiter detection so the ordinary Markdown and typed JSON bytes remain
// unchanged in the returned segments.
const fenceLine = (line: string): string => (line.endsWith('\r') ? line.slice(0, -1) : line);

/**
* Split an answer string into text and typed-block segments.
*
* Every fence is opaque until a bare closing fence with the same marker and
* at least the opening length arrives. This keeps `folio-block` examples
* inside ordinary Markdown code fences as literal text.
*
* An unclosed `folio-block` fence produces a `closed: false` block segment for
* streaming; unclosed ordinary fences remain verbatim Markdown.
*/
export function parseAnswerSegments(content: string): AnswerSegment[] {
const lines = content.split('\n');
const segments: AnswerSegment[] = [];
let textLines: string[] = [];

const flushText = () => {
if (textLines.length > 0) {
segments.push({ kind: 'text', text: textLines.join('\n') });
textLines = [];
}
};

let index = 0;
while (index < lines.length) {
const line = lines[index] ?? '';
const open = fenceLine(line).match(FENCE_OPEN);
const marker = open?.[1];
const info = open?.[2] ?? '';

// Backtick info strings cannot contain another backtick. Such a line is
// prose rather than a fence and must not swallow later answer blocks.
if (!marker || (marker[0] === '`' && info.includes('`'))) {
textLines.push(line);
index += 1;
continue;
}

const typed = info.trim() === ANSWER_BLOCK_FENCE_LANG;
if (typed) flushText();
else textLines.push(line);

const bodyLines: string[] = [];
let closed = false;
index += 1;
while (index < lines.length) {
const bodyLine = lines[index] ?? '';
const close = fenceLine(bodyLine).match(FENCE_CLOSE)?.[1];
if (close && close[0] === marker[0] && close.length >= marker.length) {
if (!typed) textLines.push(bodyLine);
closed = true;
index += 1;
break;
}
if (typed) bodyLines.push(bodyLine);
else textLines.push(bodyLine);
index += 1;
}
if (typed) segments.push({ kind: 'block', body: bodyLines.join('\n'), closed });
}
flushText();
return segments;
}
10 changes: 10 additions & 0 deletions packages/core/src/evidence-contract.ts
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,11 @@ export interface EvidenceFreshness {
/** Market-time of the fact itself, when known. */
asOf?: number;
stale: boolean;
/**
* True when the producing provider reported a delayed or end-of-day value.
* Omitted when the provider did not say so — absence is not "live".
*/
delayed?: boolean;
}

export interface EvidenceItem {
Expand All @@ -116,6 +121,11 @@ export interface EvidenceItem {
provider?: string;
/** Scoping instrument for non-financial evidence (news about a listing). */
instrumentId?: string;
/**
* Persisted `fe_*` envelope id. Copilot answers cite either this id or the
* tool-call id; both must join back to the same observation.
*/
envelopeId?: string;
};
}

Expand Down
24 changes: 22 additions & 2 deletions packages/core/src/financial-evidence.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
import type { FinancialFactCandidate, ReconciliationState } from './reconciliation.ts';

/** Machine-readable lineage for structured financial facts used by Copilot. */

export const FINANCIAL_EVIDENCE_SCHEMA_VERSION = 'financial-evidence/v1' as const;
Expand Down Expand Up @@ -34,7 +36,25 @@ export interface FinancialEvidenceMetadata {
dataset?: string;
cacheHit?: boolean;
fallback?: FinancialEvidenceEnvelope['fallback'];
reconciliation?: FinancialEvidenceEnvelope['reconciliation'];
reconciliation?: FinancialEvidenceReconciliation;
}

/**
* Provider disagreement recorded on a persisted fact.
*
* `providers` + `method` is the original shape. `state` and `candidates` are
* additive: a material conflict keeps every candidate and never replaces them
* with an average. Older records that only stored providers/method stay valid.
*/
export interface FinancialEvidenceReconciliation {
providers: string[];
method: string;
state?: ReconciliationState;
candidates?: FinancialFactCandidate[];
discrepancy?: number;
selectedProvider?: string;
/** Verbatim reconciler explanation, when the producer supplied one. */
reason?: string;
}

export interface FinancialEvidenceEnvelope {
Expand All @@ -59,7 +79,7 @@ export interface FinancialEvidenceEnvelope {
delayed?: boolean;
cacheHit: boolean;
fallback?: { from: string; to: string; reason?: string };
reconciliation?: { providers: string[]; method: string };
reconciliation?: FinancialEvidenceReconciliation;
resultSnapshot: unknown;
resultHash: string;
lineage: FinancialEvidenceLineageStep[];
Expand Down
1 change: 1 addition & 0 deletions packages/core/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -586,6 +586,7 @@ export interface Skill {

// ── Folio V3 domains ───────────────────────────────────────────────────────
export * from './answer-blocks.ts';
export * from './answer-segments.ts';
export * from './citations.ts';
export * from './capability.ts';
export * from './research.ts';
Expand Down
167 changes: 167 additions & 0 deletions packages/shared/src/evidence/citation-trace.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,167 @@
import { describe, expect, it } from 'bun:test';
import { renderCitationMarker } from '@finagent/core';
import { appendBlocksToAnswer, renderBlockFence } from '../agent/answer-block-emitter.ts';
import { buildFinancialEvidence } from './financial-evidence.ts';
import { resolveCopilotCitationTrace } from './citation-trace.ts';
import type { ToolCall } from '@finagent/core';

function quoteCall(overrides: Partial<ToolCall> = {}): ToolCall {
return {
id: 'quote-1',
toolName: 'get_quote',
args: { symbol: 'AAPL.US' },
startedAt: 1_710_000_000_000,
completedAt: 1_710_000_000_123,
status: 'success',
result: {
data: { symbol: 'AAPL.US', instrumentId: 'XNAS:AAPL', lastPrice: 210, currency: 'USD' },
provenance: {
providerId: 'massive',
provider: 'massive',
instrumentId: 'XNAS:AAPL',
fetchedAt: 1_710_000_000_123,
marketTime: 1_710_000_000_456,
delayed: true,
stale: false,
},
evidence: {
currencies: { lastPrice: 'USD' },
units: { lastPrice: 'price' },
fallback: { from: 'longbridge', to: 'massive', reason: 'TIMEOUT' },
reconciliation: {
state: 'material-conflict',
policyVersion: 'financial-facts.v1',
candidates: [
{ provider: 'longbridge', value: 100, currency: 'USD', unit: 'price' },
{ provider: 'massive', value: 300, currency: 'USD', unit: 'price' },
],
selected: { provider: 'massive', value: 300, currency: 'USD', unit: 'price' },
discrepancy: 200,
reason: 'Material disagreement; candidates are preserved and never averaged.',
},
},
},
...overrides,
};
}

describe('resolveCopilotCitationTrace', () => {
it('joins a tool-call citation and an fe_ citation to the same persisted fact', () => {
const evidence = buildFinancialEvidence({
sessionId: 'session-1',
runId: 'run-1',
toolCalls: [quoteCall()],
});
const envelope = evidence[0]!;
const block = renderBlockFence({
version: 1,
type: 'metric_grid',
title: 'AAPL',
evidenceIds: [envelope.id],
metrics: [{
label: 'Last',
value: 210,
unit: 'price',
currency: 'USD',
evidenceIds: ['quote-1'],
}],
});
const answer = appendBlocksToAnswer(
`Apple last traded at 210 USD. ${renderCitationMarker('quote-1')} ${renderCitationMarker('missing-tool')}`,
[JSON.parse(block.slice(block.indexOf('\n') + 1, block.lastIndexOf('\n```')))]
);
const trace = resolveCopilotCitationTrace({
question: "What is Apple's current price, and where did it come from?",
answer,
financialEvidence: evidence,
});

expect(trace.schemaVersion).toBe('folio-citation-trace/v1');
expect(trace.citations.map((citation) => citation.citationId)).toEqual(['quote-1', 'missing-tool', envelope.id]);

const byTool = trace.citations[0]!;
const gap = trace.citations[1]!;
const byEnvelope = trace.citations[2]!;

expect(byTool).toMatchObject({
origin: 'inline',
resolution: 'tool_call',
envelopeId: envelope.id,
toolCallId: 'quote-1',
toolName: 'get_quote',
runId: 'run-1',
provider: 'massive',
instrumentId: 'XNAS:AAPL',
metric: 'lastPrice',
value: 210,
currency: 'USD',
unit: 'price',
asOf: 1_710_000_000_456,
retrievedAt: 1_710_000_000_123,
delayed: true,
stale: false,
fallback: { from: 'longbridge', to: 'massive', reason: 'TIMEOUT' },
availability: 'conflicted',
});
expect(byTool.asOf).not.toBe(byTool.retrievedAt);
expect(byTool.value).not.toBe(200);
expect(byTool.reconciliation?.candidates).toEqual([
{ provider: 'longbridge', value: 100, currency: 'USD', unit: 'price' },
{ provider: 'massive', value: 300, currency: 'USD', unit: 'price' },
]);
expect(byTool.values.find((value) => value.metric === 'lastPrice')?.evidenceId).toMatch(/^ev_/);

expect(gap).toMatchObject({ resolution: 'unmapped', availability: 'unmapped', values: [] });
expect(gap.provider).toBeUndefined();
expect(gap.value).toBeUndefined();

expect(byEnvelope.resolution).toBe('envelope');
expect(byEnvelope.envelopeId).toBe(byTool.envelopeId);
expect(byEnvelope.values.map((value) => value.evidenceId)).toEqual(byTool.values.map((value) => value.evidenceId));
expect(byEnvelope.availability).toBe('conflicted');
});

it('is unchanged when the same persisted messages are reloaded', () => {
const evidence = buildFinancialEvidence({
sessionId: 'session-1',
runId: 'run-1',
toolCalls: [quoteCall()],
});
const answer = `Price ${renderCitationMarker('quote-1')}`;
const question = 'What is AAPL worth?';
const first = resolveCopilotCitationTrace({ question, answer, financialEvidence: evidence });
const reloaded = JSON.parse(JSON.stringify({ question, answer, financialEvidence: evidence })) as {
question: string;
answer: string;
financialEvidence: typeof evidence;
};
const second = resolveCopilotCitationTrace(reloaded);
expect(second).toEqual(first);

const reworded = resolveCopilotCitationTrace({ ...reloaded, question: 'Ignore the saved price and use today\'s quote.' });
expect(reworded.citations).toEqual(first.citations);
expect(reworded.question).not.toBe(first.question);
});

it('ignores evidence ids inside an unclosed or non-typed fence', () => {
const evidence = buildFinancialEvidence({
sessionId: 'session-1',
runId: 'run-1',
toolCalls: [quoteCall()],
});
const answer = [
'```',
'folio-block example',
'{"evidenceIds":["not-a-real-block"]}',
'```',
'```folio-block',
'{"version":1,"type":"metric_grid","metrics":[]}',
].join('\n');
const trace = resolveCopilotCitationTrace({
question: 'quote',
answer,
financialEvidence: evidence,
});
expect(trace.citations).toEqual([]);
});
});
Loading